2024-03-12T04:45:12.408 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T04:45:12.408 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} --archive /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379 --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/orig.config.yaml 2024-03-12T04:45:12.464 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T04:45:12.560 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591379' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 94a539707733325322964c6a187c6a1b41be1daf subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad owner: scheduled_teuthology@teuthology priority: 250 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8179 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 297/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFUDKe8UrWEjD/7h7It4GWED7M2xqJWeWgD7FPFX0UqaWjfKMS9KCvppBRfDQ35LWkBNqLnDk41PWpelOlWO94k= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkwGVDinsQ1NjwoRq7GGQdA4YfQkcmYB4zicYnObp/VXjYy7TVEfujAtazWuh8CHnVqT6kly+oZxLY3VkjR2NE= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiHTZqak80E/y9pbSQZkkBJnx5LZluORpfqRWHVXSGqY9+n6fx5xyj2Yxe0NIRZa5MIKI+Kph5ebWAa8jH9kFg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_14:02:35 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T04:45:12.560 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa; will attempt to use it 2024-03-12T04:45:12.561 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks 2024-03-12T04:45:12.561 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T04:45:12.562 INFO:teuthology.task.internal:Checking packages... 2024-03-12T04:45:12.585 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-12T04:45:12.585 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T04:45:12.585 INFO:teuthology.packaging:ref: None 2024-03-12T04:45:12.586 INFO:teuthology.packaging:tag: None 2024-03-12T04:45:12.586 INFO:teuthology.packaging:branch: squid 2024-03-12T04:45:12.586 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:45:12.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-12T04:45:12.727 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-12T04:45:12.729 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T04:45:12.739 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T04:45:12.739 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T04:45:12.807 INFO:teuthology.task.internal:Saving configuration 2024-03-12T04:45:12.828 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T04:45:12.837 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:45:12.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.359289', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFUDKe8UrWEjD/7h7It4GWED7M2xqJWeWgD7FPFX0UqaWjfKMS9KCvppBRfDQ35LWkBNqLnDk41PWpelOlWO94k='} 2024-03-12T04:45:12.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.361079', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkwGVDinsQ1NjwoRq7GGQdA4YfQkcmYB4zicYnObp/VXjYy7TVEfujAtazWuh8CHnVqT6kly+oZxLY3VkjR2NE='} 2024-03-12T04:45:12.898 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.357326', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiHTZqak80E/y9pbSQZkkBJnx5LZluORpfqRWHVXSGqY9+n6fx5xyj2Yxe0NIRZa5MIKI+Kph5ebWAa8jH9kFg='} 2024-03-12T04:45:12.898 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T04:45:12.911 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-12T04:45:12.911 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-12T04:45:12.911 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-12T04:45:12.911 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T04:45:13.014 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f628fac2160>, signals=[15]) 2024-03-12T04:45:13.015 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T04:45:13.026 INFO:teuthology.task.internal:Opening connections... 2024-03-12T04:45:13.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T04:45:13.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:45:13.138 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-12T04:45:13.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:45:13.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-03-12T04:45:13.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:45:13.326 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T04:45:13.337 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-03-12T04:45:13.355 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-03-12T04:45:13.356 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-03-12T04:45:13.413 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T04:45:13.414 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T04:45:13.415 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-03-12T04:45:13.434 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-12T04:45:13.451 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-12T04:45:13.452 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T04:45:13.508 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T04:45:13.509 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T04:45:13.509 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-12T04:45:13.525 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-03-12T04:45:13.542 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-03-12T04:45:13.543 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T04:45:13.600 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T04:45:13.601 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T04:45:13.601 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-03-12T04:45:13.626 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T04:45:13.639 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T04:45:13.646 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T04:45:13.646 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-03-12T04:45:13.649 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-12T04:45:13.651 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-03-12T04:45:13.667 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T04:45:13.676 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T04:45:13.676 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-03-12T04:45:13.706 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-12T04:45:13.708 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-03-12T04:45:13.794 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T04:45:13.872 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T04:45:13.889 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T04:45:13.890 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T04:45:13.890 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T04:45:13.891 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T04:45:13.891 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T04:45:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:45:13.908 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-12T04:45:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:45:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:45:13.909 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-12T04:45:13.910 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-12T04:45:13.964 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-12T04:45:13.964 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-12T04:45:13.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-12T04:45:13.967 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-361.el9.x86_64 2024-03-12T04:45:13.967 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.14.0-361.el9.x86_64 2024-03-12T04:45:13.967 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2024-03-12T04:45:13.968 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-03-12T04:45:13.968 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-361.el9.x86_64 2024-03-12T04:45:13.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-03-12T04:45:16.119 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T04:45:16.124 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T04:45:16.140 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-12T04:45:17.558 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-12T04:45:17.573 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-12T04:45:17.602 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-12T04:45:18.364 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-12T04:45:19.776 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-12T04:45:21.790 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-03-12T04:45:25.683 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-12T04:45:27.192 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-03-12T04:45:28.080 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 807 kB/s | 8.0 MB 00:10 2024-03-12T04:45:29.835 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-12T04:45:34.751 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-12T04:45:36.874 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T04:45:36.874 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T04:45:36.961 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-12T04:45:36.973 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:45:36.974 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 100 M 2024-03-12T04:45:36.975 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-12T04:45:36.975 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-12T04:45:37.811 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 8.6 MB/s | 6.0 MB 00:00 2024-03-12T04:45:38.636 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 14 MB/s | 21 MB 00:01 2024-03-12T04:45:39.469 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-03-12T04:45:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 17 MB/s | 33 MB 00:02 2024-03-12T04:45:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:45:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:Total 35 MB/s | 100 MB 00:02 2024-03-12T04:45:40.727 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-12T04:45:41.099 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-12T04:45:41.099 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-12T04:45:42.758 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-12T04:45:42.758 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-12T04:45:45.457 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-12T04:45:46.113 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:45:46.244 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:45:49.025 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:45:49.205 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:45:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:45:54.108 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:45:56.148 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 547 kB/s | 19 MB 00:35 2024-03-12T04:45:58.902 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:46:00.181 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-12T04:46:01.540 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 15 kB 00:00 2024-03-12T04:46:03.385 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 564 kB/s | 19 MB 00:34 2024-03-12T04:46:15.268 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 748 kB/s | 6.1 MB 00:08 2024-03-12T04:46:16.869 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-12T04:46:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:46:28.197 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T04:46:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:46:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:46:28.685 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:46:28.685 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:46:28.685 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:46:28.685 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:46:29.121 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-12T04:46:29.615 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:46:29.762 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:46:29.763 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T04:46:29.763 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T04:46:29.763 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T04:46:29.763 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T04:46:29.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T04:46:29.764 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:46:29.885 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:46:29.885 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:46:29.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-12T04:46:29.919 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-12T04:46:29.919 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-12T04:46:29.919 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T04:46:29.920 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:46:29.921 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T04:46:29.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T04:46:29.922 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-427.el9.x86_64 2024-03-12T04:46:29.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T04:46:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-12T04:46:31.986 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T04:46:32.021 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-12T04:46:32.023 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-12T04:46:32.038 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.7HNE3htTgj 2024-03-12T04:46:32.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7HNE3htTgj 2024-03-12T04:46:32.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.7HNE3htTgj 2024-03-12T04:46:32.250 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.7HNE3htTgj is 6KB 2024-03-12T04:46:32.298 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.7HNE3htTgj 2024-03-12T04:46:32.312 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T04:46:32.343 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 21 MB 00:15 2024-03-12T04:46:32.380 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T04:46:32.381 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T04:46:32.381 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T04:46:32.381 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T04:46:32.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T04:46:32.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-12T04:46:37.227 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-12T04:46:39.400 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T04:46:39.400 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T04:46:39.497 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-12T04:46:39.502 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-12T04:46:39.503 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-03-12T04:46:39.504 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:46:39.504 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 100 M 2024-03-12T04:46:39.504 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 123 M 2024-03-12T04:46:39.504 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-12T04:46:40.095 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-03-12T04:46:40.495 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-03-12T04:46:41.570 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-03-12T04:46:41.870 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 19 MB/s | 33 MB 00:01 2024-03-12T04:46:41.871 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:46:41.871 INFO:teuthology.orchestra.run.smithi130.stdout:Total 42 MB/s | 100 MB 00:02 2024-03-12T04:46:42.755 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-12T04:46:43.145 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-12T04:46:43.145 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-12T04:46:44.492 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-12T04:46:44.492 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-12T04:46:47.438 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-12T04:46:48.085 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:46:48.222 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:46:50.979 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:46:51.151 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:46:55.902 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:46:56.087 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:00.876 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:02.564 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T04:47:02.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-12T04:47:02.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:47:30.066 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:30.067 INFO:teuthology.orchestra.run.smithi130.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T04:47:30.067 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:47:30.151 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:30.563 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:47:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:47:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:47:30.994 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-12T04:47:31.512 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:47:31.699 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:47:31.700 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:47:31.700 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T04:47:31.700 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T04:47:31.700 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T04:47:31.700 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-12T04:47:31.700 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T04:47:31.700 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:47:31.821 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:47:31.822 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:47:31.822 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.06 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 68.el9 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Group : Unspecified 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 8264254 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2024-03-12T04:47:31.855 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : builder@centos.org 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2024-03-12T04:47:31.856 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:47:31.857 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T04:47:31.857 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T04:47:31.858 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 5.14.0-427.el9.x86_64 2024-03-12T04:47:31.858 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T04:47:32.444 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-03-12T04:47:33.793 INFO:teuthology.orchestra.run.smithi130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T04:47:33.822 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-03-12T04:47:33.840 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-03-12T04:47:33.864 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.4CCoPl84Jn 2024-03-12T04:47:33.864 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4CCoPl84Jn 2024-03-12T04:47:33.932 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.4CCoPl84Jn 2024-03-12T04:47:34.073 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.4CCoPl84Jn is 6KB 2024-03-12T04:47:34.121 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.4CCoPl84Jn 2024-03-12T04:47:34.135 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T04:47:34.200 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T04:47:34.200 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T04:47:34.200 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T04:47:34.201 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T04:47:34.202 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T04:47:34.350 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-03-12T04:48:02.568 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T04:48:04.353 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-12T04:48:04.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-12T04:48:04.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:48:11.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-12T04:48:11.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:48:12.194 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-12T04:48:12.632 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-12T04:48:12.632 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T04:48:12.632 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-12T04:48:12.647 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-427.el9.x86_64 2024-03-12T04:48:12.647 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T04:48:12.647 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T04:48:12.647 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T04:48:13.648 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T04:48:13.649 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T04:48:13.712 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-12T04:48:13.758 DEBUG:teuthology.parallel:result is None 2024-03-12T04:48:45.714 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 128 kB/s | 21 MB 02:44 2024-03-12T04:48:50.763 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-12T04:48:53.053 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T04:48:53.053 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T04:48:53.154 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-12T04:48:53.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-12T04:48:53.165 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-12T04:48:53.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-12T04:48:53.165 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-03-12T04:48:53.166 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:48:53.167 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 100 M 2024-03-12T04:48:53.167 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 123 M 2024-03-12T04:48:53.167 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-12T04:48:53.781 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-03-12T04:48:54.415 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-03-12T04:48:55.182 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-03-12T04:48:55.723 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 17 MB/s | 33 MB 00:01 2024-03-12T04:48:55.724 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:48:55.724 INFO:teuthology.orchestra.run.smithi045.stdout:Total 39 MB/s | 100 MB 00:02 2024-03-12T04:48:56.642 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-12T04:48:57.117 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-12T04:48:57.117 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-12T04:48:58.503 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-12T04:48:58.504 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-12T04:49:01.356 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-12T04:49:02.047 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:49:02.174 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:49:04.357 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T04:49:05.082 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:49:05.258 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:49:10.357 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:49:10.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:13.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-12T04:49:13.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:49:13.697 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-12T04:49:14.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-12T04:49:14.128 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-12T04:49:14.128 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-12T04:49:14.142 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-427.el9.x86_64 2024-03-12T04:49:14.142 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T04:49:14.142 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T04:49:14.143 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T04:49:15.143 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-12T04:49:15.143 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T04:49:15.232 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-03-12T04:49:15.273 DEBUG:teuthology.parallel:result is None 2024-03-12T04:49:15.614 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:46.438 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:46.439 INFO:teuthology.orchestra.run.smithi045.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T04:49:46.439 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:49:46.524 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:47.025 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:47.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T04:49:47.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T04:49:47.025 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:49:47.467 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-12T04:49:47.962 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:49:48.114 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:49:48.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:49:48.115 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T04:49:48.115 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T04:49:48.115 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T04:49:48.115 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T04:49:48.115 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T04:49:48.115 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T04:49:48.241 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T04:49:48.241 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T04:49:48.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-03-12T04:49:48.279 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:49:48.280 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T04:49:48.281 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T04:49:48.281 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T04:49:48.281 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-03-12T04:49:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:49:48.281 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T04:49:48.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T04:49:48.282 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-427.el9.x86_64 2024-03-12T04:49:48.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T04:49:48.899 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-03-12T04:49:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T04:49:50.337 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-03-12T04:49:50.339 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-12T04:49:50.355 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.tKOq1MoOZa 2024-03-12T04:49:50.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tKOq1MoOZa 2024-03-12T04:49:50.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.tKOq1MoOZa 2024-03-12T04:49:50.579 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.tKOq1MoOZa is 6KB 2024-03-12T04:49:50.627 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.tKOq1MoOZa 2024-03-12T04:49:50.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T04:49:50.710 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T04:49:50.710 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T04:49:50.710 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T04:49:50.711 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T04:49:50.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T04:49:50.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-03-12T04:50:20.864 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T04:50:20.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-12T04:50:20.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:51:20.868 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T04:51:29.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-12T04:51:29.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:51:30.243 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-12T04:51:30.666 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-12T04:51:30.666 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T04:51:30.666 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-12T04:51:30.680 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-427.el9.x86_64 2024-03-12T04:51:30.680 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T04:51:30.681 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T04:51:30.681 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T04:51:31.681 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T04:51:31.682 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T04:51:31.750 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-03-12T04:51:31.784 DEBUG:teuthology.parallel:result is None 2024-03-12T04:51:31.784 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T04:51:31.792 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T04:51:31.792 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T04:51:31.794 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T04:51:31.797 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T04:51:31.815 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T04:51:31.822 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T04:51:31.833 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T04:51:31.833 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T04:51:31.845 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T04:51:31.854 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T04:51:32.043 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T04:51:32.049 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T04:51:32.049 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T04:51:32.052 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T04:51:32.054 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T04:51:32.089 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.094 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.103 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.108 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.299 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.312 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T04:51:32.314 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T04:51:32.321 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T04:51:32.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T04:51:32.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T04:51:32.326 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T04:51:32.358 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T04:51:32.367 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T04:51:32.367 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T04:51:32.389 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T04:51:32.398 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T04:51:32.415 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:32.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:32.564 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:32.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:32.674 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:51:32.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T04:51:32.716 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:32.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:32.812 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:32.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:32.939 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:51:32.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T04:51:33.007 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:33.053 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:51:33.106 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:33.189 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:51:33.237 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:51:33.237 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T04:51:33.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-03-12T04:51:33.309 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-12T04:51:33.311 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-03-12T04:51:33.354 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:51:33.379 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:51:33.404 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:51:33.791 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T04:51:33.802 INFO:teuthology.task.internal:Starting timer... 2024-03-12T04:51:33.803 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T04:51:33.837 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T04:51:33.863 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T04:51:33.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-03-12T04:51:34.034 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-03-12T04:51:34.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-12T04:51:34.103 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-12T04:51:34.105 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-03-12T04:51:34.160 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-03-12T04:51:34.162 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T04:51:34.162 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-03-12T04:51:34.196 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-03-12T04:51:34.196 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-12T04:51:34.221 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-12T04:51:34.222 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-03-12T04:51:34.244 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-03-12T04:51:34.244 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-03-12T04:51:34.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T04:51:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:51:34.271 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T04:51:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:51:34.300 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T04:51:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:51:34.330 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T04:51:34.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-03-12T04:51:34.381 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-12T04:51:34.415 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-03-12T04:51:34.446 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T04:51:34.455 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T04:51:34.476 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T04:51:34.477 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T04:55:49.013 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-03-12T04:55:49.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-12T04:55:49.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:55:49.108 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-12T04:55:49.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-12T04:55:49.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-12T04:55:49.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:55:49.290 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-12T04:55:49.368 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-12T04:55:49.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-12T04:55:49.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:55:49.466 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-12T04:55:49.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-12T04:55:49.546 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T04:55:49.557 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T04:55:49.557 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T04:55:49.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:55:49.560 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T04:55:49.560 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:55:49.563 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T04:55:49.563 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:55:49.591 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T04:55:49.592 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T04:55:49.608 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T04:55:49.610 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T04:55:49.618 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T04:55:49.635 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T04:55:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-12T04:55:49.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-03-12T04:55:49.650 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-12T04:55:49.655 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-03-12T04:55:49.665 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-03-12T04:55:49.666 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T04:55:49.671 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T04:55:49.678 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-03-12T04:55:49.683 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T04:55:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T04:55:49.695 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T04:55:49.711 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T04:55:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:55:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.740 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.761 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:55:49.765 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:55:49.765 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-12T04:55:49.765 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.765 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.765 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.766 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:55:49.766 INFO:teuthology.run_tasks:Running task install... 2024-03-12T04:55:49.774 DEBUG:teuthology.task.install:project ceph 2024-03-12T04:55:49.774 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-12T04:55:49.774 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-12T04:55:49.774 INFO:teuthology.task.install:Using flavor: default 2024-03-12T04:55:49.781 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-12T04:55:49.781 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-12T04:55:49.781 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T04:55:49.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:55:49.784 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T04:55:49.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:55:49.785 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T04:55:49.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:55:49.941 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T04:55:49.942 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T04:55:49.946 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T04:55:49.946 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T04:55:49.950 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T04:55:49.950 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T04:55:50.061 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T04:55:50.061 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:55:50.061 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T04:55:50.089 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T04:55:50.090 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T04:55:50.121 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T04:55:50.121 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:55:50.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T04:55:50.125 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T04:55:50.125 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:55:50.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T04:55:50.149 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T04:55:50.149 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T04:55:50.154 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T04:55:50.154 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T04:55:50.158 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T04:55:50.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T04:55:50.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T04:55:50.238 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T04:55:50.296 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T04:55:50.302 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T04:55:50.303 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-03-12T04:55:50.305 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-03-12T04:55:50.367 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-03-12T04:55:50.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-03-12T04:55:50.368 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-12T04:55:50.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-12T04:55:50.539 INFO:teuthology.orchestra.run.smithi130.stdout:74 files removed 2024-03-12T04:55:50.566 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T04:55:50.615 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-03-12T04:55:50.619 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-12T04:55:50.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T04:55:50.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T04:55:51.231 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 235 kB/s | 82 kB 00:00 2024-03-12T04:55:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 201 kB/s | 82 kB 00:00 2024-03-12T04:55:51.383 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 196 kB/s | 82 kB 00:00 2024-03-12T04:55:51.656 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-12T04:55:51.798 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-12T04:55:51.808 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-12T04:55:51.998 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-12T04:55:52.168 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-12T04:55:52.233 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-12T04:55:52.451 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-12T04:55:52.685 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-12T04:55:52.979 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-03-12T04:55:53.893 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-12T04:55:54.160 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-12T04:55:54.537 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-12T04:55:55.670 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-12T04:55:56.442 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-12T04:56:01.558 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-12T04:56:03.659 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-12T04:56:09.726 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-12T04:56:11.585 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-12T04:56:13.175 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-12T04:56:18.600 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-12T04:56:19.809 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 838 kB/s | 19 MB 00:23 2024-03-12T04:56:20.385 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-12T04:56:22.444 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T04:56:23.977 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-12T04:56:25.595 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-12T04:56:27.361 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-12T04:56:29.847 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:56:29.848 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:56:29.860 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T04:56:29.861 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T04:56:29.862 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T04:56:29.863 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T04:56:29.865 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T04:56:29.867 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T04:56:29.868 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T04:56:29.950 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-12T04:56:29.977 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-12T04:56:29.977 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-12T04:56:29.977 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-12T04:56:29.977 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-12T04:56:29.977 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T04:56:29.978 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T04:56:29.979 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T04:56:29.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T04:56:29.981 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T04:56:29.982 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T04:56:29.983 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T04:56:29.984 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T04:56:29.985 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T04:56:29.986 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T04:56:29.986 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T04:56:29.986 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T04:56:29.987 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T04:56:29.988 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T04:56:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T04:56:29.990 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T04:56:29.990 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T04:56:29.990 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T04:56:29.990 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T04:56:29.990 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T04:56:29.991 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T04:56:29.991 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T04:56:29.991 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T04:56:29.991 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T04:56:29.992 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T04:56:29.992 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T04:56:29.992 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T04:56:29.992 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T04:56:29.992 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T04:56:29.993 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T04:56:29.994 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T04:56:29.994 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T04:56:29.994 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T04:56:29.994 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T04:56:29.994 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T04:56:29.995 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T04:56:29.995 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T04:56:29.995 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T04:56:29.995 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T04:56:29.995 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T04:56:29.996 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T04:56:29.997 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T04:56:29.998 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 24 Packages 2024-03-12T04:56:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:56:30.000 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 223 M 2024-03-12T04:56:30.000 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-12T04:56:30.512 INFO:teuthology.orchestra.run.smithi116.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-12T04:56:30.796 INFO:teuthology.orchestra.run.smithi116.stdout:(2/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 10 MB/s | 5.3 MB 00:00 2024-03-12T04:56:30.846 INFO:teuthology.orchestra.run.smithi116.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.9 MB/s | 145 kB 00:00 2024-03-12T04:56:30.996 INFO:teuthology.orchestra.run.smithi116.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 14 MB/s | 2.1 MB 00:00 2024-03-12T04:56:31.063 INFO:teuthology.orchestra.run.smithi116.stdout:(5/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.6 MB/s | 886 kB 00:00 2024-03-12T04:56:31.097 INFO:teuthology.orchestra.run.smithi116.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 15 MB/s | 1.5 MB 00:00 2024-03-12T04:56:31.997 INFO:teuthology.orchestra.run.smithi116.stdout:(7/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 19 MB/s | 17 MB 00:00 2024-03-12T04:56:32.648 INFO:teuthology.orchestra.run.smithi116.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 15 MB/s | 9.8 MB 00:00 2024-03-12T04:56:32.714 INFO:teuthology.orchestra.run.smithi116.stdout:(9/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-12T04:56:32.739 INFO:teuthology.orchestra.run.smithi116.stdout:(10/174): ceph-selinux-19.0.0-1607.g94a53970.el 274 kB/s | 25 kB 00:00 2024-03-12T04:56:32.815 INFO:teuthology.orchestra.run.smithi116.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.8 MB/s | 217 kB 00:00 2024-03-12T04:56:32.898 INFO:teuthology.orchestra.run.smithi116.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 324 kB/s | 27 kB 00:00 2024-03-12T04:56:33.001 INFO:teuthology.orchestra.run.smithi116.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 312 kB/s | 31 kB 00:00 2024-03-12T04:56:33.749 INFO:teuthology.orchestra.run.smithi116.stdout:(14/174): ceph-common-19.0.0-1607.g94a53970.el9 5.7 MB/s | 20 MB 00:03 2024-03-12T04:56:33.774 INFO:teuthology.orchestra.run.smithi116.stdout:(15/174): libcephfs2-19.0.0-1607.g94a53970.el9. 958 kB/s | 742 kB 00:00 2024-03-12T04:56:33.833 INFO:teuthology.orchestra.run.smithi116.stdout:(16/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.9 MB/s | 163 kB 00:00 2024-03-12T04:56:33.858 INFO:teuthology.orchestra.run.smithi116.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 1.5 MB/s | 127 kB 00:00 2024-03-12T04:56:33.925 INFO:teuthology.orchestra.run.smithi116.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 5.3 MB/s | 498 kB 00:00 2024-03-12T04:56:34.009 INFO:teuthology.orchestra.run.smithi116.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 541 kB/s | 45 kB 00:00 2024-03-12T04:56:34.100 INFO:teuthology.orchestra.run.smithi116.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.4 MB/s | 131 kB 00:00 2024-03-12T04:56:34.393 INFO:teuthology.orchestra.run.smithi116.stdout:(21/174): librgw2-19.0.0-1607.g94a53970.el9.x86 9.6 MB/s | 5.1 MB 00:00 2024-03-12T04:56:34.476 INFO:teuthology.orchestra.run.smithi116.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.8 MB/s | 320 kB 00:00 2024-03-12T04:56:34.551 INFO:teuthology.orchestra.run.smithi116.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.9 MB/s | 298 kB 00:00 2024-03-12T04:56:34.628 INFO:teuthology.orchestra.run.smithi116.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-12T04:56:34.652 INFO:teuthology.orchestra.run.smithi116.stdout:(25/174): python3-cephfs-19.0.0-1607.g94a53970. 293 kB/s | 161 kB 00:00 2024-03-12T04:56:34.702 INFO:teuthology.orchestra.run.smithi116.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-12T04:56:34.794 INFO:teuthology.orchestra.run.smithi116.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-12T04:56:34.878 INFO:teuthology.orchestra.run.smithi116.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 377 kB/s | 31 kB 00:00 2024-03-12T04:56:34.969 INFO:teuthology.orchestra.run.smithi116.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.5 MB/s | 138 kB 00:00 2024-03-12T04:56:35.345 INFO:teuthology.orchestra.run.smithi116.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 4.7 MB/s | 1.8 MB 00:00 2024-03-12T04:56:35.403 INFO:teuthology.orchestra.run.smithi116.stdout:(31/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 4.1 MB/s | 3.1 MB 00:00 2024-03-12T04:56:35.520 INFO:teuthology.orchestra.run.smithi116.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.1 MB/s | 247 kB 00:00 2024-03-12T04:56:35.621 INFO:teuthology.orchestra.run.smithi116.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 492 kB/s | 49 kB 00:00 2024-03-12T04:56:35.721 INFO:teuthology.orchestra.run.smithi116.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 153 kB/s | 15 kB 00:00 2024-03-12T04:56:35.838 INFO:teuthology.orchestra.run.smithi116.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 2.2 MB/s | 264 kB 00:00 2024-03-12T04:56:36.014 INFO:teuthology.orchestra.run.smithi116.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-12T04:56:36.080 INFO:teuthology.orchestra.run.smithi116.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 274 kB/s | 18 kB 00:00 2024-03-12T04:56:36.222 INFO:teuthology.orchestra.run.smithi116.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-03-12T04:56:36.381 INFO:teuthology.orchestra.run.smithi116.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-12T04:56:36.490 INFO:teuthology.orchestra.run.smithi116.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-12T04:56:36.515 INFO:teuthology.orchestra.run.smithi116.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 796 kB/s | 106 kB 00:00 2024-03-12T04:56:36.549 INFO:teuthology.orchestra.run.smithi116.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-12T04:56:36.574 INFO:teuthology.orchestra.run.smithi116.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 543 kB/s | 32 kB 00:00 2024-03-12T04:56:36.766 INFO:teuthology.orchestra.run.smithi116.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-12T04:56:36.791 INFO:teuthology.orchestra.run.smithi116.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 79 kB/s | 17 kB 00:00 2024-03-12T04:56:36.833 INFO:teuthology.orchestra.run.smithi116.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-12T04:56:36.858 INFO:teuthology.orchestra.run.smithi116.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-12T04:56:36.883 INFO:teuthology.orchestra.run.smithi116.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-12T04:56:36.925 INFO:teuthology.orchestra.run.smithi116.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-12T04:56:36.951 INFO:teuthology.orchestra.run.smithi116.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-12T04:56:36.976 INFO:teuthology.orchestra.run.smithi116.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 46 kB 00:00 2024-03-12T04:56:37.001 INFO:teuthology.orchestra.run.smithi116.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-12T04:56:37.018 INFO:teuthology.orchestra.run.smithi116.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-12T04:56:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-12T04:56:37.068 INFO:teuthology.orchestra.run.smithi116.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-12T04:56:37.236 INFO:teuthology.orchestra.run.smithi116.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-12T04:56:37.277 INFO:teuthology.orchestra.run.smithi116.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.3 MB/s | 1.5 MB 00:00 2024-03-12T04:56:37.311 INFO:teuthology.orchestra.run.smithi116.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-12T04:56:37.336 INFO:teuthology.orchestra.run.smithi116.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-12T04:56:37.478 INFO:teuthology.orchestra.run.smithi116.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-12T04:56:37.512 INFO:teuthology.orchestra.run.smithi116.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-12T04:56:37.537 INFO:teuthology.orchestra.run.smithi116.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-12T04:56:37.671 INFO:teuthology.orchestra.run.smithi116.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-12T04:56:37.696 INFO:teuthology.orchestra.run.smithi116.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-12T04:56:37.738 INFO:teuthology.orchestra.run.smithi116.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-12T04:56:37.763 INFO:teuthology.orchestra.run.smithi116.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 271 kB/s | 18 kB 00:00 2024-03-12T04:56:37.789 INFO:teuthology.orchestra.run.smithi116.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 629 kB/s | 32 kB 00:00 2024-03-12T04:56:37.897 INFO:teuthology.orchestra.run.smithi116.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-12T04:56:37.922 INFO:teuthology.orchestra.run.smithi116.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-12T04:56:37.948 INFO:teuthology.orchestra.run.smithi116.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-12T04:56:37.973 INFO:teuthology.orchestra.run.smithi116.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-12T04:56:37.998 INFO:teuthology.orchestra.run.smithi116.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-12T04:56:38.024 INFO:teuthology.orchestra.run.smithi116.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-12T04:56:38.049 INFO:teuthology.orchestra.run.smithi116.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-12T04:56:38.074 INFO:teuthology.orchestra.run.smithi116.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-12T04:56:38.100 INFO:teuthology.orchestra.run.smithi116.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-12T04:56:38.125 INFO:teuthology.orchestra.run.smithi116.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 853 kB/s | 43 kB 00:00 2024-03-12T04:56:38.275 INFO:teuthology.orchestra.run.smithi116.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 262 kB/s | 39 kB 00:00 2024-03-12T04:56:38.442 INFO:teuthology.orchestra.run.smithi116.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-12T04:56:38.467 INFO:teuthology.orchestra.run.smithi116.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-03-12T04:56:38.493 INFO:teuthology.orchestra.run.smithi116.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-12T04:56:38.518 INFO:teuthology.orchestra.run.smithi116.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-12T04:56:38.543 INFO:teuthology.orchestra.run.smithi116.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-12T04:56:38.569 INFO:teuthology.orchestra.run.smithi116.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-12T04:56:38.594 INFO:teuthology.orchestra.run.smithi116.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-12T04:56:38.619 INFO:teuthology.orchestra.run.smithi116.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T04:56:38.653 INFO:teuthology.orchestra.run.smithi116.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-12T04:56:38.678 INFO:teuthology.orchestra.run.smithi116.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-12T04:56:38.703 INFO:teuthology.orchestra.run.smithi116.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-12T04:56:38.728 INFO:teuthology.orchestra.run.smithi116.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-12T04:56:39.237 INFO:teuthology.orchestra.run.smithi116.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 351 kB/s | 187 kB 00:00 2024-03-12T04:56:39.421 INFO:teuthology.orchestra.run.smithi116.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-12T04:56:39.521 INFO:teuthology.orchestra.run.smithi116.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-12T04:56:39.538 INFO:teuthology.orchestra.run.smithi116.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-12T04:56:39.630 INFO:teuthology.orchestra.run.smithi116.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-12T04:56:39.656 INFO:teuthology.orchestra.run.smithi116.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 218 kB/s | 25 kB 00:00 2024-03-12T04:56:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-12T04:56:39.881 INFO:teuthology.orchestra.run.smithi116.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-12T04:56:40.006 INFO:teuthology.orchestra.run.smithi116.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.7 MB/s | 216 kB 00:00 2024-03-12T04:56:40.115 INFO:teuthology.orchestra.run.smithi116.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 322 kB/s | 35 kB 00:00 2024-03-12T04:56:40.223 INFO:teuthology.orchestra.run.smithi116.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-12T04:56:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 865 kB/s | 94 kB 00:00 2024-03-12T04:56:40.424 INFO:teuthology.orchestra.run.smithi116.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-12T04:56:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-12T04:56:41.091 INFO:teuthology.orchestra.run.smithi116.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.8 MB/s | 4.1 MB 00:01 2024-03-12T04:56:41.483 INFO:teuthology.orchestra.run.smithi116.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 765 kB/s | 548 kB 00:00 2024-03-12T04:56:41.600 INFO:teuthology.orchestra.run.smithi116.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-12T04:56:41.700 INFO:teuthology.orchestra.run.smithi116.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 507 kB/s | 309 kB 00:00 2024-03-12T04:56:41.834 INFO:teuthology.orchestra.run.smithi116.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-12T04:56:41.984 INFO:teuthology.orchestra.run.smithi116.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 324 kB/s | 49 kB 00:00 2024-03-12T04:56:42.118 INFO:teuthology.orchestra.run.smithi116.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-03-12T04:56:42.268 INFO:teuthology.orchestra.run.smithi116.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-12T04:56:42.668 INFO:teuthology.orchestra.run.smithi116.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-12T04:56:42.735 INFO:teuthology.orchestra.run.smithi116.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-03-12T04:56:43.244 INFO:teuthology.orchestra.run.smithi116.stdout:(115/174): ceph-test-19.0.0-1607.g94a53970.el9. 4.5 MB/s | 47 MB 00:10 2024-03-12T04:56:43.269 INFO:teuthology.orchestra.run.smithi116.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 81 kB/s | 43 kB 00:00 2024-03-12T04:56:43.302 INFO:teuthology.orchestra.run.smithi116.stdout:(117/174): python3-asyncssh-2.13.2-4.el9.noarch 864 kB/s | 548 kB 00:00 2024-03-12T04:56:43.386 INFO:teuthology.orchestra.run.smithi116.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-12T04:56:43.428 INFO:teuthology.orchestra.run.smithi116.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-12T04:56:43.562 INFO:teuthology.orchestra.run.smithi116.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-12T04:56:43.595 INFO:teuthology.orchestra.run.smithi116.stdout:(121/174): python3-cachetools-4.2.4-1.el9.noarc 92 kB/s | 32 kB 00:00 2024-03-12T04:56:43.621 INFO:teuthology.orchestra.run.smithi116.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-03-12T04:56:43.679 INFO:teuthology.orchestra.run.smithi116.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 300 kB/s | 35 kB 00:00 2024-03-12T04:56:43.805 INFO:teuthology.orchestra.run.smithi116.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-12T04:56:43.913 INFO:teuthology.orchestra.run.smithi116.stdout:(125/174): python3-google-auth-2.28.0-1.el9.noa 638 kB/s | 203 kB 00:00 2024-03-12T04:56:43.930 INFO:teuthology.orchestra.run.smithi116.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-03-12T04:56:44.022 INFO:teuthology.orchestra.run.smithi116.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-12T04:56:44.047 INFO:teuthology.orchestra.run.smithi116.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-12T04:56:44.131 INFO:teuthology.orchestra.run.smithi116.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-03-12T04:56:44.148 INFO:teuthology.orchestra.run.smithi116.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-03-12T04:56:44.165 INFO:teuthology.orchestra.run.smithi116.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-12T04:56:44.265 INFO:teuthology.orchestra.run.smithi116.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-03-12T04:56:44.290 INFO:teuthology.orchestra.run.smithi116.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 629 kB/s | 79 kB 00:00 2024-03-12T04:56:44.358 INFO:teuthology.orchestra.run.smithi116.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-03-12T04:56:44.374 INFO:teuthology.orchestra.run.smithi116.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-03-12T04:56:44.466 INFO:teuthology.orchestra.run.smithi116.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-12T04:56:44.492 INFO:teuthology.orchestra.run.smithi116.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 769 kB/s | 90 kB 00:00 2024-03-12T04:56:44.517 INFO:teuthology.orchestra.run.smithi116.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-12T04:56:44.575 INFO:teuthology.orchestra.run.smithi116.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-12T04:56:44.601 INFO:teuthology.orchestra.run.smithi116.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-12T04:56:44.634 INFO:teuthology.orchestra.run.smithi116.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-12T04:56:44.693 INFO:teuthology.orchestra.run.smithi116.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-12T04:56:44.718 INFO:teuthology.orchestra.run.smithi116.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-12T04:56:44.752 INFO:teuthology.orchestra.run.smithi116.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 435 kB/s | 51 kB 00:00 2024-03-12T04:56:44.810 INFO:teuthology.orchestra.run.smithi116.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-12T04:56:44.835 INFO:teuthology.orchestra.run.smithi116.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 765 kB/s | 90 kB 00:00 2024-03-12T04:56:44.927 INFO:teuthology.orchestra.run.smithi116.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-12T04:56:44.953 INFO:teuthology.orchestra.run.smithi116.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-12T04:56:44.986 INFO:teuthology.orchestra.run.smithi116.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.8 MB/s | 427 kB 00:00 2024-03-12T04:56:45.245 INFO:teuthology.orchestra.run.smithi116.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-03-12T04:56:45.304 INFO:teuthology.orchestra.run.smithi116.stdout:(151/174): librbd1-19.0.0-1607.g94a53970.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-03-12T04:56:45.337 INFO:teuthology.orchestra.run.smithi116.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-03-12T04:56:45.646 INFO:teuthology.orchestra.run.smithi116.stdout:(153/174): librados2-19.0.0-1607.g94a53970.el9. 4.9 MB/s | 3.4 MB 00:00 2024-03-12T04:56:46.013 INFO:teuthology.orchestra.run.smithi116.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 115 kB/s | 42 kB 00:00 2024-03-12T04:56:46.263 INFO:teuthology.orchestra.run.smithi116.stdout:(155/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.4 MB/s | 2.3 MB 00:00 2024-03-12T04:56:46.325 INFO:teuthology.orchestra.run.smithi116.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 1.8 MB/s | 1.8 MB 00:00 2024-03-12T04:56:46.388 INFO:teuthology.orchestra.run.smithi116.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 570 kB/s | 71 kB 00:00 2024-03-12T04:56:46.447 INFO:teuthology.orchestra.run.smithi116.stdout:(158/174): NetworkManager-tui-1.46.0-1.el9.x86_ 572 kB/s | 248 kB 00:00 2024-03-12T04:56:46.472 INFO:teuthology.orchestra.run.smithi116.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 248 kB/s | 36 kB 00:00 2024-03-12T04:56:46.514 INFO:teuthology.orchestra.run.smithi116.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-12T04:56:46.598 INFO:teuthology.orchestra.run.smithi116.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 184 kB/s | 23 kB 00:00 2024-03-12T04:56:46.632 INFO:teuthology.orchestra.run.smithi116.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 926 kB/s | 108 kB 00:00 2024-03-12T04:56:46.732 INFO:teuthology.orchestra.run.smithi116.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-12T04:56:46.757 INFO:teuthology.orchestra.run.smithi116.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-12T04:56:46.782 INFO:teuthology.orchestra.run.smithi116.stdout:(165/174): elfutils-libs-0.190-2.el9.x86_64.rpm 763 kB/s | 256 kB 00:00 2024-03-12T04:56:46.966 INFO:teuthology.orchestra.run.smithi116.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 273 kB/s | 64 kB 00:00 2024-03-12T04:56:46.991 INFO:teuthology.orchestra.run.smithi116.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 123 kB/s | 29 kB 00:00 2024-03-12T04:56:47.216 INFO:teuthology.orchestra.run.smithi116.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-12T04:56:47.375 INFO:teuthology.orchestra.run.smithi116.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.0 MB/s | 2.3 MB 00:00 2024-03-12T04:56:47.400 INFO:teuthology.orchestra.run.smithi116.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 465 kB 00:00 2024-03-12T04:56:47.500 INFO:teuthology.orchestra.run.smithi116.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 248 kB/s | 31 kB 00:00 2024-03-12T04:56:47.551 INFO:teuthology.orchestra.run.smithi116.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-12T04:56:47.626 INFO:teuthology.orchestra.run.smithi116.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-12T04:56:47.801 INFO:teuthology.orchestra.run.smithi116.stdout:(174/174): openssl-3.0.7-27.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:01 2024-03-12T04:56:47.805 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:56:47.805 INFO:teuthology.orchestra.run.smithi116.stdout:Total 13 MB/s | 223 MB 00:17 2024-03-12T04:56:48.262 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-12T04:56:48.570 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-12T04:56:48.570 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-12T04:56:50.622 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-12T04:56:50.622 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-12T04:56:57.552 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-12T04:56:57.627 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T04:56:57.662 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T04:56:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:56:57.710 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:56:57.750 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T04:56:57.804 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T04:56:57.837 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T04:56:57.871 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T04:56:57.923 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T04:56:58.258 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T04:56:58.322 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:56:58.427 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:56:58.456 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:56:58.547 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:56:58.582 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:56:58.625 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T04:56:58.807 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T04:56:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T04:56:58.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:56:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:56:59.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:56:59.069 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:56:59.117 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T04:56:59.140 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:56:59.155 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:56:59.348 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:56:59.386 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:56:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T04:56:59.619 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T04:56:59.747 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:56:59.799 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:56:59.907 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:56:59.986 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:57:00.211 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:57:00.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T04:57:00.426 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T04:57:00.455 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T04:57:00.482 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T04:57:00.517 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T04:57:00.591 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T04:57:00.645 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T04:57:00.692 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T04:57:00.756 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T04:57:00.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T04:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T04:57:00.869 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T04:57:00.897 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T04:57:00.932 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 428 kB/s | 21 MB 00:49 2024-03-12T04:57:00.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T04:57:01.002 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T04:57:01.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T04:57:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T04:57:01.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T04:57:01.205 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T04:57:01.278 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T04:57:01.419 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T04:57:01.474 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T04:57:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T04:57:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T04:57:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T04:57:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:57:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:57:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T04:57:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T04:57:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T04:57:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T04:57:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T04:57:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T04:57:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T04:57:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T04:57:03.169 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T04:57:03.228 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T04:57:03.282 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T04:57:03.355 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T04:57:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T04:57:03.423 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T04:57:03.470 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T04:57:03.506 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T04:57:03.549 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T04:57:03.591 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T04:57:03.725 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T04:57:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T04:57:03.819 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T04:57:03.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T04:57:03.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T04:57:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T04:57:04.018 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T04:57:04.043 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T04:57:04.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T04:57:04.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T04:57:04.284 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T04:57:04.384 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T04:57:04.425 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T04:57:04.474 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T04:57:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T04:57:04.721 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T04:57:04.835 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T04:57:05.118 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T04:57:05.245 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T04:57:05.284 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T04:57:05.324 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T04:57:05.353 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T04:57:05.386 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T04:57:05.425 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T04:57:05.456 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T04:57:05.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T04:57:05.562 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T04:57:05.597 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T04:57:05.630 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T04:57:05.671 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T04:57:05.714 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T04:57:05.802 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T04:57:05.806 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-12T04:57:05.893 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T04:57:05.937 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T04:57:05.969 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T04:57:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T04:57:06.112 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T04:57:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T04:57:06.881 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T04:57:06.942 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T04:57:06.976 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T04:57:07.024 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T04:57:07.120 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T04:57:07.160 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T04:57:07.336 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T04:57:07.366 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T04:57:07.443 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T04:57:08.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T04:57:08.117 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 497 kB/s | 21 MB 00:42 2024-03-12T04:57:08.204 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T04:57:08.297 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:57:08.297 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:57:08.311 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T04:57:08.312 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T04:57:08.313 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T04:57:08.314 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T04:57:08.316 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T04:57:08.319 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T04:57:08.320 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T04:57:08.400 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-12T04:57:08.425 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-12T04:57:08.425 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T04:57:08.426 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T04:57:08.427 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:57:08.428 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T04:57:08.429 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T04:57:08.430 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T04:57:08.431 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T04:57:08.432 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T04:57:08.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T04:57:08.434 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T04:57:08.435 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T04:57:08.436 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T04:57:08.437 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T04:57:08.438 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:Install 150 Packages 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 24 Packages 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 223 M 2024-03-12T04:57:08.439 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-12T04:57:09.423 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T04:57:09.508 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T04:57:09.539 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T04:57:09.570 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T04:57:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T04:57:09.800 INFO:teuthology.orchestra.run.smithi130.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-12T04:57:09.904 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T04:57:09.936 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:57:09.991 INFO:teuthology.orchestra.run.smithi130.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 4.5 MB/s | 886 kB 00:00 2024-03-12T04:57:10.009 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:57:10.033 INFO:teuthology.orchestra.run.smithi130.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 3.4 MB/s | 145 kB 00:00 2024-03-12T04:57:10.040 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T04:57:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T04:57:10.242 INFO:teuthology.orchestra.run.smithi130.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 10 MB/s | 2.1 MB 00:00 2024-03-12T04:57:10.401 INFO:teuthology.orchestra.run.smithi130.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-12T04:57:10.533 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T04:57:10.564 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:57:10.601 INFO:teuthology.orchestra.run.smithi130.stdout:(6/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 4.3 MB/s | 5.3 MB 00:01 2024-03-12T04:57:10.634 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:57:10.657 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T04:57:10.909 INFO:teuthology.orchestra.run.smithi130.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 9.2 MB/s | 4.7 MB 00:00 2024-03-12T04:57:11.968 INFO:teuthology.orchestra.run.smithi130.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 9.2 MB/s | 9.8 MB 00:01 2024-03-12T04:57:12.026 INFO:teuthology.orchestra.run.smithi130.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 433 kB/s | 25 kB 00:00 2024-03-12T04:57:12.668 INFO:teuthology.orchestra.run.smithi130.stdout:(10/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 8.1 MB/s | 17 MB 00:02 2024-03-12T04:57:12.760 INFO:teuthology.orchestra.run.smithi130.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.3 MB/s | 217 kB 00:00 2024-03-12T04:57:12.797 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:12.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:12.843 INFO:teuthology.orchestra.run.smithi130.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 325 kB/s | 27 kB 00:00 2024-03-12T04:57:12.919 INFO:teuthology.orchestra.run.smithi130.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 416 kB/s | 31 kB 00:00 2024-03-12T04:57:13.202 INFO:teuthology.orchestra.run.smithi130.stdout:(14/174): ceph-common-19.0.0-1607.g94a53970.el9 5.2 MB/s | 20 MB 00:03 2024-03-12T04:57:13.217 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-12T04:57:13.227 INFO:teuthology.orchestra.run.smithi130.stdout:(15/174): libcephfs2-19.0.0-1607.g94a53970.el9. 2.4 MB/s | 742 kB 00:00 2024-03-12T04:57:13.294 INFO:teuthology.orchestra.run.smithi130.stdout:(16/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.7 MB/s | 163 kB 00:00 2024-03-12T04:57:13.320 INFO:teuthology.orchestra.run.smithi130.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 1.3 MB/s | 127 kB 00:00 2024-03-12T04:57:13.390 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:13.423 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:57:13.445 INFO:teuthology.orchestra.run.smithi130.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 3.2 MB/s | 498 kB 00:00 2024-03-12T04:57:13.514 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:57:13.514 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T04:57:13.514 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T04:57:13.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:13.528 INFO:teuthology.orchestra.run.smithi130.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 543 kB/s | 45 kB 00:00 2024-03-12T04:57:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:57:13.612 INFO:teuthology.orchestra.run.smithi130.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.5 MB/s | 131 kB 00:00 2024-03-12T04:57:13.704 INFO:teuthology.orchestra.run.smithi130.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.7 MB/s | 161 kB 00:00 2024-03-12T04:57:13.796 INFO:teuthology.orchestra.run.smithi130.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.4 MB/s | 320 kB 00:00 2024-03-12T04:57:13.930 INFO:teuthology.orchestra.run.smithi130.stdout:(23/174): librgw2-19.0.0-1607.g94a53970.el9.x86 8.4 MB/s | 5.1 MB 00:00 2024-03-12T04:57:13.955 INFO:teuthology.orchestra.run.smithi130.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 1.8 MB/s | 298 kB 00:00 2024-03-12T04:57:13.997 INFO:teuthology.orchestra.run.smithi130.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.5 MB/s | 99 kB 00:00 2024-03-12T04:57:14.022 INFO:teuthology.orchestra.run.smithi130.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-12T04:57:14.115 INFO:teuthology.orchestra.run.smithi130.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-12T04:57:14.198 INFO:teuthology.orchestra.run.smithi130.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 378 kB/s | 31 kB 00:00 2024-03-12T04:57:14.290 INFO:teuthology.orchestra.run.smithi130.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.5 MB/s | 138 kB 00:00 2024-03-12T04:57:14.407 INFO:teuthology.orchestra.run.smithi130.stdout:(30/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 7.5 MB/s | 3.1 MB 00:00 2024-03-12T04:57:14.682 INFO:teuthology.orchestra.run.smithi130.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 4.5 MB/s | 1.8 MB 00:00 2024-03-12T04:57:14.791 INFO:teuthology.orchestra.run.smithi130.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.2 MB/s | 247 kB 00:00 2024-03-12T04:57:14.874 INFO:teuthology.orchestra.run.smithi130.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 593 kB/s | 49 kB 00:00 2024-03-12T04:57:14.966 INFO:teuthology.orchestra.run.smithi130.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 167 kB/s | 15 kB 00:00 2024-03-12T04:57:15.067 INFO:teuthology.orchestra.run.smithi130.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 2.6 MB/s | 264 kB 00:00 2024-03-12T04:57:15.267 INFO:teuthology.orchestra.run.smithi130.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 3.7 MB/s | 748 kB 00:00 2024-03-12T04:57:15.692 INFO:teuthology.orchestra.run.smithi130.stdout:(37/174): ceph-test-19.0.0-1607.g94a53970.el9.x 13 MB/s | 47 MB 00:03 2024-03-12T04:57:15.718 INFO:teuthology.orchestra.run.smithi130.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 40 kB/s | 18 kB 00:00 2024-03-12T04:57:15.801 INFO:teuthology.orchestra.run.smithi130.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-03-12T04:57:15.805 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:57:15.807 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T04:57:15.821 INFO:teuthology.orchestra.run.smithi045.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T04:57:15.822 INFO:teuthology.orchestra.run.smithi045.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T04:57:15.823 INFO:teuthology.orchestra.run.smithi045.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T04:57:15.824 INFO:teuthology.orchestra.run.smithi045.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T04:57:15.826 INFO:teuthology.orchestra.run.smithi045.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T04:57:15.829 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T04:57:15.831 INFO:teuthology.orchestra.run.smithi045.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T04:57:15.915 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-12T04:57:15.935 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-12T04:57:15.935 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-12T04:57:15.935 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T04:57:15.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T04:57:15.937 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:57:15.938 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T04:57:15.939 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T04:57:15.940 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T04:57:15.941 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T04:57:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T04:57:15.943 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T04:57:15.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T04:57:15.945 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T04:57:15.946 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T04:57:15.947 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T04:57:15.948 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T04:57:15.949 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:Install 150 Packages 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 24 Packages 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 223 M 2024-03-12T04:57:15.950 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-12T04:57:15.984 INFO:teuthology.orchestra.run.smithi130.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 800 kB/s | 106 kB 00:00 2024-03-12T04:57:16.026 INFO:teuthology.orchestra.run.smithi130.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-03-12T04:57:16.060 INFO:teuthology.orchestra.run.smithi130.stdout:(42/174): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-03-12T04:57:16.094 INFO:teuthology.orchestra.run.smithi130.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 307 kB 00:00 2024-03-12T04:57:16.119 INFO:teuthology.orchestra.run.smithi130.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 344 kB/s | 32 kB 00:00 2024-03-12T04:57:16.245 INFO:teuthology.orchestra.run.smithi130.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 114 kB/s | 17 kB 00:00 2024-03-12T04:57:16.361 INFO:teuthology.orchestra.run.smithi130.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.9 MB/s | 3.0 MB 00:00 2024-03-12T04:57:16.386 INFO:teuthology.orchestra.run.smithi130.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 38 kB/s | 10 kB 00:00 2024-03-12T04:57:16.412 INFO:teuthology.orchestra.run.smithi130.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 956 kB/s | 159 kB 00:00 2024-03-12T04:57:16.429 INFO:teuthology.orchestra.run.smithi130.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-03-12T04:57:16.504 INFO:teuthology.orchestra.run.smithi130.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-12T04:57:16.529 INFO:teuthology.orchestra.run.smithi130.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 578 kB 00:00 2024-03-12T04:57:16.546 INFO:teuthology.orchestra.run.smithi130.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 396 kB/s | 46 kB 00:00 2024-03-12T04:57:16.551 INFO:teuthology.orchestra.run.smithi045.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-12T04:57:16.571 INFO:teuthology.orchestra.run.smithi130.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-12T04:57:16.597 INFO:teuthology.orchestra.run.smithi130.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-12T04:57:16.622 INFO:teuthology.orchestra.run.smithi130.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-12T04:57:16.647 INFO:teuthology.orchestra.run.smithi130.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 454 kB/s | 34 kB 00:00 2024-03-12T04:57:16.847 INFO:teuthology.orchestra.run.smithi130.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-03-12T04:57:16.881 INFO:teuthology.orchestra.run.smithi130.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 3.4 MB/s | 811 kB 00:00 2024-03-12T04:57:16.931 INFO:teuthology.orchestra.run.smithi130.stdout:(59/174): postgresql-13.11-1.el9.x86_64.rpm 5.4 MB/s | 1.5 MB 00:00 2024-03-12T04:57:16.948 INFO:teuthology.orchestra.run.smithi130.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-12T04:57:16.976 INFO:teuthology.orchestra.run.smithi045.stdout:(2/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 8.1 MB/s | 5.3 MB 00:00 2024-03-12T04:57:17.037 INFO:teuthology.orchestra.run.smithi045.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.4 MB/s | 145 kB 00:00 2024-03-12T04:57:17.098 INFO:teuthology.orchestra.run.smithi130.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-12T04:57:17.124 INFO:teuthology.orchestra.run.smithi130.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 451 kB/s | 78 kB 00:00 2024-03-12T04:57:17.152 INFO:teuthology.orchestra.run.smithi045.stdout:(4/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.4 MB/s | 886 kB 00:00 2024-03-12T04:57:17.165 INFO:teuthology.orchestra.run.smithi130.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-03-12T04:57:17.277 INFO:teuthology.orchestra.run.smithi045.stdout:(5/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 8.8 MB/s | 2.1 MB 00:00 2024-03-12T04:57:17.291 INFO:teuthology.orchestra.run.smithi130.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-12T04:57:17.316 INFO:teuthology.orchestra.run.smithi130.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-12T04:57:17.341 INFO:teuthology.orchestra.run.smithi130.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 105 kB/s | 18 kB 00:00 2024-03-12T04:57:17.383 INFO:teuthology.orchestra.run.smithi130.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 766 kB/s | 32 kB 00:00 2024-03-12T04:57:17.408 INFO:teuthology.orchestra.run.smithi130.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-12T04:57:17.502 INFO:teuthology.orchestra.run.smithi130.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-12T04:57:17.534 INFO:teuthology.orchestra.run.smithi130.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-12T04:57:17.559 INFO:teuthology.orchestra.run.smithi130.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-12T04:57:17.584 INFO:teuthology.orchestra.run.smithi130.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-12T04:57:17.609 INFO:teuthology.orchestra.run.smithi130.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 550 kB/s | 41 kB 00:00 2024-03-12T04:57:17.635 INFO:teuthology.orchestra.run.smithi130.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-12T04:57:17.661 INFO:teuthology.orchestra.run.smithi130.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-12T04:57:17.685 INFO:teuthology.orchestra.run.smithi130.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-12T04:57:17.710 INFO:teuthology.orchestra.run.smithi130.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 629 kB/s | 47 kB 00:00 2024-03-12T04:57:17.736 INFO:teuthology.orchestra.run.smithi130.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-12T04:57:17.752 INFO:teuthology.orchestra.run.smithi045.stdout:(6/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-03-12T04:57:17.953 INFO:teuthology.orchestra.run.smithi045.stdout:(7/174): ceph-common-19.0.0-1607.g94a53970.el9. 12 MB/s | 20 MB 00:01 2024-03-12T04:57:17.994 INFO:teuthology.orchestra.run.smithi130.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-12T04:57:17.995 INFO:teuthology.orchestra.run.smithi045.stdout:(8/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-03-12T04:57:18.020 INFO:teuthology.orchestra.run.smithi130.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 125 kB/s | 39 kB 00:00 2024-03-12T04:57:18.045 INFO:teuthology.orchestra.run.smithi130.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 134 kB/s | 41 kB 00:00 2024-03-12T04:57:18.070 INFO:teuthology.orchestra.run.smithi130.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-12T04:57:18.095 INFO:teuthology.orchestra.run.smithi130.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-12T04:57:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 233 kB/s | 25 kB 00:00 2024-03-12T04:57:18.120 INFO:teuthology.orchestra.run.smithi130.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2024-03-12T04:57:18.146 INFO:teuthology.orchestra.run.smithi130.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-12T04:57:18.171 INFO:teuthology.orchestra.run.smithi130.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-12T04:57:18.196 INFO:teuthology.orchestra.run.smithi130.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-12T04:57:18.238 INFO:teuthology.orchestra.run.smithi130.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-12T04:57:18.272 INFO:teuthology.orchestra.run.smithi130.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-12T04:57:18.297 INFO:teuthology.orchestra.run.smithi130.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-03-12T04:57:18.323 INFO:teuthology.orchestra.run.smithi130.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-12T04:57:18.564 INFO:teuthology.orchestra.run.smithi130.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 703 kB/s | 187 kB 00:00 2024-03-12T04:57:18.589 INFO:teuthology.orchestra.run.smithi130.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 731 kB/s | 195 kB 00:00 2024-03-12T04:57:18.640 INFO:teuthology.orchestra.run.smithi130.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-12T04:57:18.770 INFO:teuthology.orchestra.run.smithi045.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 12 MB/s | 9.8 MB 00:00 2024-03-12T04:57:18.837 INFO:teuthology.orchestra.run.smithi045.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 3.2 MB/s | 217 kB 00:00 2024-03-12T04:57:18.840 INFO:teuthology.orchestra.run.smithi130.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-12T04:57:18.907 INFO:teuthology.orchestra.run.smithi130.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 68 kB/s | 21 kB 00:00 2024-03-12T04:57:18.957 INFO:teuthology.orchestra.run.smithi130.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 80 kB/s | 25 kB 00:00 2024-03-12T04:57:18.996 INFO:teuthology.orchestra.run.smithi045.stdout:(12/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 14 MB/s | 17 MB 00:01 2024-03-12T04:57:19.021 INFO:teuthology.orchestra.run.smithi045.stdout:(13/174): cephfs-shell-19.0.0-1607.g94a53970.el 147 kB/s | 27 kB 00:00 2024-03-12T04:57:19.055 INFO:teuthology.orchestra.run.smithi045.stdout:(14/174): libcephfs-devel-19.0.0-1607.g94a53970 533 kB/s | 31 kB 00:00 2024-03-12T04:57:19.097 INFO:teuthology.orchestra.run.smithi045.stdout:(15/174): libcephfs2-19.0.0-1607.g94a53970.el9. 9.6 MB/s | 742 kB 00:00 2024-03-12T04:57:19.122 INFO:teuthology.orchestra.run.smithi045.stdout:(16/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.4 MB/s | 163 kB 00:00 2024-03-12T04:57:19.147 INFO:teuthology.orchestra.run.smithi045.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 2.5 MB/s | 127 kB 00:00 2024-03-12T04:57:19.172 INFO:teuthology.orchestra.run.smithi045.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 9.7 MB/s | 498 kB 00:00 2024-03-12T04:57:19.222 INFO:teuthology.orchestra.run.smithi045.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 904 kB/s | 45 kB 00:00 2024-03-12T04:57:19.290 INFO:teuthology.orchestra.run.smithi045.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.9 MB/s | 131 kB 00:00 2024-03-12T04:57:19.357 INFO:teuthology.orchestra.run.smithi045.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 2.4 MB/s | 161 kB 00:00 2024-03-12T04:57:19.391 INFO:teuthology.orchestra.run.smithi130.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 536 kB/s | 245 kB 00:00 2024-03-12T04:57:19.416 INFO:teuthology.orchestra.run.smithi130.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 540 kB/s | 248 kB 00:00 2024-03-12T04:57:19.465 INFO:teuthology.orchestra.run.smithi045.stdout:(22/174): librgw2-19.0.0-1607.g94a53970.el9.x86 16 MB/s | 5.1 MB 00:00 2024-03-12T04:57:19.490 INFO:teuthology.orchestra.run.smithi045.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 2.3 MB/s | 320 kB 00:00 2024-03-12T04:57:19.516 INFO:teuthology.orchestra.run.smithi045.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 5.8 MB/s | 298 kB 00:00 2024-03-12T04:57:19.566 INFO:teuthology.orchestra.run.smithi045.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-12T04:57:19.591 INFO:teuthology.orchestra.run.smithi045.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-12T04:57:19.650 INFO:teuthology.orchestra.run.smithi130.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 149 kB/s | 35 kB 00:00 2024-03-12T04:57:19.658 INFO:teuthology.orchestra.run.smithi045.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 2.5 MB/s | 171 kB 00:00 2024-03-12T04:57:19.717 INFO:teuthology.orchestra.run.smithi045.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 541 kB/s | 31 kB 00:00 2024-03-12T04:57:19.783 INFO:teuthology.orchestra.run.smithi130.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.4 MB/s | 4.1 MB 00:00 2024-03-12T04:57:19.792 INFO:teuthology.orchestra.run.smithi045.stdout:(29/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 14 MB/s | 3.1 MB 00:00 2024-03-12T04:57:19.808 INFO:teuthology.orchestra.run.smithi130.stdout:(102/174): qatlib-23.11.0-1.el9.x86_64.rpm 519 kB/s | 216 kB 00:00 2024-03-12T04:57:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 973 kB/s | 138 kB 00:00 2024-03-12T04:57:19.892 INFO:teuthology.orchestra.run.smithi130.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 271 kB/s | 22 kB 00:00 2024-03-12T04:57:19.935 INFO:teuthology.orchestra.run.smithi045.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 13 MB/s | 1.8 MB 00:00 2024-03-12T04:57:20.017 INFO:teuthology.orchestra.run.smithi130.stdout:(104/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 121 kB/s | 44 kB 00:00 2024-03-12T04:57:20.018 INFO:teuthology.orchestra.run.smithi045.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.9 MB/s | 247 kB 00:00 2024-03-12T04:57:20.077 INFO:teuthology.orchestra.run.smithi045.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 847 kB/s | 49 kB 00:00 2024-03-12T04:57:20.135 INFO:teuthology.orchestra.run.smithi045.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 264 kB/s | 15 kB 00:00 2024-03-12T04:57:20.202 INFO:teuthology.orchestra.run.smithi045.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.9 MB/s | 264 kB 00:00 2024-03-12T04:57:20.209 INFO:teuthology.orchestra.run.smithi130.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 220 kB/s | 94 kB 00:00 2024-03-12T04:57:20.251 INFO:teuthology.orchestra.run.smithi130.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-12T04:57:20.276 INFO:teuthology.orchestra.run.smithi130.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-03-12T04:57:20.294 INFO:teuthology.orchestra.run.smithi045.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 8.0 MB/s | 748 kB 00:00 2024-03-12T04:57:20.301 INFO:teuthology.orchestra.run.smithi130.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-03-12T04:57:20.352 INFO:teuthology.orchestra.run.smithi130.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-03-12T04:57:20.410 INFO:teuthology.orchestra.run.smithi130.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-03-12T04:57:20.419 INFO:teuthology.orchestra.run.smithi045.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-12T04:57:20.435 INFO:teuthology.orchestra.run.smithi130.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 196 kB/s | 26 kB 00:00 2024-03-12T04:57:20.436 INFO:teuthology.orchestra.run.smithi045.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 129 kB/s | 18 kB 00:00 2024-03-12T04:57:20.461 INFO:teuthology.orchestra.run.smithi130.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 455 kB/s | 49 kB 00:00 2024-03-12T04:57:20.486 INFO:teuthology.orchestra.run.smithi130.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-12T04:57:20.512 INFO:teuthology.orchestra.run.smithi130.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-12T04:57:20.545 INFO:teuthology.orchestra.run.smithi130.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-12T04:57:20.570 INFO:teuthology.orchestra.run.smithi130.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 763 kB/s | 43 kB 00:00 2024-03-12T04:57:20.595 INFO:teuthology.orchestra.run.smithi130.stdout:(117/174): python3-asyncssh-2.13.2-4.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-03-12T04:57:20.620 INFO:teuthology.orchestra.run.smithi045.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 662 kB 00:00 2024-03-12T04:57:20.621 INFO:teuthology.orchestra.run.smithi130.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 430 kB/s | 32 kB 00:00 2024-03-12T04:57:20.646 INFO:teuthology.orchestra.run.smithi130.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-03-12T04:57:20.662 INFO:teuthology.orchestra.run.smithi045.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-12T04:57:20.671 INFO:teuthology.orchestra.run.smithi130.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-12T04:57:20.687 INFO:teuthology.orchestra.run.smithi045.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-12T04:57:20.696 INFO:teuthology.orchestra.run.smithi130.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-12T04:57:20.712 INFO:teuthology.orchestra.run.smithi045.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-12T04:57:20.721 INFO:teuthology.orchestra.run.smithi130.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-03-12T04:57:20.738 INFO:teuthology.orchestra.run.smithi045.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 632 kB/s | 32 kB 00:00 2024-03-12T04:57:20.746 INFO:teuthology.orchestra.run.smithi130.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 712 kB/s | 35 kB 00:00 2024-03-12T04:57:20.772 INFO:teuthology.orchestra.run.smithi130.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-12T04:57:20.813 INFO:teuthology.orchestra.run.smithi045.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-12T04:57:20.822 INFO:teuthology.orchestra.run.smithi130.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-12T04:57:20.838 INFO:teuthology.orchestra.run.smithi045.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 174 kB/s | 17 kB 00:00 2024-03-12T04:57:20.848 INFO:teuthology.orchestra.run.smithi130.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 147 kB/s | 11 kB 00:00 2024-03-12T04:57:20.873 INFO:teuthology.orchestra.run.smithi130.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-12T04:57:20.880 INFO:teuthology.orchestra.run.smithi045.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 243 kB/s | 10 kB 00:00 2024-03-12T04:57:20.898 INFO:teuthology.orchestra.run.smithi130.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-12T04:57:20.906 INFO:teuthology.orchestra.run.smithi045.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-12T04:57:20.923 INFO:teuthology.orchestra.run.smithi130.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 309 kB/s | 23 kB 00:00 2024-03-12T04:57:20.931 INFO:teuthology.orchestra.run.smithi045.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-12T04:57:20.949 INFO:teuthology.orchestra.run.smithi130.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-03-12T04:57:20.956 INFO:teuthology.orchestra.run.smithi045.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-12T04:57:20.974 INFO:teuthology.orchestra.run.smithi130.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-03-12T04:57:20.990 INFO:teuthology.orchestra.run.smithi045.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-12T04:57:21.008 INFO:teuthology.orchestra.run.smithi130.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-12T04:57:21.015 INFO:teuthology.orchestra.run.smithi045.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 807 kB/s | 46 kB 00:00 2024-03-12T04:57:21.033 INFO:teuthology.orchestra.run.smithi130.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 820 kB/s | 46 kB 00:00 2024-03-12T04:57:21.040 INFO:teuthology.orchestra.run.smithi045.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-12T04:57:21.066 INFO:teuthology.orchestra.run.smithi045.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-12T04:57:21.091 INFO:teuthology.orchestra.run.smithi045.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-12T04:57:21.116 INFO:teuthology.orchestra.run.smithi045.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-12T04:57:21.118 INFO:teuthology.orchestra.run.smithi130.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 557 kB/s | 79 kB 00:00 2024-03-12T04:57:21.183 INFO:teuthology.orchestra.run.smithi130.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-03-12T04:57:21.200 INFO:teuthology.orchestra.run.smithi045.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 44 MB/s | 4.7 MB 00:00 2024-03-12T04:57:21.209 INFO:teuthology.orchestra.run.smithi130.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 188 kB/s | 16 kB 00:00 2024-03-12T04:57:21.234 INFO:teuthology.orchestra.run.smithi045.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 13 MB/s | 1.5 MB 00:00 2024-03-12T04:57:21.234 INFO:teuthology.orchestra.run.smithi130.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-12T04:57:21.259 INFO:teuthology.orchestra.run.smithi130.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-12T04:57:21.275 INFO:teuthology.orchestra.run.smithi045.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-12T04:57:21.284 INFO:teuthology.orchestra.run.smithi130.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 378 kB/s | 28 kB 00:00 2024-03-12T04:57:21.301 INFO:teuthology.orchestra.run.smithi045.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-12T04:57:21.309 INFO:teuthology.orchestra.run.smithi130.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-03-12T04:57:21.326 INFO:teuthology.orchestra.run.smithi130.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-12T04:57:21.352 INFO:teuthology.orchestra.run.smithi130.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 889 kB/s | 59 kB 00:00 2024-03-12T04:57:21.377 INFO:teuthology.orchestra.run.smithi130.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-03-12T04:57:21.393 INFO:teuthology.orchestra.run.smithi045.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 49 MB/s | 5.7 MB 00:00 2024-03-12T04:57:21.402 INFO:teuthology.orchestra.run.smithi130.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-03-12T04:57:21.427 INFO:teuthology.orchestra.run.smithi130.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-12T04:57:21.434 INFO:teuthology.orchestra.run.smithi045.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-12T04:57:21.452 INFO:teuthology.orchestra.run.smithi130.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-12T04:57:21.460 INFO:teuthology.orchestra.run.smithi045.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-12T04:57:21.486 INFO:teuthology.orchestra.run.smithi130.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-12T04:57:21.511 INFO:teuthology.orchestra.run.smithi130.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 241 kB/s | 20 kB 00:00 2024-03-12T04:57:21.537 INFO:teuthology.orchestra.run.smithi130.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-12T04:57:21.560 INFO:teuthology.orchestra.run.smithi045.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 46 MB/s | 5.8 MB 00:00 2024-03-12T04:57:21.577 INFO:teuthology.orchestra.run.smithi045.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 214 kB/s | 24 kB 00:00 2024-03-12T04:57:21.578 INFO:teuthology.orchestra.run.smithi130.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-12T04:57:21.611 INFO:teuthology.orchestra.run.smithi045.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 544 kB/s | 18 kB 00:00 2024-03-12T04:57:21.620 INFO:teuthology.orchestra.run.smithi130.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-12T04:57:21.636 INFO:teuthology.orchestra.run.smithi045.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-12T04:57:21.653 INFO:teuthology.orchestra.run.smithi045.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 761 kB/s | 32 kB 00:00 2024-03-12T04:57:21.736 INFO:teuthology.orchestra.run.smithi045.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2024-03-12T04:57:21.762 INFO:teuthology.orchestra.run.smithi045.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-12T04:57:21.787 INFO:teuthology.orchestra.run.smithi045.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-12T04:57:21.812 INFO:teuthology.orchestra.run.smithi045.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-12T04:57:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 978 kB/s | 41 kB 00:00 2024-03-12T04:57:21.855 INFO:teuthology.orchestra.run.smithi045.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-12T04:57:21.880 INFO:teuthology.orchestra.run.smithi045.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-12T04:57:21.897 INFO:teuthology.orchestra.run.smithi045.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 5.0 MB/s | 211 kB 00:00 2024-03-12T04:57:21.922 INFO:teuthology.orchestra.run.smithi045.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-12T04:57:21.948 INFO:teuthology.orchestra.run.smithi045.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 853 kB/s | 43 kB 00:00 2024-03-12T04:57:21.962 INFO:teuthology.orchestra.run.smithi130.stdout:(152/174): librados2-19.0.0-1607.g94a53970.el9. 7.5 MB/s | 3.4 MB 00:00 2024-03-12T04:57:22.027 INFO:teuthology.orchestra.run.smithi045.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 504 kB/s | 39 kB 00:00 2024-03-12T04:57:22.173 INFO:teuthology.orchestra.run.smithi045.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-03-12T04:57:22.199 INFO:teuthology.orchestra.run.smithi045.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 242 kB/s | 41 kB 00:00 2024-03-12T04:57:22.204 INFO:teuthology.orchestra.run.smithi130.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.9 MB/s | 2.3 MB 00:00 2024-03-12T04:57:22.224 INFO:teuthology.orchestra.run.smithi045.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-12T04:57:22.249 INFO:teuthology.orchestra.run.smithi045.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-12T04:57:22.254 INFO:teuthology.orchestra.run.smithi130.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 851 kB/s | 42 kB 00:00 2024-03-12T04:57:22.274 INFO:teuthology.orchestra.run.smithi045.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-12T04:57:22.300 INFO:teuthology.orchestra.run.smithi045.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-12T04:57:22.313 INFO:teuthology.orchestra.run.smithi130.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-03-12T04:57:22.325 INFO:teuthology.orchestra.run.smithi045.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-12T04:57:22.350 INFO:teuthology.orchestra.run.smithi045.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T04:57:22.363 INFO:teuthology.orchestra.run.smithi130.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-12T04:57:22.384 INFO:teuthology.orchestra.run.smithi045.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-12T04:57:22.409 INFO:teuthology.orchestra.run.smithi045.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-12T04:57:22.434 INFO:teuthology.orchestra.run.smithi045.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-12T04:57:22.438 INFO:teuthology.orchestra.run.smithi130.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 3.8 MB/s | 1.8 MB 00:00 2024-03-12T04:57:22.460 INFO:teuthology.orchestra.run.smithi045.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-12T04:57:22.464 INFO:teuthology.orchestra.run.smithi130.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 364 kB/s | 36 kB 00:00 2024-03-12T04:57:22.498 INFO:teuthology.orchestra.run.smithi130.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-12T04:57:22.531 INFO:teuthology.orchestra.run.smithi130.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-12T04:57:22.551 INFO:teuthology.orchestra.run.smithi045.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-12T04:57:22.556 INFO:teuthology.orchestra.run.smithi130.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-12T04:57:22.590 INFO:teuthology.orchestra.run.smithi130.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-12T04:57:22.615 INFO:teuthology.orchestra.run.smithi130.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-12T04:57:22.649 INFO:teuthology.orchestra.run.smithi130.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-12T04:57:22.674 INFO:teuthology.orchestra.run.smithi130.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-12T04:57:22.685 INFO:teuthology.orchestra.run.smithi045.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-12T04:57:22.708 INFO:teuthology.orchestra.run.smithi130.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 489 kB/s | 29 kB 00:00 2024-03-12T04:57:22.769 INFO:teuthology.orchestra.run.smithi045.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-12T04:57:22.794 INFO:teuthology.orchestra.run.smithi045.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-12T04:57:22.850 INFO:teuthology.orchestra.run.smithi130.stdout:(167/174): librbd1-19.0.0-1607.g94a53970.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-12T04:57:22.917 INFO:teuthology.orchestra.run.smithi130.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-03-12T04:57:23.034 INFO:teuthology.orchestra.run.smithi130.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-12T04:57:23.092 INFO:teuthology.orchestra.run.smithi130.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 2.2 MB 00:00 2024-03-12T04:57:23.118 INFO:teuthology.orchestra.run.smithi130.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 372 kB/s | 31 kB 00:00 2024-03-12T04:57:23.220 INFO:teuthology.orchestra.run.smithi045.stdout:(95/174): ceph-test-19.0.0-1607.g94a53970.el9.x 9.2 MB/s | 47 MB 00:05 2024-03-12T04:57:23.268 INFO:teuthology.orchestra.run.smithi130.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-03-12T04:57:23.385 INFO:teuthology.orchestra.run.smithi130.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 279 kB/s | 74 kB 00:00 2024-03-12T04:57:23.560 INFO:teuthology.orchestra.run.smithi130.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 2.3 MB 00:00 2024-03-12T04:57:23.567 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:57:23.567 INFO:teuthology.orchestra.run.smithi130.stdout:Total 15 MB/s | 223 MB 00:15 2024-03-12T04:57:23.845 INFO:teuthology.orchestra.run.smithi045.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 24 kB/s | 25 kB 00:01 2024-03-12T04:57:23.862 INFO:teuthology.orchestra.run.smithi045.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 20 kB/s | 21 kB 00:01 2024-03-12T04:57:24.033 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-12T04:57:24.342 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-12T04:57:24.343 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-12T04:57:24.854 INFO:teuthology.orchestra.run.smithi045.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 243 kB/s | 245 kB 00:01 2024-03-12T04:57:24.897 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:57:24.897 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-12T04:57:24.897 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-12T04:57:24.897 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-12T04:57:24.898 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-12T04:57:24.898 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-12T04:57:24.898 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-12T04:57:24.898 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-12T04:57:24.898 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:24.937 INFO:teuthology.orchestra.run.smithi045.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 231 kB/s | 248 kB 00:01 2024-03-12T04:57:25.052 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:57:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:57:25.067 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:25.067 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T04:57:25.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:25.645 INFO:teuthology.orchestra.run.smithi045.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 49 kB/s | 35 kB 00:00 2024-03-12T04:57:25.863 INFO:teuthology.orchestra.run.smithi045.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 215 kB/s | 216 kB 00:01 2024-03-12T04:57:26.338 INFO:teuthology.orchestra.run.smithi045.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 64 kB/s | 44 kB 00:00 2024-03-12T04:57:26.356 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-12T04:57:26.356 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-12T04:57:26.413 INFO:teuthology.orchestra.run.smithi045.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2024-03-12T04:57:26.538 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:57:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:26.638 INFO:teuthology.orchestra.run.smithi045.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-03-12T04:57:26.689 INFO:teuthology.orchestra.run.smithi045.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 113 kB/s | 94 kB 00:00 2024-03-12T04:57:26.781 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:57:26.809 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:57:26.809 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:26.810 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T04:57:26.810 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:57:26.810 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:57:26.810 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:27.266 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:57:27.288 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:57:27.288 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:27.288 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T04:57:27.288 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:57:27.289 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:57:27.289 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:27.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T04:57:27.752 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:57:27.783 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:57:27.814 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T04:57:27.874 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T04:57:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:57:28.128 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:57:28.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:57:29.113 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:57:29.140 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:57:29.241 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:57:29.402 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T04:57:29.434 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:57:29.456 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:57:29.456 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:29.456 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T04:57:29.456 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:57:29.457 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:57:29.457 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:29.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:57:29.511 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:57:29.561 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T04:57:29.647 INFO:teuthology.orchestra.run.smithi045.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 658 kB/s | 4.1 MB 00:06 2024-03-12T04:57:29.889 INFO:teuthology.orchestra.run.smithi045.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-03-12T04:57:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 149 kB/s | 548 kB 00:03 2024-03-12T04:57:30.409 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T04:57:30.437 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:57:30.459 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:57:30.459 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:30.459 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T04:57:30.460 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:57:30.460 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:57:30.460 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:30.504 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:57:30.526 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:57:30.526 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:30.526 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T04:57:30.526 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:30.564 INFO:teuthology.orchestra.run.smithi045.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-12T04:57:30.571 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:57:30.593 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:57:30.593 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:30.593 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T04:57:30.593 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:57:30.593 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:57:30.594 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:30.840 INFO:teuthology.orchestra.run.smithi045.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 177 kB/s | 49 kB 00:00 2024-03-12T04:57:30.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:57:30.927 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:57:30.928 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:57:30.928 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T04:57:30.928 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:57:30.928 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:57:30.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:31.223 INFO:teuthology.orchestra.run.smithi045.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 176 kB/s | 67 kB 00:00 2024-03-12T04:57:31.448 INFO:teuthology.orchestra.run.smithi045.stdout:(112/174): gperftools-libs-2.9.1-2.el9.x86_64.r 65 kB/s | 309 kB 00:04 2024-03-12T04:57:32.032 INFO:teuthology.orchestra.run.smithi045.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 187 kB/s | 151 kB 00:00 2024-03-12T04:57:33.223 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-12T04:57:33.343 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T04:57:33.382 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T04:57:33.429 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:57:33.438 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:57:33.478 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T04:57:33.532 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T04:57:33.574 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T04:57:33.599 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T04:57:33.639 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T04:57:33.971 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T04:57:33.999 INFO:teuthology.orchestra.run.smithi045.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 279 kB/s | 548 kB 00:01 2024-03-12T04:57:34.042 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:57:34.057 INFO:teuthology.orchestra.run.smithi045.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 746 kB/s | 43 kB 00:00 2024-03-12T04:57:34.116 INFO:teuthology.orchestra.run.smithi045.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-03-12T04:57:34.160 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:57:34.182 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:57:34.261 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:57:34.297 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:57:34.342 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T04:57:34.391 INFO:teuthology.orchestra.run.smithi045.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 54 kB/s | 15 kB 00:00 2024-03-12T04:57:34.521 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T04:57:34.591 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T04:57:34.612 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:57:34.715 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:57:34.738 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:57:34.803 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:57:34.842 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T04:57:34.865 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:57:34.880 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:57:35.065 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:57:35.096 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:57:35.128 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T04:57:35.295 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T04:57:35.299 INFO:teuthology.orchestra.run.smithi045.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 189 kB/s | 172 kB 00:00 2024-03-12T04:57:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:57:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T04:57:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T04:57:35.423 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:57:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T04:57:35.469 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:57:35.574 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:57:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T04:57:35.634 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:57:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T04:57:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T04:57:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T04:57:35.853 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:57:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T04:57:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T04:57:35.961 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T04:57:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T04:57:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T04:57:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T04:57:36.053 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T04:57:36.089 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T04:57:36.094 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T04:57:36.117 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T04:57:36.160 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T04:57:36.161 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T04:57:36.201 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T04:57:36.235 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T04:57:36.297 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T04:57:36.335 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T04:57:36.400 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T04:57:36.435 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T04:57:36.478 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T04:57:36.511 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T04:57:36.538 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T04:57:36.609 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T04:57:36.644 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T04:57:36.687 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T04:57:36.715 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T04:57:36.762 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:57:36.785 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T04:57:36.801 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:57:36.820 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:57:36.838 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T04:57:36.910 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T04:57:37.045 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T04:57:37.098 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T04:57:37.149 INFO:teuthology.orchestra.run.smithi045.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 194 kB/s | 358 kB 00:01 2024-03-12T04:57:37.620 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T04:57:37.657 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T04:57:37.697 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T04:57:37.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T04:57:37.777 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T04:57:37.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T04:57:37.824 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:57:37.834 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:57:37.866 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T04:57:37.869 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T04:57:37.905 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T04:57:37.907 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T04:57:37.933 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T04:57:37.942 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T04:57:37.958 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T04:57:37.973 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T04:57:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T04:57:37.984 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:57:38.072 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:57:38.102 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T04:57:38.102 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:57:38.186 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:57:38.212 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T04:57:38.246 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T04:57:38.248 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T04:57:38.279 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T04:57:38.304 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T04:57:38.338 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T04:57:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T04:57:38.418 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T04:57:38.422 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T04:57:38.443 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T04:57:38.468 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T04:57:38.493 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:57:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T04:57:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:57:38.525 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:57:38.571 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T04:57:38.674 INFO:teuthology.orchestra.run.smithi045.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 199 kB/s | 303 kB 00:01 2024-03-12T04:57:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:57:38.713 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:57:38.787 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:57:38.812 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:57:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T04:57:38.837 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T04:57:38.861 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T04:57:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T04:57:38.900 INFO:teuthology.orchestra.run.smithi045.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 157 kB/s | 35 kB 00:00 2024-03-12T04:57:38.954 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:57:39.231 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T04:57:39.371 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T04:57:39.426 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T04:57:39.499 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T04:57:39.530 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T04:57:39.566 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T04:57:39.613 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T04:57:39.649 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T04:57:39.692 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T04:57:39.743 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T04:57:39.874 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T04:57:39.923 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T04:57:39.953 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T04:57:39.994 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T04:57:40.033 INFO:teuthology.orchestra.run.smithi045.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 179 kB/s | 203 kB 00:01 2024-03-12T04:57:40.088 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T04:57:40.128 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T04:57:40.168 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T04:57:40.194 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T04:57:40.218 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T04:57:40.302 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T04:57:40.426 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T04:57:40.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T04:57:40.584 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T04:57:40.632 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T04:57:40.688 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T04:57:40.880 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T04:57:40.930 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T04:57:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T04:57:41.094 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T04:57:41.127 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T04:57:41.157 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T04:57:41.185 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T04:57:41.220 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T04:57:41.259 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T04:57:41.290 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T04:57:41.320 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T04:57:41.396 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T04:57:41.431 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T04:57:41.464 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T04:57:41.505 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T04:57:41.540 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T04:57:41.628 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T04:57:41.719 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T04:57:41.762 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T04:57:41.793 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T04:57:41.841 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T04:57:41.939 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T04:57:41.986 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T04:57:42.694 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T04:57:42.742 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T04:57:42.772 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T04:57:42.828 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T04:57:42.915 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T04:57:42.916 INFO:teuthology.orchestra.run.smithi045.stdout:(123/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 73 kB/s | 838 kB 00:11 2024-03-12T04:57:42.948 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T04:57:43.123 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T04:57:43.154 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T04:57:43.231 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T04:57:43.862 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:57:43.862 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T04:57:43.866 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T04:57:43.867 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T04:57:43.868 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T04:57:43.875 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T04:57:43.885 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T04:57:43.885 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T04:57:43.885 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T04:57:43.886 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T04:57:43.887 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T04:57:43.888 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T04:57:43.889 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T04:57:43.890 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T04:57:43.891 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T04:57:43.892 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T04:57:43.893 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T04:57:43.894 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T04:57:43.895 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T04:57:43.896 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T04:57:43.897 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T04:57:43.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:57:43.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T04:57:43.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T04:57:43.898 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T04:57:43.983 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T04:57:44.775 INFO:teuthology.orchestra.run.smithi045.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 78 kB/s | 144 kB 00:01 2024-03-12T04:57:44.950 INFO:teuthology.orchestra.run.smithi045.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 61 kB/s | 11 kB 00:00 2024-03-12T04:57:45.203 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T04:57:45.267 INFO:teuthology.orchestra.run.smithi045.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 56 kB/s | 18 kB 00:00 2024-03-12T04:57:45.294 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T04:57:45.326 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T04:57:45.356 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T04:57:45.384 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T04:57:45.691 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T04:57:45.709 INFO:teuthology.orchestra.run.smithi045.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 52 kB/s | 23 kB 00:00 2024-03-12T04:57:45.722 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:57:45.821 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:57:45.852 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T04:57:45.888 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T04:57:46.051 INFO:teuthology.orchestra.run.smithi045.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 57 kB/s | 19 kB 00:00 2024-03-12T04:57:46.350 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T04:57:46.384 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:57:46.409 INFO:teuthology.orchestra.run.smithi045.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 55 kB/s | 20 kB 00:00 2024-03-12T04:57:46.437 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:57:46.460 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T04:57:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T04:57:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T04:57:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T04:57:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T04:57:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T04:57:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T04:57:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T04:57:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T04:57:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T04:57:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T04:57:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T04:57:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T04:57:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T04:57:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T04:57:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T04:57:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T04:57:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T04:57:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T04:57:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-12T04:57:46.489 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-12T04:57:46.960 DEBUG:teuthology.parallel:result is None 2024-03-12T04:57:48.460 INFO:teuthology.orchestra.run.smithi045.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 248 kB/s | 2.0 MB 00:08 2024-03-12T04:57:48.643 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:48.672 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:48.744 INFO:teuthology.orchestra.run.smithi045.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 163 kB/s | 46 kB 00:00 2024-03-12T04:57:49.217 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:57:49.227 INFO:teuthology.orchestra.run.smithi045.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 163 kB/s | 79 kB 00:00 2024-03-12T04:57:49.250 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:57:49.343 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:57:49.343 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T04:57:49.343 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T04:57:49.343 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:57:49.376 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:57:49.586 INFO:teuthology.orchestra.run.smithi045.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 161 kB/s | 58 kB 00:00 2024-03-12T04:57:51.202 INFO:teuthology.orchestra.run.smithi045.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 168 kB/s | 272 kB 00:01 2024-03-12T04:57:51.353 INFO:teuthology.orchestra.run.smithi045.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2024-03-12T04:57:51.928 INFO:teuthology.orchestra.run.smithi045.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 156 kB/s | 90 kB 00:00 2024-03-12T04:57:52.111 INFO:teuthology.orchestra.run.smithi045.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 154 kB/s | 28 kB 00:00 2024-03-12T04:57:52.303 INFO:teuthology.orchestra.run.smithi045.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 160 kB/s | 31 kB 00:00 2024-03-12T04:57:53.370 INFO:teuthology.orchestra.run.smithi045.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 177 kB/s | 188 kB 00:01 2024-03-12T04:57:53.737 INFO:teuthology.orchestra.run.smithi045.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 161 kB/s | 59 kB 00:00 2024-03-12T04:57:53.971 INFO:teuthology.orchestra.run.smithi045.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 153 kB/s | 36 kB 00:00 2024-03-12T04:57:54.288 INFO:teuthology.orchestra.run.smithi045.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 161 kB/s | 51 kB 00:00 2024-03-12T04:57:55.080 INFO:teuthology.orchestra.run.smithi045.stdout:(143/174): python3-webob-1.8.7-6.el9.noarch.rpm 290 kB/s | 230 kB 00:00 2024-03-12T04:57:55.213 INFO:teuthology.orchestra.run.smithi045.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 672 kB/s | 90 kB 00:00 2024-03-12T04:57:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:(145/174): python3-werkzeug-2.0.3-3.el9.1.noarc 222 kB/s | 427 kB 00:01 2024-03-12T04:57:57.289 INFO:teuthology.orchestra.run.smithi045.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-03-12T04:57:58.238 INFO:teuthology.orchestra.run.smithi045.stdout:(147/174): re2-20211101-3.el9.x86_64.rpm 202 kB/s | 192 kB 00:00 2024-03-12T04:58:01.314 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:58:01.314 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-03-12T04:58:01.314 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-03-12T04:58:01.314 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-03-12T04:58:01.315 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-03-12T04:58:01.315 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-03-12T04:58:01.315 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-03-12T04:58:01.315 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-03-12T04:58:01.315 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:01.481 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:58:01.494 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:58:01.495 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:01.495 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T04:58:01.495 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:02.982 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:58:03.007 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:58:03.008 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:03.008 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T04:58:03.008 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:58:03.008 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:58:03.008 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:03.147 INFO:teuthology.orchestra.run.smithi045.stdout:(148/174): python3-kubernetes-26.1.0-2.el9.noar 62 kB/s | 1.0 MB 00:16 2024-03-12T04:58:03.216 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:58:03.238 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:03.533 INFO:teuthology.orchestra.run.smithi045.stdout:(149/174): librados2-19.0.0-1607.g94a53970.el9. 8.8 MB/s | 3.4 MB 00:00 2024-03-12T04:58:03.683 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:58:03.705 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:58:03.705 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:03.705 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T04:58:03.706 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:58:03.706 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:58:03.706 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:03.722 INFO:teuthology.orchestra.run.smithi045.stdout:(150/174): librbd1-19.0.0-1607.g94a53970.el9.x8 16 MB/s | 3.1 MB 00:00 2024-03-12T04:58:03.764 INFO:teuthology.orchestra.run.smithi045.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-12T04:58:03.801 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T04:58:04.139 INFO:teuthology.orchestra.run.smithi045.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 6.1 MB/s | 2.3 MB 00:00 2024-03-12T04:58:04.236 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:58:04.266 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:58:04.306 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T04:58:04.365 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T04:58:04.389 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:58:04.620 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:58:04.648 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:58:04.906 INFO:teuthology.orchestra.run.smithi045.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.4 MB/s | 1.8 MB 00:00 2024-03-12T04:58:04.965 INFO:teuthology.orchestra.run.smithi045.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 728 kB/s | 42 kB 00:00 2024-03-12T04:58:05.090 INFO:teuthology.orchestra.run.smithi045.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-12T04:58:05.166 INFO:teuthology.orchestra.run.smithi045.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 949 kB/s | 71 kB 00:00 2024-03-12T04:58:05.216 INFO:teuthology.orchestra.run.smithi045.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 731 kB/s | 36 kB 00:00 2024-03-12T04:58:05.308 INFO:teuthology.orchestra.run.smithi045.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-12T04:58:05.433 INFO:teuthology.orchestra.run.smithi045.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-12T04:58:05.483 INFO:teuthology.orchestra.run.smithi045.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 460 kB/s | 23 kB 00:00 2024-03-12T04:58:05.567 INFO:teuthology.orchestra.run.smithi045.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-12T04:58:05.608 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:58:05.640 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:58:05.650 INFO:teuthology.orchestra.run.smithi045.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-12T04:58:05.726 INFO:teuthology.orchestra.run.smithi045.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-12T04:58:05.740 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:58:05.793 INFO:teuthology.orchestra.run.smithi045.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 960 kB/s | 64 kB 00:00 2024-03-12T04:58:05.843 INFO:teuthology.orchestra.run.smithi045.stdout:(165/174): thrift-0.15.0-2.el9.x86_64.rpm 213 kB/s | 1.6 MB 00:07 2024-03-12T04:58:05.868 INFO:teuthology.orchestra.run.smithi045.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-03-12T04:58:05.902 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T04:58:05.934 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:58:05.956 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:58:05.957 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:05.957 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T04:58:05.957 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:58:05.957 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:58:05.957 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:06.001 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:58:06.012 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:58:06.061 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T04:58:06.127 INFO:teuthology.orchestra.run.smithi045.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 4.3 MB/s | 1.2 MB 00:00 2024-03-12T04:58:06.186 INFO:teuthology.orchestra.run.smithi045.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-12T04:58:06.327 INFO:teuthology.orchestra.run.smithi045.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-12T04:58:06.386 INFO:teuthology.orchestra.run.smithi045.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 534 kB/s | 31 kB 00:00 2024-03-12T04:58:06.561 INFO:teuthology.orchestra.run.smithi045.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.3 MB/s | 2.3 MB 00:00 2024-03-12T04:58:06.670 INFO:teuthology.orchestra.run.smithi045.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.8 MB/s | 1.1 MB 00:00 2024-03-12T04:58:06.911 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T04:58:06.944 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:58:06.967 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:58:06.968 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:06.968 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T04:58:06.968 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:58:06.968 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:58:06.968 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:07.014 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:58:07.034 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:58:07.034 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:07.035 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T04:58:07.035 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:07.079 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:58:07.101 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:58:07.101 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:07.101 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T04:58:07.101 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:58:07.101 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:58:07.102 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:07.387 INFO:teuthology.orchestra.run.smithi045.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 90 kB/s | 74 kB 00:00 2024-03-12T04:58:07.404 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:58:07.428 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:58:07.428 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:58:07.428 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T04:58:07.428 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:58:07.428 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:58:07.429 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:11.851 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:58:11.891 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T04:58:11.925 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T04:58:11.958 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T04:58:12.137 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T04:58:12.247 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T04:58:12.288 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T04:58:12.344 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T04:58:12.388 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T04:58:12.423 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T04:58:12.455 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T04:58:12.488 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T04:58:12.518 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T04:58:12.581 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T04:58:12.655 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T04:58:12.690 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T04:58:13.310 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:58:13.346 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:58:13.367 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:58:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T04:58:14.266 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T04:58:14.307 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T04:58:14.388 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T04:58:14.419 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T04:58:14.444 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T04:58:14.469 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T04:58:14.495 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T04:58:14.496 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:58:14.584 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:58:14.613 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T04:58:14.613 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:58:14.681 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:58:14.707 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T04:58:14.741 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T04:58:14.766 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T04:58:14.791 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T04:58:14.825 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T04:58:14.871 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T04:58:14.912 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T04:58:14.937 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T04:58:14.962 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T04:58:14.987 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:58:15.014 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T04:58:15.019 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:58:15.020 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:58:15.199 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:58:15.199 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:58:15.265 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:58:15.292 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:58:15.317 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T04:58:15.350 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T04:58:15.385 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T04:58:15.450 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T04:58:20.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T04:58:20.210 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T04:58:20.211 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T04:58:20.211 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T04:58:20.211 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T04:58:20.211 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T04:58:20.211 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T04:58:20.227 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T04:58:20.228 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T04:58:20.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T04:58:20.230 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T04:58:20.231 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T04:58:20.232 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T04:58:20.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T04:58:20.234 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T04:58:20.235 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T04:58:20.236 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T04:58:20.237 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:58:20.238 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T04:58:20.239 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T04:58:23.229 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.230 INFO:teuthology.orchestra.run.smithi130.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.231 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.232 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-16.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T04:58:23.233 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T04:58:23.234 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T04:58:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T04:58:23.236 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T04:58:23.237 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T04:58:23.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T04:58:23.239 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T04:58:23.240 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T04:58:23.241 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T04:58:23.242 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T04:58:23.242 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T04:58:23.242 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-12T04:58:23.706 DEBUG:teuthology.parallel:result is None 2024-03-12T04:58:33.076 INFO:teuthology.orchestra.run.smithi045.stdout:(174/174): libarrow-9.0.0-10.el9.x86_64.rpm 72 kB/s | 4.4 MB 01:03 2024-03-12T04:58:33.080 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-12T04:58:33.081 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.9 MB/s | 223 MB 01:17 2024-03-12T04:58:33.535 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-12T04:58:33.866 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-12T04:58:33.866 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-12T04:58:35.966 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-12T04:58:35.966 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-12T04:58:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-12T04:58:43.113 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T04:58:43.148 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T04:58:43.194 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:58:43.203 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T04:58:43.251 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T04:58:43.304 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T04:58:43.346 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T04:58:43.379 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T04:58:43.426 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T04:58:43.766 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T04:58:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:58:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:58:43.981 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:58:44.077 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:58:44.107 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:58:44.150 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T04:58:44.328 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T04:58:44.384 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T04:58:44.411 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:58:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:58:44.529 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:58:44.594 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T04:58:44.633 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T04:58:44.656 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:58:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T04:58:44.857 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:58:44.892 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T04:58:44.931 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T04:58:45.110 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T04:58:45.231 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:58:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T04:58:45.392 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:58:45.437 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:58:45.649 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T04:58:45.764 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T04:58:45.859 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T04:58:45.893 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T04:58:45.920 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T04:58:45.955 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T04:58:46.032 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T04:58:46.093 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T04:58:46.130 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T04:58:46.206 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T04:58:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T04:58:46.290 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T04:58:46.323 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T04:58:46.350 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T04:58:46.422 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T04:58:46.455 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T04:58:46.490 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T04:58:46.510 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T04:58:46.582 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T04:58:46.634 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T04:58:46.711 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T04:58:46.857 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T04:58:46.912 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T04:58:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T04:58:47.551 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T04:58:47.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T04:58:47.685 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:58:47.693 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T04:58:47.765 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T04:58:47.817 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T04:58:47.855 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T04:58:47.885 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T04:58:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T04:58:48.326 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T04:58:48.396 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T04:58:48.441 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T04:58:48.581 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T04:58:48.635 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T04:58:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T04:58:48.765 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T04:58:48.802 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T04:58:48.831 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T04:58:48.878 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T04:58:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T04:58:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T04:58:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T04:58:49.139 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T04:58:49.188 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T04:58:49.218 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T04:58:49.260 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T04:58:49.358 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T04:58:49.401 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T04:58:49.449 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T04:58:49.483 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T04:58:49.508 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T04:58:49.590 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T04:58:49.715 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T04:58:49.831 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T04:58:49.876 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T04:58:49.935 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T04:58:49.998 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T04:58:50.191 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T04:58:50.239 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T04:58:50.381 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T04:58:50.429 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T04:58:50.461 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T04:58:50.491 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T04:58:50.519 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T04:58:50.554 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T04:58:50.594 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T04:58:50.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T04:58:50.663 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T04:58:50.742 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T04:58:50.774 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T04:58:50.807 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T04:58:50.840 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T04:58:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T04:58:50.960 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T04:58:51.050 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T04:58:51.098 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T04:58:51.148 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T04:58:51.194 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T04:58:51.304 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T04:58:51.366 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T04:58:52.135 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T04:58:52.194 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T04:58:52.227 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T04:58:52.286 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T04:58:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T04:58:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T04:58:52.596 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T04:58:52.626 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T04:58:52.706 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T04:58:53.382 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T04:58:53.497 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T04:58:54.782 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T04:58:54.862 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T04:58:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T04:58:54.924 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T04:58:54.952 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T04:58:55.277 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T04:58:55.306 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:58:55.389 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T04:58:55.420 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T04:58:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T04:58:55.940 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T04:58:55.985 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:58:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T04:58:56.062 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T04:58:58.344 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:58:58.384 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:58:58.953 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T04:58:59.232 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:58:59.336 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T04:58:59.336 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T04:58:59.336 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T04:58:59.337 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:58:59.380 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-03-12T04:59:11.425 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:11.621 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:59:11.635 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T04:59:11.635 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:11.636 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T04:59:11.636 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:13.159 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T04:59:13.184 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:13.401 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:59:13.422 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T04:59:13.422 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:13.423 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T04:59:13.423 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:59:13.423 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T04:59:13.423 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:13.885 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T04:59:13.907 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:14.004 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T04:59:14.363 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:59:14.394 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T04:59:14.426 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T04:59:14.488 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T04:59:14.517 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:59:14.760 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T04:59:14.793 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:59:15.806 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T04:59:15.835 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:59:15.942 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T04:59:16.114 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T04:59:16.138 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T04:59:16.160 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:16.214 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:59:16.225 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T04:59:16.274 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T04:59:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T04:59:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T04:59:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:59:17.296 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T04:59:17.296 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:17.296 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T04:59:17.296 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:17.342 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T04:59:17.364 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:17.694 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T04:59:17.715 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:22.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:59:22.459 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T04:59:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T04:59:22.524 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T04:59:22.726 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T04:59:22.844 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T04:59:22.890 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T04:59:22.942 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T04:59:23.006 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T04:59:23.050 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T04:59:23.082 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T04:59:23.123 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T04:59:23.161 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T04:59:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T04:59:23.292 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T04:59:23.335 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T04:59:23.900 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:59:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:59:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T04:59:24.846 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T04:59:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T04:59:24.972 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T04:59:25.058 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T04:59:25.092 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T04:59:25.117 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T04:59:25.143 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T04:59:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T04:59:25.198 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:59:25.291 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T04:59:25.329 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T04:59:25.329 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:59:25.414 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T04:59:25.440 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T04:59:25.473 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T04:59:25.499 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T04:59:25.524 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T04:59:25.558 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T04:59:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T04:59:25.637 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T04:59:25.662 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T04:59:25.688 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T04:59:25.713 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:59:25.739 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T04:59:25.745 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:59:25.745 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:59:25.933 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T04:59:25.933 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:59:26.016 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T04:59:26.041 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:59:26.066 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T04:59:26.100 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T04:59:26.126 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T04:59:26.194 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T04:59:31.044 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T04:59:31.045 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T04:59:31.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T04:59:31.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T04:59:31.048 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T04:59:31.049 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T04:59:31.050 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T04:59:31.051 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T04:59:31.052 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T04:59:31.053 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T04:59:31.054 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T04:59:31.055 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T04:59:31.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T04:59:31.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T04:59:31.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T04:59:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T04:59:33.742 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.743 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.744 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T04:59:33.745 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T04:59:33.746 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T04:59:33.747 INFO:teuthology.orchestra.run.smithi045.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T04:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T04:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T04:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T04:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T04:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T04:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T04:59:33.754 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-12T04:59:34.224 DEBUG:teuthology.parallel:result is None 2024-03-12T04:59:34.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:34.446 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T04:59:34.468 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.469 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.469 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T04:59:34.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:34.694 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T04:59:34.719 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.720 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.720 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T04:59:34.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:34.938 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T04:59:34.964 INFO:teuthology.orchestra.run.smithi130.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.964 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T04:59:34.964 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T04:59:34.966 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-12T04:59:34.966 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:59:34.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T04:59:35.001 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:59:35.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T04:59:35.035 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:59:35.036 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T04:59:35.070 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-12T04:59:35.070 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:59:35.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T04:59:35.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T04:59:35.169 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:59:35.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T04:59:35.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T04:59:35.268 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:59:35.268 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T04:59:35.297 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T04:59:35.364 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-12T04:59:35.364 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:59:35.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T04:59:35.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T04:59:35.461 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:59:35.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T04:59:35.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T04:59:35.556 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:59:35.556 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T04:59:35.598 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T04:59:35.665 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-12T04:59:35.665 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T04:59:35.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T04:59:35.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T04:59:35.762 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T04:59:35.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T04:59:35.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T04:59:35.855 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T04:59:35.855 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T04:59:35.882 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T04:59:35.948 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T04:59:36.039 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'subvols': {'create': 2}} 2024-03-12T04:59:36.040 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:36.040 INFO:tasks.cephadm:Cluster fsid is 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T04:59:36.040 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T04:59:36.040 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.45', 'mon.b': '172.21.15.116', 'mon.c': '172.21.15.130'} 2024-03-12T04:59:36.040 INFO:tasks.cephadm:First mon is mon.a on smithi045 2024-03-12T04:59:36.040 INFO:tasks.cephadm:First mgr is x 2024-03-12T04:59:36.041 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T04:59:36.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-03-12T04:59:36.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-12T04:59:36.097 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-03-12T04:59:36.124 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-12T04:59:36.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:36.340 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-12T04:59:36.479 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T04:59:36.480 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-12T04:59:36.480 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:37.707 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 04:59 /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:37.708 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 04:59 /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.057 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.960 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 04:59 /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.960 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.979 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:39.997 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T04:59:40.032 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-12T04:59:40.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T04:59:40.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T04:59:40.040 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T04:59:40.826 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T04:59:40.915 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T04:59:41.149 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-03-12T05:00:07.005 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-12T05:00:20.314 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:00:20.315 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:00:32.074 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-12T05:00:32.074 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:00:32.075 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:00:32.075 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-12T05:00:32.075 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:00:32.075 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-12T05:00:32.075 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-12T05:00:32.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-03-12T05:00:32.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-12T05:00:32.164 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-03-12T05:00:32.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-03-12T05:00:32.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-12T05:00:32.257 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-03-12T05:00:32.287 INFO:tasks.cephadm:Writing seed config... 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-12T05:00:32.289 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-12T05:00:32.290 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T05:00:32.291 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-12T05:00:32.292 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T05:00:32.293 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-12T05:00:32.294 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:00:32.294 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T05:00:32.310 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 52aed436-e02d-11ee-95c8-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-12T05:00:32.311 DEBUG:teuthology.orchestra.run.smithi045:mon.a> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T05:00:32.353 DEBUG:teuthology.orchestra.run.smithi045:mgr.x> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T05:00:32.396 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T05:00:32.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:00:32.592 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:00:32.592 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '52aed436-e02d-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.45', '--skip-admin-label'] 2024-03-12T05:00:32.619 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:00:32.619 INFO:teuthology.orchestra.run.smithi045.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T05:00:32.619 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying podman|docker is present... 2024-03-12T05:00:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:00:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying lvm2 is present... 2024-03-12T05:00:32.643 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying time synchronization is in place... 2024-03-12T05:00:32.651 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T05:00:32.651 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T05:00:32.657 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T05:00:32.657 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:00:32.664 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-12T05:00:32.670 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-12T05:00:32.671 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-12T05:00:32.671 INFO:teuthology.orchestra.run.smithi045.stdout:Repeating the final host check... 2024-03-12T05:00:32.694 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:00:32.694 INFO:teuthology.orchestra.run.smithi045.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T05:00:32.695 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl is present 2024-03-12T05:00:32.695 INFO:teuthology.orchestra.run.smithi045.stdout:lvcreate is present 2024-03-12T05:00:32.701 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T05:00:32.701 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T05:00:32.707 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T05:00:32.708 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:00:32.714 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-12T05:00:32.722 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Host looks OK 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Cluster fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Acquiring lock 140083133593200 on /run/cephadm/52aed436-e02d-11ee-95c8-87774f69a715.lock 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Lock 140083133593200 acquired on /run/cephadm/52aed436-e02d-11ee-95c8-87774f69a715.lock 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 3300 ... 2024-03-12T05:00:32.723 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 6789 ... 2024-03-12T05:00:32.724 INFO:teuthology.orchestra.run.smithi045.stdout:Base mon IP(s) is [172.21.15.45:3300, 172.21.15.45:6789], mon addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-12T05:00:32.726 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2024-03-12T05:00:32.726 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2024-03-12T05:00:32.729 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T05:00:32.729 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-12T05:00:32.729 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-12T05:00:32.732 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T05:00:32.732 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T05:00:32.732 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T05:00:32.732 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-12T05:00:32.733 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-03-12T05:00:32.733 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T05:00:32.733 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-12T05:00:32.733 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-12T05:00:32.733 INFO:teuthology.orchestra.run.smithi045.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T05:00:32.734 INFO:teuthology.orchestra.run.smithi045.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T05:00:32.734 INFO:teuthology.orchestra.run.smithi045.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-12T05:00:33.904 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T05:00:34.775 INFO:teuthology.orchestra.run.smithi045.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-12T05:00:34.775 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-12T05:00:34.775 INFO:teuthology.orchestra.run.smithi045.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T05:00:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:stat: stdout 167 167 2024-03-12T05:00:35.540 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial keys... 2024-03-12T05:00:36.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQBz4e9lMPltMhAAic46vPl6XkUuGjW1J+LaTA== 2024-03-12T05:00:37.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQB04e9lzn7FLRAAl5fTvdhoBk3EdT6rD/ZE7Q== 2024-03-12T05:00:39.285 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQB14e9lzHCqJxAAe6NyhdoXiv9XYhF+rVixVw== 2024-03-12T05:00:39.286 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial monmap... 2024-03-12T05:00:45.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T05:00:45.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T05:00:45.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:45.899 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T05:00:45.899 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool for a [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T05:00:45.900 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-03-12T05:00:45.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: set fsid to 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:45.900 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T05:00:45.900 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:00:45.900 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mon... 2024-03-12T05:00:47.196 INFO:teuthology.orchestra.run.smithi045.stdout:create mon.a on 2024-03-12T05:00:47.406 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-12T05:00:47.542 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T05:00:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target. 2024-03-12T05:00:47.729 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target. 2024-03-12T05:00:47.967 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a 2024-03-12T05:00:47.967 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T05:00:48.124 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target.wants/ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715@.service. 2024-03-12T05:00:48.629 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:00:48.629 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon to start... 2024-03-12T05:00:48.637 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon... 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout id: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout services: 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.539143s) 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T05:00:49.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout data: 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.073+0000 7fae16aa0700 1 Processor -- start 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.073+0000 7fae16aa0700 1 -- start start 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.073+0000 7fae16aa0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.073+0000 7fae16aa0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1007ae60 con 0x7fae10107f10 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae1483c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae1483c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52654/0 (socket says 172.21.15.45:52654) 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae1483c700 1 -- 172.21.15.45:0/2934654211 learned_addr learned my addr 172.21.15.45:0/2934654211 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:49.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae1483c700 1 -- 172.21.15.45:0/2934654211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1010a840 con 0x7fae10107f10 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae1483c700 1 --2- 172.21.15.45:0/2934654211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae040098a0 tx=0x7fae04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa97f98bb4a499a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae0f7fe700 1 -- 172.21.15.45:0/2934654211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae04004160 con 0x7fae10107f10 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae0f7fe700 1 -- 172.21.15.45:0/2934654211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae040042c0 con 0x7fae10107f10 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.074+0000 7fae0f7fe700 1 -- 172.21.15.45:0/2934654211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae040045a0 con 0x7fae10107f10 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- 172.21.15.45:0/2934654211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 msgr2=0x7fae1010a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 --2- 172.21.15.45:0/2934654211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae040098a0 tx=0x7fae04009bb0 comp rx=0 tx=0).stop 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- 172.21.15.45:0/2934654211 shutdown_connections 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 --2- 172.21.15.45:0/2934654211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae1010a300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- 172.21.15.45:0/2934654211 >> 172.21.15.45:0/2934654211 conn(0x7fae10101920 msgr2=0x7fae10103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- 172.21.15.45:0/2934654211 shutdown_connections 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- 172.21.15.45:0/2934654211 wait complete. 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 Processor -- start 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.075+0000 7fae16aa0700 1 -- start start 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae16aa0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:49.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae16aa0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae10107cd0 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae1483c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae1483c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52660/0 (socket says 172.21.15.45:52660) 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae1483c700 1 -- 172.21.15.45:0/3411972383 learned_addr learned my addr 172.21.15.45:0/3411972383 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae1483c700 1 -- 172.21.15.45:0/3411972383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae04009550 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae1483c700 1 --2- 172.21.15.45:0/3411972383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae1007a2a0 tx=0x7fae04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae04011630 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae04011790 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae04011a30 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.076+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae10105ec0 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.077+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae10106360 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.077+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fae04021070 con 0x7fae10107f10 2024-03-12T05:00:49.734 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.077+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae0401b7d0 con 0x7fae10107f10 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.078+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae100632a0 con 0x7fae10107f10 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.080+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fae0401ba30 con 0x7fae10107f10 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.155+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fae1010ace0 con 0x7fae10107f10 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.155+0000 7fae0dffb700 1 -- 172.21.15.45:0/3411972383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fae0401bbd0 con 0x7fae10107f10 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 msgr2=0x7fae10107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 --2- 172.21.15.45:0/3411972383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae1007a2a0 tx=0x7fae04004750 comp rx=0 tx=0).stop 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 shutdown_connections 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 --2- 172.21.15.45:0/3411972383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae10107f10 0x7fae10107790 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 >> 172.21.15.45:0/3411972383 conn(0x7fae10101920 msgr2=0x7fae10103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 shutdown_connections 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:49.156+0000 7fae16aa0700 1 -- 172.21.15.45:0/3411972383 wait complete. 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:mon is available 2024-03-12T05:00:49.735 INFO:teuthology.orchestra.run.smithi045.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T05:00:50.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:50 smithi045 ceph-mon[31800]: from='client.? 172.21.15.45:0/1271024357' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T05:00:50.778 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:50.778 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T05:00:50.778 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T05:00:50.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.112+0000 7faa24867700 1 Processor -- start 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa24867700 1 -- start start 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa24867700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa24867700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20106dc0 con 0x7faa201063d0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa1e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa1e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52674/0 (socket says 172.21.15.45:52674) 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.113+0000 7faa1e59c700 1 -- 172.21.15.45:0/269284179 learned_addr learned my addr 172.21.15.45:0/269284179 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa1e59c700 1 -- 172.21.15.45:0/269284179 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa201075d0 con 0x7faa201063d0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa1e59c700 1 --2- 172.21.15.45:0/269284179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faa18009e80 tx=0x7faa1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb99a849db927f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa1d59a700 1 -- 172.21.15.45:0/269284179 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa1800e070 con 0x7faa201063d0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa1d59a700 1 -- 172.21.15.45:0/269284179 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18003a20 con 0x7faa201063d0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa1d59a700 1 -- 172.21.15.45:0/269284179 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa18003ce0 con 0x7faa201063d0 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 -- 172.21.15.45:0/269284179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 msgr2=0x7faa201067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:50.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 --2- 172.21.15.45:0/269284179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faa18009e80 tx=0x7faa1800b330 comp rx=0 tx=0).stop 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 -- 172.21.15.45:0/269284179 shutdown_connections 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 --2- 172.21.15.45:0/269284179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa201067f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 -- 172.21.15.45:0/269284179 >> 172.21.15.45:0/269284179 conn(0x7faa201019b0 msgr2=0x7faa20103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.114+0000 7faa24867700 1 -- 172.21.15.45:0/269284179 shutdown_connections 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa24867700 1 -- 172.21.15.45:0/269284179 wait complete. 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa24867700 1 Processor -- start 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa24867700 1 -- start start 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa24867700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa24867700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2019b000 con 0x7faa201063d0 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.115+0000 7faa1e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa1e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52682/0 (socket says 172.21.15.45:52682) 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa1e59c700 1 -- 172.21.15.45:0/1271024357 learned_addr learned my addr 172.21.15.45:0/1271024357 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:50.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa1e59c700 1 -- 172.21.15.45:0/1271024357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18009b30 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa1e59c700 1 --2- 172.21.15.45:0/1271024357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7faa18004000 tx=0x7faa18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa18004370 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18004510 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2019b200 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa180047f0 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.116+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2019b6a0 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.117+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7faa180127c0 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.117+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faa1801dbf0 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.117+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa201946e0 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.119+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7faa1802b060 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.189+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7faa20107940 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.192+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7faa18021030 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.192+0000 7faa177fe700 1 -- 172.21.15.45:0/1271024357 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7faa18004950 con 0x7faa201063d0 2024-03-12T05:00:50.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 msgr2=0x7faa2019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 --2- 172.21.15.45:0/1271024357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7faa18004000 tx=0x7faa18004ad0 comp rx=0 tx=0).stop 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 shutdown_connections 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 --2- 172.21.15.45:0/1271024357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa201063d0 0x7faa2019aac0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 >> 172.21.15.45:0/1271024357 conn(0x7faa201019b0 msgr2=0x7faa20191820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 shutdown_connections 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:50.193+0000 7faa24867700 1 -- 172.21.15.45:0/1271024357 wait complete. 2024-03-12T05:00:50.783 INFO:teuthology.orchestra.run.smithi045.stdout:Generating new minimal ceph.conf... 2024-03-12T05:00:52.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:52 smithi045 ceph-mon[31800]: from='client.? 172.21.15.45:0/404149287' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:00:52.550 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d9952d700 1 Processor -- start 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d9952d700 1 -- start start 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d9952d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d9952d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d94106dc0 con 0x7f2d941063d0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d92ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d92ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52684/0 (socket says 172.21.15.45:52684) 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d92ffd700 1 -- 172.21.15.45:0/4105319268 learned_addr learned my addr 172.21.15.45:0/4105319268 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.950+0000 7f2d92ffd700 1 -- 172.21.15.45:0/4105319268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d941075d0 con 0x7f2d941063d0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d92ffd700 1 --2- 172.21.15.45:0/4105319268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d8400bb40 tx=0x7f2d8400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267f944be9c46cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d91ffb700 1 -- 172.21.15.45:0/4105319268 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d8400dbe0 con 0x7f2d941063d0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d91ffb700 1 -- 172.21.15.45:0/4105319268 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f2d840044c0 con 0x7f2d941063d0 2024-03-12T05:00:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d91ffb700 1 -- 172.21.15.45:0/4105319268 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d84003e50 con 0x7f2d941063d0 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d9952d700 1 -- 172.21.15.45:0/4105319268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 msgr2=0x7f2d941067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d9952d700 1 --2- 172.21.15.45:0/4105319268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d8400bb40 tx=0x7f2d8400be50 comp rx=0 tx=0).stop 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d9952d700 1 -- 172.21.15.45:0/4105319268 shutdown_connections 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d9952d700 1 --2- 172.21.15.45:0/4105319268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d941067f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.951+0000 7f2d9952d700 1 -- 172.21.15.45:0/4105319268 >> 172.21.15.45:0/4105319268 conn(0x7f2d941019b0 msgr2=0x7f2d94103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 -- 172.21.15.45:0/4105319268 shutdown_connections 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 -- 172.21.15.45:0/4105319268 wait complete. 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 Processor -- start 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 -- start start 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.952+0000 7f2d9952d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9419ca80 con 0x7f2d941063d0 2024-03-12T05:00:52.552 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d92ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d92ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52688/0 (socket says 172.21.15.45:52688) 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d92ffd700 1 -- 172.21.15.45:0/404149287 learned_addr learned my addr 172.21.15.45:0/404149287 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d92ffd700 1 -- 172.21.15.45:0/404149287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8400b7f0 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d92ffd700 1 --2- 172.21.15.45:0/404149287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d8400b340 tx=0x7f2d84003fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d840106c0 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f2d84010cb0 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9419cc80 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d84019d60 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d94199080 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2d8402a450 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.953+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2d84021c00 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.954+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d940632a0 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:51.956+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2d84021e60 con 0x7f2d941063d0 2024-03-12T05:00:52.553 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.027+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2d94192130 con 0x7f2d941063d0 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d7bfff700 1 -- 172.21.15.45:0/404149287 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2d84010e20 con 0x7f2d941063d0 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 msgr2=0x7f2d9419c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d9952d700 1 --2- 172.21.15.45:0/404149287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d8400b340 tx=0x7f2d84003fa0 comp rx=0 tx=0).stop 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 shutdown_connections 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d9952d700 1 --2- 172.21.15.45:0/404149287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d941063d0 0x7f2d9419c540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.028+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 >> 172.21.15.45:0/404149287 conn(0x7f2d941019b0 msgr2=0x7f2d9418f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.029+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 shutdown_connections 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:52.029+0000 7f2d9952d700 1 -- 172.21.15.45:0/404149287 wait complete. 2024-03-12T05:00:52.554 INFO:teuthology.orchestra.run.smithi045.stdout:Restarting the monitor... 2024-03-12T05:00:52.831 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:52 smithi045 systemd[1]: Stopping Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:00:53.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a[31792]: 2024-03-12T05:00:52.826+0000 7f10e33e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:00:53.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a[31792]: 2024-03-12T05:00:52.826+0000 7f10e33e9700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T05:00:53.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 podman[32048]: 2024-03-12 05:00:53.031639772 +0000 UTC m=+0.320053332 container died 6d8070c2db33ef3e4bb0e20486d9e3446283859ca5a039ca90e002d574e7798e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-12T05:00:53.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 podman[32048]: 2024-03-12 05:00:53.402102191 +0000 UTC m=+0.690515750 container cleanup 6d8070c2db33ef3e4bb0e20486d9e3446283859ca5a039ca90e002d574e7798e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:00:53.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 bash[32048]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a 2024-03-12T05:00:54.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 podman[32062]: 2024-03-12 05:00:53.661692136 +0000 UTC m=+0.629711389 container remove 6d8070c2db33ef3e4bb0e20486d9e3446283859ca5a039ca90e002d574e7798e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-12T05:00:54.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T05:00:54.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 systemd[1]: Stopped Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:00:54.042 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:53 smithi045 systemd[1]: Starting Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:00:54.263 INFO:teuthology.orchestra.run.smithi045.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 podman[32155]: 2024-03-12 05:00:53.941892836 +0000 UTC m=+0.017217381 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 podman[32155]: 2024-03-12 05:00:54.045931021 +0000 UTC m=+0.121255561 container create 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 podman[32155]: 2024-03-12 05:00:54.1980423 +0000 UTC m=+0.273366841 container init 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 podman[32155]: 2024-03-12 05:00:54.202561324 +0000 UTC m=+0.277885865 container start 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: pidfile_write: ignore empty --pid-file 2024-03-12T05:00:54.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: load: jerasure load: lrc 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: RocksDB version: 7.9.2 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Git sha 0 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: DB SUMMARY 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: DB Session ID: P68RVU36YAGAWRGY6A84 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: CURRENT file: CURRENT 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99784 ; 2024-03-12T05:00:54.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.error_if_exists: 0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.create_if_missing: 0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.paranoid_checks: 1 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.env: 0x55b76a5bcae0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.info_log: 0x55b76bd3d8c0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.statistics: (nil) 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.use_fsync: 0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_log_file_size: 0 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T05:00:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_fallocate: 1 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.use_direct_reads: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.db_log_dir: 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.wal_dir: 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T05:00:54.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.write_buffer_manager: 0x55b76bd11400 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T05:00:54.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T05:00:54.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.unordered_write: 0 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.row_cache: None 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.wal_filter: None 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T05:00:54.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.two_write_queues: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.wal_compression: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.atomic_flush: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.log_readahead_size: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T05:00:54.376 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_background_jobs: 2 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_background_compactions: -1 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_subcompactions: 1 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_open_files: -1 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T05:00:54.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_background_flushes: -1 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Compression algorithms supported: 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kZSTD supported: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kXpressCompression supported: 0 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kZlibCompression supported: 1 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kSnappyCompression supported: 1 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kLZ4Compression supported: 1 2024-03-12T05:00:54.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: kBZip2Compression supported: 0 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.merge_operator: 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_filter: None 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b76bd3d9c0) 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T05:00:54.379 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_type: 0 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_index_type: 0 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_shortening: 1 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: checksum: 4 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: no_block_cache: 0 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache: 0x55b76bd41090 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_options: 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: capacity : 536870912 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_shard_bits : 4 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: strict_capacity_limit : 0 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-03-12T05:00:54.380 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_compressed: (nil) 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: persistent_cache: (nil) 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size: 4096 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size_deviation: 10 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_restart_interval: 16 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_block_restart_interval: 1 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: metadata_block_size: 4096 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: partition_filters: 0 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: use_delta_encoding: 1 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: filter_policy: bloomfilter 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: whole_key_filtering: 1 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: verify_compression: 0 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: format_version: 5 2024-03-12T05:00:54.381 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_index_compression: 1 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_align: 0 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout: max_auto_readahead_size: 262144 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout: prepopulate_block_cache: 0 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout: initial_auto_readahead_size: 8192 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression: NoCompression 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.num_levels: 7 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T05:00:54.382 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T05:00:54.383 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T05:00:54.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T05:00:54.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.table_properties_collectors: 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.inplace_update_support: 0 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T05:00:54.386 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.bloom_locality: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.max_successive_merges: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.ttl: 2592000 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T05:00:54.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enable_blob_files: false 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.min_blob_size: 0 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T05:00:54.388 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ba775e68-bbfc-4d7f-a3bc-f95329dcebd5 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219654230542, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219654231480, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94971, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 92969, "index_size": 284, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13777, "raw_average_key_size": 50, "raw_value_size": 85468, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710219654, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ba775e68-bbfc-4d7f-a3bc-f95329dcebd5", "db_session_id": "P68RVU36YAGAWRGY6A84", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219654231540, "job": 1, "event": "recovery_finished"} 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b76be32000 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: DB pointer 0x55b76be1c000 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** DB Stats ** 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:00:54.389 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: L0 2/0 94.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Sum 2/0 94.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:00:54.390 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T05:00:54.391 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative compaction: 0.00 GB write, 31.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval compaction: 0.00 GB write, 31.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache BinnedLRUCache@0x55b76bd41090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: starting mon.a rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???) e1 preinit fsid 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).mds e1 new map 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).mds e1 print_map 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: e1 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: legacy client fscid: -1 2024-03-12T05:00:54.392 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout: No filesystems configured 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 ceph-mon[32174]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 bash[32155]: 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 2024-03-12T05:00:54.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:54 smithi045 systemd[1]: Started Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:00:55.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.753+0000 7ff604876700 1 Processor -- start 2024-03-12T05:00:55.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff604876700 1 -- start start 2024-03-12T05:00:55.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff604876700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:55.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff604876700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff600107f00 con 0x7ff600107510 2024-03-12T05:00:55.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff5fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff5fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52696/0 (socket says 172.21.15.45:52696) 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.754+0000 7ff5fe59c700 1 -- 172.21.15.45:0/2049956216 learned_addr learned my addr 172.21.15.45:0/2049956216 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.755+0000 7ff5fe59c700 1 -- 172.21.15.45:0/2049956216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600108670 con 0x7ff600107510 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.755+0000 7ff5fe59c700 1 --2- 172.21.15.45:0/2049956216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00098a0 tx=0x7ff5f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06a108a0522cc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.755+0000 7ff5fd59a700 1 -- 172.21.15.45:0/2049956216 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0004160 con 0x7ff600107510 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.755+0000 7ff5fd59a700 1 -- 172.21.15.45:0/2049956216 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff5f00042c0 con 0x7ff600107510 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.755+0000 7ff5fd59a700 1 -- 172.21.15.45:0/2049956216 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0011a50 con 0x7ff600107510 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 -- 172.21.15.45:0/2049956216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 msgr2=0x7ff600107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 --2- 172.21.15.45:0/2049956216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00098a0 tx=0x7ff5f0009bb0 comp rx=0 tx=0).stop 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 -- 172.21.15.45:0/2049956216 shutdown_connections 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 --2- 172.21.15.45:0/2049956216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff600107930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 -- 172.21.15.45:0/2049956216 >> 172.21.15.45:0/2049956216 conn(0x7ff600102b70 msgr2=0x7ff600104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 -- 172.21.15.45:0/2049956216 shutdown_connections 2024-03-12T05:00:55.307 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.756+0000 7ff604876700 1 -- 172.21.15.45:0/2049956216 wait complete. 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff604876700 1 Processor -- start 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff604876700 1 -- start start 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff604876700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff604876700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60019b710 con 0x7ff600107510 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff5fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff5fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52706/0 (socket says 172.21.15.45:52706) 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff5fe59c700 1 -- 172.21.15.45:0/3216751452 learned_addr learned my addr 172.21.15.45:0/3216751452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff5fe59c700 1 -- 172.21.15.45:0/3216751452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009550 con 0x7ff600107510 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.757+0000 7ff5fe59c700 1 --2- 172.21.15.45:0/3216751452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006af0 tx=0x7ff5f0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0003930 con 0x7ff600107510 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff5f0003a90 con 0x7ff600107510 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60019b910 con 0x7ff600107510 2024-03-12T05:00:55.308 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5f0019670 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60019bdb0 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff5f0019c70 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.758+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff5f00249a0 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.759+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6000632a0 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.761+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff5f001aa30 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.833+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ff60010b8a0 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.835+0000 7ff5ef7fe700 1 -- 172.21.15.45:0/3216751452 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff5f001abd0 con 0x7ff600107510 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 msgr2=0x7ff60019b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 --2- 172.21.15.45:0/3216751452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006af0 tx=0x7ff5f0011ab0 comp rx=0 tx=0).stop 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 shutdown_connections 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 --2- 172.21.15.45:0/3216751452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff600107510 0x7ff60019b1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 >> 172.21.15.45:0/3216751452 conn(0x7ff600102b70 msgr2=0x7ff600103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 shutdown_connections 2024-03-12T05:00:55.309 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:54.836+0000 7ff604876700 1 -- 172.21.15.45:0/3216751452 wait complete. 2024-03-12T05:00:55.310 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T05:00:55.324 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:00:55.324 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mgr... 2024-03-12T05:00:55.324 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T05:00:55.325 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T05:00:55.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T05:00:55.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:00:55.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: fsmap 2024-03-12T05:00:55.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T05:00:55.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: mgrmap e1: no daemons active 2024-03-12T05:00:55.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:55 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3216751452' entity='client.admin' 2024-03-12T05:00:55.578 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x 2024-03-12T05:00:55.578 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x.service: Unit ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x.service not loaded. 2024-03-12T05:00:55.721 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715.target.wants/ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-52aed436-e02d-11ee-95c8-87774f69a715@.service. 2024-03-12T05:00:56.251 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:00:56.251 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-12T05:00:56.258 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:00:56.258 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:00:56.258 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-12T05:00:56.258 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T05:00:56.266 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:00:56.266 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr to start... 2024-03-12T05:00:56.274 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr... 2024-03-12T05:00:56.373 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:56 smithi045 systemd[1]: Started Ceph mgr.x for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:00:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:56 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2009380520' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:00:57.038 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:56.716+0000 7fbfc3943200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:00:57.038 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:56.815+0000 7fbfc3943200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:00:57.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:00:57.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:00:57.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:00:57.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:00:57.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:00:57.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:00:57.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:00:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:00:48.630767+0000", 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b161ef700 1 Processor -- start 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b161ef700 1 -- start start 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b161ef700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b161ef700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b10107ee0 con 0x7f0b101074f0 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b0f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b0f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52732/0 (socket says 172.21.15.45:52732) 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.697+0000 7f0b0f7fe700 1 -- 172.21.15.45:0/427385873 learned_addr learned my addr 172.21.15.45:0/427385873 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.698+0000 7f0b0f7fe700 1 -- 172.21.15.45:0/427385873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10108650 con 0x7f0b101074f0 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.698+0000 7f0b0f7fe700 1 --2- 172.21.15.45:0/427385873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0b000098a0 tx=0x7f0b00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847318502eb84a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:57.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.698+0000 7f0b0e7fc700 1 -- 172.21.15.45:0/427385873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b00004160 con 0x7f0b101074f0 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.698+0000 7f0b0e7fc700 1 -- 172.21.15.45:0/427385873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f0b000042c0 con 0x7f0b101074f0 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.698+0000 7f0b0e7fc700 1 -- 172.21.15.45:0/427385873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b00011a50 con 0x7f0b101074f0 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 -- 172.21.15.45:0/427385873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 msgr2=0x7f0b10107910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 --2- 172.21.15.45:0/427385873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0b000098a0 tx=0x7f0b00009bb0 comp rx=0 tx=0).stop 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 -- 172.21.15.45:0/427385873 shutdown_connections 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 --2- 172.21.15.45:0/427385873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b10107910 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 -- 172.21.15.45:0/427385873 >> 172.21.15.45:0/427385873 conn(0x7f0b10102a90 msgr2=0x7f0b10104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 -- 172.21.15.45:0/427385873 shutdown_connections 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 -- 172.21.15.45:0/427385873 wait complete. 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.699+0000 7f0b161ef700 1 Processor -- start 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b161ef700 1 -- start start 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b161ef700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b161ef700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1007d0a0 con 0x7f0b101074f0 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b0f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:00:57.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b0f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52744/0 (socket says 172.21.15.45:52744) 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b0f7fe700 1 -- 172.21.15.45:0/2009380520 learned_addr learned my addr 172.21.15.45:0/2009380520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b0f7fe700 1 -- 172.21.15.45:0/2009380520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b00009550 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.700+0000 7f0b0f7fe700 1 --2- 172.21.15.45:0/2009380520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b00003f40 tx=0x7f0b00011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b00003a40 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1007b290 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f0b00003ba0 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1007b730 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b000196f0 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0b00019d40 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.701+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0b00024a00 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.702+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b100632a0 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.704+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0b0001abe0 con 0x7f0b101074f0 2024-03-12T05:00:57.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.777+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0b1010b950 con 0x7f0b101074f0 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.778+0000 7f0b0cff9700 1 -- 172.21.15.45:0/2009380520 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f0b00024c60 con 0x7f0b101074f0 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 msgr2=0x7f0b1007cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 --2- 172.21.15.45:0/2009380520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b00003f40 tx=0x7f0b00011ab0 comp rx=0 tx=0).stop 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 shutdown_connections 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 --2- 172.21.15.45:0/2009380520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b101074f0 0x7f0b1007cb60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 >> 172.21.15.45:0/2009380520 conn(0x7f0b10102a90 msgr2=0x7f0b10103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 shutdown_connections 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:56.779+0000 7f0b161ef700 1 -- 172.21.15.45:0/2009380520 wait complete. 2024-03-12T05:00:57.347 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (1/15)... 2024-03-12T05:00:57.578 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:57.481+0000 7fbfc3943200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:00:57.578 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:57.574+0000 7fbfc3943200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:00:57.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:57.716+0000 7fbfc3943200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:00:57.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:57.813+0000 7fbfc3943200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:00:58.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:57.896+0000 7fbfc3943200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:00:58.165 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:58.161+0000 7fbfc3943200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:00:58.458 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:58.272+0000 7fbfc3943200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:00:58.458 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:58.359+0000 7fbfc3943200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:00:58.459 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:58.455+0000 7fbfc3943200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:00:58.858 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:58.631+0000 7fbfc3943200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:00:59.730 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:59.629+0000 7fbfc3943200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:01:00.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:00:59 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1454205795' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:01:00.100 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:00:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:00:59.767+0000 7fbfc3943200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:01:00.351 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:00.108+0000 7fbfc3943200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:01:00.351 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:00.200+0000 7fbfc3943200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:01:00.355 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:01:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:01:00.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:01:00.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:01:00.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:00:48.630767+0000", 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.725+0000 7f3a5f960700 1 Processor -- start 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.725+0000 7f3a5f960700 1 -- start start 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.725+0000 7f3a5f960700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.725+0000 7f3a5f960700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5807ae40 con 0x7f3a58107f30 2024-03-12T05:01:00.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a5d6fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a5d6fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60320/0 (socket says 172.21.15.45:60320) 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a5d6fc700 1 -- 172.21.15.45:0/3819281608 learned_addr learned my addr 172.21.15.45:0/3819281608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a5d6fc700 1 -- 172.21.15.45:0/3819281608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5810a860 con 0x7f3a58107f30 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a5d6fc700 1 --2- 172.21.15.45:0/3819281608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3a480098a0 tx=0x7f3a48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4b58a012222df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a4ffff700 1 -- 172.21.15.45:0/3819281608 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a48004160 con 0x7f3a58107f30 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a4ffff700 1 -- 172.21.15.45:0/3819281608 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3a480042c0 con 0x7f3a58107f30 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.726+0000 7f3a4ffff700 1 -- 172.21.15.45:0/3819281608 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a48011a50 con 0x7f3a58107f30 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 -- 172.21.15.45:0/3819281608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 msgr2=0x7f3a5810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 --2- 172.21.15.45:0/3819281608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3a480098a0 tx=0x7f3a48009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 -- 172.21.15.45:0/3819281608 shutdown_connections 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 --2- 172.21.15.45:0/3819281608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5810a320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 -- 172.21.15.45:0/3819281608 >> 172.21.15.45:0/3819281608 conn(0x7f3a581018e0 msgr2=0x7f3a58103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 -- 172.21.15.45:0/3819281608 shutdown_connections 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.727+0000 7f3a5f960700 1 -- 172.21.15.45:0/3819281608 wait complete. 2024-03-12T05:01:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5f960700 1 Processor -- start 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5f960700 1 -- start start 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5f960700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5f960700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5819b720 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5d6fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5d6fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60332/0 (socket says 172.21.15.45:60332) 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5d6fc700 1 -- 172.21.15.45:0/1454205795 learned_addr learned my addr 172.21.15.45:0/1454205795 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.728+0000 7f3a5d6fc700 1 -- 172.21.15.45:0/1454205795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a48009550 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a5d6fc700 1 --2- 172.21.15.45:0/1454205795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006af0 tx=0x7f3a48011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a48003930 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f3a48003a90 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a48019670 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5819b920 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5819bdc0 con 0x7f3a58107f30 2024-03-12T05:01:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3a48019c70 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.729+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3a48024a80 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.730+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5810b600 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.732+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3a4801aa30 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.806+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3a5810b810 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.806+0000 7f3a4e7fc700 1 -- 172.21.15.45:0/1454205795 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f3a4801abd0 con 0x7f3a58107f30 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 msgr2=0x7f3a5819b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 --2- 172.21.15.45:0/1454205795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006af0 tx=0x7f3a48011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 shutdown_connections 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 --2- 172.21.15.45:0/1454205795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a58107f30 0x7f3a5819b1e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 >> 172.21.15.45:0/1454205795 conn(0x7f3a581018e0 msgr2=0x7f3a58102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 shutdown_connections 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:00:59.807+0000 7f3a5f960700 1 -- 172.21.15.45:0/1454205795 wait complete. 2024-03-12T05:01:00.370 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (2/15)... 2024-03-12T05:01:00.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:00.401+0000 7fbfc3943200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:00.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:00.589+0000 7fbfc3943200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:01:01.141 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:00.877+0000 7fbfc3943200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:01.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:01.137+0000 7fbfc3943200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:01:01.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:01.378+0000 7fbfc3943200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:01:02.477 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:02.196+0000 7fbfc3943200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:01:03.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:02 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3019952380' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:01:03.032 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:02.788+0000 7fbfc3943200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:01:03.032 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:02.895+0000 7fbfc3943200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:01:03.373 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:03.122+0000 7fbfc3943200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:01:03.477 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:03.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:01:03.489 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.490 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:03.491 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:00:48.630767+0000", 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f6961d43700 1 Processor -- start 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f6961d43700 1 -- start start 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f6961d43700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f6961d43700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c108330 con 0x7f695c107940 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f695b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f695b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60348/0 (socket says 172.21.15.45:60348) 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.751+0000 7f695b7fe700 1 -- 172.21.15.45:0/2891553845 learned_addr learned my addr 172.21.15.45:0/2891553845 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:03.492 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.752+0000 7f695b7fe700 1 -- 172.21.15.45:0/2891553845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c1065a0 con 0x7f695c107940 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.752+0000 7f695b7fe700 1 --2- 172.21.15.45:0/2891553845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f694c00bb40 tx=0x7f694c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921d621fc5b71eac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.752+0000 7f695a7fc700 1 -- 172.21.15.45:0/2891553845 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f694c00dbe0 con 0x7f695c107940 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.752+0000 7f695a7fc700 1 -- 172.21.15.45:0/2891553845 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f694c0044c0 con 0x7f695c107940 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.752+0000 7f695a7fc700 1 -- 172.21.15.45:0/2891553845 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f694c003e50 con 0x7f695c107940 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 -- 172.21.15.45:0/2891553845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 msgr2=0x7f695c107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 --2- 172.21.15.45:0/2891553845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f694c00bb40 tx=0x7f694c00be50 comp rx=0 tx=0).stop 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 -- 172.21.15.45:0/2891553845 shutdown_connections 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 --2- 172.21.15.45:0/2891553845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c107940 0x7f695c107d60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 -- 172.21.15.45:0/2891553845 >> 172.21.15.45:0/2891553845 conn(0x7f695c101090 msgr2=0x7f695c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 -- 172.21.15.45:0/2891553845 shutdown_connections 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.753+0000 7f6961d43700 1 -- 172.21.15.45:0/2891553845 wait complete. 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f6961d43700 1 Processor -- start 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f6961d43700 1 -- start start 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f6961d43700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f6961d43700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c199c60 con 0x7f695c199300 2024-03-12T05:01:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f695b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f695b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60358/0 (socket says 172.21.15.45:60358) 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.754+0000 7f695b7fe700 1 -- 172.21.15.45:0/3019952380 learned_addr learned my addr 172.21.15.45:0/3019952380 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f695b7fe700 1 -- 172.21.15.45:0/3019952380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c00b7f0 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f695b7fe700 1 --2- 172.21.15.45:0/3019952380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f694c003b90 tx=0x7f694c003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f694c01e070 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f694c0109d0 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c199e60 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f694c0228e0 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c19c980 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.755+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f694c022a40 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.756+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f694c0194e0 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.756+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c105490 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.758+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f694c027070 con 0x7f695c199300 2024-03-12T05:01:03.494 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.832+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f695c1928b0 con 0x7f695c199300 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.833+0000 7f6958ff9700 1 -- 172.21.15.45:0/3019952380 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f694c010b40 con 0x7f695c199300 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 msgr2=0x7f695c199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 --2- 172.21.15.45:0/3019952380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f694c003b90 tx=0x7f694c003f50 comp rx=0 tx=0).stop 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 shutdown_connections 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 --2- 172.21.15.45:0/3019952380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f695c199300 0x7f695c199720 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 >> 172.21.15.45:0/3019952380 conn(0x7f695c101090 msgr2=0x7f695c18f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 shutdown_connections 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:02.834+0000 7f6961d43700 1 -- 172.21.15.45:0/3019952380 wait complete. 2024-03-12T05:01:03.495 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (3/15)... 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: Activating manager daemon x 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: mgrmap e2: x(active, starting, since 0.0031409s) 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: Manager daemon x is now available 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' 2024-03-12T05:01:04.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:03 smithi045 ceph-mon[32174]: from='mgr.14100 172.21.15.45:0/327639875' entity='mgr.x' 2024-03-12T05:01:05.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:05 smithi045 ceph-mon[32174]: mgrmap e3: x(active, since 1.00608s) 2024-03-12T05:01:06.769 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:06.769 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:06.769 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-12T05:01:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:01:06.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:01:06.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:01:06.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:01:06.779 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:01:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:00:48.630767+0000", 2024-03-12T05:01:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb7a696f700 1 Processor -- start 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb7a696f700 1 -- start start 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb7a696f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb7a696f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a007ae60 con 0x7fb7a0107f40 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60432/0 (socket says 172.21.15.45:60432) 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.882+0000 7fb79ffff700 1 -- 172.21.15.45:0/1272924953 learned_addr learned my addr 172.21.15.45:0/1272924953 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb79ffff700 1 -- 172.21.15.45:0/1272924953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a010a870 con 0x7fb7a0107f40 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb79ffff700 1 --2- 172.21.15.45:0/1272924953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7900098a0 tx=0x7fb790009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d97f5b850c682e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb79effd700 1 -- 172.21.15.45:0/1272924953 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb790004160 con 0x7fb7a0107f40 2024-03-12T05:01:06.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb79effd700 1 -- 172.21.15.45:0/1272924953 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb7900042c0 con 0x7fb7a0107f40 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb79effd700 1 -- 172.21.15.45:0/1272924953 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb790011a50 con 0x7fb7a0107f40 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb7a696f700 1 -- 172.21.15.45:0/1272924953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 msgr2=0x7fb7a010a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1272924953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7900098a0 tx=0x7fb790009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb7a696f700 1 -- 172.21.15.45:0/1272924953 shutdown_connections 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1272924953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a010a330 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.883+0000 7fb7a696f700 1 -- 172.21.15.45:0/1272924953 >> 172.21.15.45:0/1272924953 conn(0x7fb7a0101910 msgr2=0x7fb7a0103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.884+0000 7fb7a696f700 1 -- 172.21.15.45:0/1272924953 shutdown_connections 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.884+0000 7fb7a696f700 1 -- 172.21.15.45:0/1272924953 wait complete. 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.884+0000 7fb7a696f700 1 Processor -- start 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.884+0000 7fb7a696f700 1 -- start start 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb7a696f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb7a696f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a0107d00 con 0x7fb7a0107f40 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60438/0 (socket says 172.21.15.45:60438) 2024-03-12T05:01:06.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79ffff700 1 -- 172.21.15.45:0/1568633480 learned_addr learned my addr 172.21.15.45:0/1568633480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79ffff700 1 -- 172.21.15.45:0/1568633480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790009550 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79ffff700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb790011a00 tx=0x7fb790011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb790003a40 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb790003ba0 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79001a6e0 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a0105ef0 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.885+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a0106390 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.886+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 3) v1 ==== 45652+0+0 (secure 0 0 0) 0x7fb79001a950 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.886+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a010b610 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.886+0000 7fb79d7fa700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 0x7fb78803b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.886+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb79004c980 con 0x7fb7a0107f40 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.887+0000 7fb79f7fe700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 0x7fb78803b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.887+0000 7fb79f7fe700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 0x7fb78803b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb794006fd0 tx=0x7fb794006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:05.892+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb790018dc0 con 0x7fb7a0107f40 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.142+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7fb79002d910 con 0x7fb7a0107f40 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.180+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb7a0106920 con 0x7fb7a0107f40 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.180+0000 7fb79d7fa700 1 -- 172.21.15.45:0/1568633480 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb790018530 con 0x7fb7a0107f40 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 msgr2=0x7fb78803b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 0x7fb78803b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb794006fd0 tx=0x7fb794006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 msgr2=0x7fb7a01077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb790011a00 tx=0x7fb790011ae0 comp rx=0 tx=0).stop 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 shutdown_connections 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fb788038b90 0x7fb78803b050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 --2- 172.21.15.45:0/1568633480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7a0107f40 0x7fb7a01077c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.182+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 >> 172.21.15.45:0/1568633480 conn(0x7fb7a0101910 msgr2=0x7fb7a0102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.183+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 shutdown_connections 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:06.183+0000 7fb7a696f700 1 -- 172.21.15.45:0/1568633480 wait complete. 2024-03-12T05:01:06.785 INFO:teuthology.orchestra.run.smithi045.stdout:mgr is available 2024-03-12T05:01:07.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:07 smithi045 ceph-mon[32174]: mgrmap e4: x(active, since 3s) 2024-03-12T05:01:07.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:07 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1568633480' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-12T05:01:07.888 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff804fad700 1 Processor -- start 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff804fad700 1 -- start start 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff804fad700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff804fad700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800107ef0 con 0x7ff800107500 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60446/0 (socket says 172.21.15.45:60446) 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.130+0000 7ff7fe59c700 1 -- 172.21.15.45:0/2489888735 learned_addr learned my addr 172.21.15.45:0/2489888735 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff7fe59c700 1 -- 172.21.15.45:0/2489888735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800108660 con 0x7ff800107500 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff7fe59c700 1 --2- 172.21.15.45:0/2489888735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009e80 tx=0x7ff7f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a249ee1fbe3ba743 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff7fd59a700 1 -- 172.21.15.45:0/2489888735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f000e070 con 0x7ff800107500 2024-03-12T05:01:07.889 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff7fd59a700 1 -- 172.21.15.45:0/2489888735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff7f000b8c0 con 0x7ff800107500 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff7fd59a700 1 -- 172.21.15.45:0/2489888735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f00044b0 con 0x7ff800107500 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff804fad700 1 -- 172.21.15.45:0/2489888735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 msgr2=0x7ff800107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.131+0000 7ff804fad700 1 --2- 172.21.15.45:0/2489888735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0009e80 tx=0x7ff7f000b330 comp rx=0 tx=0).stop 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 -- 172.21.15.45:0/2489888735 shutdown_connections 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 --2- 172.21.15.45:0/2489888735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff800107920 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 -- 172.21.15.45:0/2489888735 >> 172.21.15.45:0/2489888735 conn(0x7ff800102b20 msgr2=0x7ff800104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 -- 172.21.15.45:0/2489888735 shutdown_connections 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 -- 172.21.15.45:0/2489888735 wait complete. 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 Processor -- start 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.132+0000 7ff804fad700 1 -- start start 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff804fad700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff804fad700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80007d0a0 con 0x7ff800107500 2024-03-12T05:01:07.890 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60448/0 (socket says 172.21.15.45:60448) 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7fe59c700 1 -- 172.21.15.45:0/2601309183 learned_addr learned my addr 172.21.15.45:0/2601309183 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7fe59c700 1 -- 172.21.15.45:0/2601309183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0009b30 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7fe59c700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0004070 tx=0x7ff7f00093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f000e040 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7ff7f000bd70 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7f001c7a0 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80007b290 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.133+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80007b790 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.134+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7ff7f0012430 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.134+0000 7ff7f77fe700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 0x7ff7ec03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.134+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff7f004ccd0 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.134+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8000632a0 con 0x7ff800107500 2024-03-12T05:01:07.891 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.134+0000 7ff7fdd9b700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 0x7ff7ec03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.135+0000 7ff7fdd9b700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 0x7ff7ec03b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.140+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff7f0020080 con 0x7ff800107500 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.341+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff80007baf0 con 0x7ff800107500 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.342+0000 7ff7f77fe700 1 -- 172.21.15.45:0/2601309183 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7ff7f0017020 con 0x7ff800107500 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 msgr2=0x7ff7ec03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 0x7ff7ec03b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006fd0 tx=0x7ff7e8006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 msgr2=0x7ff80007cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0004070 tx=0x7ff7f00093c0 comp rx=0 tx=0).stop 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 shutdown_connections 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7ff7ec038c60 0x7ff7ec03b120 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 --2- 172.21.15.45:0/2601309183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff800107500 0x7ff80007cb60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 >> 172.21.15.45:0/2601309183 conn(0x7ff800102b20 msgr2=0x7ff800103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 shutdown_connections 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:07.344+0000 7ff804fad700 1 -- 172.21.15.45:0/2601309183 wait complete. 2024-03-12T05:01:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling cephadm module... 2024-03-12T05:01:08.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:08 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2601309183' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T05:01:09.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:09 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3982169508' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T05:01:09.435 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:09 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setuser ceph since I am not root 2024-03-12T05:01:09.435 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:09 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setgroup ceph since I am not root 2024-03-12T05:01:09.694 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.299+0000 7f476a7df700 1 Processor -- start 2024-03-12T05:01:09.694 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.299+0000 7f476a7df700 1 -- start start 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.299+0000 7f476a7df700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.299+0000 7f476a7df700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476407ae60 con 0x7f4764107f00 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.299+0000 7f4763fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4763fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32886/0 (socket says 172.21.15.45:32886) 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4763fff700 1 -- 172.21.15.45:0/120526288 learned_addr learned my addr 172.21.15.45:0/120526288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4763fff700 1 -- 172.21.15.45:0/120526288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476410a830 con 0x7f4764107f00 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4763fff700 1 --2- 172.21.15.45:0/120526288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47540098a0 tx=0x7f4754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e27cd0af59ce26 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4762ffd700 1 -- 172.21.15.45:0/120526288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4754004160 con 0x7f4764107f00 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4762ffd700 1 -- 172.21.15.45:0/120526288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f47540042c0 con 0x7f4764107f00 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.300+0000 7f4762ffd700 1 -- 172.21.15.45:0/120526288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4754011a50 con 0x7f4764107f00 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 -- 172.21.15.45:0/120526288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 msgr2=0x7f476410a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 --2- 172.21.15.45:0/120526288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47540098a0 tx=0x7f4754009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 -- 172.21.15.45:0/120526288 shutdown_connections 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 --2- 172.21.15.45:0/120526288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f476410a2f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 -- 172.21.15.45:0/120526288 >> 172.21.15.45:0/120526288 conn(0x7f4764101930 msgr2=0x7f4764103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 -- 172.21.15.45:0/120526288 shutdown_connections 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 -- 172.21.15.45:0/120526288 wait complete. 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.301+0000 7f476a7df700 1 Processor -- start 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f476a7df700 1 -- start start 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f476a7df700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f476a7df700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764107cc0 con 0x7f4764107f00 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f4763fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f4763fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32892/0 (socket says 172.21.15.45:32892) 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f4763fff700 1 -- 172.21.15.45:0/3982169508 learned_addr learned my addr 172.21.15.45:0/3982169508 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f4763fff700 1 -- 172.21.15.45:0/3982169508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754009550 con 0x7f4764107f00 2024-03-12T05:01:09.696 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.302+0000 7f4763fff700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4754003f40 tx=0x7f4754011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.303+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4754003a40 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.303+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f4754003ba0 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.303+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764105eb0 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.303+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4754019710 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.303+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4764106350 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f475401abc0 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f47617fa700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 0x7f474c03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476410b5d0 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f475404cf30 con 0x7f4764107f00 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f47637fe700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 0x7f474c03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.304+0000 7f47637fe700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 0x7f474c03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4758006fd0 tx=0x7f4758006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:09.697 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.309+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f475402d430 con 0x7f4764107f00 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:08.571+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4764106980 con 0x7f4764107f00 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.146+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f475404b090 con 0x7f4764107f00 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.147+0000 7f47617fa700 1 -- 172.21.15.45:0/3982169508 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f475401a430 con 0x7f4764107f00 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.149+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 msgr2=0x7f474c03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.149+0000 7f476a7df700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 0x7f474c03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4758006fd0 tx=0x7f4758006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.149+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 msgr2=0x7f4764107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.149+0000 7f476a7df700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4754003f40 tx=0x7f4754011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 shutdown_connections 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7f474c038c60 0x7f474c03b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 --2- 172.21.15.45:0/3982169508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4764107f00 0x7f4764107780 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 >> 172.21.15.45:0/3982169508 conn(0x7f4764101930 msgr2=0x7f4764102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 shutdown_connections 2024-03-12T05:01:09.698 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:09.150+0000 7f476a7df700 1 -- 172.21.15.45:0/3982169508 wait complete. 2024-03-12T05:01:10.021 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:09 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:09.733+0000 7f8d47a9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:01:10.022 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:09 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:09.829+0000 7f8d47a9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:01:10.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3982169508' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T05:01:10.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-mon[32174]: mgrmap e5: x(active, since 6s) 2024-03-12T05:01:10.606 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:10.507+0000 7f8d47a9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:01:10.606 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:10.602+0000 7f8d47a9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:01:10.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:10.752+0000 7f8d47a9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:01:10.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:10.851+0000 7f8d47a9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:01:11.017 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef97510700 1 Processor -- start 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef97510700 1 -- start start 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef97510700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef97510700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9007aea0 con 0x7fef90107ed0 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef952ac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:11.018 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef952ac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32916/0 (socket says 172.21.15.45:32916) 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.118+0000 7fef952ac700 1 -- 172.21.15.45:0/2135213341 learned_addr learned my addr 172.21.15.45:0/2135213341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.119+0000 7fef952ac700 1 -- 172.21.15.45:0/2135213341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9010a800 con 0x7fef90107ed0 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.119+0000 7fef952ac700 1 --2- 172.21.15.45:0/2135213341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0098a0 tx=0x7fef8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23101da36206eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.119+0000 7fef87fff700 1 -- 172.21.15.45:0/2135213341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef8c004160 con 0x7fef90107ed0 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.119+0000 7fef87fff700 1 -- 172.21.15.45:0/2135213341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fef8c0042c0 con 0x7fef90107ed0 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.119+0000 7fef87fff700 1 -- 172.21.15.45:0/2135213341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef8c011a50 con 0x7fef90107ed0 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 -- 172.21.15.45:0/2135213341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 msgr2=0x7fef9010a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 --2- 172.21.15.45:0/2135213341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0098a0 tx=0x7fef8c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 -- 172.21.15.45:0/2135213341 shutdown_connections 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 --2- 172.21.15.45:0/2135213341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9010a2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 -- 172.21.15.45:0/2135213341 >> 172.21.15.45:0/2135213341 conn(0x7fef90101920 msgr2=0x7fef90103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 -- 172.21.15.45:0/2135213341 shutdown_connections 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 -- 172.21.15.45:0/2135213341 wait complete. 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.120+0000 7fef97510700 1 Processor -- start 2024-03-12T05:01:11.019 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef97510700 1 -- start start 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef97510700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef97510700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9019b770 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef952ac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef952ac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32932/0 (socket says 172.21.15.45:32932) 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef952ac700 1 -- 172.21.15.45:0/3231471262 learned_addr learned my addr 172.21.15.45:0/3231471262 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef952ac700 1 -- 172.21.15.45:0/3231471262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c009550 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef952ac700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef8c011710 tx=0x7fef8c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef8c003af0 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.121+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fef8c003c50 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.122+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef8c0197a0 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.122+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9019b970 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.122+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9019be10 con 0x7fef90107ed0 2024-03-12T05:01:11.020 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.122+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fef8c019a10 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.122+0000 7fef867fc700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 0x7fef7c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.123+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fef8c04cc60 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.123+0000 7fef94aab700 1 -- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 msgr2=0x7fef7c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.123+0000 7fef94aab700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 0x7fef7c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.123+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9010b600 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.128+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef8c018c70 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.324+0000 7fef94aab700 1 -- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 msgr2=0x7fef7c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.324+0000 7fef94aab700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 0x7fef7c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.389+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fef9019e8d0 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.390+0000 7fef867fc700 1 -- 172.21.15.45:0/3231471262 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fef8c04b070 con 0x7fef90107ed0 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 msgr2=0x7fef7c03b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 0x7fef7c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.021 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 msgr2=0x7fef9019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef8c011710 tx=0x7fef8c011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 shutdown_connections 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fef7c038d00 0x7fef7c03b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 --2- 172.21.15.45:0/3231471262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef90107ed0 0x7fef9019b230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 >> 172.21.15.45:0/3231471262 conn(0x7fef90101920 msgr2=0x7fef901024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 shutdown_connections 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:10.393+0000 7fef97510700 1 -- 172.21.15.45:0/3231471262 wait complete. 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-12T05:01:11.022 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 5... 2024-03-12T05:01:11.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3231471262' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T05:01:11.214 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:10.937+0000 7f8d47a9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:01:11.522 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:11.211+0000 7f8d47a9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:01:11.522 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:11.329+0000 7f8d47a9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:01:11.522 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:11.420+0000 7f8d47a9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:01:11.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:11.519+0000 7f8d47a9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:01:11.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:11.699+0000 7f8d47a9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:01:13.123 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:12 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:12.686+0000 7f8d47a9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:01:13.123 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:12 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:12.820+0000 7f8d47a9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:01:13.457 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:13.157+0000 7f8d47a9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:01:13.457 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:13.249+0000 7f8d47a9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:01:13.457 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:13.453+0000 7f8d47a9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:13.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:13.642+0000 7f8d47a9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:01:14.202 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:13.937+0000 7f8d47a9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:14.202 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:14 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:14.199+0000 7f8d47a9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:01:14.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:14 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:14.440+0000 7f8d47a9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:01:15.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:15.264+0000 7f8d47a9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:01:16.123 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:15.847+0000 7f8d47a9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:01:16.123 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:15.947+0000 7f8d47a9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:01:16.623 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:16.170+0000 7f8d47a9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:01:16.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: Active manager daemon x restarted 2024-03-12T05:01:16.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: Activating manager daemon x 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: mgrmap e6: x(active, starting, since 0.0034727s) 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: Manager daemon x is now available 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:16.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:16 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:17.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: Found migration_current of "None". Setting to last migration. 2024-03-12T05:01:17.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:17.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:01:17.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:01:17.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:16] ENGINE Bus STARTING 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:16] ENGINE Serving on https://172.21.15.45:7150 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:16] ENGINE Serving on http://172.21.15.45:8765 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:16] ENGINE Bus STARTED 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: mgrmap e7: x(active, since 1.00653s) 2024-03-12T05:01:17.524 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:17 smithi045 ceph-mon[32174]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.425+0000 7fa84856c700 1 Processor -- start 2024-03-12T05:01:17.809 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.425+0000 7fa84856c700 1 -- start start 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa84856c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa84856c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840106d50 con 0x7fa840106360 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa846308700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa846308700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32944/0 (socket says 172.21.15.45:32944) 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa846308700 1 -- 172.21.15.45:0/3277910756 learned_addr learned my addr 172.21.15.45:0/3277910756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa846308700 1 -- 172.21.15.45:0/3277910756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa840107560 con 0x7fa840106360 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa846308700 1 --2- 172.21.15.45:0/3277910756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0098a0 tx=0x7fa83c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe4157b6eefde58 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa845306700 1 -- 172.21.15.45:0/3277910756 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa83c004160 con 0x7fa840106360 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa845306700 1 -- 172.21.15.45:0/3277910756 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa83c0042c0 con 0x7fa840106360 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.426+0000 7fa845306700 1 -- 172.21.15.45:0/3277910756 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa83c011a50 con 0x7fa840106360 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 -- 172.21.15.45:0/3277910756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 msgr2=0x7fa840106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 --2- 172.21.15.45:0/3277910756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0098a0 tx=0x7fa83c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.810 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 -- 172.21.15.45:0/3277910756 shutdown_connections 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 --2- 172.21.15.45:0/3277910756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840106780 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 -- 172.21.15.45:0/3277910756 >> 172.21.15.45:0/3277910756 conn(0x7fa840101920 msgr2=0x7fa840103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 -- 172.21.15.45:0/3277910756 shutdown_connections 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.427+0000 7fa84856c700 1 -- 172.21.15.45:0/3277910756 wait complete. 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa84856c700 1 Processor -- start 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa84856c700 1 -- start start 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa84856c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa84856c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840191950 con 0x7fa840106360 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa846308700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa846308700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32950/0 (socket says 172.21.15.45:32950) 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa846308700 1 -- 172.21.15.45:0/816715984 learned_addr learned my addr 172.21.15.45:0/816715984 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa846308700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c009550 con 0x7fa840106360 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.428+0000 7fa846308700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa83c003f40 tx=0x7fa83c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.429+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa83c003a40 con 0x7fa840106360 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.429+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fa83c003ba0 con 0x7fa840106360 2024-03-12T05:01:17.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.429+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84018fb40 con 0x7fa840106360 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.429+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa83c0196f0 con 0x7fa840106360 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.429+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84018ffe0 con 0x7fa840106360 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fa83c019960 con 0x7fa840106360 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa8377fe700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa82c03b880 con 0x7fa82c038cb0 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa83c04cf90 con 0x7fa840106360 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa845b07700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.430+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.631+0000 7fa845b07700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:11.631+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:12.032+0000 7fa845b07700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:12.032+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:12.833+0000 7fa845b07700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:17.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:12.833+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:14.434+0000 7fa845b07700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1305486259 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:14.434+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:16.185+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7fa83c01abe0 con 0x7fa840106360 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:16.185+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 msgr2=0x7fa82c03b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:16.185+0000 7fa8377fe700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.186+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fa83c04e860 con 0x7fa840106360 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.187+0000 7fa8377fe700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 0x7fa82c03e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.187+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa82c03b880 con 0x7fa82c03c5a0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.187+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 0x7fa82c03e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.188+0000 7fa845b07700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 0x7fa82c03e990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa830003a10 tx=0x7fa8300092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.188+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fa82c03b880 con 0x7fa82c03c5a0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.195+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa8401083d0 con 0x7fa82c03c5a0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.195+0000 7fa8377fe700 1 -- 172.21.15.45:0/816715984 <== mgr.14120 v2:172.21.15.45:6800/2524215406 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa83000a050 con 0x7fa82c03c5a0 2024-03-12T05:01:17.813 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 msgr2=0x7fa82c03e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 0x7fa82c03e990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa830003a10 tx=0x7fa8300092b0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 msgr2=0x7fa840191410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa83c003f40 tx=0x7fa83c011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 shutdown_connections 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fa82c03c5a0 0x7fa82c03e990 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa840106360 0x7fa840191410 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 --2- 172.21.15.45:0/816715984 >> [v2:172.21.15.45:6800/1305486259,v1:172.21.15.45:6801/1305486259] conn(0x7fa82c038cb0 0x7fa82c03b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 >> 172.21.15.45:0/816715984 conn(0x7fa840101920 msgr2=0x7fa840102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 shutdown_connections 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:17.196+0000 7fa84856c700 1 -- 172.21.15.45:0/816715984 wait complete. 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 5 is available 2024-03-12T05:01:17.814 INFO:teuthology.orchestra.run.smithi045.stdout:Setting orchestrator backend to cephadm... 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.219+0000 7fb5ed26e700 1 Processor -- start 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.219+0000 7fb5ed26e700 1 -- start start 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.219+0000 7fb5ed26e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.219+0000 7fb5ed26e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e807ae20 con 0x7fb5e8107fc0 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:18.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51018/0 (socket says 172.21.15.45:51018) 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e6ffd700 1 -- 172.21.15.45:0/826093695 learned_addr learned my addr 172.21.15.45:0/826093695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e6ffd700 1 -- 172.21.15.45:0/826093695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e810a8f0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e6ffd700 1 --2- 172.21.15.45:0/826093695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009e80 tx=0x7fb5e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b406695651c47c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e5ffb700 1 -- 172.21.15.45:0/826093695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5e000e070 con 0x7fb5e8107fc0 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e5ffb700 1 -- 172.21.15.45:0/826093695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb5e000b8c0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.220+0000 7fb5e5ffb700 1 -- 172.21.15.45:0/826093695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5e00044b0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 -- 172.21.15.45:0/826093695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 msgr2=0x7fb5e810a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/826093695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009e80 tx=0x7fb5e000b330 comp rx=0 tx=0).stop 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 -- 172.21.15.45:0/826093695 shutdown_connections 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/826093695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e810a3b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 -- 172.21.15.45:0/826093695 >> 172.21.15.45:0/826093695 conn(0x7fb5e8101990 msgr2=0x7fb5e8103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 -- 172.21.15.45:0/826093695 shutdown_connections 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 -- 172.21.15.45:0/826093695 wait complete. 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.221+0000 7fb5ed26e700 1 Processor -- start 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5ed26e700 1 -- start start 2024-03-12T05:01:18.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5ed26e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5ed26e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e819b670 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51026/0 (socket says 172.21.15.45:51026) 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5e6ffd700 1 -- 172.21.15.45:0/1612526835 learned_addr learned my addr 172.21.15.45:0/1612526835 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5e6ffd700 1 -- 172.21.15.45:0/1612526835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0009b30 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.222+0000 7fb5e6ffd700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0004070 tx=0x7fb5e00093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5e000e040 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb5e000bd70 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e819b870 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5e001c7a0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e819bd70 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.223+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fb5e001c990 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.224+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e810b640 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.224+0000 7fb5dffff700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 0x7fb5d403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.224+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb5e004cdb0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.224+0000 7fb5e67fc700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 0x7fb5d403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:18.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.224+0000 7fb5e67fc700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 0x7fb5d403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006fd0 tx=0x7fb5d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.229+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb5e0011980 con 0x7fb5e8107fc0 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.237+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fb5e0012be0 con 0x7fb5e8107fc0 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.448+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5e8102aa0 con 0x7fb5d4038ba0 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.453+0000 7fb5dffff700 1 -- 172.21.15.45:0/1612526835 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb5e8102aa0 con 0x7fb5d4038ba0 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 msgr2=0x7fb5d403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 0x7fb5d403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006fd0 tx=0x7fb5d0006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 msgr2=0x7fb5e819b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0004070 tx=0x7fb5e00093c0 comp rx=0 tx=0).stop 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 shutdown_connections 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb5d4038ba0 0x7fb5d403b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 --2- 172.21.15.45:0/1612526835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e8107fc0 0x7fb5e819b130 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 >> 172.21.15.45:0/1612526835 conn(0x7fb5e8101990 msgr2=0x7fb5e8102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 shutdown_connections 2024-03-12T05:01:18.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:18.455+0000 7fb5ed26e700 1 -- 172.21.15.45:0/1612526835 wait complete. 2024-03-12T05:01:19.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:19 smithi045 ceph-mon[32174]: mgrmap e8: x(active, since 2s) 2024-03-12T05:01:19.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:19 smithi045 ceph-mon[32174]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:19.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:19 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:19.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:19 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.326+0000 7f2646ac3700 1 Processor -- start 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.326+0000 7f2646ac3700 1 -- start start 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.326+0000 7f2646ac3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.326+0000 7f2646ac3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264007b870 con 0x7f264007ce30 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f264485f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f264485f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51030/0 (socket says 172.21.15.45:51030) 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f264485f700 1 -- 172.21.15.45:0/2456057442 learned_addr learned my addr 172.21.15.45:0/2456057442 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f264485f700 1 -- 172.21.15.45:0/2456057442 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264007b9b0 con 0x7f264007ce30 2024-03-12T05:01:20.079 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f264485f700 1 --2- 172.21.15.45:0/2456057442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e80 tx=0x7f263800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d188774be681c763 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f263f7fe700 1 -- 172.21.15.45:0/2456057442 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f263800e070 con 0x7f264007ce30 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f263f7fe700 1 -- 172.21.15.45:0/2456057442 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f263800b8c0 con 0x7f264007ce30 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.327+0000 7f263f7fe700 1 -- 172.21.15.45:0/2456057442 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26380044b0 con 0x7f264007ce30 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 -- 172.21.15.45:0/2456057442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 msgr2=0x7f264007b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2456057442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2638009e80 tx=0x7f263800b330 comp rx=0 tx=0).stop 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 -- 172.21.15.45:0/2456057442 shutdown_connections 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2456057442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264007b330 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 -- 172.21.15.45:0/2456057442 >> 172.21.15.45:0/2456057442 conn(0x7f2640102ac0 msgr2=0x7f2640104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 -- 172.21.15.45:0/2456057442 shutdown_connections 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.328+0000 7f2646ac3700 1 -- 172.21.15.45:0/2456057442 wait complete. 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f2646ac3700 1 Processor -- start 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f2646ac3700 1 -- start start 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f2646ac3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f2646ac3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264019d900 con 0x7f264007ce30 2024-03-12T05:01:20.080 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f264485f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f264485f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51046/0 (socket says 172.21.15.45:51046) 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f264485f700 1 -- 172.21.15.45:0/2654001849 learned_addr learned my addr 172.21.15.45:0/2654001849 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f264485f700 1 -- 172.21.15.45:0/2654001849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2638009b30 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f264485f700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2638004240 tx=0x7f2638004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.329+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f263800e040 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.330+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f263800bd70 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.330+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264019db00 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.330+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f263801c6b0 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.330+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264019e000 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.330+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f263801c8a0 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.331+0000 7f263dffb700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 0x7f262803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.331+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f263804cdf0 con 0x7f264007ce30 2024-03-12T05:01:20.081 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.331+0000 7f263ffff700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 0x7f262803b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.331+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264010d7d0 con 0x7f264007ce30 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.331+0000 7f263ffff700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 0x7f262803b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.336+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2638017070 con 0x7f264007ce30 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.539+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f2640103d10 con 0x7f2628038d00 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.539+0000 7f263dffb700 1 -- 172.21.15.45:0/2654001849 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f2640103d10 con 0x7f2628038d00 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 msgr2=0x7f262803b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 0x7f262803b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 msgr2=0x7f264019d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2638004240 tx=0x7f2638004600 comp rx=0 tx=0).stop 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 shutdown_connections 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2628038d00 0x7f262803b1c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 --2- 172.21.15.45:0/2654001849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f264007ce30 0x7f264019d3c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.541+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 >> 172.21.15.45:0/2654001849 conn(0x7f2640102ac0 msgr2=0x7f2640103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:20.082 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.542+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 shutdown_connections 2024-03-12T05:01:20.083 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:19.542+0000 7f2646ac3700 1 -- 172.21.15.45:0/2654001849 wait complete. 2024-03-12T05:01:20.083 INFO:teuthology.orchestra.run.smithi045.stdout:Generating ssh key... 2024-03-12T05:01:20.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:20 smithi045 ceph-mon[32174]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:21.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-mon[32174]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:21.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-mon[32174]: Generating ssh key... 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: Generating public/private rsa key pair. 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: Your identification has been saved in /tmp/tmpvgujy67s/key. 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: Your public key has been saved in /tmp/tmpvgujy67s/key.pub. 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: The key fingerprint is: 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: SHA256:GIA61f3D0uBWDqfL8g1aS9ilZ05yYzKQ2q/lnwB/wGo ceph-52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: The key's randomart image is: 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: +---[RSA 3072]----+ 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | o.. | 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | o ..+ o | 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | o o.@ | 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: |o o.=oB | 2024-03-12T05:01:21.561 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | . o.*+=S. | 2024-03-12T05:01:21.562 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | . ++%.B | 2024-03-12T05:01:21.562 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | EB+#.. | 2024-03-12T05:01:21.562 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | ..o+oo. | 2024-03-12T05:01:21.562 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: | ....o | 2024-03-12T05:01:21.562 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: +----[SHA256]-----+ 2024-03-12T05:01:21.827 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f2170aaf700 1 Processor -- start 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f2170aaf700 1 -- start start 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f2170aaf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f2170aaf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c106d50 con 0x7f216c106360 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f216a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f216a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51054/0 (socket says 172.21.15.45:51054) 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.518+0000 7f216a59c700 1 -- 172.21.15.45:0/3963337983 learned_addr learned my addr 172.21.15.45:0/3963337983 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f216a59c700 1 -- 172.21.15.45:0/3963337983 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c107560 con 0x7f216c106360 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f216a59c700 1 --2- 172.21.15.45:0/3963337983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f215c0098a0 tx=0x7f215c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e887a2999f6d4d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f216959a700 1 -- 172.21.15.45:0/3963337983 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f215c004160 con 0x7f216c106360 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f216959a700 1 -- 172.21.15.45:0/3963337983 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f215c0042c0 con 0x7f216c106360 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f216959a700 1 -- 172.21.15.45:0/3963337983 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f215c011a50 con 0x7f216c106360 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f2170aaf700 1 -- 172.21.15.45:0/3963337983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 msgr2=0x7f216c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:21.828 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.519+0000 7f2170aaf700 1 --2- 172.21.15.45:0/3963337983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f215c0098a0 tx=0x7f215c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 -- 172.21.15.45:0/3963337983 shutdown_connections 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 --2- 172.21.15.45:0/3963337983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c106780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 -- 172.21.15.45:0/3963337983 >> 172.21.15.45:0/3963337983 conn(0x7f216c101920 msgr2=0x7f216c103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 -- 172.21.15.45:0/3963337983 shutdown_connections 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 -- 172.21.15.45:0/3963337983 wait complete. 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.520+0000 7f2170aaf700 1 Processor -- start 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f2170aaf700 1 -- start start 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f2170aaf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f2170aaf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c19b7c0 con 0x7f216c106360 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f216a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:21.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f216a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51070/0 (socket says 172.21.15.45:51070) 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f216a59c700 1 -- 172.21.15.45:0/1170946089 learned_addr learned my addr 172.21.15.45:0/1170946089 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f216a59c700 1 -- 172.21.15.45:0/1170946089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c009550 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f216a59c700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f215c003f40 tx=0x7f215c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f215c003a40 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f215c003ba0 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.521+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f215c0196f0 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.522+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c19b9c0 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.522+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c19be60 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.522+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f215c019960 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.523+0000 7f215b7fe700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 0x7f215403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.523+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f215c04ce40 con 0x7f216c106360 2024-03-12T05:01:21.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.523+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c10a790 con 0x7f216c106360 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.523+0000 7f2169d9b700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 0x7f215403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.523+0000 7f2169d9b700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 0x7f215403b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2160006fd0 tx=0x7f2160006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.528+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f215c019e40 con 0x7f216c106360 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:20.731+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f216c102c00 con 0x7f2154038cb0 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.322+0000 7f215b7fe700 1 -- 172.21.15.45:0/1170946089 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f216c102c00 con 0x7f2154038cb0 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 msgr2=0x7f215403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 0x7f215403b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2160006fd0 tx=0x7f2160006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 msgr2=0x7f216c19b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f215c003f40 tx=0x7f215c011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 shutdown_connections 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2154038cb0 0x7f215403b170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 --2- 172.21.15.45:0/1170946089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f216c106360 0x7f216c19b280 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:21.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.324+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 >> 172.21.15.45:0/1170946089 conn(0x7f216c101920 msgr2=0x7f216c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:21.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.325+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 shutdown_connections 2024-03-12T05:01:21.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:21.325+0000 7f2170aaf700 1 -- 172.21.15.45:0/1170946089 wait complete. 2024-03-12T05:01:22.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:22 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:22.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:22 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:23.000 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:23.000 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.242+0000 7fb1a44a9700 1 Processor -- start 2024-03-12T05:01:23.000 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.242+0000 7fb1a44a9700 1 -- start start 2024-03-12T05:01:23.000 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.242+0000 7fb1a44a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:23.000 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.242+0000 7fb1a44a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c106150 con 0x7fb19c105760 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a2245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a2245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51074/0 (socket says 172.21.15.45:51074) 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a2245700 1 -- 172.21.15.45:0/2389120024 learned_addr learned my addr 172.21.15.45:0/2389120024 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a2245700 1 -- 172.21.15.45:0/2389120024 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c1086f0 con 0x7fb19c105760 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a2245700 1 --2- 172.21.15.45:0/2389120024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009e80 tx=0x7fb18c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea1c5cb42fbc767 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a1243700 1 -- 172.21.15.45:0/2389120024 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c00e070 con 0x7fb19c105760 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a1243700 1 -- 172.21.15.45:0/2389120024 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb18c00b8c0 con 0x7fb19c105760 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.243+0000 7fb1a1243700 1 -- 172.21.15.45:0/2389120024 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c0044b0 con 0x7fb19c105760 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 -- 172.21.15.45:0/2389120024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 msgr2=0x7fb19c105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/2389120024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009e80 tx=0x7fb18c00b330 comp rx=0 tx=0).stop 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 -- 172.21.15.45:0/2389120024 shutdown_connections 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/2389120024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c105b80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 -- 172.21.15.45:0/2389120024 >> 172.21.15.45:0/2389120024 conn(0x7fb19c101090 msgr2=0x7fb19c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:23.001 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 -- 172.21.15.45:0/2389120024 shutdown_connections 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.244+0000 7fb1a44a9700 1 -- 172.21.15.45:0/2389120024 wait complete. 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 Processor -- start 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 -- start start 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19c1084b0 con 0x7fb19c105760 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a2245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a2245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51088/0 (socket says 172.21.15.45:51088) 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a2245700 1 -- 172.21.15.45:0/1962273155 learned_addr learned my addr 172.21.15.45:0/1962273155 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a2245700 1 -- 172.21.15.45:0/1962273155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009b30 con 0x7fb19c105760 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a2245700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb18c004100 tx=0x7fb18c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c00e040 con 0x7fb19c105760 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7fb18c00bd70 con 0x7fb19c105760 2024-03-12T05:01:23.002 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c1066a0 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c01c670 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.245+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c106ba0 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.246+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fb18c012430 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.246+0000 7fb19b7fe700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 0x7fb18403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.246+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c0632a0 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.246+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb18c04cb00 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.247+0000 7fb1a1a44700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 0x7fb18403b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.247+0000 7fb1a1a44700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 0x7fb18403b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.252+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb18c012b10 con 0x7fb19c105760 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.455+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb19c1071a0 con 0x7fb184038d00 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.455+0000 7fb19b7fe700 1 -- 172.21.15.45:0/1962273155 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb19c1071a0 con 0x7fb184038d00 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 msgr2=0x7fb18403b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 0x7fb18403b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 msgr2=0x7fb19c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:23.003 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb18c004100 tx=0x7fb18c004600 comp rx=0 tx=0).stop 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 shutdown_connections 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fb184038d00 0x7fb18403b1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 --2- 172.21.15.45:0/1962273155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb19c105760 0x7fb19c107f70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 >> 172.21.15.45:0/1962273155 conn(0x7fb19c101090 msgr2=0x7fb19c101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 shutdown_connections 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:22.458+0000 7fb1a44a9700 1 -- 172.21.15.45:0/1962273155 wait complete. 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T05:01:23.004 INFO:teuthology.orchestra.run.smithi045.stdout:Adding host smithi045... 2024-03-12T05:01:23.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:23 smithi045 ceph-mon[32174]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:24.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:24 smithi045 ceph-mon[32174]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:25.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:25 smithi045 ceph-mon[32174]: Deploying cephadm binary to smithi045 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Added host 'smithi045' with addr '172.21.15.45' 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a2b16f700 1 Processor -- start 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a2b16f700 1 -- start start 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a2b16f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a2b16f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2407ae40 con 0x7f6a24107f90 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a28f0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a28f0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51092/0 (socket says 172.21.15.45:51092) 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a28f0b700 1 -- 172.21.15.45:0/2422693846 learned_addr learned my addr 172.21.15.45:0/2422693846 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.384+0000 7f6a28f0b700 1 -- 172.21.15.45:0/2422693846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2410a8c0 con 0x7f6a24107f90 2024-03-12T05:01:28.441 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a28f0b700 1 --2- 172.21.15.45:0/2422693846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a180098a0 tx=0x7f6a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757b1ebdf653ddac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a237fe700 1 -- 172.21.15.45:0/2422693846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18004160 con 0x7f6a24107f90 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a237fe700 1 -- 172.21.15.45:0/2422693846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6a180042c0 con 0x7f6a24107f90 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a237fe700 1 -- 172.21.15.45:0/2422693846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18011a50 con 0x7f6a24107f90 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a2b16f700 1 -- 172.21.15.45:0/2422693846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 msgr2=0x7f6a2410a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/2422693846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a180098a0 tx=0x7f6a18009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a2b16f700 1 -- 172.21.15.45:0/2422693846 shutdown_connections 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/2422693846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a2410a380 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.385+0000 7f6a2b16f700 1 -- 172.21.15.45:0/2422693846 >> 172.21.15.45:0/2422693846 conn(0x7f6a241019a0 msgr2=0x7f6a24103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 -- 172.21.15.45:0/2422693846 shutdown_connections 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 -- 172.21.15.45:0/2422693846 wait complete. 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 Processor -- start 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 -- start start 2024-03-12T05:01:28.442 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a2b16f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24107d50 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a28f0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a28f0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51106/0 (socket says 172.21.15.45:51106) 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.386+0000 7f6a28f0b700 1 -- 172.21.15.45:0/3860138280 learned_addr learned my addr 172.21.15.45:0/3860138280 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a28f0b700 1 -- 172.21.15.45:0/3860138280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009550 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a28f0b700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a18003f40 tx=0x7f6a18011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18003a40 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f6a18003ba0 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a24105f40 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a18019710 con 0x7f6a24107f90 2024-03-12T05:01:28.443 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.387+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a241063e0 con 0x7f6a24107f90 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.388+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f6a18019980 con 0x7f6a24107f90 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.388+0000 7f6a21ffb700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 0x7f6a1403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.388+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6a1804cc10 con 0x7f6a24107f90 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.388+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2410b660 con 0x7f6a24107f90 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.388+0000 7f6a23fff700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 0x7f6a1403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.389+0000 7f6a23fff700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 0x7f6a1403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.393+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a180243f0 con 0x7f6a24107f90 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:23.600+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a241069d0 con 0x7f6a14038cb0 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.863+0000 7f6a21ffb700 1 -- 172.21.15.45:0/3860138280 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f6a241069d0 con 0x7f6a14038cb0 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 msgr2=0x7f6a1403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 0x7f6a1403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:28.444 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 msgr2=0x7f6a24107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a18003f40 tx=0x7f6a18011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 shutdown_connections 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f6a14038cb0 0x7f6a1403b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 --2- 172.21.15.45:0/3860138280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a24107f90 0x7f6a24107810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 >> 172.21.15.45:0/3860138280 conn(0x7f6a241019a0 msgr2=0x7f6a241024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 shutdown_connections 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:27.866+0000 7f6a2b16f700 1 -- 172.21.15.45:0/3860138280 wait complete. 2024-03-12T05:01:28.445 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mon service... 2024-03-12T05:01:29.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:28 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:29.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:28 smithi045 ceph-mon[32174]: Added host smithi045 2024-03-12T05:01:29.065 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:28 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:30.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:30 smithi045 ceph-mon[32174]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:30.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:30 smithi045 ceph-mon[32174]: Saving service mon spec with placement count:5 2024-03-12T05:01:30.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:30 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.879+0000 7f6037efb700 1 Processor -- start 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.879+0000 7f6037efb700 1 -- start start 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.879+0000 7f6037efb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6037efb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603007ae60 con 0x7f6030107f00 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6035c97700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6035c97700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59798/0 (socket says 172.21.15.45:59798) 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6035c97700 1 -- 172.21.15.45:0/2200299692 learned_addr learned my addr 172.21.15.45:0/2200299692 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6035c97700 1 -- 172.21.15.45:0/2200299692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603010a830 con 0x7f6030107f00 2024-03-12T05:01:30.599 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6035c97700 1 --2- 172.21.15.45:0/2200299692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f602c0098a0 tx=0x7f602c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0659571b8d9ad8f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6034c95700 1 -- 172.21.15.45:0/2200299692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f602c004160 con 0x7f6030107f00 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6034c95700 1 -- 172.21.15.45:0/2200299692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f602c0042c0 con 0x7f6030107f00 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.880+0000 7f6034c95700 1 -- 172.21.15.45:0/2200299692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f602c011a50 con 0x7f6030107f00 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 -- 172.21.15.45:0/2200299692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 msgr2=0x7f603010a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 --2- 172.21.15.45:0/2200299692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f602c0098a0 tx=0x7f602c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 -- 172.21.15.45:0/2200299692 shutdown_connections 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 --2- 172.21.15.45:0/2200299692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603010a2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 -- 172.21.15.45:0/2200299692 >> 172.21.15.45:0/2200299692 conn(0x7f6030101930 msgr2=0x7f6030103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 -- 172.21.15.45:0/2200299692 shutdown_connections 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 -- 172.21.15.45:0/2200299692 wait complete. 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.881+0000 7f6037efb700 1 Processor -- start 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6037efb700 1 -- start start 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6037efb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6037efb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603019b770 con 0x7f6030107f00 2024-03-12T05:01:30.600 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6035c97700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6035c97700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59806/0 (socket says 172.21.15.45:59806) 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6035c97700 1 -- 172.21.15.45:0/2503236736 learned_addr learned my addr 172.21.15.45:0/2503236736 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6035c97700 1 -- 172.21.15.45:0/2503236736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c009550 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6035c97700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f602c003f40 tx=0x7f602c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f602c003a40 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1071+0+0 (secure 0 0 0) 0x7f602c003ba0 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603019b970 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f602c019710 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.882+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603019be10 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.883+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f602c01a430 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.883+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603010b5d0 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.884+0000 7f6026ffd700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 0x7f601c03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.884+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f602c04cc30 con 0x7f6030107f00 2024-03-12T05:01:30.601 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.884+0000 7f6035496700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 0x7f601c03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.884+0000 7f6035496700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 0x7f601c03b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6020006fd0 tx=0x7f6020006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:28.888+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f602c023750 con 0x7f6030107f00 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.108+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6030102c40 con 0x7f601c038cb0 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.113+0000 7f6026ffd700 1 -- 172.21.15.45:0/2503236736 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6030102c40 con 0x7f601c038cb0 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 msgr2=0x7f601c03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 0x7f601c03b170 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6020006fd0 tx=0x7f6020006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 msgr2=0x7f603019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f602c003f40 tx=0x7f602c011ab0 comp rx=0 tx=0).stop 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 shutdown_connections 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f601c038cb0 0x7f601c03b170 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 --2- 172.21.15.45:0/2503236736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6030107f00 0x7f603019b230 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 >> 172.21.15.45:0/2503236736 conn(0x7f6030101930 msgr2=0x7f6030102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 shutdown_connections 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:29.116+0000 7f6037efb700 1 -- 172.21.15.45:0/2503236736 wait complete. 2024-03-12T05:01:30.602 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mgr service... 2024-03-12T05:01:31.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:31 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:31.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:31 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.017+0000 7f5c60a4b700 1 Processor -- start 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.017+0000 7f5c60a4b700 1 -- start start 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c60a4b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c60a4b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c07ae20 con 0x7f5c5c105d30 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59808/0 (socket says 172.21.15.45:59808) 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c5a59c700 1 -- 172.21.15.45:0/464109206 learned_addr learned my addr 172.21.15.45:0/464109206 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:31.780 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c5a59c700 1 -- 172.21.15.45:0/464109206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c106690 con 0x7f5c5c105d30 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.018+0000 7f5c5a59c700 1 --2- 172.21.15.45:0/464109206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098a0 tx=0x7f5c4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d37c2673a99ea83 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c5959a700 1 -- 172.21.15.45:0/464109206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c004160 con 0x7f5c5c105d30 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c5959a700 1 -- 172.21.15.45:0/464109206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c4c0042c0 con 0x7f5c5c105d30 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c5959a700 1 -- 172.21.15.45:0/464109206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c0119b0 con 0x7f5c5c105d30 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 -- 172.21.15.45:0/464109206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 msgr2=0x7f5c5c106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/464109206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098a0 tx=0x7f5c4c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 -- 172.21.15.45:0/464109206 shutdown_connections 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/464109206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c106150 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 -- 172.21.15.45:0/464109206 >> 172.21.15.45:0/464109206 conn(0x7f5c5c101930 msgr2=0x7f5c5c103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 -- 172.21.15.45:0/464109206 shutdown_connections 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.019+0000 7f5c60a4b700 1 -- 172.21.15.45:0/464109206 wait complete. 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c60a4b700 1 Processor -- start 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c60a4b700 1 -- start start 2024-03-12T05:01:31.781 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c60a4b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c60a4b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19d990 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59810/0 (socket says 172.21.15.45:59810) 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.020+0000 7f5c5a59c700 1 -- 172.21.15.45:0/3743644287 learned_addr learned my addr 172.21.15.45:0/3743644287 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c5a59c700 1 -- 172.21.15.45:0/3743644287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009550 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c5a59c700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c011670 tx=0x7f5c4c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c0038b0 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5c4c003a10 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c19db90 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c019550 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.021+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c19e030 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.022+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f5c4c019810 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.022+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c1970a0 con 0x7f5c5c105d30 2024-03-12T05:01:31.782 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.022+0000 7f5c4b7fe700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 0x7f5c4403b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.022+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5c4c04cef0 con 0x7f5c5c105d30 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.022+0000 7f5c59d9b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 0x7f5c4403b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.023+0000 7f5c59d9b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 0x7f5c4403b120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5c50006fd0 tx=0x7f5c50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.027+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c4c018a40 con 0x7f5c5c105d30 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.248+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5c02d950 con 0x7f5c44038c60 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.253+0000 7f5c4b7fe700 1 -- 172.21.15.45:0/3743644287 <== mgr.14120 v2:172.21.15.45:6800/2524215406 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5c5c02d950 con 0x7f5c44038c60 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.255+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 msgr2=0x7f5c4403b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.255+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 0x7f5c4403b120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5c50006fd0 tx=0x7f5c50006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.255+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 msgr2=0x7f5c5c19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.255+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c011670 tx=0x7f5c4c011a30 comp rx=0 tx=0).stop 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 shutdown_connections 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f5c44038c60 0x7f5c4403b120 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 --2- 172.21.15.45:0/3743644287 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5c5c105d30 0x7f5c5c19d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:31.783 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 >> 172.21.15.45:0/3743644287 conn(0x7f5c5c101930 msgr2=0x7f5c5c1941e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:31.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 shutdown_connections 2024-03-12T05:01:31.784 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:31.256+0000 7f5c60a4b700 1 -- 172.21.15.45:0/3743644287 wait complete. 2024-03-12T05:01:32.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:32 smithi045 ceph-mon[32174]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:32.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:32 smithi045 ceph-mon[32174]: Saving service mgr spec with placement count:2 2024-03-12T05:01:32.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:32 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3216cf6700 1 Processor -- start 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3216cf6700 1 -- start start 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3216cf6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3216cf6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210106da0 con 0x7f32101063b0 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3214a92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3214a92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59816/0 (socket says 172.21.15.45:59816) 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3214a92700 1 -- 172.21.15.45:0/959056957 learned_addr learned my addr 172.21.15.45:0/959056957 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3214a92700 1 -- 172.21.15.45:0/959056957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32101075b0 con 0x7f32101063b0 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.191+0000 7f3214a92700 1 --2- 172.21.15.45:0/959056957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a850d61904ef6811 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f320f7fe700 1 -- 172.21.15.45:0/959056957 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3204004160 con 0x7f32101063b0 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f320f7fe700 1 -- 172.21.15.45:0/959056957 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f32040042c0 con 0x7f32101063b0 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f320f7fe700 1 -- 172.21.15.45:0/959056957 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32040119b0 con 0x7f32101063b0 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 -- 172.21.15.45:0/959056957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 msgr2=0x7f32101067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 --2- 172.21.15.45:0/959056957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 -- 172.21.15.45:0/959056957 shutdown_connections 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 --2- 172.21.15.45:0/959056957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f32101067d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 -- 172.21.15.45:0/959056957 >> 172.21.15.45:0/959056957 conn(0x7f3210101950 msgr2=0x7f3210103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 -- 172.21.15.45:0/959056957 shutdown_connections 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.192+0000 7f3216cf6700 1 -- 172.21.15.45:0/959056957 wait complete. 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3216cf6700 1 Processor -- start 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3216cf6700 1 -- start start 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3216cf6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3216cf6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321007af80 con 0x7f32101063b0 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3214a92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:33.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3214a92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59820/0 (socket says 172.21.15.45:59820) 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3214a92700 1 -- 172.21.15.45:0/519192413 learned_addr learned my addr 172.21.15.45:0/519192413 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.193+0000 7f3214a92700 1 -- 172.21.15.45:0/519192413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3204009550 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f3214a92700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3204011980 tx=0x7f3204011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3204003970 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3204003ad0 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321019df60 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3204019610 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.194+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321019e400 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f32040198d0 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321010a7e0 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f320dffb700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 0x7f320003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f320404cd50 con 0x7f32101063b0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f320ffff700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 0x7f320003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.195+0000 7f320ffff700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 0x7f320003b170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006fd0 tx=0x7f31fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:33.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.200+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3204023c20 con 0x7f32101063b0 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.419+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f321010a9f0 con 0x7f32101063b0 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.421+0000 7f320dffb700 1 -- 172.21.15.45:0/519192413 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3204023c20 con 0x7f32101063b0 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 msgr2=0x7f320003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 0x7f320003b170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006fd0 tx=0x7f31fc006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 msgr2=0x7f321007aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3204011980 tx=0x7f3204011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 shutdown_connections 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f3200038cb0 0x7f320003b170 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 --2- 172.21.15.45:0/519192413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f32101063b0 0x7f321007aa40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 >> 172.21.15.45:0/519192413 conn(0x7f3210101950 msgr2=0x7f3210102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 shutdown_connections 2024-03-12T05:01:33.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:32.423+0000 7f3216cf6700 1 -- 172.21.15.45:0/519192413 wait complete. 2024-03-12T05:01:33.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:33 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/519192413' entity='client.admin' 2024-03-12T05:01:33.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:33 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:34.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.772+0000 7f950e86a700 1 Processor -- start 2024-03-12T05:01:34.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.772+0000 7f950e86a700 1 -- start start 2024-03-12T05:01:34.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.772+0000 7f950e86a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:34.631 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.772+0000 7f950e86a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508106d70 con 0x7f9508106380 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9507fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9507fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59834/0 (socket says 172.21.15.45:59834) 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9507fff700 1 -- 172.21.15.45:0/2018683321 learned_addr learned my addr 172.21.15.45:0/2018683321 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9507fff700 1 -- 172.21.15.45:0/2018683321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508107580 con 0x7f9508106380 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9507fff700 1 --2- 172.21.15.45:0/2018683321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94f80098a0 tx=0x7f94f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bb8737b6682909 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9506ffd700 1 -- 172.21.15.45:0/2018683321 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f8004160 con 0x7f9508106380 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9506ffd700 1 -- 172.21.15.45:0/2018683321 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f94f80042c0 con 0x7f9508106380 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.773+0000 7f9506ffd700 1 -- 172.21.15.45:0/2018683321 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f80119b0 con 0x7f9508106380 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 -- 172.21.15.45:0/2018683321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 msgr2=0x7f95081067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 --2- 172.21.15.45:0/2018683321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94f80098a0 tx=0x7f94f8009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 -- 172.21.15.45:0/2018683321 shutdown_connections 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 --2- 172.21.15.45:0/2018683321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f95081067a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 -- 172.21.15.45:0/2018683321 >> 172.21.15.45:0/2018683321 conn(0x7f9508101940 msgr2=0x7f9508103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 -- 172.21.15.45:0/2018683321 shutdown_connections 2024-03-12T05:01:34.632 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.774+0000 7f950e86a700 1 -- 172.21.15.45:0/2018683321 wait complete. 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.775+0000 7f950e86a700 1 Processor -- start 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.775+0000 7f950e86a700 1 -- start start 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.775+0000 7f950e86a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.775+0000 7f950e86a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950819b7c0 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f9507fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f9507fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59838/0 (socket says 172.21.15.45:59838) 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f9507fff700 1 -- 172.21.15.45:0/3951079692 learned_addr learned my addr 172.21.15.45:0/3951079692 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f9507fff700 1 -- 172.21.15.45:0/3951079692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8009550 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f9507fff700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94f8011670 tx=0x7f94f8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f80038b0 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950819b9c0 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f94f8003a10 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f8019530 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.776+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950819be60 con 0x7f9508106380 2024-03-12T05:01:34.633 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.777+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f94f80197a0 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.778+0000 7f95057fa700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 0x7f94f003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.778+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f94f804d190 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.778+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9508195420 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.778+0000 7f95077fe700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 0x7f94f003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.778+0000 7f95077fe700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 0x7f94f003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94fc006fd0 tx=0x7f94fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.783+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f94f8019aa0 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.989+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f950810a7b0 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.991+0000 7f95057fa700 1 -- 172.21.15.45:0/3951079692 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f94f8018be0 con 0x7f9508106380 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 msgr2=0x7f94f003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 0x7f94f003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94fc006fd0 tx=0x7f94fc006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 msgr2=0x7f950819b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94f8011670 tx=0x7f94f8011a30 comp rx=0 tx=0).stop 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 shutdown_connections 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f94f0038cb0 0x7f94f003b170 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:34.634 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 --2- 172.21.15.45:0/3951079692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9508106380 0x7f950819b280 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:34.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 >> 172.21.15.45:0/3951079692 conn(0x7f9508101940 msgr2=0x7f9508102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:34.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 shutdown_connections 2024-03-12T05:01:34.635 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:33.994+0000 7f950e86a700 1 -- 172.21.15.45:0/3951079692 wait complete. 2024-03-12T05:01:34.635 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling the dashboard module... 2024-03-12T05:01:35.243 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3951079692' entity='client.admin' 2024-03-12T05:01:35.243 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:34 smithi045 ceph-mon[32174]: from='mgr.14120 172.21.15.45:0/1754983275' entity='mgr.x' 2024-03-12T05:01:36.318 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3096262312' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T05:01:36.318 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setuser ceph since I am not root 2024-03-12T05:01:36.318 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setgroup ceph since I am not root 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.017+0000 7f25d79f3700 1 Processor -- start 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.017+0000 7f25d79f3700 1 -- start start 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.017+0000 7f25d79f3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.017+0000 7f25d79f3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0107ea0 con 0x7f25d01074b0 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25d578f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25d578f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59852/0 (socket says 172.21.15.45:59852) 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25d578f700 1 -- 172.21.15.45:0/641760513 learned_addr learned my addr 172.21.15.45:0/641760513 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25d578f700 1 -- 172.21.15.45:0/641760513 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0108660 con 0x7f25d01074b0 2024-03-12T05:01:36.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25d578f700 1 --2- 172.21.15.45:0/641760513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25c80098a0 tx=0x7f25c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42279772e1f2a90d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25cffff700 1 -- 172.21.15.45:0/641760513 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c8004160 con 0x7f25d01074b0 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25cffff700 1 -- 172.21.15.45:0/641760513 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f25c80042c0 con 0x7f25d01074b0 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.018+0000 7f25cffff700 1 -- 172.21.15.45:0/641760513 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c80119b0 con 0x7f25d01074b0 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 -- 172.21.15.45:0/641760513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 msgr2=0x7f25d01078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 --2- 172.21.15.45:0/641760513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25c80098a0 tx=0x7f25c8009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 -- 172.21.15.45:0/641760513 shutdown_connections 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 --2- 172.21.15.45:0/641760513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d01078d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 -- 172.21.15.45:0/641760513 >> 172.21.15.45:0/641760513 conn(0x7f25d0102a90 msgr2=0x7f25d0104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 -- 172.21.15.45:0/641760513 shutdown_connections 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.019+0000 7f25d79f3700 1 -- 172.21.15.45:0/641760513 wait complete. 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 Processor -- start 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 -- start start 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:36.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d019d9c0 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d578f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d578f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59858/0 (socket says 172.21.15.45:59858) 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d578f700 1 -- 172.21.15.45:0/3096262312 learned_addr learned my addr 172.21.15.45:0/3096262312 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d578f700 1 -- 172.21.15.45:0/3096262312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c8009550 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d578f700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25c8011670 tx=0x7f25c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c80038b0 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f25c8003a10 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c801a630 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d019dbc0 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.020+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d019e060 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.021+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f25c8024430 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.021+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d010b870 con 0x7f25d01074b0 2024-03-12T05:01:36.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.021+0000 7f25ce7fc700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 0x7f25b803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.022+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f25c804ce20 con 0x7f25d01074b0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.022+0000 7f25d4f8e700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 0x7f25b803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.022+0000 7f25d4f8e700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 0x7f25b803b170 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f25c4006fd0 tx=0x7f25c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.026+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f25c8018530 con 0x7f25d01074b0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.299+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f25d01a0940 con 0x7f25d01074b0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.993+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f25c8018350 con 0x7f25d01074b0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.995+0000 7f25ce7fc700 1 -- 172.21.15.45:0/3096262312 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f25c802ad40 con 0x7f25d01074b0 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 msgr2=0x7f25b803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 0x7f25b803b170 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f25c4006fd0 tx=0x7f25c4006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 msgr2=0x7f25d019d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25c8011670 tx=0x7f25c8011a30 comp rx=0 tx=0).stop 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 shutdown_connections 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f25b8038cb0 0x7f25b803b170 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:36.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 --2- 172.21.15.45:0/3096262312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25d01074b0 0x7f25d019d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:36.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 >> 172.21.15.45:0/3096262312 conn(0x7f25d0102a90 msgr2=0x7f25d0103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:36.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.997+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 shutdown_connections 2024-03-12T05:01:36.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:35.998+0000 7f25d79f3700 1 -- 172.21.15.45:0/3096262312 wait complete. 2024-03-12T05:01:36.835 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:36.581+0000 7f125675a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:01:36.836 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:36.681+0000 7f125675a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:01:37.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3096262312' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T05:01:37.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:36 smithi045 ceph-mon[32174]: mgrmap e9: x(active, since 19s) 2024-03-12T05:01:37.599 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:37.353+0000 7f125675a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:01:37.599 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:37.451+0000 7f125675a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:01:37.599 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:37.596+0000 7f125675a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:01:37.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:37.695+0000 7f125675a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:01:37.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:37.780+0000 7f125675a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.052+0000 7fc952f61700 1 Processor -- start 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.052+0000 7fc952f61700 1 -- start start 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.052+0000 7fc952f61700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.052+0000 7fc952f61700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c107ee0 con 0x7fc94c1074f0 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc950cfd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:37.917 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc950cfd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59884/0 (socket says 172.21.15.45:59884) 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc950cfd700 1 -- 172.21.15.45:0/610224950 learned_addr learned my addr 172.21.15.45:0/610224950 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc950cfd700 1 -- 172.21.15.45:0/610224950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c1086a0 con 0x7fc94c1074f0 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc950cfd700 1 --2- 172.21.15.45:0/610224950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9400098a0 tx=0x7fc940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a839b7bc29c252a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc94b7fe700 1 -- 172.21.15.45:0/610224950 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc940004160 con 0x7fc94c1074f0 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc94b7fe700 1 -- 172.21.15.45:0/610224950 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9400042c0 con 0x7fc94c1074f0 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.053+0000 7fc94b7fe700 1 -- 172.21.15.45:0/610224950 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9400119b0 con 0x7fc94c1074f0 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 -- 172.21.15.45:0/610224950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 msgr2=0x7fc94c107910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 --2- 172.21.15.45:0/610224950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9400098a0 tx=0x7fc940009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 -- 172.21.15.45:0/610224950 shutdown_connections 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 --2- 172.21.15.45:0/610224950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c107910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 -- 172.21.15.45:0/610224950 >> 172.21.15.45:0/610224950 conn(0x7fc94c102b50 msgr2=0x7fc94c104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 -- 172.21.15.45:0/610224950 shutdown_connections 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.054+0000 7fc952f61700 1 -- 172.21.15.45:0/610224950 wait complete. 2024-03-12T05:01:37.918 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc952f61700 1 Processor -- start 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc952f61700 1 -- start start 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc952f61700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc952f61700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c19b740 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc950cfd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc950cfd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59896/0 (socket says 172.21.15.45:59896) 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc950cfd700 1 -- 172.21.15.45:0/859304467 learned_addr learned my addr 172.21.15.45:0/859304467 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc950cfd700 1 -- 172.21.15.45:0/859304467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc940009550 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.055+0000 7fc950cfd700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc940011980 tx=0x7fc940011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.056+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc940003970 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.056+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc940003ad0 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.056+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c19b940 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.056+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc940019610 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.056+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c19bde0 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c10b8d0 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fc9400198d0 con 0x7fc94c1074f0 2024-03-12T05:01:37.919 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc949ffb700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 0x7fc93c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc94004cee0 con 0x7fc94c1074f0 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc94bfff700 1 -- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 msgr2=0x7fc93c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.057+0000 7fc94bfff700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 0x7fc93c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.066+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc94002b340 con 0x7fc94c1074f0 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.258+0000 7fc94bfff700 1 -- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 msgr2=0x7fc93c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.258+0000 7fc94bfff700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 0x7fc93c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.331+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc94c19e720 con 0x7fc94c1074f0 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.331+0000 7fc949ffb700 1 -- 172.21.15.45:0/859304467 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fc94c19e720 con 0x7fc94c1074f0 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.333+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 msgr2=0x7fc93c03b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 0x7fc93c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 msgr2=0x7fc94c19b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc940011980 tx=0x7fc940011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 shutdown_connections 2024-03-12T05:01:37.920 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7fc93c038d00 0x7fc93c03b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 --2- 172.21.15.45:0/859304467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc94c1074f0 0x7fc94c19b200 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 >> 172.21.15.45:0/859304467 conn(0x7fc94c102b50 msgr2=0x7fc94c1036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 shutdown_connections 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:37.334+0000 7fc952f61700 1 -- 172.21.15.45:0/859304467 wait complete. 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-12T05:01:37.921 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 9... 2024-03-12T05:01:38.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:37 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/859304467' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T05:01:38.178 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:38.057+0000 7f125675a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:01:38.509 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:38.175+0000 7f125675a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:01:38.510 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:38.265+0000 7f125675a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:01:38.510 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:38.365+0000 7f125675a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:01:38.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:38.543+0000 7f125675a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:01:39.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:39.527+0000 7f125675a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:01:39.873 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:39.661+0000 7f125675a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:01:40.295 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:39.999+0000 7f125675a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:01:40.295 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:40.091+0000 7f125675a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:01:40.295 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:40.292+0000 7f125675a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:40.622 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:40.482+0000 7f125675a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:01:41.038 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:40.775+0000 7f125675a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:01:41.038 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:41.035+0000 7f125675a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:01:41.313 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:41.278+0000 7f125675a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:01:42.373 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:42.104+0000 7f125675a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:01:43.010 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:42.685+0000 7f125675a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:01:43.010 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:42.786+0000 7f125675a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:01:43.010 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:01:43.007+0000 7f125675a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: Active manager daemon x restarted 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: Activating manager daemon x 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: mgrmap e10: x(active, starting, since 0.00310372s) 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: Manager daemon x is now available 2024-03-12T05:01:43.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:43.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:44.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:01:44.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:01:44.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:43] ENGINE Bus STARTING 2024-03-12T05:01:44.231 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:43] ENGINE Serving on http://172.21.15.45:8765 2024-03-12T05:01:44.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T05:01:44.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: mgrmap e11: x(active, since 1.00571s) 2024-03-12T05:01:44.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:44 smithi045 ceph-mon[32174]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26b0372700 1 Processor -- start 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26b0372700 1 -- start start 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26b0372700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26b0372700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a8106150 con 0x7f26a8105760 2024-03-12T05:01:45.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26ae10e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26ae10e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40578/0 (socket says 172.21.15.45:40578) 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26ae10e700 1 -- 172.21.15.45:0/4172050293 learned_addr learned my addr 172.21.15.45:0/4172050293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.302+0000 7f26ae10e700 1 -- 172.21.15.45:0/4172050293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a81086f0 con 0x7f26a8105760 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26ae10e700 1 --2- 172.21.15.45:0/4172050293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26a40098a0 tx=0x7f26a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988438253ef5d354 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26ad10c700 1 -- 172.21.15.45:0/4172050293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a4004160 con 0x7f26a8105760 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26ad10c700 1 -- 172.21.15.45:0/4172050293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26a40042c0 con 0x7f26a8105760 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26ad10c700 1 -- 172.21.15.45:0/4172050293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a40119b0 con 0x7f26a8105760 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26b0372700 1 -- 172.21.15.45:0/4172050293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 msgr2=0x7f26a8105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26b0372700 1 --2- 172.21.15.45:0/4172050293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f26a40098a0 tx=0x7f26a4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26b0372700 1 -- 172.21.15.45:0/4172050293 shutdown_connections 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26b0372700 1 --2- 172.21.15.45:0/4172050293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a8105b80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.303+0000 7f26b0372700 1 -- 172.21.15.45:0/4172050293 >> 172.21.15.45:0/4172050293 conn(0x7f26a8101090 msgr2=0x7f26a81034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 -- 172.21.15.45:0/4172050293 shutdown_connections 2024-03-12T05:01:45.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 -- 172.21.15.45:0/4172050293 wait complete. 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 Processor -- start 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 -- start start 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26b0372700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a819b810 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26ae10e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26ae10e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40580/0 (socket says 172.21.15.45:40580) 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.304+0000 7f26ae10e700 1 -- 172.21.15.45:0/3268616039 learned_addr learned my addr 172.21.15.45:0/3268616039 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f26ae10e700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a4009550 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f26ae10e700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26a4011670 tx=0x7f26a4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a40038b0 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f26b0372700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a819ba10 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26a4003a10 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a401a630 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.305+0000 7f26b0372700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a819beb0 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f26a401a8f0 con 0x7f26a8105760 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f269f7fe700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:45.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2690000d10 con 0x7f2694038cb0 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f26a404d150 con 0x7f26a8105760 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f26ad90d700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.306+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.507+0000 7f26ad90d700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.507+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.908+0000 7f26ad90d700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:38.908+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:39.709+0000 7f26ad90d700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:39.709+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:41.310+0000 7f26ad90d700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2524215406 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:41.310+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:43.022+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f26a402da00 con 0x7f26a8105760 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:43.022+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 msgr2=0x7f269403b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:01:45.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:43.022+0000 7f269f7fe700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.023+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f26a404ea30 con 0x7f26a8105760 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.024+0000 7f269f7fe700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 0x7f269403e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.024+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2690000d10 con 0x7f269403c580 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.024+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 0x7f269403e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.025+0000 7f26ad90d700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 0x7f269403e970 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2698003a10 tx=0x7f26980092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.025+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f2690000d10 con 0x7f269403c580 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f26900027d0 con 0x7f269403c580 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269f7fe700 1 -- 172.21.15.45:0/3268616039 <== mgr.14152 v2:172.21.15.45:6800/1134390880 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f26900027d0 con 0x7f269403c580 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 msgr2=0x7f269403e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 0x7f269403e970 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2698003a10 tx=0x7f26980092b0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 msgr2=0x7f26a819b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f26a4011670 tx=0x7f26a4011a30 comp rx=0 tx=0).stop 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 shutdown_connections 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f269403c580 0x7f269403e970 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26a8105760 0x7f26a819b2d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 --2- 172.21.15.45:0/3268616039 >> [v2:172.21.15.45:6800/2524215406,v1:172.21.15.45:6801/2524215406] conn(0x7f2694038cb0 0x7f269403b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 >> 172.21.15.45:0/3268616039 conn(0x7f26a8101090 msgr2=0x7f26a8101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 shutdown_connections 2024-03-12T05:01:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:44.032+0000 7f269d7fa700 1 -- 172.21.15.45:0/3268616039 wait complete. 2024-03-12T05:01:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 9 is available 2024-03-12T05:01:45.126 INFO:teuthology.orchestra.run.smithi045.stdout:Generating a dashboard self-signed certificate... 2024-03-12T05:01:46.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:45 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:44] ENGINE Serving on https://172.21.15.45:7150 2024-03-12T05:01:46.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:45 smithi045 ceph-mon[32174]: [12/Mar/2024:05:01:44] ENGINE Bus STARTED 2024-03-12T05:01:46.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:45 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:46.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T05:01:46.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5e06137700 1 Processor -- start 2024-03-12T05:01:46.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5e06137700 1 -- start start 2024-03-12T05:01:46.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5e06137700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:46.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5e06137700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00106da0 con 0x7f5e001063b0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5dff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5dff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40648/0 (socket says 172.21.15.45:40648) 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5dff7fe700 1 -- 172.21.15.45:0/4283402811 learned_addr learned my addr 172.21.15.45:0/4283402811 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5dff7fe700 1 -- 172.21.15.45:0/4283402811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e001075b0 con 0x7f5e001063b0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.501+0000 7f5dff7fe700 1 --2- 172.21.15.45:0/4283402811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5680cce128cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5dfe7fc700 1 -- 172.21.15.45:0/4283402811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0004160 con 0x7f5e001063b0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5dfe7fc700 1 -- 172.21.15.45:0/4283402811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5df00042c0 con 0x7f5e001063b0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5dfe7fc700 1 -- 172.21.15.45:0/4283402811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df00119b0 con 0x7f5e001063b0 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 -- 172.21.15.45:0/4283402811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 msgr2=0x7f5e001067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 --2- 172.21.15.45:0/4283402811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 -- 172.21.15.45:0/4283402811 shutdown_connections 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 --2- 172.21.15.45:0/4283402811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e001067d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 -- 172.21.15.45:0/4283402811 >> 172.21.15.45:0/4283402811 conn(0x7f5e00101950 msgr2=0x7f5e00103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:46.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 -- 172.21.15.45:0/4283402811 shutdown_connections 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.502+0000 7f5e06137700 1 -- 172.21.15.45:0/4283402811 wait complete. 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5e06137700 1 Processor -- start 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5e06137700 1 -- start start 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5e06137700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5e06137700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0019a690 con 0x7f5e001063b0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5dff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5dff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40662/0 (socket says 172.21.15.45:40662) 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.503+0000 7f5dff7fe700 1 -- 172.21.15.45:0/2210839034 learned_addr learned my addr 172.21.15.45:0/2210839034 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5dff7fe700 1 -- 172.21.15.45:0/2210839034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df0009550 con 0x7f5e001063b0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5dff7fe700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5df0011980 tx=0x7f5df0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0003970 con 0x7f5e001063b0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5df0003ad0 con 0x7f5e001063b0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0019a890 con 0x7f5e001063b0 2024-03-12T05:01:46.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5df0019610 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.504+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0019acb0 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f5df001a430 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5dfcff9700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 0x7f5de803d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0010a7e0 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5df004ce30 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5dfeffd700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 0x7f5de803d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.505+0000 7f5dfeffd700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 0x7f5de803d2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.510+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5df002d430 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.721+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e00102bf0 con 0x7f5de803adf0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.742+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f5df001aa70 con 0x7f5e001063b0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.799+0000 7f5dfcff9700 1 -- 172.21.15.45:0/2210839034 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5e00102bf0 con 0x7f5de803adf0 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 msgr2=0x7f5de803d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 0x7f5de803d2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:46.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 msgr2=0x7f5e0019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5df0011980 tx=0x7f5df0011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 shutdown_connections 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5de803adf0 0x7f5de803d2b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 --2- 172.21.15.45:0/2210839034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e001063b0 0x7f5e0019a150 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 >> 172.21.15.45:0/2210839034 conn(0x7f5e00101950 msgr2=0x7f5e001024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 shutdown_connections 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:45.801+0000 7f5e06137700 1 -- 172.21.15.45:0/2210839034 wait complete. 2024-03-12T05:01:46.360 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial admin user... 2024-03-12T05:01:46.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:46 smithi045 ceph-mon[32174]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:46.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:46 smithi045 ceph-mon[32174]: mgrmap e12: x(active, since 2s) 2024-03-12T05:01:46.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:46.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:46.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$leP6oFvw7pXQwPMms7a4.OJOcnzNi07eSjhVeYhoaBJ.Tl89Y1GuG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710219707, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4ed107700 1 Processor -- start 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4ed107700 1 -- start start 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4ed107700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4ed107700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e807b900 con 0x7fd4e807ce80 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40678/0 (socket says 172.21.15.45:40678) 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4e6d9d700 1 -- 172.21.15.45:0/696955025 learned_addr learned my addr 172.21.15.45:0/696955025 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:48.485 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.821+0000 7fd4e6d9d700 1 -- 172.21.15.45:0/696955025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e807ba40 con 0x7fd4e807ce80 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4e6d9d700 1 --2- 172.21.15.45:0/696955025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0098a0 tx=0x7fd4dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ddc52dc2039e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4e5d9b700 1 -- 172.21.15.45:0/696955025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4dc004160 con 0x7fd4e807ce80 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4e5d9b700 1 -- 172.21.15.45:0/696955025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4dc0042c0 con 0x7fd4e807ce80 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4e5d9b700 1 -- 172.21.15.45:0/696955025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4dc0119b0 con 0x7fd4e807ce80 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 -- 172.21.15.45:0/696955025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 msgr2=0x7fd4e807b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 --2- 172.21.15.45:0/696955025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0098a0 tx=0x7fd4dc009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 -- 172.21.15.45:0/696955025 shutdown_connections 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 --2- 172.21.15.45:0/696955025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e807b330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 -- 172.21.15.45:0/696955025 >> 172.21.15.45:0/696955025 conn(0x7fd4e8102b50 msgr2=0x7fd4e8104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 -- 172.21.15.45:0/696955025 shutdown_connections 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.822+0000 7fd4ed107700 1 -- 172.21.15.45:0/696955025 wait complete. 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.823+0000 7fd4ed107700 1 Processor -- start 2024-03-12T05:01:48.486 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.823+0000 7fd4ed107700 1 -- start start 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4ed107700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4ed107700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a1d10 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40690/0 (socket says 172.21.15.45:40690) 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4e6d9d700 1 -- 172.21.15.45:0/3670826501 learned_addr learned my addr 172.21.15.45:0/3670826501 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4e6d9d700 1 -- 172.21.15.45:0/3670826501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc009550 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4e6d9d700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc011980 tx=0x7fd4dc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4dc0039c0 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4dc003b20 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4dc019650 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e81a1f10 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.824+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e81a23b0 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.825+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fd4dc01a430 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.825+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e819af90 con 0x7fd4e807ce80 2024-03-12T05:01:48.487 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.826+0000 7fd4cffff700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 0x7fd4d003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.826+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4dc04d120 con 0x7fd4e807ce80 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.826+0000 7fd4e659c700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 0x7fd4d003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.826+0000 7fd4e659c700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 0x7fd4d003b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8006fd0 tx=0x7fd4d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:46.831+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd4dc018350 con 0x7fd4e807ce80 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.093+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd4e80630c0 con 0x7fd4d0038d00 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.339+0000 7fd4cffff700 1 -- 172.21.15.45:0/3670826501 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fd4e80630c0 con 0x7fd4d0038d00 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 msgr2=0x7fd4d003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 0x7fd4d003b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8006fd0 tx=0x7fd4d8006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 msgr2=0x7fd4e81a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc011980 tx=0x7fd4dc011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 shutdown_connections 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd4d0038d00 0x7fd4d003b1c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 --2- 172.21.15.45:0/3670826501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4e807ce80 0x7fd4e81a17d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 >> 172.21.15.45:0/3670826501 conn(0x7fd4e8102b50 msgr2=0x7fd4e8104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 shutdown_connections 2024-03-12T05:01:48.488 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:47.341+0000 7fd4ed107700 1 -- 172.21.15.45:0/3670826501 wait complete. 2024-03-12T05:01:48.489 INFO:teuthology.orchestra.run.smithi045.stdout:Fetching dashboard port number... 2024-03-12T05:01:48.594 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:48 smithi045 ceph-mon[32174]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:48.595 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:49.458 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:49 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/656090424' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T05:01:49.741 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T05:01:49.741 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bfaa2f700 1 Processor -- start 2024-03-12T05:01:49.741 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bfaa2f700 1 -- start start 2024-03-12T05:01:49.741 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bfaa2f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:49.741 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bfaa2f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf407aea0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36518/0 (socket says 172.21.15.45:36518) 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.896+0000 7f8bf3fff700 1 -- 172.21.15.45:0/3278772900 learned_addr learned my addr 172.21.15.45:0/3278772900 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bf3fff700 1 -- 172.21.15.45:0/3278772900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf410a7f0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bf3fff700 1 --2- 172.21.15.45:0/3278772900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098a0 tx=0x7f8be4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3475f37f85b35a77 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bf2ffd700 1 -- 172.21.15.45:0/3278772900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4004160 con 0x7f8bf4107ec0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bf2ffd700 1 -- 172.21.15.45:0/3278772900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8be40042c0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bf2ffd700 1 -- 172.21.15.45:0/3278772900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be40119b0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/3278772900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 msgr2=0x7f8bf410a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/3278772900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8be40098a0 tx=0x7f8be4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/3278772900 shutdown_connections 2024-03-12T05:01:49.742 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/3278772900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf410a2b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.897+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/3278772900 >> 172.21.15.45:0/3278772900 conn(0x7f8bf4101930 msgr2=0x7f8bf4103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.898+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/3278772900 shutdown_connections 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.898+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/3278772900 wait complete. 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.898+0000 7f8bfaa2f700 1 Processor -- start 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.898+0000 7f8bfaa2f700 1 -- start start 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bfaa2f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bfaa2f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf419b780 con 0x7f8bf4107ec0 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36530/0 (socket says 172.21.15.45:36530) 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf3fff700 1 -- 172.21.15.45:0/656090424 learned_addr learned my addr 172.21.15.45:0/656090424 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf3fff700 1 -- 172.21.15.45:0/656090424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4009550 con 0x7f8bf4107ec0 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf3fff700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8be4011980 tx=0x7f8be4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4003970 con 0x7f8bf4107ec0 2024-03-12T05:01:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf419b980 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8be4003ad0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be40195e0 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.899+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf419be20 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.900+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f8be401a430 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.900+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf410b630 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.900+0000 7f8bf17fa700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 0x7f8bdc03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.900+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8be404d150 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.900+0000 7f8bf37fe700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 0x7f8bdc03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.901+0000 7f8bf37fe700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 0x7f8bdc03b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8be8006fd0 tx=0x7f8be8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:48.905+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8be4023390 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.108+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8bf410b830 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.108+0000 7f8bf17fa700 1 -- 172.21.15.45:0/656090424 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8be4023390 con 0x7f8bf4107ec0 2024-03-12T05:01:49.744 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 msgr2=0x7f8bdc03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 0x7f8bdc03b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8be8006fd0 tx=0x7f8be8006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 msgr2=0x7f8bf419b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8be4011980 tx=0x7f8be4011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 shutdown_connections 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8bdc038cb0 0x7f8bdc03b170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 --2- 172.21.15.45:0/656090424 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf4107ec0 0x7f8bf419b240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 >> 172.21.15.45:0/656090424 conn(0x7f8bf4101930 msgr2=0x7f8bf41024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 shutdown_connections 2024-03-12T05:01:49.745 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:49.111+0000 7f8bfaa2f700 1 -- 172.21.15.45:0/656090424 wait complete. 2024-03-12T05:01:49.748 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:01:49.748 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-12T05:01:49.755 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:01:49.755 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-12T05:01:49.755 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-12T05:01:49.755 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T05:01:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph Dashboard is now available at: 2024-03-12T05:01:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: URL: https://smithi045.front.sepia.ceph.com:8443/ 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: User: admin 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: Password: 4xq7bngklx 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Saving cluster configuration to /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config directory 2024-03-12T05:01:49.757 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling autotune for osd_memory_target 2024-03-12T05:01:50.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:50 smithi045 ceph-mon[32174]: mgrmap e13: x(active, since 6s) 2024-03-12T05:01:50.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cc590f700 1 Processor -- start 2024-03-12T05:01:50.946 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cc590f700 1 -- start start 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cc590f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cc590f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0106d70 con 0x7f1cc0106380 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cbeffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cbeffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36532/0 (socket says 172.21.15.45:36532) 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cbeffd700 1 -- 172.21.15.45:0/3027974892 learned_addr learned my addr 172.21.15.45:0/3027974892 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.169+0000 7f1cbeffd700 1 -- 172.21.15.45:0/3027974892 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc0107580 con 0x7f1cc0106380 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cbeffd700 1 --2- 172.21.15.45:0/3027974892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00098a0 tx=0x7f1cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b388362319165476 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cbdffb700 1 -- 172.21.15.45:0/3027974892 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cb0004160 con 0x7f1cc0106380 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cbdffb700 1 -- 172.21.15.45:0/3027974892 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cb00042c0 con 0x7f1cc0106380 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cbdffb700 1 -- 172.21.15.45:0/3027974892 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cb00119b0 con 0x7f1cc0106380 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cc590f700 1 -- 172.21.15.45:0/3027974892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 msgr2=0x7f1cc01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3027974892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00098a0 tx=0x7f1cb0009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:50.947 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cc590f700 1 -- 172.21.15.45:0/3027974892 shutdown_connections 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3027974892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01067a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.170+0000 7f1cc590f700 1 -- 172.21.15.45:0/3027974892 >> 172.21.15.45:0/3027974892 conn(0x7f1cc0101940 msgr2=0x7f1cc0103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 -- 172.21.15.45:0/3027974892 shutdown_connections 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 -- 172.21.15.45:0/3027974892 wait complete. 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 Processor -- start 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 -- start start 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cc590f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0191a30 con 0x7f1cc0106380 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cbeffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.171+0000 7f1cbeffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36538/0 (socket says 172.21.15.45:36538) 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cbeffd700 1 -- 172.21.15.45:0/3294349668 learned_addr learned my addr 172.21.15.45:0/3294349668 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cbeffd700 1 -- 172.21.15.45:0/3294349668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0009550 con 0x7f1cc0106380 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cbeffd700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0011980 tx=0x7f1cb0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cb0003970 con 0x7f1cc0106380 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1cb0003ad0 con 0x7f1cc0106380 2024-03-12T05:01:50.948 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc018fc20 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cb0019610 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.172+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc01900c0 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.173+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f1cb0019880 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.173+0000 7f1cc490d700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 0x7f1ca803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.173+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1cb004d3e0 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.173+0000 7f1cbe7fc700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 0x7f1ca803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.173+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc0190ad0 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.174+0000 7f1cbe7fc700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 0x7f1ca803b2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4006fd0 tx=0x7f1cb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.178+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1cb0019e00 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.380+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f1cc010a7b0 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.380+0000 7f1cc490d700 1 -- 172.21.15.45:0/3294349668 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1cb00350c0 con 0x7f1cc0106380 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 msgr2=0x7f1ca803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 0x7f1ca803b2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4006fd0 tx=0x7f1cb4006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 msgr2=0x7f1cc01914f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:50.949 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0011980 tx=0x7f1cb0011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 shutdown_connections 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1ca8038e00 0x7f1ca803b2c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 --2- 172.21.15.45:0/3294349668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1cc0106380 0x7f1cc01914f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 >> 172.21.15.45:0/3294349668 conn(0x7f1cc0101940 msgr2=0x7f1cc0102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 shutdown_connections 2024-03-12T05:01:50.950 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:50.382+0000 7f1cc590f700 1 -- 172.21.15.45:0/3294349668 wait complete. 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.345+0000 7efcb1912700 1 Processor -- start 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.345+0000 7efcb1912700 1 -- start start 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.345+0000 7efcb1912700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.345+0000 7efcb1912700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac107f00 con 0x7efcac107510 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efcaaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:52.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efcaaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36554/0 (socket says 172.21.15.45:36554) 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efcaaffd700 1 -- 172.21.15.45:0/1745124945 learned_addr learned my addr 172.21.15.45:0/1745124945 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efcaaffd700 1 -- 172.21.15.45:0/1745124945 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac108670 con 0x7efcac107510 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efcaaffd700 1 --2- 172.21.15.45:0/1745124945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0098a0 tx=0x7efc9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8547eeb955f600bb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efca9ffb700 1 -- 172.21.15.45:0/1745124945 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc9c004160 con 0x7efcac107510 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efca9ffb700 1 -- 172.21.15.45:0/1745124945 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc9c0042c0 con 0x7efcac107510 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.346+0000 7efca9ffb700 1 -- 172.21.15.45:0/1745124945 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc9c0119b0 con 0x7efcac107510 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 -- 172.21.15.45:0/1745124945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 msgr2=0x7efcac107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 --2- 172.21.15.45:0/1745124945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0098a0 tx=0x7efc9c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 -- 172.21.15.45:0/1745124945 shutdown_connections 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 --2- 172.21.15.45:0/1745124945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac107930 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 -- 172.21.15.45:0/1745124945 >> 172.21.15.45:0/1745124945 conn(0x7efcac102b50 msgr2=0x7efcac104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 -- 172.21.15.45:0/1745124945 shutdown_connections 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 -- 172.21.15.45:0/1745124945 wait complete. 2024-03-12T05:01:52.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.347+0000 7efcb1912700 1 Processor -- start 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb1912700 1 -- start start 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb1912700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb1912700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac07d0a0 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcaaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcaaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36568/0 (socket says 172.21.15.45:36568) 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcaaffd700 1 -- 172.21.15.45:0/1249211261 learned_addr learned my addr 172.21.15.45:0/1249211261 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcaaffd700 1 -- 172.21.15.45:0/1249211261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c009550 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcaaffd700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc9c011980 tx=0x7efc9c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc9c003970 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc9c003ad0 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac07b290 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.348+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc9c019610 con 0x7efcac107510 2024-03-12T05:01:52.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.349+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac07b730 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.349+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7efc9c0198d0 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.349+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac10b8a0 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.350+0000 7efcb0910700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 0x7efc9803b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.350+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efc9c04d0e0 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.350+0000 7efcaa7fc700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 0x7efc9803b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.350+0000 7efcaa7fc700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 0x7efc9803b260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc94006fd0 tx=0x7efc94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.355+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc9c023c20 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.669+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7efcac10bab0 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.671+0000 7efcb0910700 1 -- 172.21.15.45:0/1249211261 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7efc9c024e40 con 0x7efcac107510 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.673+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 msgr2=0x7efc9803b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.673+0000 7efcb1912700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 0x7efc9803b260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc94006fd0 tx=0x7efc94006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.673+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 msgr2=0x7efcac07cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.673+0000 7efcb1912700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc9c011980 tx=0x7efc9c011a60 comp rx=0 tx=0).stop 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 shutdown_connections 2024-03-12T05:01:52.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efc98038da0 0x7efc9803b260 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 --2- 172.21.15.45:0/1249211261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcac107510 0x7efcac07cb60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 >> 172.21.15.45:0/1249211261 conn(0x7efcac102b50 msgr2=0x7efcac103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 shutdown_connections 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2024-03-12T05:01:51.674+0000 7efcb1912700 1 -- 172.21.15.45:0/1249211261 wait complete. 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.124 INFO:teuthology.orchestra.run.smithi045.stdout: ceph telemetry on 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout:For more information see: 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:01:52.125 INFO:teuthology.orchestra.run.smithi045.stdout:Bootstrap complete. 2024-03-12T05:01:52.160 INFO:tasks.cephadm:Fetching config... 2024-03-12T05:01:52.161 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:01:52.161 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T05:01:52.177 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T05:01:52.177 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:01:52.177 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T05:01:52.232 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T05:01:52.232 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:01:52.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T05:01:52.299 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T05:01:52.299 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:01:52.300 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T05:01:52.354 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T05:01:52.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:01:52.435 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:52.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:01:52.497 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:52.512 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:01:52.560 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdNnUhMM4AjLUCVgIspGvP6ARd/cOfUH/oruVg6r291BGEsBv+TmaxwreOFSZRAe9rn3WJEX7sdeyDJvq2htLWWeF/DVY1RiBK5Xc1TZi61ZkcgKoVQqA/95vKVY4Sib7NpK13/0Z/eFgtRbRQ/DNOEVNX3iXAVVxIfK6L171w6isgBXkqNqg5wBYqbVrzlIqCF9Kwi1rRg5e24JOBQYg4CeEsQHhanfhxKeTb908LSEuG5L/zQ4yqAceT2WpgeyN1hGqd/bcr4MT5MQOoC4J42kPQbXYRVO6NWNQsD7xH8iXyFv/c3a9n+39xz7mhXmOaifSfTI+GRIrzpB/BYmglhnWx7VPLxFUhyT+cTcYlEYXCXPAmFrVPPI8Dd3okTEcu1qI4PdVHARDzYxoZhfvmSQbDRsOGksHwCb2dSL/cTEpCsV437PlJuuOX7lLFFfscnUAoLsP1TAvzhyty3X1htJImv5vucEcUiV3qhw+zE3yHn9lGlVfc3gF04ZVC/5c= ceph-52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:01:52.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T05:01:52.820 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:01:52.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:52 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1249211261' entity='client.admin' 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.964+0000 7f0013612700 1 -- 172.21.15.45:0/3989741509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 msgr2=0x7f000c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.964+0000 7f0013612700 1 --2- 172.21.15.45:0/3989741509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c101ae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0008009910 tx=0x7f0008009c20 comp rx=0 tx=0).stop 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.964+0000 7f0013612700 1 -- 172.21.15.45:0/3989741509 shutdown_connections 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.964+0000 7f0013612700 1 --2- 172.21.15.45:0/3989741509 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c101ae0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.964+0000 7f0013612700 1 -- 172.21.15.45:0/3989741509 >> 172.21.15.45:0/3989741509 conn(0x7f000c0fcc60 msgr2=0x7f000c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 -- 172.21.15.45:0/3989741509 shutdown_connections 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 -- 172.21.15.45:0/3989741509 wait complete. 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 Processor -- start 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 -- start start 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f0013612700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c199290 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.965+0000 7f00113ae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00113ae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36596/0 (socket says 172.21.15.45:36596) 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00113ae700 1 -- 172.21.15.45:0/773999866 learned_addr learned my addr 172.21.15.45:0/773999866 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00113ae700 1 -- 172.21.15.45:0/773999866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080095f0 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00113ae700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0008005950 tx=0x7f00080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f000801d070 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f000800bc00 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c199490 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00080039e0 con 0x7f000c1016c0 2024-03-12T05:01:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.966+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c199930 con 0x7f000c1016c0 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.967+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f0008003c50 con 0x7f000c1016c0 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.967+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c1927d0 con 0x7f000c1016c0 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.967+0000 7f00027fc700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 0x7efff803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.967+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f000804cbc0 con 0x7f000c1016c0 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.967+0000 7f0010bad700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 0x7efff803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:54.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.968+0000 7f0010bad700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 0x7efff803b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efffc006fd0 tx=0x7efffc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:54.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:54.972+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0008029900 con 0x7f000c1016c0 2024-03-12T05:01:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.183+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f000c0632a0 con 0x7f000c1016c0 2024-03-12T05:01:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.186+0000 7f00027fc700 1 -- 172.21.15.45:0/773999866 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0008026030 con 0x7f000c1016c0 2024-03-12T05:01:55.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 msgr2=0x7efff803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 0x7efff803b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efffc006fd0 tx=0x7efffc006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 msgr2=0x7f000c198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0008005950 tx=0x7f00080057f0 comp rx=0 tx=0).stop 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 shutdown_connections 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efff8038db0 0x7efff803b270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 --2- 172.21.15.45:0/773999866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f000c1016c0 0x7f000c198d50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 >> 172.21.15.45:0/773999866 conn(0x7f000c0fcc60 msgr2=0x7f000c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 shutdown_connections 2024-03-12T05:01:55.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:55.188+0000 7f0013612700 1 -- 172.21.15.45:0/773999866 wait complete. 2024-03-12T05:01:55.875 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T05:01:55.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T05:01:56.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:56 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/773999866' entity='client.admin' 2024-03-12T05:01:56.580 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:01:57.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:57.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:57.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:01:57.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:57.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:57.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 -- 172.21.15.45:0/3967893779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 msgr2=0x7f5b74104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 --2- 172.21.15.45:0/3967893779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74104580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009910 tx=0x7f5b64009c20 comp rx=0 tx=0).stop 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 -- 172.21.15.45:0/3967893779 shutdown_connections 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 --2- 172.21.15.45:0/3967893779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74104580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 -- 172.21.15.45:0/3967893779 >> 172.21.15.45:0/3967893779 conn(0x7f5b740fbb40 msgr2=0x7f5b740fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 -- 172.21.15.45:0/3967893779 shutdown_connections 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.085+0000 7f5b79c39700 1 -- 172.21.15.45:0/3967893779 wait complete. 2024-03-12T05:01:58.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b79c39700 1 Processor -- start 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b79c39700 1 -- start start 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b79c39700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b79c39700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b74197040 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b737fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b737fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43258/0 (socket says 172.21.15.45:43258) 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b737fe700 1 -- 172.21.15.45:0/2731857743 learned_addr learned my addr 172.21.15.45:0/2731857743 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b737fe700 1 -- 172.21.15.45:0/2731857743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b640095f0 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.086+0000 7f5b737fe700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b64005b40 tx=0x7f5b64004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b6401d070 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b74197240 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5b6400be10 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b64022db0 con 0x7f5b74102190 2024-03-12T05:01:58.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b741976e0 con 0x7f5b74102190 2024-03-12T05:01:58.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.087+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f5b6400f4b0 con 0x7f5b74102190 2024-03-12T05:01:58.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.088+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b741903b0 con 0x7f5b74102190 2024-03-12T05:01:58.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.088+0000 7f5b70ff9700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 0x7f5b5c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:01:58.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.088+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b6404cfd0 con 0x7f5b74102190 2024-03-12T05:01:58.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.088+0000 7f5b72ffd700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 0x7f5b5c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:01:58.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.088+0000 7f5b72ffd700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 0x7f5b5c03b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5b68006fd0 tx=0x7f5b68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:01:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.093+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b64080d20 con 0x7f5b74102190 2024-03-12T05:01:58.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.301+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b7402d7d0 con 0x7f5b5c038d50 2024-03-12T05:01:58.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.305+0000 7f5b70ff9700 1 -- 172.21.15.45:0/2731857743 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b7402d7d0 con 0x7f5b5c038d50 2024-03-12T05:01:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 msgr2=0x7f5b5c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 0x7f5b5c03b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5b68006fd0 tx=0x7f5b68006e40 comp rx=0 tx=0).stop 2024-03-12T05:01:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 msgr2=0x7f5b74196b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:01:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b64005b40 tx=0x7f5b64004f40 comp rx=0 tx=0).stop 2024-03-12T05:01:58.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 shutdown_connections 2024-03-12T05:01:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b5c038d50 0x7f5b5c03b210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 --2- 172.21.15.45:0/2731857743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b74102190 0x7f5b74196b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:01:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 >> 172.21.15.45:0/2731857743 conn(0x7f5b740fbb40 msgr2=0x7f5b740fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:01:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 shutdown_connections 2024-03-12T05:01:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:01:58.307+0000 7f5b79c39700 1 -- 172.21.15.45:0/2731857743 wait complete. 2024-03-12T05:01:59.010 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-03-12T05:01:59.010 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T05:01:59.010 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-03-12T05:01:59.032 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T05:01:59.032 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:01:59.089 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-03-12T05:01:59.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch host add smithi116 2024-03-12T05:01:59.334 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:59.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:01:59.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:01:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 -- 172.21.15.45:0/23027293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 msgr2=0x7fc1ac107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 --2- 172.21.15.45:0/23027293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac107a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc194009910 tx=0x7fc194009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 -- 172.21.15.45:0/23027293 shutdown_connections 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 --2- 172.21.15.45:0/23027293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac107a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 -- 172.21.15.45:0/23027293 >> 172.21.15.45:0/23027293 conn(0x7fc1ac0fd180 msgr2=0x7fc1ac0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 -- 172.21.15.45:0/23027293 shutdown_connections 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.789+0000 7fc1b146a700 1 -- 172.21.15.45:0/23027293 wait complete. 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1b146a700 1 Processor -- start 2024-03-12T05:02:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1b146a700 1 -- start start 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1b146a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1b146a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac075e60 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1aaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1aaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43284/0 (socket says 172.21.15.45:43284) 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.790+0000 7fc1aaffd700 1 -- 172.21.15.45:0/3367404524 learned_addr learned my addr 172.21.15.45:0/3367404524 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1aaffd700 1 -- 172.21.15.45:0/3367404524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940095f0 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1aaffd700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc194004f40 tx=0x7fc194005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19401d070 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc19400b6f0 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac0744b0 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1940037d0 con 0x7fc1ac107650 2024-03-12T05:02:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.791+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac074930 con 0x7fc1ac107650 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.792+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fc194003a40 con 0x7fc1ac107650 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.792+0000 7fc1a3fff700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 0x7fc19803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.792+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc19404ccd0 con 0x7fc1ac107650 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.792+0000 7fc1aa7fc700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 0x7fc19803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.792+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac0632a0 con 0x7fc1ac107650 2024-03-12T05:02:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.793+0000 7fc1aa7fc700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 0x7fc19803b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc19c006fd0 tx=0x7fc19c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:00.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:00.797+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc194026020 con 0x7fc1ac107650 2024-03-12T05:02:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:01.003+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1ac074b90 con 0x7fc198038db0 2024-03-12T05:02:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:01 smithi045 ceph-mon[32174]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:02.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:02 smithi045 ceph-mon[32174]: Deploying cephadm binary to smithi116 2024-03-12T05:02:04.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:04 smithi045 ceph-mon[32174]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.421+0000 7fc1a3fff700 1 -- 172.21.15.45:0/3367404524 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc1ac074b90 con 0x7fc198038db0 2024-03-12T05:02:05.424 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 msgr2=0x7fc19803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 0x7fc19803b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc19c006fd0 tx=0x7fc19c006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 msgr2=0x7fc1ac073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc194004f40 tx=0x7fc194005020 comp rx=0 tx=0).stop 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 shutdown_connections 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc198038db0 0x7fc19803b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 --2- 172.21.15.45:0/3367404524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1ac107650 0x7fc1ac073f70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.424+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 >> 172.21.15.45:0/3367404524 conn(0x7fc1ac0fd180 msgr2=0x7fc1ac1091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.425+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 shutdown_connections 2024-03-12T05:02:05.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:05.425+0000 7fc1b146a700 1 -- 172.21.15.45:0/3367404524 wait complete. 2024-03-12T05:02:05.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T05:02:06.218 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:06.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:06 smithi045 ceph-mon[32174]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:06.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:06.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:06.339 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:07.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:07 smithi045 ceph-mon[32174]: Added host smithi116 2024-03-12T05:02:07.326 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 -- 172.21.15.45:0/867365315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 msgr2=0x7fa3681069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 --2- 172.21.15.45:0/867365315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3681069e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009910 tx=0x7fa35c009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 -- 172.21.15.45:0/867365315 shutdown_connections 2024-03-12T05:02:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 --2- 172.21.15.45:0/867365315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3681069e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:07.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 -- 172.21.15.45:0/867365315 >> 172.21.15.45:0/867365315 conn(0x7fa3680fc020 msgr2=0x7fa3680fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 -- 172.21.15.45:0/867365315 shutdown_connections 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.630+0000 7fa36f814700 1 -- 172.21.15.45:0/867365315 wait complete. 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36f814700 1 Processor -- start 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36f814700 1 -- start start 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36f814700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:07.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36f814700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680fe3c0 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36d5b0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36d5b0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49812/0 (socket says 172.21.15.45:49812) 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36d5b0700 1 -- 172.21.15.45:0/3844341149 learned_addr learned my addr 172.21.15.45:0/3844341149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.631+0000 7fa36d5b0700 1 -- 172.21.15.45:0/3844341149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0095f0 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa36d5b0700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa35c004f40 tx=0x7fa35c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa35c01d070 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa35c00b750 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3680fca10 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa35c0037d0 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.632+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3680fce90 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fa35c004240 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3680632a0 con 0x7fa368106600 2024-03-12T05:02:07.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa35a7fc700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 0x7fa35403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa35c04caa0 con 0x7fa368106600 2024-03-12T05:02:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa36cdaf700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 0x7fa35403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:07.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.633+0000 7fa36cdaf700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 0x7fa35403b210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa364006fd0 tx=0x7fa364006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:07.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.638+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa35c02a730 con 0x7fa368106600 2024-03-12T05:02:07.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.847+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3680fd220 con 0x7fa354038d50 2024-03-12T05:02:07.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.848+0000 7fa35a7fc700 1 -- 172.21.15.45:0/3844341149 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fa3680fd220 con 0x7fa354038d50 2024-03-12T05:02:07.850 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:02:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 msgr2=0x7fa35403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 0x7fa35403b210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa364006fd0 tx=0x7fa364006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 msgr2=0x7fa3680fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa35c004f40 tx=0x7fa35c005020 comp rx=0 tx=0).stop 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 shutdown_connections 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa354038d50 0x7fa35403b210 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 --2- 172.21.15.45:0/3844341149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa368106600 0x7fa3680fc4d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 >> 172.21.15.45:0/3844341149 conn(0x7fa3680fc020 msgr2=0x7fa3680698f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 shutdown_connections 2024-03-12T05:02:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:07.850+0000 7fa36f814700 1 -- 172.21.15.45:0/3844341149 wait complete. 2024-03-12T05:02:08.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:08 smithi045 ceph-mon[32174]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:08.487 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2024-03-12T05:02:08.487 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T05:02:08.487 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2024-03-12T05:02:08.506 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T05:02:08.506 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:08.563 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2024-03-12T05:02:08.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch host add smithi130 2024-03-12T05:02:08.808 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:09.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:09 smithi045 ceph-mon[32174]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:02:09.194 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:09 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 -- 172.21.15.45:0/3489569413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 msgr2=0x7f1a0c102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 --2- 172.21.15.45:0/3489569413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c102cb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009910 tx=0x7f19fc009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 -- 172.21.15.45:0/3489569413 shutdown_connections 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 --2- 172.21.15.45:0/3489569413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c102cb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 -- 172.21.15.45:0/3489569413 >> 172.21.15.45:0/3489569413 conn(0x7f1a0c0fc040 msgr2=0x7f1a0c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 -- 172.21.15.45:0/3489569413 shutdown_connections 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.213+0000 7f1a112ba700 1 -- 172.21.15.45:0/3489569413 wait complete. 2024-03-12T05:02:10.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a112ba700 1 Processor -- start 2024-03-12T05:02:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a112ba700 1 -- start start 2024-03-12T05:02:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a112ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a112ba700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c197420 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a0affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a0affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49826/0 (socket says 172.21.15.45:49826) 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.214+0000 7f1a0affd700 1 -- 172.21.15.45:0/639344840 learned_addr learned my addr 172.21.15.45:0/639344840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a0affd700 1 -- 172.21.15.45:0/639344840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0095f0 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a0affd700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004f40 tx=0x7f19fc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc01d070 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c197620 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f19fc00bc00 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc003920 con 0x7f1a0c1028d0 2024-03-12T05:02:10.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.215+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c197a40 con 0x7f1a0c1028d0 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.216+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f19fc003be0 con 0x7f1a0c1028d0 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.216+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0632a0 con 0x7f1a0c1028d0 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.216+0000 7f1a03fff700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 0x7f19f803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.216+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f19fc04cdb0 con 0x7f1a0c1028d0 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.216+0000 7f1a0a7fc700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 0x7f19f803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.217+0000 7f1a0a7fc700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 0x7f19f803b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f19f4006fd0 tx=0x7f19f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:10.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.221+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f19fc00fd00 con 0x7f1a0c1028d0 2024-03-12T05:02:10.224 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:10 smithi045 ceph-mon[32174]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:10.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:10.427+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a0c10ba70 con 0x7f19f8038d50 2024-03-12T05:02:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:12.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:13.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:13 smithi045 ceph-mon[32174]: Deploying cephadm binary to smithi130 2024-03-12T05:02:13.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:13 smithi045 ceph-mon[32174]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-12T05:02:13.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:13 smithi045 ceph-mon[32174]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:13.312 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:13 smithi045 ceph-mon[32174]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:14.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:14 smithi045 ceph-mon[32174]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:14.963 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi130' with addr '172.21.15.130' 2024-03-12T05:02:14.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.959+0000 7f1a03fff700 1 -- 172.21.15.45:0/639344840 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1a0c10ba70 con 0x7f19f8038d50 2024-03-12T05:02:14.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 msgr2=0x7f19f803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 0x7f19f803b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f19f4006fd0 tx=0x7f19f4006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 msgr2=0x7f1a0c196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f19fc004f40 tx=0x7f19fc005020 comp rx=0 tx=0).stop 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 shutdown_connections 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f19f8038d50 0x7f19f803b210 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 --2- 172.21.15.45:0/639344840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a0c1028d0 0x7f1a0c196ee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.961+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 >> 172.21.15.45:0/639344840 conn(0x7f1a0c0fc040 msgr2=0x7f1a0c103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.962+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 shutdown_connections 2024-03-12T05:02:14.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:14.962+0000 7f1a112ba700 1 -- 172.21.15.45:0/639344840 wait complete. 2024-03-12T05:02:15.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T05:02:15.748 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: Added host smithi130 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:15.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 -- 172.21.15.45:0/3799410232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 msgr2=0x7faea0101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 --2- 172.21.15.45:0/3799410232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0101380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae9000b0a0 tx=0x7fae9000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:02:17.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 -- 172.21.15.45:0/3799410232 shutdown_connections 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 --2- 172.21.15.45:0/3799410232 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0101380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 -- 172.21.15.45:0/3799410232 >> 172.21.15.45:0/3799410232 conn(0x7faea0076c20 msgr2=0x7faea0077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 -- 172.21.15.45:0/3799410232 shutdown_connections 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.010+0000 7faea49c3700 1 -- 172.21.15.45:0/3799410232 wait complete. 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7faea49c3700 1 Processor -- start 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7faea49c3700 1 -- start start 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7faea49c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7faea49c3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea01962d0 con 0x7faea0100fa0 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae9e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae9e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:49850/0 (socket says 172.21.15.45:49850) 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae9e59c700 1 -- 172.21.15.45:0/1073456694 learned_addr learned my addr 172.21.15.45:0/1073456694 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae9e59c700 1 -- 172.21.15.45:0/1073456694 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90009d00 con 0x7faea0100fa0 2024-03-12T05:02:17.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae9e59c700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae900062a0 tx=0x7fae9000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae9000e040 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae9001d070 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea01964d0 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.011+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae90024480 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.012+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea01969d0 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea00632a0 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fae90012430 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7fae977fe700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 0x7fae8c03b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae9001f070 con 0x7faea0100fa0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7fae9dd9b700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 0x7fae8c03b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.013+0000 7fae9dd9b700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 0x7fae8c03b260 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:17.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.022+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae90023c70 con 0x7faea0100fa0 2024-03-12T05:02:17.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.231+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faea010a970 con 0x7fae8c038da0 2024-03-12T05:02:17.235 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:17.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.232+0000 7fae977fe700 1 -- 172.21.15.45:0/1073456694 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7faea010a970 con 0x7fae8c038da0 2024-03-12T05:02:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:02:17.236 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.234+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 msgr2=0x7fae8c03b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.234+0000 7faea49c3700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 0x7fae8c03b260 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.234+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 msgr2=0x7faea0195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.234+0000 7faea49c3700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae900062a0 tx=0x7fae9000bd40 comp rx=0 tx=0).stop 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 shutdown_connections 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae8c038da0 0x7fae8c03b260 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 --2- 172.21.15.45:0/1073456694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faea0100fa0 0x7faea0195d90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:17.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 >> 172.21.15.45:0/1073456694 conn(0x7faea0076c20 msgr2=0x7faea00fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:17.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 shutdown_connections 2024-03-12T05:02:17.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:17.235+0000 7faea49c3700 1 -- 172.21.15.45:0/1073456694 wait complete. 2024-03-12T05:02:17.807 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T05:02:17.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T05:02:18.042 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:18.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:17 smithi045 ceph-mon[32174]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:18.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:19.317 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:18 smithi045 ceph-mon[32174]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 -- 172.21.15.45:0/1645608635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 msgr2=0x7f563c107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 --2- 172.21.15.45:0/1645608635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c107a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5634009910 tx=0x7f5634009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:19.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 -- 172.21.15.45:0/1645608635 shutdown_connections 2024-03-12T05:02:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 --2- 172.21.15.45:0/1645608635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c107a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 -- 172.21.15.45:0/1645608635 >> 172.21.15.45:0/1645608635 conn(0x7f563c077b70 msgr2=0x7f563c077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 -- 172.21.15.45:0/1645608635 shutdown_connections 2024-03-12T05:02:19.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.530+0000 7f564190f700 1 -- 172.21.15.45:0/1645608635 wait complete. 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f564190f700 1 Processor -- start 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f564190f700 1 -- start start 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f564190f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f564190f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c197400 con 0x7f563c107690 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f563affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:19.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f563affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34462/0 (socket says 172.21.15.45:34462) 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f563affd700 1 -- 172.21.15.45:0/868843197 learned_addr learned my addr 172.21.15.45:0/868843197 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.531+0000 7f563affd700 1 -- 172.21.15.45:0/868843197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340095f0 con 0x7f563c107690 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f563affd700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5634004f40 tx=0x7f5634005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f563401d070 con 0x7f563c107690 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f563400b6f0 con 0x7f563c107690 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c197660 con 0x7f563c107690 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56340037d0 con 0x7f563c107690 2024-03-12T05:02:19.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.532+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c197ae0 con 0x7f563c107690 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f5634003a90 con 0x7f563c107690 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c0632a0 con 0x7f563c107690 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f564090d700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 0x7f562803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f563404cb90 con 0x7f563c107690 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f563a7fc700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 0x7f562803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:19.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.533+0000 7f563a7fc700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 0x7f562803b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:19.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.538+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5634029720 con 0x7f563c107690 2024-03-12T05:02:19.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.744+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f563c10bab0 con 0x7f563c107690 2024-03-12T05:02:19.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.997+0000 7f564090d700 1 -- 172.21.15.45:0/868843197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5634026070 con 0x7f563c107690 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.999+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 msgr2=0x7f562803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.999+0000 7f564190f700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 0x7f562803b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.999+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 msgr2=0x7f563c196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:19.999+0000 7f564190f700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5634004f40 tx=0x7f5634005020 comp rx=0 tx=0).stop 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 shutdown_connections 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5628038d50 0x7f562803b210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 --2- 172.21.15.45:0/868843197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f563c107690 0x7f563c196ec0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 >> 172.21.15.45:0/868843197 conn(0x7f563c077b70 msgr2=0x7f563c0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 shutdown_connections 2024-03-12T05:02:20.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:20.000+0000 7f564190f700 1 -- 172.21.15.45:0/868843197 wait complete. 2024-03-12T05:02:20.003 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-03-12T05:02:20.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:19 smithi045 ceph-mon[32174]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:20.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:19 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/868843197' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T05:02:20.542 INFO:tasks.cephadm:Adding mon.a on smithi045 2024-03-12T05:02:20.542 INFO:tasks.cephadm:Adding mon.b on smithi116 2024-03-12T05:02:20.542 INFO:tasks.cephadm:Adding mon.c on smithi130 2024-03-12T05:02:20.542 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c' 2024-03-12T05:02:20.879 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T05:02:20.925 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T05:02:21.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/868843197' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T05:02:21.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:21.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:22.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:22 smithi045 ceph-mon[32174]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-12T05:02:22.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:22 smithi045 ceph-mon[32174]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:22.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:22 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:22.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:22 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:22.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:22 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:22.551 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- 172.21.15.130:0/3685475545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 msgr2=0x7f6ed8102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 --2- 172.21.15.130:0/3685475545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8102c50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009910 tx=0x7f6ec8009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- 172.21.15.130:0/3685475545 shutdown_connections 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 --2- 172.21.15.130:0/3685475545 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8102c50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- 172.21.15.130:0/3685475545 >> 172.21.15.130:0/3685475545 conn(0x7f6ed80fc040 msgr2=0x7f6ed80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- 172.21.15.130:0/3685475545 shutdown_connections 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- 172.21.15.130:0/3685475545 wait complete. 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 Processor -- start 2024-03-12T05:02:22.552 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.549+0000 7f6ede786700 1 -- start start 2024-03-12T05:02:22.553 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ede786700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:22.553 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ede786700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed8197440 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ed7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ed7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.130:59596/0 (socket says 172.21.15.130:59596) 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ed7fff700 1 -- 172.21.15.130:0/3386148144 learned_addr learned my addr 172.21.15.130:0/3386148144 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.550+0000 7f6ed7fff700 1 -- 172.21.15.130:0/3386148144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80095f0 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ed7fff700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006010 tx=0x7f6ec8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec801d070 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed8197640 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ec800bc00 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec80039a0 con 0x7f6ed8102870 2024-03-12T05:02:22.554 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.551+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed8197a60 con 0x7f6ed8102870 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.552+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed80632a0 con 0x7f6ed8102870 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.552+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6ec8003bd0 con 0x7f6ed8102870 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.552+0000 7f6ed57fa700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 0x7f6ec003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.552+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6ec804cab0 con 0x7f6ed8102870 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.552+0000 7f6ed77fe700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 0x7f6ec003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:22.555 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.553+0000 7f6ed77fe700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 0x7f6ec003b270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc006fd0 tx=0x7f6ecc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:22.559 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.558+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ec8005c00 con 0x7f6ed8102870 2024-03-12T05:02:22.761 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.758+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ed810ba30 con 0x7f6ec0038db0 2024-03-12T05:02:22.767 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.765+0000 7f6ed57fa700 1 -- 172.21.15.130:0/3386148144 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6ed810ba30 con 0x7f6ec0038db0 2024-03-12T05:02:22.767 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mon update... 2024-03-12T05:02:22.769 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 msgr2=0x7f6ec003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:22.769 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 0x7f6ec003b270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc006fd0 tx=0x7f6ecc006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:22.769 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 msgr2=0x7f6ed8196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:22.769 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006010 tx=0x7f6ec8004ff0 comp rx=0 tx=0).stop 2024-03-12T05:02:22.769 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 shutdown_connections 2024-03-12T05:02:22.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6ec0038db0 0x7f6ec003b270 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:22.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 --2- 172.21.15.130:0/3386148144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ed8102870 0x7f6ed8196f00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:22.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 >> 172.21.15.130:0/3386148144 conn(0x7f6ed80fc040 msgr2=0x7f6ed8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:22.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 shutdown_connections 2024-03-12T05:02:22.770 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:22.767+0000 7f6ede786700 1 -- 172.21.15.130:0/3386148144 wait complete. 2024-03-12T05:02:23.352 DEBUG:teuthology.orchestra.run.smithi116:mon.b> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T05:02:23.355 DEBUG:teuthology.orchestra.run.smithi130:mon.c> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T05:02:23.357 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:02:23.357 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:23.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:23.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:23.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:24.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:24 smithi045 ceph-mon[32174]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:24.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:24 smithi045 ceph-mon[32174]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c;count:3 2024-03-12T05:02:24.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:24 smithi045 ceph-mon[32174]: Deploying daemon mon.c on smithi130 2024-03-12T05:02:24.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:24 smithi045 ceph-mon[32174]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:25.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:25 smithi045 ceph-mon[32174]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:25.466 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:02:27.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:27 smithi045 ceph-mon[32174]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:27.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:27 smithi130 systemd[1]: Started Ceph mon.c for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:02:27.496 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 -- 172.21.15.130:0/1509859177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 msgr2=0x7feb58107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:27.496 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 --2- 172.21.15.130:0/1509859177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58107b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb54009910 tx=0x7feb54009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 -- 172.21.15.130:0/1509859177 shutdown_connections 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 --2- 172.21.15.130:0/1509859177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58107b90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 -- 172.21.15.130:0/1509859177 >> 172.21.15.130:0/1509859177 conn(0x7feb580fbd80 msgr2=0x7feb580fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 -- 172.21.15.130:0/1509859177 shutdown_connections 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.309+0000 7feb5ff05700 1 -- 172.21.15.130:0/1509859177 wait complete. 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5ff05700 1 Processor -- start 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5ff05700 1 -- start start 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5ff05700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5ff05700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58197490 con 0x7feb581077b0 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5dca1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5dca1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.130:51896/0 (socket says 172.21.15.130:51896) 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.310+0000 7feb5dca1700 1 -- 172.21.15.130:0/891416190 learned_addr learned my addr 172.21.15.130:0/891416190 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:02:27.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb5dca1700 1 -- 172.21.15.130:0/891416190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb540095f0 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb5dca1700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb54004f40 tx=0x7feb54005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb5401d070 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb581976f0 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb5400b6f0 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb58197b70 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.311+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb540037b0 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.312+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7feb54004240 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.312+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb580632a0 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.312+0000 7feb4effd700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 0x7feb4403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.312+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feb5404cb70 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.313+0000 7feb5d4a0700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 0x7feb4403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.313+0000 7feb5d4a0700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 0x7feb4403b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb48006fd0 tx=0x7feb48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.318+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feb5400b860 con 0x7feb581077b0 2024-03-12T05:02:27.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.363+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7feb54051020 con 0x7feb581077b0 2024-03-12T05:02:27.617 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:27.614+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb5810bbd0 con 0x7feb581077b0 2024-03-12T05:02:32.412 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.409+0000 7feb4effd700 1 -- 172.21.15.130:0/891416190 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7feb54026070 con 0x7feb581077b0 2024-03-12T05:02:32.413 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T05:02:32.413 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":2,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","modified":"2024-03-12T05:02:27.364669Z","created":"2024-03-12T05:00:42.185744Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.412+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 msgr2=0x7feb4403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.412+0000 7feb5ff05700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 0x7feb4403b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb48006fd0 tx=0x7feb48006e40 comp rx=0 tx=0).stop 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.412+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 msgr2=0x7feb58196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.412+0000 7feb5ff05700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb54004f40 tx=0x7feb54005020 comp rx=0 tx=0).stop 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 shutdown_connections 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feb44038d50 0x7feb4403b210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 --2- 172.21.15.130:0/891416190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb581077b0 0x7feb58196f50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 >> 172.21.15.130:0/891416190 conn(0x7feb580fbd80 msgr2=0x7feb580fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 shutdown_connections 2024-03-12T05:02:32.416 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:32.413+0000 7feb5ff05700 1 -- 172.21.15.130:0/891416190 wait complete. 2024-03-12T05:02:32.417 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 2 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: mon.a calling monitor election 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/891416190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: mon.c calling monitor election 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: fsmap 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: mgrmap e13: x(active, since 49s) 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: overall HEALTH_OK 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: mon.a calling monitor election 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/891416190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: mon.c calling monitor election 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: fsmap 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: mgrmap e13: x(active, since 49s) 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: overall HEALTH_OK 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:33 smithi045 ceph-mon[32174]: Deploying daemon mon.b on smithi116 2024-03-12T05:02:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:33 smithi045 ceph-mon[32174]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:33.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:33.965 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:02:33.965 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:02:33.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:33 smithi130 ceph-mon[35398]: Deploying daemon mon.b on smithi116 2024-03-12T05:02:33.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:33 smithi130 ceph-mon[35398]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:33.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:34.215 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:02:35.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:35 smithi045 ceph-mon[32174]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:35.417 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:35 smithi130 ceph-mon[35398]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 -- 172.21.15.130:0/2412213662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110107740 msgr2=0x7f7110107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 --2- 172.21.15.130:0/2412213662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110107740 0x7f7110107b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7100009910 tx=0x7f7100009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 -- 172.21.15.130:0/2412213662 shutdown_connections 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 --2- 172.21.15.130:0/2412213662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110107740 0x7f7110107b20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 -- 172.21.15.130:0/2412213662 >> 172.21.15.130:0/2412213662 conn(0x7f71100fd210 msgr2=0x7f71100ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 -- 172.21.15.130:0/2412213662 shutdown_connections 2024-03-12T05:02:35.731 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.728+0000 7f7116788700 1 -- 172.21.15.130:0/2412213662 wait complete. 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 Processor -- start 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 -- start start 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 0x7f711019b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7110197be0 con 0x7f7110197650 2024-03-12T05:02:35.733 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f7116788700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7110197d50 con 0x7f7110107740 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39186/0 (socket says 172.21.15.130:39186) 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 -- 172.21.15.130:0/4224341802 learned_addr learned my addr 172.21.15.130:0/4224341802 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710f7fe700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 0x7f711019b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 -- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 msgr2=0x7f7110197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 -- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 msgr2=0x7f7110197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.729+0000 7f710ffff700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710ffff700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710f7fe700 1 -- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 msgr2=0x7f7110197110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710f7fe700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710f7fe700 1 -- 172.21.15.130:0/4224341802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000095f0 con 0x7f7110197650 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710f7fe700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 0x7f711019b9e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f710400b4d0 tx=0x7f710400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:35.734 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7104011630 con 0x7f7110197650 2024-03-12T05:02:35.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7104011c40 con 0x7f7110197650 2024-03-12T05:02:35.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f711019bf80 con 0x7f7110197650 2024-03-12T05:02:35.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7104010cc0 con 0x7f7110197650 2024-03-12T05:02:35.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.730+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711019c4d0 con 0x7f7110197650 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.731+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7110102e80 con 0x7f7110197650 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.731+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f7104011790 con 0x7f7110197650 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.732+0000 7f710d7fa700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 0x7f70f803b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.732+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f710404b410 con 0x7f7110197650 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.732+0000 7f710ffff700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 0x7f70f803b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:35.736 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.732+0000 7f710ffff700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 0x7f70f803b000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f710000b5c0 tx=0x7f7100009ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:35.740 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:35.737+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f71040179b0 con 0x7f7110197650 2024-03-12T05:02:36.008 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.005+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71100632a0 con 0x7f7110197650 2024-03-12T05:02:36.009 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.006+0000 7f710d7fa700 1 -- 172.21.15.130:0/4224341802 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f71040172c0 con 0x7f7110197650 2024-03-12T05:02:36.009 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T05:02:36.009 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":2,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","modified":"2024-03-12T05:02:27.364669Z","created":"2024-03-12T05:00:42.185744Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 msgr2=0x7f70f803b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 0x7f70f803b000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f710000b5c0 tx=0x7f7100009ec0 comp rx=0 tx=0).stop 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 msgr2=0x7f711019b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 0x7f711019b9e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f710400b4d0 tx=0x7f710400b890 comp rx=0 tx=0).stop 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 shutdown_connections 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f70f8038b40 0x7f70f803b000 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:36.011 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7110107740 0x7f7110197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:36.012 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 --2- 172.21.15.130:0/4224341802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7110197650 0x7f711019b9e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:36.012 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 >> 172.21.15.130:0/4224341802 conn(0x7f71100fd210 msgr2=0x7f7110103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:36.012 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 shutdown_connections 2024-03-12T05:02:36.012 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:36.008+0000 7f7116788700 1 -- 172.21.15.130:0/4224341802 wait complete. 2024-03-12T05:02:36.012 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 2 2024-03-12T05:02:36.055 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 podman[35467]: 2024-03-12 05:02:35.926615932 +0000 UTC m=+0.020362570 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:02:36.055 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 podman[35467]: 2024-03-12 05:02:36.053878696 +0000 UTC m=+0.147625323 container create 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 podman[35467]: 2024-03-12 05:02:36.264415333 +0000 UTC m=+0.358161976 container init 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 podman[35467]: 2024-03-12 05:02:36.267242697 +0000 UTC m=+0.360989327 container start 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pidfile_write: ignore empty --pid-file 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: load: jerasure load: lrc 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: RocksDB version: 7.9.2 2024-03-12T05:02:36.318 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Git sha 0 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: DB SUMMARY 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: DB Session ID: UKGTO5VRMIKKD9TT8XRV 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: CURRENT file: CURRENT 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.error_if_exists: 0 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.create_if_missing: 0 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.paranoid_checks: 1 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.env: 0x5567dd788ae0 2024-03-12T05:02:36.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T05:02:36.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.info_log: 0x5567dfb298c0 2024-03-12T05:02:36.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T05:02:36.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.statistics: (nil) 2024-03-12T05:02:36.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.use_fsync: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_log_file_size: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_fallocate: 1 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.use_direct_reads: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T05:02:36.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.db_log_dir: 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.wal_dir: 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.write_buffer_manager: 0x5567dfafd400 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T05:02:36.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.unordered_write: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.row_cache: None 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.wal_filter: None 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.two_write_queues: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T05:02:36.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.wal_compression: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.atomic_flush: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.log_readahead_size: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_background_jobs: 2 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_background_compactions: -1 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_subcompactions: 1 2024-03-12T05:02:36.325 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_open_files: -1 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_background_flushes: -1 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Compression algorithms supported: 2024-03-12T05:02:36.326 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kZSTD supported: 0 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kXpressCompression supported: 0 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kZlibCompression supported: 1 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kSnappyCompression supported: 1 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kLZ4Compression supported: 1 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: kBZip2Compression supported: 0 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.merge_operator: 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_filter: None 2024-03-12T05:02:36.327 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567dfb299c0) 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_type: 0 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: data_block_index_type: 0 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_shortening: 1 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: checksum: 4 2024-03-12T05:02:36.328 INFO:journalctl@ceph.mon.b.smithi116.stdout: no_block_cache: 0 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache: 0x5567dfb2d090 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_options: 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: capacity : 536870912 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: num_shard_bits : 4 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: strict_capacity_limit : 0 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_cache_compressed: (nil) 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: persistent_cache: (nil) 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_size: 4096 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_size_deviation: 10 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_restart_interval: 16 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: index_block_restart_interval: 1 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: metadata_block_size: 4096 2024-03-12T05:02:36.329 INFO:journalctl@ceph.mon.b.smithi116.stdout: partition_filters: 0 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: use_delta_encoding: 1 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: filter_policy: bloomfilter 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: whole_key_filtering: 1 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: verify_compression: 0 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: format_version: 5 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: enable_index_compression: 1 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: block_align: 0 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: prepopulate_block_cache: 0 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T05:02:36.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression: NoCompression 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.num_levels: 7 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T05:02:36.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T05:02:36.332 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T05:02:36.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.table_properties_collectors: 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.inplace_update_support: 0 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T05:02:36.334 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.bloom_locality: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.max_successive_merges: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.ttl: 2592000 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enable_blob_files: false 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.min_blob_size: 0 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T05:02:36.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b581cb0f-3e5b-4df6-a317-dc10132bf976 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219756296666, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219756297065, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710219756, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b581cb0f-3e5b-4df6-a317-dc10132bf976", "db_session_id": "UKGTO5VRMIKKD9TT8XRV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710219756297155, "job": 1, "event": "recovery_finished"} 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T05:02:36.336 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5567dfc1e000 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: DB pointer 0x5567dfc08000 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** DB Stats ** 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:02:36.337 INFO:journalctl@ceph.mon.b.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T05:02:36.338 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: Block cache BinnedLRUCache@0x5567dfb2d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: using public_addr v2:172.21.15.116:0/0 -> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: starting mon.b rank -1 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(???) e0 preinit fsid 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).mds e1 new map 2024-03-12T05:02:36.339 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: e1 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: legacy client fscid: -1 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout: No filesystems configured 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:02:36.340 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Deploying cephadm binary to smithi130 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:36.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Added host smithi130 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/868843197' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T05:02:36.342 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/868843197' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:36.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c;count:3 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Deploying daemon mon.c on smithi130 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:36.344 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.a calling monitor election 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/891416190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.c calling monitor election 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: fsmap 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mgrmap e13: x(active, since 49s) 2024-03-12T05:02:36.345 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: overall HEALTH_OK 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: Deploying daemon mon.b on smithi116 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/4224341802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.346 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 ceph-mon[35486]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T05:02:36.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:36 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/4224341802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:36 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/4224341802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:02:36.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 bash[35467]: 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 2024-03-12T05:02:36.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:36 smithi116 systemd[1]: Started Ceph mon.b for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:02:37.603 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:02:37.603 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:02:37.844 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: mon.c calling monitor election 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: mon.a calling monitor election 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: fsmap 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: mgrmap e13: x(active, since 58s) 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: mon.c calling monitor election 2024-03-12T05:02:41.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:41.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: mon.a calling monitor election 2024-03-12T05:02:41.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: fsmap 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: mgrmap e13: x(active, since 58s) 2024-03-12T05:02:41.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:41.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:42.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 -- 172.21.15.130:0/2936689676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 msgr2=0x7f04e4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:42.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/2936689676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04e4005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f04f8100560 tx=0x7f04f4009e80 comp rx=0 tx=0).stop 2024-03-12T05:02:42.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 -- 172.21.15.130:0/2936689676 shutdown_connections 2024-03-12T05:02:42.278 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/2936689676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04e4005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 -- 172.21.15.130:0/2936689676 >> 172.21.15.130:0/2936689676 conn(0x7f04f80fc120 msgr2=0x7f04f80fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 -- 172.21.15.130:0/2936689676 shutdown_connections 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.273+0000 7f04ffeeb700 1 -- 172.21.15.130:0/2936689676 wait complete. 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 Processor -- start 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 -- start start 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04f8197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f8197da0 con 0x7f04e40035c0 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f81a3ca0 con 0x7f04f819bf30 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04ffeeb700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f81a3de0 con 0x7f04f81976b0 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:32894/0 (socket says 172.21.15.130:32894) 2024-03-12T05:02:42.279 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 -- 172.21.15.130:0/1570073790 learned_addr learned my addr 172.21.15.130:0/1570073790 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 msgr2=0x7f04f819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 msgr2=0x7f04f819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fd486700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fdc87700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04f8197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fe488700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fe488700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 msgr2=0x7f04f819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fe488700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 msgr2=0x7f04f819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fe488700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.274+0000 7f04fe488700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 msgr2=0x7f04f819b9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.280 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 msgr2=0x7f04f819c3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 -- 172.21.15.130:0/1570073790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40098d0 con 0x7f04e40035c0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04fdc87700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04f8197170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f04f4002230 tx=0x7f04f4004de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4017780 con 0x7f04e40035c0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f81a4060 con 0x7f04e40035c0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f04f40208f0 con 0x7f04e40035c0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40054c0 con 0x7f04e40035c0 2024-03-12T05:02:42.281 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.275+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f81a4560 con 0x7f04e40035c0 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.276+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f80632a0 con 0x7f04e40035c0 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.276+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f04f4020440 con 0x7f04e40035c0 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.276+0000 7f04eeffd700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 0x7f04dc03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.276+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f04f404da10 con 0x7f04e40035c0 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.277+0000 7f04fd486700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 0x7f04dc03b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:42.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.277+0000 7f04fd486700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 0x7f04dc03b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04e8009180 tx=0x7f04e8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:42.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.283+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04f4030350 con 0x7f04e40035c0 2024-03-12T05:02:42.554 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:42.554 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:42.567 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.565+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04f8108390 con 0x7f04e40035c0 2024-03-12T05:02:42.567 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.566+0000 7f04eeffd700 1 -- 172.21.15.130:0/1570073790 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7f04f4030350 con 0x7f04e40035c0 2024-03-12T05:02:42.568 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-12T05:02:42.568 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":3,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","modified":"2024-03-12T05:02:36.321814Z","created":"2024-03-12T05:00:42.185744Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 msgr2=0x7f04dc03b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 0x7f04dc03b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04e8009180 tx=0x7f04e8008040 comp rx=0 tx=0).stop 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 msgr2=0x7f04f8197170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04f8197170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f04f4002230 tx=0x7f04f4004de0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 shutdown_connections 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04dc0391e0 0x7f04dc03b6a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.570 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04f819bf30 0x7f04f819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04e40035c0 0x7f04f8197170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 --2- 172.21.15.130:0/1570073790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04f81976b0 0x7f04f819b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 >> 172.21.15.130:0/1570073790 conn(0x7f04f80fc120 msgr2=0x7f04f8105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 shutdown_connections 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:42.568+0000 7f04ffeeb700 1 -- 172.21.15.130:0/1570073790 wait complete. 2024-03-12T05:02:42.571 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 3 2024-03-12T05:02:42.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:42.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.487 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T05:02:43.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T05:02:43.735 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:43.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: mon.c calling monitor election 2024-03-12T05:02:43.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: mon.a calling monitor election 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: fsmap 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: mgrmap e13: x(active, since 58s) 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:02:43.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:43.751 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:44.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mon.b calling monitor election 2024-03-12T05:02:44.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.438 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mon.b calling monitor election 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mon.c calling monitor election 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mon.a calling monitor election 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: fsmap 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: mgrmap e13: x(active, since 60s) 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Cluster is now healthy 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: overall HEALTH_OK 2024-03-12T05:02:44.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: Reconfiguring daemon mon.a on smithi045 2024-03-12T05:02:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mon.b calling monitor election 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mon.b calling monitor election 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mon.c calling monitor election 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mon.a calling monitor election 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: fsmap 2024-03-12T05:02:44.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: mgrmap e13: x(active, since 60s) 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Cluster is now healthy 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: overall HEALTH_OK 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:44.725 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:44.726 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: Reconfiguring daemon mon.a on smithi045 2024-03-12T05:02:44.726 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:44.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mon.b calling monitor election 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mon.b calling monitor election 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mon.c calling monitor election 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mon.a calling monitor election 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: fsmap 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: mgrmap e13: x(active, since 60s) 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Cluster is now healthy 2024-03-12T05:02:44.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: overall HEALTH_OK 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: Reconfiguring daemon mon.a on smithi045 2024-03-12T05:02:44.750 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:02:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:45 smithi045 ceph-mon[32174]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:45.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.652+0000 7f0483ec3700 1 -- 172.21.15.45:0/109992898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 msgr2=0x7f047c107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.652+0000 7f0483ec3700 1 --2- 172.21.15.45:0/109992898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c107a30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f046c00b0a0 tx=0x7f046c00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 -- 172.21.15.45:0/109992898 shutdown_connections 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 --2- 172.21.15.45:0/109992898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c107a30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 -- 172.21.15.45:0/109992898 >> 172.21.15.45:0/109992898 conn(0x7f047c0fd180 msgr2=0x7f047c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 -- 172.21.15.45:0/109992898 shutdown_connections 2024-03-12T05:02:45.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 -- 172.21.15.45:0/109992898 wait complete. 2024-03-12T05:02:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 Processor -- start 2024-03-12T05:02:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.653+0000 7f0483ec3700 1 -- start start 2024-03-12T05:02:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:45.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 0x7f047c19b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f047c19be90 0x7f047c19c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c197cb0 con 0x7f047c107650 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c1a3c00 con 0x7f047c1975c0 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0483ec3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c1a3d40 con 0x7f047c19be90 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52292/0 (socket says 172.21.15.45:52292) 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 -- 172.21.15.45:0/1285963661 learned_addr learned my addr 172.21.15.45:0/1285963661 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f048145e700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 0x7f047c19b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 -- 172.21.15.45:0/1285963661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f047c19be90 msgr2=0x7f047c19c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f047c19be90 0x7f047c19c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 -- 172.21.15.45:0/1285963661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 msgr2=0x7f047c19b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 0x7f047c19b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 -- 172.21.15.45:0/1285963661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c009d00 con 0x7f047c107650 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f048145e700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 0x7f047c19b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:02:45.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f0481c5f700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f046c009510 tx=0x7f046c0075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c00e040 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.654+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f046c005720 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.655+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c1a3fc0 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.655+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c012b80 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.655+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c1a4430 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.655+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f046c019070 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.655+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c0632a0 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.656+0000 7f047affd700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 0x7f046403b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.656+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f046c04caf0 con 0x7f047c107650 2024-03-12T05:02:45.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.656+0000 7f048145e700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 0x7f046403b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:45.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.656+0000 7f048145e700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 0x7f046403b6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04700079b0 tx=0x7f0470008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:45.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.661+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f046c012430 con 0x7f047c107650 2024-03-12T05:02:45.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:45 smithi130 ceph-mon[35398]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:45.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:45 smithi116 ceph-mon[35486]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.863+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f047c10ba90 con 0x7f047c107650 2024-03-12T05:02:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.864+0000 7f047affd700 1 -- 172.21.15.45:0/1285963661 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f046c017070 con 0x7f047c107650 2024-03-12T05:02:45.865 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:02:45.865 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-03-12T05:02:45.866 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:02:45.866 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2024-03-12T05:02:45.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.865+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 msgr2=0x7f046403b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:45.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.865+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 0x7f046403b6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f04700079b0 tx=0x7f0470008040 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.865+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 msgr2=0x7f047c197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.865+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f046c009510 tx=0x7f046c0075c0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 shutdown_connections 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f04640391e0 0x7f046403b6a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f047c19be90 0x7f047c19c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f047c107650 0x7f047c197080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 --2- 172.21.15.45:0/1285963661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047c1975c0 0x7f047c19b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 >> 172.21.15.45:0/1285963661 conn(0x7f047c0fd180 msgr2=0x7f047c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 shutdown_connections 2024-03-12T05:02:45.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:45.866+0000 7f0483ec3700 1 -- 172.21.15.45:0/1285963661 wait complete. 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1285963661' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:46 smithi045 ceph-mon[32174]: Reconfiguring daemon mon.b on smithi116 2024-03-12T05:02:46.502 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T05:02:46.502 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:02:46.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:02:46.532 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:02:46.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:46.598 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T05:02:46.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1285963661' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:46.623 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.624 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:46 smithi116 ceph-mon[35486]: Reconfiguring daemon mon.b on smithi116 2024-03-12T05:02:46.627 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T05:02:46.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:46.694 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T05:02:46.694 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:02:46.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1285963661' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:46.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:02:46.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:46.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:46.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:46.724 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:46 smithi130 ceph-mon[35398]: Reconfiguring daemon mon.b on smithi116 2024-03-12T05:02:46.729 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T05:02:46.729 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:02:46.802 INFO:tasks.cephadm:Adding mgr.x on smithi045 2024-03-12T05:02:46.802 INFO:tasks.cephadm:Adding mgr.y on smithi116 2024-03-12T05:02:46.802 INFO:tasks.cephadm:Adding mgr.z on smithi130 2024-03-12T05:02:46.802 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi045=x;smithi116=y;smithi130=z' 2024-03-12T05:02:47.078 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:02:47.450 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:47 smithi130 ceph-mon[35398]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:47.602 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:47 smithi116 ceph-mon[35486]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:47.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:47 smithi045 ceph-mon[32174]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:48.518 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- 172.21.15.130:0/13152712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981006d0 msgr2=0x7f7598100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:48.518 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 --2- 172.21.15.130:0/13152712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981006d0 0x7f7598100af0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7594009910 tx=0x7f7594009c20 comp rx=0 tx=0).stop 2024-03-12T05:02:48.518 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- 172.21.15.130:0/13152712 shutdown_connections 2024-03-12T05:02:48.518 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 --2- 172.21.15.130:0/13152712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981006d0 0x7f7598100af0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.518 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- 172.21.15.130:0/13152712 >> 172.21.15.130:0/13152712 conn(0x7f75980fbc50 msgr2=0x7f75980fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- 172.21.15.130:0/13152712 shutdown_connections 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- 172.21.15.130:0/13152712 wait complete. 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 Processor -- start 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.515+0000 7f759f297700 1 -- start start 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7598196160 0x7f759819b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981965e0 0x7f759819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759819df10 con 0x7f75981965e0 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759819e080 con 0x7f7598196160 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759819e240 con 0x7f75981006d0 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:54040/0 (socket says 172.21.15.130:54040) 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 -- 172.21.15.130:0/881481637 learned_addr learned my addr 172.21.15.130:0/881481637 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:02:48.519 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d834700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981965e0 0x7f759819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759c832700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7598196160 0x7f759819b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 -- 172.21.15.130:0/881481637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7598196160 msgr2=0x7f759819b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7598196160 0x7f759819b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 -- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981965e0 msgr2=0x7f759819d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981965e0 0x7f759819d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 -- 172.21.15.130:0/881481637 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940095f0 con 0x7f75981006d0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759d033700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7594000c00 tx=0x7f759400bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759401d070 con 0x7f75981006d0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7594003d40 con 0x7f75981006d0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759819e4c0 con 0x7f75981006d0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.516+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400fd20 con 0x7f75981006d0 2024-03-12T05:02:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.517+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759819e9e0 con 0x7f75981006d0 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.517+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f759400f4f0 con 0x7f75981006d0 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.517+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759818f390 con 0x7f75981006d0 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.518+0000 7f758a7fc700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 0x7f758403b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.518+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f759404c910 con 0x7f75981006d0 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.518+0000 7f759c832700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 0x7f758403b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.518+0000 7f759c832700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 0x7f758403b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f758c009850 tx=0x7f758c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.523+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f75940163a0 con 0x7f75981006d0 2024-03-12T05:02:48.735 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.733+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi116=y;smithi130=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f759802d870 con 0x7f75840391e0 2024-03-12T05:02:48.741 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.739+0000 7f758a7fc700 1 -- 172.21.15.130:0/881481637 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f759802d870 con 0x7f75840391e0 2024-03-12T05:02:48.741 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mgr update... 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 msgr2=0x7f758403b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 0x7f758403b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f758c009850 tx=0x7f758c008040 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 msgr2=0x7f7598195c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7594000c00 tx=0x7f759400bcd0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 shutdown_connections 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75840391e0 0x7f758403b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75981965e0 0x7f759819d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75981006d0 0x7f7598195c20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 --2- 172.21.15.130:0/881481637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7598196160 0x7f759819b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 >> 172.21.15.130:0/881481637 conn(0x7f75980fbc50 msgr2=0x7f75980fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 shutdown_connections 2024-03-12T05:02:48.744 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:02:48.742+0000 7f759f297700 1 -- 172.21.15.130:0/881481637 wait complete. 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:48.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:48 smithi130 ceph-mon[35398]: Reconfiguring daemon mon.c on smithi130 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:48.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:48 smithi116 ceph-mon[35486]: Reconfiguring daemon mon.c on smithi130 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:49.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:48 smithi045 ceph-mon[32174]: Reconfiguring daemon mon.c on smithi130 2024-03-12T05:02:49.612 DEBUG:teuthology.orchestra.run.smithi116:mgr.y> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T05:02:49.615 DEBUG:teuthology.orchestra.run.smithi130:mgr.z> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.z.service 2024-03-12T05:02:49.618 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T05:02:49.618 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:02:49.618 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:02:49.637 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:02:49.638 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-12 05:01:55.706187882 +0000 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-12 04:55:27.183576943 +0000 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-12 04:55:27.183576943 +0000 2024-03-12T05:02:49.695 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-12 04:55:27.183576943 +0000 2024-03-12T05:02:49.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:02:49.767 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-12T05:02:49.767 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-12T05:02:49.767 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000122579 s, 4.2 MB/s 2024-03-12T05:02:49.768 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:02:49.826 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-03-12T05:02:49.882 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:02:49.882 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:49.882 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-12 05:01:55.742187365 +0000 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-12 04:55:27.541571717 +0000 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-12 04:55:27.541571717 +0000 2024-03-12T05:02:49.883 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-12 04:55:27.541571717 +0000 2024-03-12T05:02:49.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:02:49.927 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:49 smithi130 ceph-mon[35398]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi116=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:49.927 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:49 smithi130 ceph-mon[35398]: Saving service mgr spec with placement smithi045=x;smithi116=y;smithi130=z;count:3 2024-03-12T05:02:49.927 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:49 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:49.928 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:49 smithi130 ceph-mon[35398]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-12T05:02:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-12T05:02:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000108632 s, 4.7 MB/s 2024-03-12T05:02:49.951 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:02:49.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:49 smithi116 ceph-mon[35486]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi116=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:49.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:49 smithi116 ceph-mon[35486]: Saving service mgr spec with placement smithi045=x;smithi116=y;smithi130=z;count:3 2024-03-12T05:02:49.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:49 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:49.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:49 smithi116 ceph-mon[35486]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:50.008 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-12 05:01:55.775186892 +0000 2024-03-12T05:02:50.064 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-12 04:55:27.880566767 +0000 2024-03-12T05:02:50.065 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-12 04:55:27.880566767 +0000 2024-03-12T05:02:50.065 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-12 04:55:27.880566767 +0000 2024-03-12T05:02:50.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:02:50.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:49 smithi045 ceph-mon[32174]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi045=x;smithi116=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:50.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:49 smithi045 ceph-mon[32174]: Saving service mgr spec with placement smithi045=x;smithi116=y;smithi130=z;count:3 2024-03-12T05:02:50.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:49 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:50.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:49 smithi045 ceph-mon[32174]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:50.130 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-12T05:02:50.130 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-12T05:02:50.130 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000107574 s, 4.8 MB/s 2024-03-12T05:02:50.131 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:02:50.187 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-12 05:01:55.809186403 +0000 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-12 04:55:28.220561803 +0000 2024-03-12T05:02:50.242 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-12 04:55:28.220561803 +0000 2024-03-12T05:02:50.243 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-12 04:55:28.220561803 +0000 2024-03-12T05:02:50.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:02:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-12T05:02:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-12T05:02:50.308 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000104465 s, 4.9 MB/s 2024-03-12T05:02:50.309 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:02:50.365 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-12T05:02:50.365 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:02:50.383 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:02:50.384 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-12 05:02:35.729076215 +0000 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-12 04:55:22.686642423 +0000 2024-03-12T05:02:50.441 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-12 04:55:22.686642423 +0000 2024-03-12T05:02:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-12 04:55:22.686642423 +0000 2024-03-12T05:02:50.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:02:50.510 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-12T05:02:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-12T05:02:50.511 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000223073 s, 2.3 MB/s 2024-03-12T05:02:50.512 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:02:50.569 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-12 05:02:35.729076215 +0000 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-12 04:55:23.032636965 +0000 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-12 04:55:23.032636965 +0000 2024-03-12T05:02:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-12 04:55:23.032636965 +0000 2024-03-12T05:02:50.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:02:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-12T05:02:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-12T05:02:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000226142 s, 2.3 MB/s 2024-03-12T05:02:50.695 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:02:50.752 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-12 05:02:35.729076215 +0000 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-12 04:55:23.385631396 +0000 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-12 04:55:23.385631396 +0000 2024-03-12T05:02:50.808 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-12 04:55:23.385631396 +0000 2024-03-12T05:02:50.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:02:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-12T05:02:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-12T05:02:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000206392 s, 2.5 MB/s 2024-03-12T05:02:50.875 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:02:50.931 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-12 05:02:35.730076200 +0000 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-12 04:55:23.732625921 +0000 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-12 04:55:23.732625921 +0000 2024-03-12T05:02:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-12 04:55:23.732625921 +0000 2024-03-12T05:02:50.988 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:02:51.054 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-12T05:02:51.054 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-12T05:02:51.054 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000273718 s, 1.9 MB/s 2024-03-12T05:02:51.055 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:02:51.112 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-12T05:02:51.112 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:02:51.127 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:02:51.128 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-03-12T05:02:51.182 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:02:51.182 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:51.182 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-12 05:02:26.334013719 +0000 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-12 04:55:22.690683657 +0000 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-12 04:55:22.690683657 +0000 2024-03-12T05:02:51.183 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-12 04:55:22.690683657 +0000 2024-03-12T05:02:51.183 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:02:51.246 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.246 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:51.247 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:51.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.249 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-12T05:02:51.249 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-12T05:02:51.249 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000262964 s, 1.9 MB/s 2024-03-12T05:02:51.250 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:02:51.306 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-03-12T05:02:51.360 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-12 05:02:26.334013719 +0000 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-12 04:55:23.039676976 +0000 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-12 04:55:23.039676976 +0000 2024-03-12T05:02:51.361 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-12 04:55:23.039676976 +0000 2024-03-12T05:02:51.361 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:51.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:51.426 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-12T05:02:51.426 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-12T05:02:51.426 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000191397 s, 2.7 MB/s 2024-03-12T05:02:51.427 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:02:51.482 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-03-12T05:02:51.537 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-12 05:02:26.335013701 +0000 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-12 04:55:23.389670278 +0000 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-12 04:55:23.389670278 +0000 2024-03-12T05:02:51.538 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-12 04:55:23.389670278 +0000 2024-03-12T05:02:51.538 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:02:51.603 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-12T05:02:51.603 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-12T05:02:51.603 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.00023522 s, 2.2 MB/s 2024-03-12T05:02:51.604 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:02:51.659 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-12 05:02:26.335013701 +0000 2024-03-12T05:02:51.714 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-12 04:55:23.720663944 +0000 2024-03-12T05:02:51.715 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-12 04:55:23.720663944 +0000 2024-03-12T05:02:51.715 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-12 04:55:23.720663944 +0000 2024-03-12T05:02:51.715 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:02:51.780 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-12T05:02:51.780 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-12T05:02:51.780 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000187751 s, 2.7 MB/s 2024-03-12T05:02:51.782 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:02:51.838 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2024-03-12T05:02:51.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:02:52.089 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:52.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:51 smithi045 ceph-mon[32174]: Deploying daemon mgr.z on smithi130 2024-03-12T05:02:52.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:51 smithi045 ceph-mon[32174]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:52.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:51 smithi130 ceph-mon[35398]: Deploying daemon mgr.z on smithi130 2024-03-12T05:02:52.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:51 smithi130 ceph-mon[35398]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:52.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:51 smithi116 ceph-mon[35486]: Deploying daemon mgr.z on smithi130 2024-03-12T05:02:52.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:51 smithi116 ceph-mon[35486]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:53.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:53 smithi045 ceph-mon[32174]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:53.378 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:53 smithi116 ceph-mon[35486]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:53.426 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:53 smithi130 ceph-mon[35398]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:53.718 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:53.716+0000 7f96aae07200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00825853 s, 1.3 GB/s 2024-03-12T05:02:53.842 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-12T05:02:54.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-03-12T05:02:54.468 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:54.469 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:54 smithi116 ceph-mon[35486]: Deploying daemon mgr.y on smithi116 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:54.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:54 smithi045 ceph-mon[32174]: Deploying daemon mgr.y on smithi116 2024-03-12T05:02:54.611 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:54.375+0000 7f96aae07200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:02:54.611 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:54.469+0000 7f96aae07200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:02:54.611 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:54.609+0000 7f96aae07200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:02:54.611 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.611 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:54.612 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-mon[35398]: Deploying daemon mgr.y on smithi116 2024-03-12T05:02:54.699 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:02:54.973 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:54.703+0000 7f96aae07200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:02:54.973 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:54.784+0000 7f96aae07200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:02:55.232 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 systemd[1]: Starting Ceph mgr.y for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:02:55.334 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-mon[35398]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:55.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:55.045+0000 7f96aae07200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:02:55.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:55.153+0000 7f96aae07200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:02:55.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:55.238+0000 7f96aae07200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:02:55.334 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:55.332+0000 7f96aae07200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:02:55.359 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:55 smithi045 ceph-mon[32174]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:55.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:55 smithi116 ceph-mon[35486]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:55.498 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 podman[36856]: 2024-03-12 05:02:55.34059208 +0000 UTC m=+0.099875427 container create f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-12T05:02:55.498 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 podman[36856]: 2024-03-12 05:02:55.257690585 +0000 UTC m=+0.016973937 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:02:55.723 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:55.508+0000 7f96aae07200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:02:55.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 podman[36856]: 2024-03-12 05:02:55.551556177 +0000 UTC m=+0.310839535 container init f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T05:02:55.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 podman[36856]: 2024-03-12 05:02:55.554629583 +0000 UTC m=+0.313912934 container start f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T05:02:55.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 ceph-mgr[36875]: -- 172.21.15.116:0/2888053785 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8353d21a0 con 0x55c8353a9000 2024-03-12T05:02:55.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 bash[36856]: f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 2024-03-12T05:02:55.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:55 smithi116 systemd[1]: Started Ceph mgr.y for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:02:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/1613595857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8102130 msgr2=0x7f6fc8102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:56.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/1613595857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8102130 0x7f6fc8102550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009990 tx=0x7f6fb8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/1613595857 shutdown_connections 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/1613595857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8101810 0x7f6fc81090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/1613595857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8102130 0x7f6fc8102550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/1613595857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8100e70 0x7f6fc81012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/1613595857 >> 172.21.15.45:0/1613595857 conn(0x7f6fc80fb8e0 msgr2=0x7f6fc80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/1613595857 shutdown_connections 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.107+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/1613595857 wait complete. 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 Processor -- start 2024-03-12T05:02:56.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 -- start start 2024-03-12T05:02:56.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:56.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:56.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8102130 0x7f6fc81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81a18f0 con 0x7f6fc8101810 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81a1a30 con 0x7f6fc8100e70 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fcd7e1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81a1bf0 con 0x7f6fc8102130 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fc67fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fc6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fc6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:48550/0 (socket says 172.21.15.45:48550) 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fc67fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33010/0 (socket says 172.21.15.45:33010) 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.108+0000 7f6fc6ffd700 1 -- 172.21.15.45:0/3432079541 learned_addr learned my addr 172.21.15.45:0/3432079541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc77fe700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8102130 0x7f6fc81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 -- 172.21.15.45:0/3432079541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8102130 msgr2=0x7f6fc81a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8102130 0x7f6fc81a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 -- 172.21.15.45:0/3432079541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 msgr2=0x7f6fc8199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 -- 172.21.15.45:0/3432079541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80095f0 con 0x7f6fc8101810 2024-03-12T05:02:56.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc6ffd700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fc67fc700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d950 tx=0x7f6fbc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00f890 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6fbc010460 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc81a1ed0 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00fea0 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.109+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc81a22e0 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.110+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6fbc00e610 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.110+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc810d530 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.110+0000 7f6faffff700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 0x7f6fb003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.110+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6fbc04b5f0 con 0x7f6fc8101810 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.110+0000 7f6fc6ffd700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 0x7f6fb003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:02:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.111+0000 7f6fc6ffd700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 0x7f6fb003b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800b5c0 tx=0x7f6fb8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:02:56.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.115+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fbc009670 con 0x7f6fc8101810 2024-03-12T05:02:56.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:02:56.329+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fc80689d0 con 0x7f6fb0038f90 2024-03-12T05:02:56.498 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:56.110+0000 7f0d9a98f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:02:56.498 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:56.207+0000 7f0d9a98f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:56.519+0000 7f96aae07200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:56.654+0000 7f96aae07200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:02:56.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:56.895+0000 7f0d9a98f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:02:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:02:56.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.223 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:57.000+0000 7f96aae07200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:02:57.223 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:57.092+0000 7f96aae07200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:02:57.248 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:56.989+0000 7f0d9a98f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:02:57.248 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.141+0000 7f0d9a98f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:02:57.593 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.247+0000 7f0d9a98f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:02:57.594 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.331+0000 7f0d9a98f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:02:57.594 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.591+0000 7f0d9a98f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:02:57.700 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:57.295+0000 7f96aae07200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:02:57.700 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:57.480+0000 7f96aae07200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:02:57.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:57.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:57.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:57.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.701+0000 7f0d9a98f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:02:57.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.786+0000 7f0d9a98f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:02:57.881 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:57.879+0000 7f0d9a98f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.884 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.885 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:57.885 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.885 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:57.885 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:57.885 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.973 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:57.762+0000 7f96aae07200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:02:57.975 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:02:58.248 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:58.051+0000 7f0d9a98f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:02:58.473 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:58.024+0000 7f96aae07200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:02:58.473 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:58.257+0000 7f96aae07200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:02:58.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:58 smithi130 ceph-mon[35398]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:02:58.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:58 smithi130 ceph-mon[35398]: Reconfiguring daemon mgr.x on smithi045 2024-03-12T05:02:58.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:58 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:58.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:58 smithi116 ceph-mon[35486]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:02:58.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:58 smithi116 ceph-mon[35486]: Reconfiguring daemon mgr.x on smithi045 2024-03-12T05:02:58.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:58 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:59.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:58 smithi045 ceph-mon[32174]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:02:59.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:58 smithi045 ceph-mon[32174]: Reconfiguring daemon mgr.x on smithi045 2024-03-12T05:02:59.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:58 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:02:59.464 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.010+0000 7f0d9a98f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:02:59.464 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.136+0000 7f0d9a98f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:02:59.464 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.462+0000 7f0d9a98f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:02:59.473 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:59.051+0000 7f96aae07200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:02:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.551+0000 7f0d9a98f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:02:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.744+0000 7f0d9a98f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:02:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]: dispatch 2024-03-12T05:02:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]': finished 2024-03-12T05:02:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:02:59.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:02:59.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:59.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2606181554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:02:59.928 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:59.616+0000 7f96aae07200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:59.712+0000 7f96aae07200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:02:59.927+0000 7f96aae07200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]: dispatch 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]': finished 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:02:59.929 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:02:59 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2606181554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:02:59.998 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:02:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:02:59.924+0000 7f0d9a98f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]: dispatch 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1107436159' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e690af1-6483-47d3-8fcb-240b9f1bec6b"}]': finished 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:00.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:02:59 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2606181554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:00.459 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:00.204+0000 7f0d9a98f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:03:00.748 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:00.457+0000 7f0d9a98f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:03:00.748 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:00.689+0000 7f0d9a98f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:03:00.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:00.942+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 14) v1 ==== 91024+0+0 (secure 0 0 0) 0x7f6fbc019360 con 0x7f6fc8101810 2024-03-12T05:03:01.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:00 smithi130 ceph-mon[35398]: Standby manager daemon z started 2024-03-12T05:03:01.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:00 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:03:01.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:00 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:01.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:00 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:03:01.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:00 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:01.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-mon[35486]: Standby manager daemon z started 2024-03-12T05:03:01.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:03:01.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:01.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:03:01.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:00 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:00 smithi045 ceph-mon[32174]: Standby manager daemon z started 2024-03-12T05:03:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:00 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:03:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:00 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:00 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:03:01.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:00 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/3648297659' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:01.748 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:01.480+0000 7f0d9a98f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:03:02.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:01 smithi130 ceph-mon[35398]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:03:02.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:03:02.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:01 smithi130 ceph-mon[35398]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:02.248 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:02.041+0000 7f0d9a98f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:03:02.248 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:02.138+0000 7f0d9a98f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:03:02.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:01 smithi116 ceph-mon[35486]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:03:02.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:03:02.249 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:01 smithi116 ceph-mon[35486]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:02.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:01 smithi045 ceph-mon[32174]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:03:02.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:03:02.273 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:01 smithi045 ceph-mon[32174]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:02.748 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:03:02.351+0000 7f0d9a98f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:03:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:02.955+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f6fbc0105d0 con 0x7f6fc8101810 2024-03-12T05:03:03.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:02 smithi045 ceph-mon[32174]: Standby manager daemon y started 2024-03-12T05:03:03.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:02 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:03:03.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:02 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:03.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:02 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:03:03.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:02 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:03.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:02 smithi130 ceph-mon[35398]: Standby manager daemon y started 2024-03-12T05:03:03.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:02 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:03:03.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:02 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:03.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:02 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:03:03.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:02 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:03.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-mon[35486]: Standby manager daemon y started 2024-03-12T05:03:03.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:03:03.249 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:03:03.249 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:03:03.249 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:02 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/853475258' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:03:04.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:03 smithi045 ceph-mon[32174]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-12T05:03:04.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:03 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:03:04.149 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:03 smithi045 ceph-mon[32174]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:04.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:03 smithi130 ceph-mon[35398]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-12T05:03:04.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:03 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:03:04.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:03 smithi130 ceph-mon[35398]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:04.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:03 smithi116 ceph-mon[35486]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-12T05:03:04.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:03 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:03:04.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:03 smithi116 ceph-mon[35486]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.347 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:05 smithi045 ceph-mon[32174]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:05.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:05.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:05.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:05.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:05 smithi130 ceph-mon[35398]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:05.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:05 smithi116 ceph-mon[35486]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:06.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:03:06.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:06.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:03:06.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:07.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:03:07.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:07.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:07 smithi045 ceph-mon[32174]: Deploying daemon osd.0 on smithi045 2024-03-12T05:03:07.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:07 smithi045 ceph-mon[32174]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:07 smithi130 ceph-mon[35398]: Deploying daemon osd.0 on smithi045 2024-03-12T05:03:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:07 smithi130 ceph-mon[35398]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:07.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:07 smithi116 ceph-mon[35486]: Deploying daemon osd.0 on smithi045 2024-03-12T05:03:07.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:07 smithi116 ceph-mon[35486]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:09.457 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:09 smithi045 ceph-mon[32174]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:09.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:09 smithi130 ceph-mon[35398]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:09.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:09 smithi116 ceph-mon[35486]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:11.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:11 smithi045 ceph-mon[32174]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:11.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:11 smithi130 ceph-mon[35398]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:11.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:11 smithi116 ceph-mon[35486]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:12.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:12.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:12.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:12.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:12.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:12.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:13.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:13.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:13.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:13.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:13 smithi130 ceph-mon[35398]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:13.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:13 smithi116 ceph-mon[35486]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:14.076 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:13 smithi045 ceph-mon[32174]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:14.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.680+0000 7f6faffff700 1 -- 172.21.15.45:0/3432079541 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6fc80689d0 con 0x7f6fb0038f90 2024-03-12T05:03:14.681 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 msgr2=0x7f6fb003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 0x7f6fb003b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6fb800b5c0 tx=0x7f6fb8005fb0 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 msgr2=0x7f6fc819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00d950 tx=0x7f6fbc00dc60 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 shutdown_connections 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6fb0038f90 0x7f6fb003b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fc8102130 0x7f6fc81a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fc8100e70 0x7f6fc8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 --2- 172.21.15.45:0/3432079541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fc8101810 0x7f6fc819a2e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.682+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 >> 172.21.15.45:0/3432079541 conn(0x7f6fc80fb8e0 msgr2=0x7f6fc8109910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.683+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 shutdown_connections 2024-03-12T05:03:14.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:03:14.683+0000 7f6fcd7e1700 1 -- 172.21.15.45:0/3432079541 wait complete. 2024-03-12T05:03:15.240 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T05:03:15.282 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-12T05:03:15.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:03:15.524 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:03:15.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:15.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-mon[32174]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:03:15.872 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:03:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:03:15.591+0000 7f1c8cc397c0 -1 osd.0 0 log_to_monitors true 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:15.957 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:15 smithi116 ceph-mon[35486]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:15.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:15 smithi130 ceph-mon[35398]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:03:16.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:16 smithi045 ceph-mon[32174]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:03:16.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:16 smithi045 ceph-mon[32174]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:03:16.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:16 smithi045 ceph-mon[32174]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:03:16.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:16.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:16 smithi116 ceph-mon[35486]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:03:16.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:16 smithi116 ceph-mon[35486]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:03:16.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:16 smithi116 ceph-mon[35486]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:03:16.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:16 smithi130 ceph-mon[35398]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:03:17.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:16 smithi130 ceph-mon[35398]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:03:17.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:16 smithi130 ceph-mon[35398]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:03:17.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.361 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:03:17.361 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:03:17.361 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-12T05:03:17.361 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-12T05:03:17.362 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128108 s, 819 MB/s 2024-03-12T05:03:17.362 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-12T05:03:17.508 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:03:17.230+0000 7f1c82a9f700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T05:03:17.508 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:03:17.233+0000 7f1c7a089700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:03:17.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:03:17.810 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:03:17.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:17.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:17 smithi045 ceph-mon[32174]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:17.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:17 smithi116 ceph-mon[35486]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' 2024-03-12T05:03:18.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-12T05:03:18.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:03:18.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:03:18.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:18.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:18.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:03:18.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:17 smithi130 ceph-mon[35398]: from='osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477]' entity='osd.0' 2024-03-12T05:03:18.271 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:03:18.922 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] boot 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:18.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:19.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:03:19.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] boot 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:19.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: osd.0 [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] boot 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:03:19.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- 172.21.15.116:0/2429290483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 msgr2=0x7f96e4100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/2429290483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4100fc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96d40099c0 tx=0x7f96d4009cd0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- 172.21.15.116:0/2429290483 shutdown_connections 2024-03-12T05:03:19.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/2429290483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4102740 0x7f96e4108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/2429290483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4100fc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/2429290483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4101da0 0x7f96e4102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- 172.21.15.116:0/2429290483 >> 172.21.15.116:0/2429290483 conn(0x7f96e40fc160 msgr2=0x7f96e40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- 172.21.15.116:0/2429290483 shutdown_connections 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- 172.21.15.116:0/2429290483 wait complete. 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 Processor -- start 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.694+0000 7f96eb0ce700 1 -- start start 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4101da0 0x7f96e4103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:19.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 0x7f96e41036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4103e40 con 0x7f96e4102740 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4103fb0 con 0x7f96e4100ba0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41a1be0 con 0x7f96e4101da0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60874/0 (socket says 172.21.15.116:60874) 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 -- 172.21.15.116:0/1768035393 learned_addr learned my addr 172.21.15.116:0/1768035393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e3fff700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4101da0 0x7f96e4103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e966b700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 0x7f96e41036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 -- 172.21.15.116:0/1768035393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4101da0 msgr2=0x7f96e4103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4101da0 0x7f96e4103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 -- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 msgr2=0x7f96e41036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 0x7f96e41036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 -- 172.21.15.116:0/1768035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d40095f0 con 0x7f96e4100ba0 2024-03-12T05:03:19.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e966b700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 0x7f96e41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e8e6a700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d4005af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d401d070 con 0x7f96e4100ba0 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96d4004110 con 0x7f96e4100ba0 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e41a1e60 con 0x7f96e4100ba0 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.695+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4021410 con 0x7f96e4100ba0 2024-03-12T05:03:19.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.696+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e41a2210 con 0x7f96e4100ba0 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.696+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e4068960 con 0x7f96e4100ba0 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.698+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f96d400f710 con 0x7f96e4100ba0 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.698+0000 7f96e1ffb700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 0x7f96cc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.698+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f96d40ccbe0 con 0x7f96e4100ba0 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.699+0000 7f96e3fff700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 0x7f96cc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:19.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.699+0000 7f96e3fff700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 0x7f96cc0a3f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96d8005ea0 tx=0x7f96d8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:03:19.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.702+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96d4097a10 con 0x7f96e4100ba0 2024-03-12T05:03:19.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:19.904+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f96e40fe2e0 con 0x7f96cc0a1ab0 2024-03-12T05:03:20.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi045 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 17093M 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:20.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:19 smithi045 ceph-mon[32174]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi045 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 17093M 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:20.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:19 smithi130 ceph-mon[35398]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi045 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 17093M 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:19 smithi116 ceph-mon[35486]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:03:21.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:20 smithi116 ceph-mon[35486]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:21.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:21.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:21.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:21.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:20 smithi045 ceph-mon[32174]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:21.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:21.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:21.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:21.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:20 smithi130 ceph-mon[35398]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:21.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:21.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:21.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:21.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:21 smithi116 ceph-mon[35486]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:22.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:21 smithi045 ceph-mon[32174]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:22.123 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:21 smithi130 ceph-mon[35398]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:23.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:22 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/2617057716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:22 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:22 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]': finished 2024-03-12T05:03:23.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:22 smithi045 ceph-mon[32174]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:03:23.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:22 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:23.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:22 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/2617057716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:22 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:22 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]': finished 2024-03-12T05:03:23.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:22 smithi130 ceph-mon[35398]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:03:23.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:22 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:23.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:22 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/2617057716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:22 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]: dispatch 2024-03-12T05:03:23.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:22 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50765f29-2dd5-499e-aeb8-529a3344ab49"}]': finished 2024-03-12T05:03:23.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:22 smithi116 ceph-mon[35486]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:03:23.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:22 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:24.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:23 smithi045 ceph-mon[32174]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:24.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:23 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/2976232399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:24.148 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:23 smithi116 ceph-mon[35486]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:24.148 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:23 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/2976232399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:24.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:23 smithi130 ceph-mon[35398]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:24.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:23 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/2976232399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:25.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:25 smithi130 ceph-mon[35398]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:25.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:25 smithi116 ceph-mon[35486]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:25.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:25 smithi045 ceph-mon[32174]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:27.399 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:27 smithi116 ceph-mon[35486]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:27.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:27 smithi130 ceph-mon[35398]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:27.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:27 smithi045 ceph-mon[32174]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:29.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:29 smithi130 ceph-mon[35398]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:29.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:29 smithi116 ceph-mon[35486]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:29.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:29 smithi045 ceph-mon[32174]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:30.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:03:30.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:30.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:30 smithi116 ceph-mon[35486]: Deploying daemon osd.1 on smithi116 2024-03-12T05:03:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:03:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:30 smithi130 ceph-mon[35398]: Deploying daemon osd.1 on smithi116 2024-03-12T05:03:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:03:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:30.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:30 smithi045 ceph-mon[32174]: Deploying daemon osd.1 on smithi116 2024-03-12T05:03:31.313 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:31 smithi116 ceph-mon[35486]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:31.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:31 smithi130 ceph-mon[35398]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:31.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:31 smithi045 ceph-mon[32174]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:33.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:33 smithi116 ceph-mon[35486]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:33.375 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:33 smithi130 ceph-mon[35398]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:33.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:33 smithi045 ceph-mon[32174]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:35.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:35 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:35 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:35 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:35.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:35 smithi116 ceph-mon[35486]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:35.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:35 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:35 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:35 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:35.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:35 smithi045 ceph-mon[32174]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:35.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:35 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:35 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:35.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:35 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:35.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:35 smithi130 ceph-mon[35398]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:37.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:37 smithi116 ceph-mon[35486]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:37.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:37 smithi130 ceph-mon[35398]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:37.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:37 smithi045 ceph-mon[32174]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:38.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.066+0000 7f96e1ffb700 1 -- 172.21.15.116:0/1768035393 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f96e40fe2e0 con 0x7f96cc0a1ab0 2024-03-12T05:03:38.067 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 msgr2=0x7f96cc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 0x7f96cc0a3f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f96d8005ea0 tx=0x7f96d8005e30 comp rx=0 tx=0).stop 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 msgr2=0x7f96e4104b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d4005af0 comp rx=0 tx=0).stop 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 shutdown_connections 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f96cc0a1ab0 0x7f96cc0a3f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96e4102740 0x7f96e41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:38.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96e4100ba0 0x7f96e4104b30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 --2- 172.21.15.116:0/1768035393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96e4101da0 0x7f96e4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.067+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 >> 172.21.15.116:0/1768035393 conn(0x7f96e40fc160 msgr2=0x7f96e40fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:03:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.068+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 shutdown_connections 2024-03-12T05:03:38.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:03:38.068+0000 7f96eb0ce700 1 -- 172.21.15.116:0/1768035393 wait complete. 2024-03-12T05:03:38.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:38.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:38.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:38.677 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T05:03:38.680 INFO:tasks.cephadm:Deploying osd.2 on smithi130 with /dev/vg_nvme/lv_4... 2024-03-12T05:03:38.680 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:03:38.925 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:03:39.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:39 smithi116 ceph-mon[35486]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:39 smithi116 ceph-mon[35486]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:39 smithi116 ceph-mon[35486]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:39.374 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:39 smithi130 ceph-mon[35398]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.375 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:39 smithi130 ceph-mon[35398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.375 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:39 smithi130 ceph-mon[35398]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:39.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:39 smithi045 ceph-mon[32174]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:39 smithi045 ceph-mon[32174]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:03:39.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:39 smithi045 ceph-mon[32174]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:40.324 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:40 smithi130 ceph-mon[35398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:03:40.324 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:40 smithi130 ceph-mon[35398]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.324 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:40 smithi130 ceph-mon[35398]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:03:40.324 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:40.324 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:40 smithi130 ceph-mon[35398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.401 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-mon[35486]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:03:40.402 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-mon[35486]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.402 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-mon[35486]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:03:40.402 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:40.402 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-mon[35486]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.402 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1[40068]: 2024-03-12T05:03:40.143+0000 7fad407a8700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T05:03:40.402 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:03:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1[40068]: 2024-03-12T05:03:40.146+0000 7fad3c59b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:03:40.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:40 smithi045 ceph-mon[32174]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:03:40.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:40 smithi045 ceph-mon[32174]: from='osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:40 smithi045 ceph-mon[32174]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:03:40.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:40.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:40 smithi045 ceph-mon[32174]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:03:40.873 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:03:40.873 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:03:40.873 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-12T05:03:40.874 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-12T05:03:40.874 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146967 s, 713 MB/s 2024-03-12T05:03:40.874 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-12T05:03:41.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:41 smithi116 ceph-mon[35486]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:03:41.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:41 smithi116 ceph-mon[35486]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:03:41.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:41 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:41 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:41 smithi116 ceph-mon[35486]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:41.441 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:41 smithi130 ceph-mon[35398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:03:41.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:41 smithi130 ceph-mon[35398]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:03:41.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:41 smithi130 ceph-mon[35398]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:41.470 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2024-03-12T05:03:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:41 smithi045 ceph-mon[32174]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:03:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:41 smithi045 ceph-mon[32174]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:03:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:41.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:41 smithi045 ceph-mon[32174]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:03:41.718 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] boot 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi116 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-12T05:03:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] boot 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi116 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-12T05:03:42.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.499 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:42.499 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:42.499 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:03:42.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:03:42.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: osd.1 [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] boot 2024-03-12T05:03:42.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi116 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:03:42.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 -- 172.21.15.130:0/4212406604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 msgr2=0x7f9dbc10a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 --2- 172.21.15.130:0/4212406604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc10a0f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9db400b0a0 tx=0x7f9db400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 -- 172.21.15.130:0/4212406604 shutdown_connections 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 --2- 172.21.15.130:0/4212406604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc10a0f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 --2- 172.21.15.130:0/4212406604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 0x7f9dbc1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 --2- 172.21.15.130:0/4212406604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 0x7f9dbc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 -- 172.21.15.130:0/4212406604 >> 172.21.15.130:0/4212406604 conn(0x7f9dbc0fd280 msgr2=0x7f9dbc0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.246+0000 7f9dc1956700 1 -- 172.21.15.130:0/4212406604 shutdown_connections 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.247+0000 7f9dc1956700 1 -- 172.21.15.130:0/4212406604 wait complete. 2024-03-12T05:03:43.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.247+0000 7f9dc1956700 1 Processor -- start 2024-03-12T05:03:43.250 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.247+0000 7f9dc1956700 1 -- start start 2024-03-12T05:03:43.250 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.247+0000 7f9dc1956700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 0x7f9dbc197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:43.250 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.247+0000 7f9dc1956700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 0x7f9dbc198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc19f850 con 0x7f9dbc102ea0 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc19f990 con 0x7f9dbc101ca0 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc19fb50 con 0x7f9dbc103840 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43358/0 (socket says 172.21.15.130:43358) 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 -- 172.21.15.130:0/3308170288 learned_addr learned my addr 172.21.15.130:0/3308170288 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dba7fc700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 0x7f9dbc198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbaffd700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 0x7f9dbc197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 -- 172.21.15.130:0/3308170288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 msgr2=0x7f9dbc197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 0x7f9dbc197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 -- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 msgr2=0x7f9dbc198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:03:43.251 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 0x7f9dbc198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 -- 172.21.15.130:0/3308170288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db4009d00 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dbb7fe700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9db400b830 tx=0x7f9db4004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db400e040 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9db401db30 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc19fdd0 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db401bc90 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.248+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc1a0250 con 0x7f9dbc103840 2024-03-12T05:03:43.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.249+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc109060 con 0x7f9dbc103840 2024-03-12T05:03:43.253 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.250+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f9db401dca0 con 0x7f9dbc103840 2024-03-12T05:03:43.253 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.251+0000 7f9dc0954700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 0x7f9da40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:03:43.253 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.251+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f9db40cdd50 con 0x7f9dbc103840 2024-03-12T05:03:43.253 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.252+0000 7f9dbaffd700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 0x7f9da40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:03:43.253 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.252+0000 7f9dbaffd700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 0x7f9da40a3f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9dac006fd0 tx=0x7f9dac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:03:43.256 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.255+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9db4098840 con 0x7f9dbc103840 2024-03-12T05:03:43.454 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:03:43.453+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dbc02da20 con 0x7f9da40a1ab0 2024-03-12T05:03:43.454 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:43 smithi130 ceph-mon[35398]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:03:43.455 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:43 smithi130 ceph-mon[35398]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:43.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:43 smithi116 ceph-mon[35486]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:03:43.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:43 smithi116 ceph-mon[35486]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:43.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:43 smithi045 ceph-mon[32174]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:03:43.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:43 smithi045 ceph-mon[32174]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:44.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:44 smithi130 ceph-mon[35398]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:44.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:44.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:44.362 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:44.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:44 smithi116 ceph-mon[35486]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:44.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:44.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:44.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:44.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:44 smithi045 ceph-mon[32174]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:03:44.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:03:44.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:03:44.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:45.327 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:45 smithi130 ceph-mon[35398]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:45.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:45 smithi116 ceph-mon[35486]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:45.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:45 smithi045 ceph-mon[32174]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:46.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/2030326079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:46 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:46 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]': finished 2024-03-12T05:03:46.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:46 smithi130 ceph-mon[35398]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:03:46.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:03:46.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/2030326079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:46 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:46 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]': finished 2024-03-12T05:03:46.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:46 smithi116 ceph-mon[35486]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:03:46.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:03:46.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/2030326079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:46 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]: dispatch 2024-03-12T05:03:46.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:46 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d3b4de1-376d-4209-bef9-a48e51f77a93"}]': finished 2024-03-12T05:03:46.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:46 smithi045 ceph-mon[32174]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:03:46.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:03:47.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:47 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/224114161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:47.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:47 smithi130 ceph-mon[35398]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:47.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:47 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/224114161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:47.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:47 smithi116 ceph-mon[35486]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:47.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:47 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/224114161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:03:47.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:47 smithi045 ceph-mon[32174]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:49.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:49 smithi130 ceph-mon[35398]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:49.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:49 smithi116 ceph-mon[35486]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:49.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:49 smithi045 ceph-mon[32174]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:51.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:51 smithi130 ceph-mon[35398]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:51.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:51 smithi116 ceph-mon[35486]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:51.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:51 smithi045 ceph-mon[32174]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:53.276 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:53 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:03:53.276 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:53 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:53.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:53 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:03:53.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:53 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:53 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:03:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:53 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:03:54.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:54 smithi130 ceph-mon[35398]: Deploying daemon osd.2 on smithi130 2024-03-12T05:03:54.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:54 smithi130 ceph-mon[35398]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:54 smithi045 ceph-mon[32174]: Deploying daemon osd.2 on smithi130 2024-03-12T05:03:54.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:54 smithi045 ceph-mon[32174]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:54.399 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:54 smithi116 ceph-mon[35486]: Deploying daemon osd.2 on smithi130 2024-03-12T05:03:54.399 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:54 smithi116 ceph-mon[35486]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:56.182 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:56 smithi130 ceph-mon[35398]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:56.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:56 smithi045 ceph-mon[32174]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:56.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:56 smithi116 ceph-mon[35486]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:58.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:58 smithi045 ceph-mon[32174]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:58.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:58 smithi130 ceph-mon[35398]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:58.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:58 smithi116 ceph-mon[35486]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:59.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:59.283 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:03:59 smithi130 ceph-mon[35398]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:59.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:59.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:03:59 smithi045 ceph-mon[32174]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:03:59.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:03:59.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:03:59.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:03:59 smithi116 ceph-mon[35486]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:01.469 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.468+0000 7f9dc0954700 1 -- 172.21.15.130:0/3308170288 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9dbc02da20 con 0x7f9da40a1ab0 2024-03-12T05:04:01.469 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 2 on host 'smithi130' 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 msgr2=0x7f9da40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 0x7f9da40a3f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9dac006fd0 tx=0x7f9dac008040 comp rx=0 tx=0).stop 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 msgr2=0x7f9dbc19f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9db400b830 tx=0x7f9db4004a50 comp rx=0 tx=0).stop 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 shutdown_connections 2024-03-12T05:04:01.472 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f9da40a1ab0 0x7f9da40a3f70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9dbc103840 0x7f9dbc19f280 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dbc101ca0 0x7f9dbc197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 --2- 172.21.15.130:0/3308170288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dbc102ea0 0x7f9dbc198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 >> 172.21.15.130:0/3308170288 conn(0x7f9dbc0fd280 msgr2=0x7f9dbc1061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 shutdown_connections 2024-03-12T05:04:01.473 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:01.471+0000 7f9dc1956700 1 -- 172.21.15.130:0/3308170288 wait complete. 2024-03-12T05:04:01.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:01.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:01.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.062 DEBUG:teuthology.orchestra.run.smithi130:osd.2> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T05:04:02.064 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_3... 2024-03-12T05:04:02.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:02.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:02.318 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:04:02.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:02 smithi045 ceph-mon[32174]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:02.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:02 smithi045 ceph-mon[32174]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:02.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:02 smithi130 ceph-mon[35398]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:02.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:02 smithi130 ceph-mon[35398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:02.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:02 smithi116 ceph-mon[35486]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:02.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:02 smithi116 ceph-mon[35486]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:03 smithi045 ceph-mon[32174]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:03.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:04:03.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:04:03.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:03.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:03 smithi130 ceph-mon[35398]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:03.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:04:03.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:04:03.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:03.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: from='osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:04:03.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:03 smithi116 ceph-mon[35486]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:04.398 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:04:04.399 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:04:04.399 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-12T05:04:04.399 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-12T05:04:04.399 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125326 s, 837 MB/s 2024-03-12T05:04:04.399 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-12T05:04:04.472 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2[40565]: 2024-03-12T05:04:04.204+0000 7fcdd6d1a700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T05:04:04.473 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2[40565]: 2024-03-12T05:04:04.209+0000 7fcdceb05700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:04:04.816 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-mon[35398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:04:04.816 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-mon[35398]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:04:04.816 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.816 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.816 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:04 smithi130 ceph-mon[35398]: from='osd.2 ' entity='osd.2' 2024-03-12T05:04:04.976 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:04 smithi045 ceph-mon[32174]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:04:04.976 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:04 smithi045 ceph-mon[32174]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:04:04.976 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:04 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.976 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:04 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.976 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:04 smithi045 ceph-mon[32174]: from='osd.2 ' entity='osd.2' 2024-03-12T05:04:04.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:04 smithi116 ceph-mon[35486]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:04:04.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:04 smithi116 ceph-mon[35486]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:04:04.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:04 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:04 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:04.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:04 smithi116 ceph-mon[35486]: from='osd.2 ' entity='osd.2' 2024-03-12T05:04:05.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-03-12T05:04:05.294 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:04:05.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:04:05.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:04:05.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:05.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:05.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] boot 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:05.974 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:04:05.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:04:05.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:05.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] boot 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:05.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: osd.2 [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] boot 2024-03-12T05:04:06.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:04:06.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:06.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:04:06.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:06.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 -- 172.21.15.45:0/4056112128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c105060 msgr2=0x7f7d9c107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/4056112128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c105060 0x7f7d9c107450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009990 tx=0x7f7d90009ca0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 -- 172.21.15.45:0/4056112128 shutdown_connections 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/4056112128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c107990 0x7f7d9c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/4056112128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 0x7f7d9c104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/4056112128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c105060 0x7f7d9c107450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 -- 172.21.15.45:0/4056112128 >> 172.21.15.45:0/4056112128 conn(0x7f7d9c0fc140 msgr2=0x7f7d9c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 -- 172.21.15.45:0/4056112128 shutdown_connections 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.874+0000 7f7da1dcf700 1 -- 172.21.15.45:0/4056112128 wait complete. 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 Processor -- start 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 -- start start 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 0x7f7d9c197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c105060 0x7f7d9c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19f830 con 0x7f7d9c107990 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19f970 con 0x7f7d9c105060 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7da1dcf700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19fb30 con 0x7f7d9c102730 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41088/0 (socket says 172.21.15.45:41088) 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9bfff700 1 -- 172.21.15.45:0/2822224230 learned_addr learned my addr 172.21.15.45:0/2822224230 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9affd700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c105060 0x7f7d9c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9b7fe700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 0x7f7d9c197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9bfff700 1 -- 172.21.15.45:0/2822224230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 msgr2=0x7f7d9c197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.875+0000 7f7d9bfff700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 0x7f7d9c197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d9bfff700 1 -- 172.21.15.45:0/2822224230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c105060 msgr2=0x7f7d9c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d9bfff700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c105060 0x7f7d9c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d9bfff700 1 -- 172.21.15.45:0/2822224230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d900095f0 con 0x7f7d9c107990 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d9bfff700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400b9e0 tx=0x7f7d9400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9400c930 con 0x7f7d9c107990 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c19fe10 con 0x7f7d9c107990 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d94014440 con 0x7f7d9c107990 2024-03-12T05:04:06.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d940125a0 con 0x7f7d9c107990 2024-03-12T05:04:06.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.876+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c1a02b0 con 0x7f7d9c107990 2024-03-12T05:04:06.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.877+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c0fdcc0 con 0x7f7d9c107990 2024-03-12T05:04:06.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.878+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f7d94014960 con 0x7f7d9c107990 2024-03-12T05:04:06.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.878+0000 7f7d98ff9700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 0x7f7d840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:06.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.879+0000 7f7d9b7fe700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 0x7f7d840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:06.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.879+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f7d94006e40 con 0x7f7d9c107990 2024-03-12T05:04:06.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.879+0000 7f7d9b7fe700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 0x7f7d840a3ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006fd0 tx=0x7f7d8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:06.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:06.882+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d94095d80 con 0x7f7d9c107990 2024-03-12T05:04:06.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:06 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi130 2024-03-12T05:04:06.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:06 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-12T05:04:06.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:06 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi130 2024-03-12T05:04:06.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:06 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-12T05:04:07.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:07.093+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d9c02d8e0 con 0x7f7d840a1a30 2024-03-12T05:04:07.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:06 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi130 2024-03-12T05:04:07.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:06 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 17088M 2024-03-12T05:04:07.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:04:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:07.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:07 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:07.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:07.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:04:07.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:07.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:07.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:07.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:07.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:07 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:08 smithi045 ceph-mon[32174]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:08 smithi045 ceph-mon[32174]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:04:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:08.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:08 smithi130 ceph-mon[35398]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:08.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:08.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:08 smithi130 ceph-mon[35398]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:04:08.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:08.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:08 smithi116 ceph-mon[35486]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:08.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:08.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:08 smithi116 ceph-mon[35486]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:04:08.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:04:09.325 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52134]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:09.325 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52134]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:09.325 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52134]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:09.325 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52134]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:09.621 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:09.621 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:09.621 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:09.621 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:04:09 smithi045 sudo[52160]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:09.853 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:09.853 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:09.853 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44029]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:09.853 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44029]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:09.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]: dispatch 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]': finished 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:04:09.854 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:09 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:09.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:09.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]: dispatch 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]': finished 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:04:09.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:09 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]: dispatch 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1846640882' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "522e40da-a98f-4f92-bc68-60f665e2f962"}]': finished 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:04:09.997 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:09 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:10.247 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:10.248 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:10.248 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44032]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:10.248 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:04:09 smithi116 sudo[44032]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:10.373 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44461]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:10.373 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44461]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:10.374 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44461]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:10.374 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44461]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:10.722 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44464]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:10.722 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44464]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:10.722 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44464]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:10.723 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:04:10 smithi130 sudo[44464]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:10.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52266]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:10.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:10.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52266]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:10.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52266]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:10.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:10.936+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7d94096150 con 0x7f7d9c107990 2024-03-12T05:04:11.185 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:10 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4213138701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:10 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44467]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:11.186 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44467]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:11.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:10 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4213138701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:11.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:10 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52452]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52452]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52452]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 sudo[52452]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4213138701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:11.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:10 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:11.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:11.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:11.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:11.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 sudo[44470]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44035]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:04:11.917 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 sudo[44038]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.224 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:11 smithi130 ceph-mon[35398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:11 smithi116 ceph-mon[35486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:04:12.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:11 smithi045 ceph-mon[32174]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:04:13.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:13 smithi130 ceph-mon[35398]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:13.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:13 smithi116 ceph-mon[35486]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:13.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:13 smithi045 ceph-mon[32174]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:15.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:15 smithi045 ceph-mon[32174]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:15.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:15 smithi130 ceph-mon[35398]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:15.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:15 smithi116 ceph-mon[35486]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:17.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:04:17.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:04:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:17.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:04:17.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:18.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:18 smithi045 ceph-mon[32174]: Deploying daemon osd.3 on smithi045 2024-03-12T05:04:18.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:18 smithi045 ceph-mon[32174]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:18.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:18 smithi130 ceph-mon[35398]: Deploying daemon osd.3 on smithi045 2024-03-12T05:04:18.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:18 smithi130 ceph-mon[35398]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:18.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:18 smithi116 ceph-mon[35486]: Deploying daemon osd.3 on smithi045 2024-03-12T05:04:18.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:18 smithi116 ceph-mon[35486]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:19.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:19 smithi045 ceph-mon[32174]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:19.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:19 smithi130 ceph-mon[35398]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:19.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:19 smithi116 ceph-mon[35486]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:21.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:21 smithi045 ceph-mon[32174]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:21.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:21 smithi130 ceph-mon[35398]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:21.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:21 smithi116 ceph-mon[35486]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:23.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:23.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:23 smithi045 ceph-mon[32174]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:23.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:23.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:23 smithi130 ceph-mon[35398]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:23.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:23.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:23.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:23 smithi116 ceph-mon[35486]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:25.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:25.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:25.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:25 smithi045 ceph-mon[32174]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:25.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:25.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:25.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:25 smithi130 ceph-mon[35398]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:25.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:25.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:25.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:25.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:25 smithi116 ceph-mon[35486]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:25.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:25.999+0000 7f7d98ff9700 1 -- 172.21.15.45:0/2822224230 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7d9c02d8e0 con 0x7f7d840a1a30 2024-03-12T05:04:26.000 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 msgr2=0x7f7d840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 0x7f7d840a3ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006fd0 tx=0x7f7d8c008040 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 msgr2=0x7f7d9c19f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400b9e0 tx=0x7f7d9400bcf0 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 shutdown_connections 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7d840a1a30 0x7f7d840a3ef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d9c107990 0x7f7d9c19f260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d9c102730 0x7f7d9c197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 --2- 172.21.15.45:0/2822224230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d9c105060 0x7f7d9c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 >> 172.21.15.45:0/2822224230 conn(0x7f7d9c0fc140 msgr2=0x7f7d9c103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 shutdown_connections 2024-03-12T05:04:26.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:04:26.001+0000 7f7da1dcf700 1 -- 172.21.15.45:0/2822224230 wait complete. 2024-03-12T05:04:26.665 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T05:04:26.667 INFO:tasks.cephadm:Deploying osd.4 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-12T05:04:26.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:04:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:04:27.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:26 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:26 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:26 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:27.218 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:26 smithi045 ceph-mon[32174]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:04:27.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:26 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:26 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:26 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:27.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:26 smithi116 ceph-mon[35486]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:04:27.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:26 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:26 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:27.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:26 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:27.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:26 smithi130 ceph-mon[35398]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:04:28.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-mon[32174]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:04:28.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-mon[32174]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:04:28.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-mon[32174]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:04:28.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:28.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-mon[32174]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:28 smithi116 ceph-mon[35486]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:04:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:28 smithi116 ceph-mon[35486]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:04:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:28 smithi116 ceph-mon[35486]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:04:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:28 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:28 smithi116 ceph-mon[35486]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:28.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:28 smithi130 ceph-mon[35398]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:04:28.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:28 smithi130 ceph-mon[35398]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:04:28.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:28 smithi130 ceph-mon[35398]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:04:28.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:28 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:28.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:28 smithi130 ceph-mon[35398]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112667 s, 931 MB/s 2024-03-12T05:04:28.833 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-12T05:04:28.871 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3[54225]: 2024-03-12T05:04:28.469+0000 7fb45de49700 -1 osd.3 0 waiting for initial osdmap 2024-03-12T05:04:28.871 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:04:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3[54225]: 2024-03-12T05:04:28.476+0000 7fb455433700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:04:29.121 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:29 smithi116 ceph-mon[35486]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:04:29.122 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:29 smithi116 ceph-mon[35486]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:04:29.122 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:29 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.122 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:29 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.122 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:29 smithi116 ceph-mon[35486]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' 2024-03-12T05:04:29.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:29 smithi045 ceph-mon[32174]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:04:29.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:29 smithi045 ceph-mon[32174]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:04:29.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:29 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:29 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:29 smithi045 ceph-mon[32174]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' 2024-03-12T05:04:29.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-12T05:04:29.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:29 smithi130 ceph-mon[35398]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:04:29.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:29 smithi130 ceph-mon[35398]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:04:29.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:29 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:29 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:29.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:29 smithi130 ceph-mon[35398]: from='osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218]' entity='osd.3' 2024-03-12T05:04:29.630 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:04:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:04:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:04:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] boot 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:30.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] boot 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:30.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:04:30.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:04:30.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: osd.3 [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] boot 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:30.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:30.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 -- 172.21.15.116:0/525989901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c102720 msgr2=0x7f858c104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 --2- 172.21.15.116:0/525989901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c102720 0x7f858c104b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f857c009990 tx=0x7f857c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 -- 172.21.15.116:0/525989901 shutdown_connections 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 --2- 172.21.15.116:0/525989901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c107980 0x7f858c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 --2- 172.21.15.116:0/525989901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c102720 0x7f858c104b10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 --2- 172.21.15.116:0/525989901 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 0x7f858c107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 -- 172.21.15.116:0/525989901 >> 172.21.15.116:0/525989901 conn(0x7f858c0fc130 msgr2=0x7f858c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 -- 172.21.15.116:0/525989901 shutdown_connections 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.117+0000 7f8591706700 1 -- 172.21.15.116:0/525989901 wait complete. 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 Processor -- start 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 -- start start 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c102720 0x7f858c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 0x7f858c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19f6e0 con 0x7f858c105050 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19f820 con 0x7f858c107980 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f8591706700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c19f9b0 con 0x7f858c102720 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f858b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f858b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36346/0 (socket says 172.21.15.116:36346) 2024-03-12T05:04:31.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f858b7fe700 1 -- 172.21.15.116:0/177826036 learned_addr learned my addr 172.21.15.116:0/177826036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f858a7fc700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 0x7f858c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.118+0000 7f858affd700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c102720 0x7f858c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 -- 172.21.15.116:0/177826036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c102720 msgr2=0x7f858c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c102720 0x7f858c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 -- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 msgr2=0x7f858c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 0x7f858c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 -- 172.21.15.116:0/177826036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c0095f0 con 0x7f858c107980 2024-03-12T05:04:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f858b7fe700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f85840094f0 tx=0x7f858400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858400ddd0 con 0x7f858c107980 2024-03-12T05:04:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f858400f040 con 0x7f858c107980 2024-03-12T05:04:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c19fc90 con 0x7f858c107980 2024-03-12T05:04:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584005a90 con 0x7f858c107980 2024-03-12T05:04:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.119+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c1a0190 con 0x7f858c107980 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.120+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c0fdcb0 con 0x7f858c107980 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.121+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8584005020 con 0x7f858c107980 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.121+0000 7f8583fff700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 0x7f85780a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.122+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f85840cba60 con 0x7f858c107980 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.122+0000 7f858affd700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 0x7f85780a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.122+0000 7f858affd700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 0x7f85780a4000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f857c00b5c0 tx=0x7f857c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.126+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8584095e70 con 0x7f858c107980 2024-03-12T05:04:31.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:31 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi045 2024-03-12T05:04:31.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:31 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 8546M 2024-03-12T05:04:31.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:31 smithi116 ceph-mon[35486]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:04:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:31.331+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f858c02d870 con 0x7f85780a1b40 2024-03-12T05:04:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:31 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi045 2024-03-12T05:04:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:31 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 8546M 2024-03-12T05:04:31.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:31 smithi045 ceph-mon[32174]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:04:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:31 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi045 2024-03-12T05:04:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:31 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 8546M 2024-03-12T05:04:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:31 smithi130 ceph-mon[35398]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:04:32.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:32 smithi116 ceph-mon[35486]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:32.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:32 smithi116 ceph-mon[35486]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:04:32.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:32 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:32.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:32 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:32.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:32 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:32 smithi045 ceph-mon[32174]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:32 smithi045 ceph-mon[32174]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:04:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:32.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:32 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:32.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:32 smithi130 ceph-mon[35398]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:32.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:32 smithi130 ceph-mon[35398]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:04:32.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:32.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:32.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:32 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:33.171 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:33 smithi116 ceph-mon[35486]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:33.171 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:33 smithi116 ceph-mon[35486]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:04:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:33 smithi045 ceph-mon[32174]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:33.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:33 smithi045 ceph-mon[32174]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:04:33.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:33 smithi130 ceph-mon[35398]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:33.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:33 smithi130 ceph-mon[35398]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/2046459142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]': finished 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:04:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/2046459142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]': finished 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:04:34.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:34.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:04:34.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/2046459142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]: dispatch 2024-03-12T05:04:34.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c172804f-1d0d-4ab9-9dc9-a67b20f80b52"}]': finished 2024-03-12T05:04:34.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:04:34.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:35.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/212296612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:35.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:35 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/212296612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:35.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:35 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/212296612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:36.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:36 smithi045 ceph-mon[32174]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-12T05:04:36.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:36 smithi130 ceph-mon[35398]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-12T05:04:36.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:36 smithi116 ceph-mon[35486]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-12T05:04:37.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:37 smithi130 ceph-mon[35398]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:04:37.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:37 smithi116 ceph-mon[35486]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:04:37.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:37 smithi045 ceph-mon[32174]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:04:39.441 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:39 smithi116 ceph-mon[35486]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-12T05:04:39.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:39 smithi130 ceph-mon[35398]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-12T05:04:39.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:39 smithi045 ceph-mon[32174]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-12T05:04:41.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:41 smithi116 ceph-mon[35486]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:41.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:41 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:04:41.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:41 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:41.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:41 smithi130 ceph-mon[35398]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:41.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:04:41.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:41 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:41 smithi045 ceph-mon[32174]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:04:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:41 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:42.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:42 smithi116 ceph-mon[35486]: Deploying daemon osd.4 on smithi116 2024-03-12T05:04:42.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:42 smithi130 ceph-mon[35398]: Deploying daemon osd.4 on smithi116 2024-03-12T05:04:42.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:42 smithi045 ceph-mon[32174]: Deploying daemon osd.4 on smithi116 2024-03-12T05:04:43.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:43 smithi130 ceph-mon[35398]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:43.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:43 smithi116 ceph-mon[35486]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:43 smithi045 ceph-mon[32174]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:45.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:45 smithi116 ceph-mon[35486]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:45.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:45 smithi130 ceph-mon[35398]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:45.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:45 smithi045 ceph-mon[32174]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:47.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:47 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:47 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:47.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:47 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:47 smithi116 ceph-mon[35486]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:47.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:47 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:47 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:47.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:47 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:47 smithi045 ceph-mon[32174]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:47.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:47 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:47 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:47.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:47 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:47.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:47 smithi130 ceph-mon[35398]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:49.319 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:49 smithi116 ceph-mon[35486]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:49.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:49 smithi130 ceph-mon[35398]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:49.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:49 smithi045 ceph-mon[32174]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:50.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.348+0000 7f8583fff700 1 -- 172.21.15.116:0/177826036 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f858c02d870 con 0x7f85780a1b40 2024-03-12T05:04:50.348 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 4 on host 'smithi116' 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 msgr2=0x7f85780a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 0x7f85780a4000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f857c00b5c0 tx=0x7f857c005fb0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 msgr2=0x7f858c19f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f85840094f0 tx=0x7f858400d3b0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 shutdown_connections 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f85780a1b40 0x7f85780a4000 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f858c107980 0x7f858c19f110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f858c102720 0x7f858c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 --2- 172.21.15.116:0/177826036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f858c105050 0x7f858c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 >> 172.21.15.116:0/177826036 conn(0x7f858c0fc130 msgr2=0x7f858c103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:50.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 shutdown_connections 2024-03-12T05:04:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:04:50.350+0000 7f8591706700 1 -- 172.21.15.116:0/177826036 wait complete. 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:50.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:04:50.942 DEBUG:teuthology.orchestra.run.smithi116:osd.4> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T05:04:50.945 INFO:tasks.cephadm:Deploying osd.5 on smithi130 with /dev/vg_nvme/lv_3... 2024-03-12T05:04:50.945 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:04:51.192 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:04:51.643 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:51 smithi130 ceph-mon[35398]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.644 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:51 smithi130 ceph-mon[35398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.644 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:51 smithi130 ceph-mon[35398]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:51.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:51 smithi116 ceph-mon[35486]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:51 smithi116 ceph-mon[35486]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:51 smithi116 ceph-mon[35486]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:51.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:51 smithi045 ceph-mon[32174]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:51 smithi045 ceph-mon[32174]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:04:51.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:51 smithi045 ceph-mon[32174]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:52.514 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:52 smithi130 ceph-mon[35398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:04:52.514 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:52 smithi130 ceph-mon[35398]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:04:52.514 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:52 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:52.514 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:52 smithi130 ceph-mon[35398]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:52.514 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:52 smithi130 ceph-mon[35398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:52.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-mon[35486]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:04:52.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-mon[35486]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:04:52.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:52.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-mon[35486]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:52.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-mon[35486]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:52.748 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:04:52.404+0000 7f18258a0700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T05:04:52.748 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:04:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:04:52.410+0000 7f1821693700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:04:52.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:52 smithi045 ceph-mon[32174]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:04:52.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:52 smithi045 ceph-mon[32174]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:04:52.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:52 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:52.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:52 smithi045 ceph-mon[32174]: from='osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:52.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:52 smithi045 ceph-mon[32174]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:04:53.068 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:04:53.069 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:04:53.069 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-12T05:04:53.069 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-12T05:04:53.069 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123867 s, 847 MB/s 2024-03-12T05:04:53.069 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-12T05:04:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:53 smithi116 ceph-mon[35486]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:04:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:53 smithi116 ceph-mon[35486]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:04:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:53 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:53 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:53 smithi116 ceph-mon[35486]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:53.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:53 smithi130 ceph-mon[35398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:04:53.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:53 smithi130 ceph-mon[35398]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:04:53.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:53 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:53 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:53 smithi130 ceph-mon[35398]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:53.830 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2024-03-12T05:04:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:53 smithi045 ceph-mon[32174]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:04:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:53 smithi045 ceph-mon[32174]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:04:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:53 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:53 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:53 smithi045 ceph-mon[32174]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 127 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:04:54.071 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:04:54.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] boot 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi116 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:54.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:54 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] boot 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi116 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:54.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:54.748 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:54 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: osd.4 [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] boot 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi116 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:54.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:54.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:04:54.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:54 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 -- 172.21.15.130:0/721036257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 msgr2=0x7f6c98109060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 --2- 172.21.15.130:0/721036257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c98109060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b0a0 tx=0x7f6c9000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 -- 172.21.15.130:0/721036257 shutdown_connections 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 --2- 172.21.15.130:0/721036257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c98109060 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 --2- 172.21.15.130:0/721036257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c98100c10 0x7f6c98101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 --2- 172.21.15.130:0/721036257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 -- 172.21.15.130:0/721036257 >> 172.21.15.130:0/721036257 conn(0x7f6c980fc1d0 msgr2=0x7f6c980fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 -- 172.21.15.130:0/721036257 shutdown_connections 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.581+0000 7f6c9f722700 1 -- 172.21.15.130:0/721036257 wait complete. 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 Processor -- start 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 -- start start 2024-03-12T05:04:55.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c98100c10 0x7f6c98197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9819f8d0 con 0x7f6c98101e10 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9819fa10 con 0x7f6c98100c10 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9f722700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9819fba0 con 0x7f6c981027b0 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:35330/0 (socket says 172.21.15.130:35330) 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 -- 172.21.15.130:0/3760574261 learned_addr learned my addr 172.21.15.130:0/3760574261 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 -- 172.21.15.130:0/3760574261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c98100c10 msgr2=0x7f6c98197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9ccbd700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c98100c10 0x7f6c98197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 -- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 msgr2=0x7f6c98198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:04:55.585 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.582+0000 7f6c9dcbf700 1 -- 172.21.15.130:0/3760574261 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009d00 con 0x7f6c981027b0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c9ccbd700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c9dcbf700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b070 tx=0x7f6c90004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9000e040 con 0x7f6c981027b0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6c90005700 con 0x7f6c981027b0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9819fe20 con 0x7f6c981027b0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9001db90 con 0x7f6c981027b0 2024-03-12T05:04:55.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.583+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c981a02b0 con 0x7f6c981027b0 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.584+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c98108050 con 0x7f6c981027b0 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.586+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6c90005870 con 0x7f6c981027b0 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.586+0000 7f6c8e7fc700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 0x7f6c840a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.586+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f6c900ce160 con 0x7f6c981027b0 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.587+0000 7f6c9d4be700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 0x7f6c840a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:04:55.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.587+0000 7f6c9d4be700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 0x7f6c840a4000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c98101c70 tx=0x7f6c94006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:04:55.590 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.590+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c90098470 con 0x7f6c981027b0 2024-03-12T05:04:55.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:55 smithi130 ceph-mon[35398]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:04:55.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:55 smithi130 ceph-mon[35398]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:55.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:55 smithi116 ceph-mon[35486]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:04:55.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:55 smithi116 ceph-mon[35486]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:55.792 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:04:55.792+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c9802d8d0 con 0x7f6c840a1b40 2024-03-12T05:04:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:55 smithi045 ceph-mon[32174]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:04:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:55 smithi045 ceph-mon[32174]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:56.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:56 smithi130 ceph-mon[35398]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:56.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:56.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:56.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:56.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:56 smithi116 ceph-mon[35486]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:56.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:56.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:56.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:56.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:56 smithi045 ceph-mon[32174]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:04:56.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:04:56.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:04:56.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:04:57.642 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:57 smithi130 ceph-mon[35398]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:57.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:57 smithi116 ceph-mon[35486]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:57.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:57 smithi045 ceph-mon[32174]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:58.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:58 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/1201769425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:58 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:58 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]': finished 2024-03-12T05:04:58.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:58 smithi116 ceph-mon[35486]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:04:58.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:58 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:04:58.807 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:58 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/1201769425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.807 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:58 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.807 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:58 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]': finished 2024-03-12T05:04:58.807 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:58 smithi130 ceph-mon[35398]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:04:58.807 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:58 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:04:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:58 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/1201769425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:58 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]: dispatch 2024-03-12T05:04:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:58 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bf734df-d459-46bb-a3a2-432bdcf349fb"}]': finished 2024-03-12T05:04:58.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:58 smithi045 ceph-mon[32174]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:04:58.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:58 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:04:59.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:59 smithi116 ceph-mon[35486]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:59.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:04:59 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/4177541302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:59 smithi045 ceph-mon[32174]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:04:59 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/4177541302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:04:59.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:59 smithi130 ceph-mon[35398]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:04:59.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:04:59 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/4177541302' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:01.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:01 smithi130 ceph-mon[35398]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:01.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:01 smithi116 ceph-mon[35486]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:01.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:01 smithi045 ceph-mon[32174]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:03.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:03 smithi130 ceph-mon[35398]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:03.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:03 smithi045 ceph-mon[32174]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:03.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:03 smithi116 ceph-mon[35486]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:05.331 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:05 smithi130 ceph-mon[35398]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:05.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:05 smithi116 ceph-mon[35486]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:05.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:05 smithi045 ceph-mon[32174]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:06.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:05:06.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:06.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:06 smithi130 ceph-mon[35398]: Deploying daemon osd.5 on smithi130 2024-03-12T05:05:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:05:06.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:06.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:06 smithi116 ceph-mon[35486]: Deploying daemon osd.5 on smithi130 2024-03-12T05:05:06.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:05:06.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:06.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:06 smithi045 ceph-mon[32174]: Deploying daemon osd.5 on smithi130 2024-03-12T05:05:07.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:07 smithi116 ceph-mon[35486]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:07.452 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:07 smithi130 ceph-mon[35398]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:07 smithi045 ceph-mon[32174]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:09.413 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:09 smithi130 ceph-mon[35398]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:09 smithi116 ceph-mon[35486]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:09 smithi045 ceph-mon[32174]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:11.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:11 smithi130 ceph-mon[35398]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:11 smithi116 ceph-mon[35486]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:11.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:11 smithi045 ceph-mon[32174]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:12.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.722 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:12.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:12.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:12.873 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:13.683 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:13 smithi130 ceph-mon[35398]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:13.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:13 smithi116 ceph-mon[35486]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:13.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:13 smithi045 ceph-mon[32174]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:14.859 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.858+0000 7f6c8e7fc700 1 -- 172.21.15.130:0/3760574261 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6c9802d8d0 con 0x7f6c840a1b40 2024-03-12T05:05:14.860 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 5 on host 'smithi130' 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.860+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 msgr2=0x7f6c840a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.860+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 0x7f6c840a4000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c98101c70 tx=0x7f6c94006d20 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.860+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 msgr2=0x7f6c9819f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.860+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b070 tx=0x7f6c90004d30 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 shutdown_connections 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c840a1b40 0x7f6c840a4000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981027b0 0x7f6c9819f300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c98100c10 0x7f6c98197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 --2- 172.21.15.130:0/3760574261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c98101e10 0x7f6c98198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 >> 172.21.15.130:0/3760574261 conn(0x7f6c980fc1d0 msgr2=0x7f6c98105190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:05:14.862 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 shutdown_connections 2024-03-12T05:05:14.863 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:05:14.861+0000 7f6c9f722700 1 -- 172.21.15.130:0/3760574261 wait complete. 2024-03-12T05:05:15.483 DEBUG:teuthology.orchestra.run.smithi130:osd.5> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T05:05:15.486 INFO:tasks.cephadm:Deploying osd.6 on smithi045 with /dev/vg_nvme/lv_2... 2024-03-12T05:05:15.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:05:15.738 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:15 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.189 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.190 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:15 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:16.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:05:16.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:16.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:15 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:17.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:16 smithi045 ceph-mon[32174]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:05:17.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:16 smithi045 ceph-mon[32174]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:05:17.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:17.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:16 smithi045 ceph-mon[32174]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:17.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:16 smithi045 ceph-mon[32174]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:17.172 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-mon[35398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:05:17.173 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-mon[35398]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:05:17.173 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:17.173 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-mon[35398]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:17.173 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-mon[35398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:17.173 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:05:16.846+0000 7fd77b1a2700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T05:05:17.173 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:05:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:05:16.852+0000 7fd776f95700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:05:17.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:16 smithi116 ceph-mon[35486]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:05:17.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:16 smithi116 ceph-mon[35486]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:05:17.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:17.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:16 smithi116 ceph-mon[35486]: from='osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:17.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:16 smithi116 ceph-mon[35486]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:05:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:05:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:05:18.012 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-12T05:05:18.012 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-12T05:05:18.012 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00835379 s, 1.3 GB/s 2024-03-12T05:05:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-12T05:05:18.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:17 smithi045 ceph-mon[32174]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:05:18.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:17 smithi045 ceph-mon[32174]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:05:18.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.121 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:17 smithi045 ceph-mon[32174]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:18.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:17 smithi130 ceph-mon[35398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:05:18.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:17 smithi130 ceph-mon[35398]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:05:18.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:17 smithi130 ceph-mon[35398]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:18.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:17 smithi116 ceph-mon[35486]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:05:18.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:17 smithi116 ceph-mon[35486]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:05:18.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:18.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:17 smithi116 ceph-mon[35486]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:05:18.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-03-12T05:05:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] boot 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:05:19.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:18 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:19.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:05:19.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:05:19.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:19.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] boot 2024-03-12T05:05:19.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:05:19.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:18 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:19.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:05:19.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:05:19.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:19.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: osd.5 [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] boot 2024-03-12T05:05:19.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:05:19.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:18 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi130 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 8544M 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.082 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:20.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:20.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.083 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi130 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 8544M 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:20.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.223 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi130 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 8544M 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.247 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 -- 172.21.15.45:0/2185696030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde381027c0 msgr2=0x7fde38104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 --2- 172.21.15.45:0/2185696030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde381027c0 0x7fde38104bb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fde34009990 tx=0x7fde34009ca0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 -- 172.21.15.45:0/2185696030 shutdown_connections 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 --2- 172.21.15.45:0/2185696030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde38107a20 0x7fde38109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 --2- 172.21.15.45:0/2185696030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde381027c0 0x7fde38104bb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 --2- 172.21.15.45:0/2185696030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381050f0 0x7fde381074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 -- 172.21.15.45:0/2185696030 >> 172.21.15.45:0/2185696030 conn(0x7fde380fc130 msgr2=0x7fde380fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 -- 172.21.15.45:0/2185696030 shutdown_connections 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.601+0000 7fde40126700 1 -- 172.21.15.45:0/2185696030 wait complete. 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 Processor -- start 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 -- start start 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381027c0 0x7fde38102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde381050f0 0x7fde38100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde38101430 con 0x7fde38107a20 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381015a0 con 0x7fde381050f0 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde40126700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381a1b90 con 0x7fde381027c0 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde3e6c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde3e6c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33902/0 (socket says 172.21.15.45:33902) 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde3e6c3700 1 -- 172.21.15.45:0/3129623201 learned_addr learned my addr 172.21.15.45:0/3129623201 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:05:20.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.602+0000 7fde3d6c1700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde381050f0 0x7fde38100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3dec2700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381027c0 0x7fde38102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 -- 172.21.15.45:0/3129623201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381027c0 msgr2=0x7fde38102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381027c0 0x7fde38102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 -- 172.21.15.45:0/3129623201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde381050f0 msgr2=0x7fde38100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde381050f0 0x7fde38100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 -- 172.21.15.45:0/3129623201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde340095f0 con 0x7fde38107a20 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde3e6c3700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fde30007ec0 tx=0x7fde3000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3000ddd0 con 0x7fde38107a20 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde3000f040 con 0x7fde38107a20 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde381a1e70 con 0x7fde38107a20 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde30012710 con 0x7fde38107a20 2024-03-12T05:05:20.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.603+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde381a2290 con 0x7fde38107a20 2024-03-12T05:05:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.604+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde38068930 con 0x7fde38107a20 2024-03-12T05:05:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.605+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fde300133f0 con 0x7fde38107a20 2024-03-12T05:05:20.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.606+0000 7fde2effd700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 0x7fde240a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.606+0000 7fde3dec2700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 0x7fde240a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.606+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fde300cc3e0 con 0x7fde38107a20 2024-03-12T05:05:20.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.606+0000 7fde3dec2700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 0x7fde240a3f80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b5c0 tx=0x7fde34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:05:20.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.610+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fde300963f0 con 0x7fde38107a20 2024-03-12T05:05:20.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:20.820+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fde3802d910 con 0x7fde240a1ac0 2024-03-12T05:05:21.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:20 smithi045 ceph-mon[32174]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:05:21.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:21.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:21.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:20 smithi130 ceph-mon[35398]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:05:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:21.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:21.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:20 smithi116 ceph-mon[35486]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:05:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:22.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:21 smithi045 ceph-mon[32174]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:22.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:21 smithi045 ceph-mon[32174]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:05:22.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:21 smithi045 ceph-mon[32174]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:22.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:21 smithi130 ceph-mon[35398]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:22.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:21 smithi130 ceph-mon[35398]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:05:22.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:21 smithi130 ceph-mon[35398]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:22.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:21 smithi116 ceph-mon[35486]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:22.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:21 smithi116 ceph-mon[35486]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:05:22.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:21 smithi116 ceph-mon[35486]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:23.020 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:22 smithi045 ceph-mon[32174]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:05:23.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:22 smithi130 ceph-mon[35398]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:05:23.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:22 smithi116 ceph-mon[35486]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:05:24.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:23 smithi130 ceph-mon[35398]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-12T05:05:24.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:23 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]: dispatch 2024-03-12T05:05:24.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:23 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]': finished 2024-03-12T05:05:24.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:23 smithi130 ceph-mon[35398]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:05:24.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:24.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:23 smithi116 ceph-mon[35486]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-12T05:05:24.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:23 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]: dispatch 2024-03-12T05:05:24.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:23 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]': finished 2024-03-12T05:05:24.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:23 smithi116 ceph-mon[35486]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:05:24.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:24.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:23 smithi045 ceph-mon[32174]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-12T05:05:24.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:23 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]: dispatch 2024-03-12T05:05:24.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:23 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/722071707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84d0438a-d41f-43a0-b705-636605037f1c"}]': finished 2024-03-12T05:05:24.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:23 smithi045 ceph-mon[32174]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:05:24.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:25.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:24 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2521057628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:25.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:24 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2521057628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:25.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:24 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2521057628' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:26.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:25 smithi130 ceph-mon[35398]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-12T05:05:26.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:25 smithi116 ceph-mon[35486]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-12T05:05:26.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:25 smithi045 ceph-mon[32174]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-12T05:05:27.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:27 smithi130 ceph-mon[35398]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-12T05:05:27.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:27 smithi116 ceph-mon[35486]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-12T05:05:27.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:27 smithi045 ceph-mon[32174]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-12T05:05:29.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:29 smithi045 ceph-mon[32174]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:05:29.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:29 smithi130 ceph-mon[35398]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:05:29.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:29 smithi116 ceph-mon[35486]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:05:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:31 smithi045 ceph-mon[32174]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T05:05:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:31 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:05:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:31 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:31.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:31 smithi045 ceph-mon[32174]: Deploying daemon osd.6 on smithi045 2024-03-12T05:05:31.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:31 smithi130 ceph-mon[35398]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T05:05:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:31 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:05:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:31 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:31.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:31 smithi130 ceph-mon[35398]: Deploying daemon osd.6 on smithi045 2024-03-12T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:31 smithi116 ceph-mon[35486]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:31 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:31 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:31.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:31 smithi116 ceph-mon[35486]: Deploying daemon osd.6 on smithi045 2024-03-12T05:05:33.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:33 smithi045 ceph-mon[32174]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:33 smithi130 ceph-mon[35398]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:33 smithi116 ceph-mon[35486]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:35.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:35 smithi045 ceph-mon[32174]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:35.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:35 smithi130 ceph-mon[35398]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:35.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:35 smithi116 ceph-mon[35486]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:36.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:36.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:36.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:36.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:36.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:36.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:37.072 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:37.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:37 smithi130 ceph-mon[35398]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:37.972 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:37 smithi045 ceph-mon[32174]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:37.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:37 smithi116 ceph-mon[35486]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:39.297 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:39 smithi045 ceph-mon[32174]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:39 smithi130 ceph-mon[35398]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:39.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:39 smithi116 ceph-mon[35486]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:40.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.188+0000 7fde2effd700 1 -- 172.21.15.45:0/3129623201 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde3802d910 con 0x7fde240a1ac0 2024-03-12T05:05:40.188 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 6 on host 'smithi045' 2024-03-12T05:05:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 msgr2=0x7fde240a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 0x7fde240a3f80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b5c0 tx=0x7fde34005fb0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 msgr2=0x7fde38100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fde30007ec0 tx=0x7fde3000d3b0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 shutdown_connections 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fde240a1ac0 0x7fde240a3f80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde38107a20 0x7fde38100cb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde381027c0 0x7fde38102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 --2- 172.21.15.45:0/3129623201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde381050f0 0x7fde38100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 >> 172.21.15.45:0/3129623201 conn(0x7fde380fc130 msgr2=0x7fde381032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 shutdown_connections 2024-03-12T05:05:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:05:40.190+0000 7fde40126700 1 -- 172.21.15.45:0/3129623201 wait complete. 2024-03-12T05:05:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.791 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:40 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:40.821 DEBUG:teuthology.orchestra.run.smithi045:osd.6> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.6.service 2024-03-12T05:05:40.823 INFO:tasks.cephadm:Deploying osd.7 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-12T05:05:40.824 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.853 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:40 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:40.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:40 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:05:41.069 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:05:41.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:41 smithi116 ceph-mon[35486]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:05:41.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:41 smithi116 ceph-mon[35486]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:41.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:41 smithi045 ceph-mon[32174]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:05:41.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:41 smithi045 ceph-mon[32174]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:41.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:41 smithi130 ceph-mon[35398]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:05:41.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:41 smithi130 ceph-mon[35398]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:42 smithi045 ceph-mon[32174]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:05:42.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:42 smithi045 ceph-mon[32174]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:05:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:42 smithi045 ceph-mon[32174]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:05:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:42 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:42.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:42 smithi116 ceph-mon[35486]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:05:42.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:42 smithi116 ceph-mon[35486]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:05:42.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:42 smithi116 ceph-mon[35486]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:05:42.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:42 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:42.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:42 smithi130 ceph-mon[35398]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:05:42.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:42 smithi130 ceph-mon[35398]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:05:42.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:42 smithi130 ceph-mon[35398]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:05:42.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:42 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:05:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:05:43.208 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-12T05:05:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-12T05:05:43.209 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112173 s, 935 MB/s 2024-03-12T05:05:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-12T05:05:43.347 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:05:43.015+0000 7fe794ccf700 -1 osd.6 0 waiting for initial osdmap 2024-03-12T05:05:43.347 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:05:43.025+0000 7fe78caba700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' 2024-03-12T05:05:43.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:43 smithi045 ceph-mon[32174]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:43.846 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:05:43.846 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:05:43.846 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.846 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.846 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' 2024-03-12T05:05:43.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:43 smithi116 ceph-mon[35486]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:43.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-12T05:05:43.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:05:43.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:05:43.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:43.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: from='osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389]' entity='osd.6' 2024-03-12T05:05:43.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:43 smithi130 ceph-mon[35398]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:05:44.155 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi045 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 5697M 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] boot 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:05:44.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:44.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi045 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 5697M 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:44.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] boot 2024-03-12T05:05:44.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:05:44.973 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:44.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:05:44.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:05:44.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:44.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi045 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 5697M 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: osd.6 [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] boot 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:05:44.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- 172.21.15.116:0/1342039224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 msgr2=0x7f6e70103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 --2- 172.21.15.116:0/1342039224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70103300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0099c0 tx=0x7f6e6c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- 172.21.15.116:0/1342039224 shutdown_connections 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 --2- 172.21.15.116:0/1342039224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70103840 0x7f6e7010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 --2- 172.21.15.116:0/1342039224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70101ca0 0x7f6e701020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 --2- 172.21.15.116:0/1342039224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70103300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- 172.21.15.116:0/1342039224 >> 172.21.15.116:0/1342039224 conn(0x7f6e700fd280 msgr2=0x7f6e700ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- 172.21.15.116:0/1342039224 shutdown_connections 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- 172.21.15.116:0/1342039224 wait complete. 2024-03-12T05:05:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 Processor -- start 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.801+0000 7f6e77465700 1 -- start start 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70101ca0 0x7f6e70075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70103840 0x7f6e700745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70074d20 con 0x7f6e70103840 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70074e90 con 0x7f6e70102ea0 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701a1c00 con 0x7f6e70101ca0 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55336/0 (socket says 172.21.15.116:55336) 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 -- 172.21.15.116:0/1584159454 learned_addr learned my addr 172.21.15.116:0/1584159454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e75a02700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70103840 0x7f6e700745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e75201700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70101ca0 0x7f6e70075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 -- 172.21.15.116:0/1584159454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70101ca0 msgr2=0x7f6e70075a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70101ca0 0x7f6e70075a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 -- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70103840 msgr2=0x7f6e700745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70103840 0x7f6e700745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 -- 172.21.15.116:0/1584159454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0095f0 con 0x7f6e70102ea0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e74a00700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00b580 tx=0x7f6e6c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c01d070 con 0x7f6e70102ea0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e6c004050 con 0x7f6e70102ea0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e701a1e80 con 0x7f6e70102ea0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c022d00 con 0x7f6e70102ea0 2024-03-12T05:05:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.802+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e701a2290 con 0x7f6e70102ea0 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.803+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e701091d0 con 0x7f6e70102ea0 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.804+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6e6c022470 con 0x7f6e70102ea0 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.805+0000 7f6e667fc700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 0x7f6e5c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.805+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f6e6c0cda00 con 0x7f6e70102ea0 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.805+0000 7f6e75201700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 0x7f6e5c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:05:45.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.806+0000 7f6e75201700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 0x7f6e5c0a3f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e70102d00 tx=0x7f6e68009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:05:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:45.809+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e6c097990 con 0x7f6e70102ea0 2024-03-12T05:05:46.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:05:46.008+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e7002da20 con 0x7f6e5c0a1ac0 2024-03-12T05:05:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:46 smithi045 ceph-mon[32174]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:05:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:46 smithi045 ceph-mon[32174]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:46.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:46 smithi130 ceph-mon[35398]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:05:46.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:46 smithi130 ceph-mon[35398]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:46.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:46.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:46.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:46.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:46 smithi116 ceph-mon[35486]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:05:46.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:46 smithi116 ceph-mon[35486]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:46.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:05:46.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:05:46.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:47.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:47 smithi116 ceph-mon[35486]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:47.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:47 smithi045 ceph-mon[32174]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:47.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:47 smithi130 ceph-mon[35398]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:05:48.251 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:48 smithi116 ceph-mon[35486]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:48.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:48 smithi045 ceph-mon[32174]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:48.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:48 smithi130 ceph-mon[35398]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:49.249 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:49 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/2288048785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.250 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:49 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.250 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:49 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]': finished 2024-03-12T05:05:49.250 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:49 smithi116 ceph-mon[35486]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:05:49.250 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:49 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:05:49.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:49 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/2288048785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:49 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:49 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]': finished 2024-03-12T05:05:49.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:49 smithi045 ceph-mon[32174]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:05:49.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:49 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:05:49.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:49 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/2288048785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:49 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]: dispatch 2024-03-12T05:05:49.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:49 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da01102a-465e-47b8-bc0a-8d4119260bac"}]': finished 2024-03-12T05:05:49.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:49 smithi130 ceph-mon[35398]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:05:49.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:49 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:05:50.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:50 smithi045 ceph-mon[32174]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:50.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:50 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/3575074824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:50.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:50 smithi130 ceph-mon[35398]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:50.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:50 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/3575074824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:50.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:50 smithi116 ceph-mon[35486]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:50.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:50 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/3575074824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:05:51.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:51 smithi130 ceph-mon[35398]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:51.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:51 smithi116 ceph-mon[35486]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:51.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:51 smithi045 ceph-mon[32174]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:53.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:53 smithi116 ceph-mon[35486]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:53 smithi130 ceph-mon[35398]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:53.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:53 smithi045 ceph-mon[32174]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:55.413 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:55 smithi116 ceph-mon[35486]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:55.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:55 smithi130 ceph-mon[35398]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:55 smithi045 ceph-mon[32174]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:56.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:05:56.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:56.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:56 smithi116 ceph-mon[35486]: Deploying daemon osd.7 on smithi116 2024-03-12T05:05:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:05:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:56 smithi130 ceph-mon[35398]: Deploying daemon osd.7 on smithi116 2024-03-12T05:05:56.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:05:56.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:05:56.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:56 smithi045 ceph-mon[32174]: Deploying daemon osd.7 on smithi116 2024-03-12T05:05:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:57 smithi130 ceph-mon[35398]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:57.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:57 smithi116 ceph-mon[35486]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:57.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:57 smithi045 ceph-mon[32174]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:59.398 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:05:59 smithi116 ceph-mon[35486]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:59.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:05:59 smithi130 ceph-mon[35398]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:05:59.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:05:59 smithi045 ceph-mon[32174]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:01.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:01 smithi116 ceph-mon[35486]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:01.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:01 smithi130 ceph-mon[35398]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:01.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:01 smithi045 ceph-mon[32174]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:02.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:02.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:03.809 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:03 smithi116 ceph-mon[35486]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:03.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:03 smithi045 ceph-mon[32174]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:03.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:03 smithi130 ceph-mon[35398]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:05.221 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 7 on host 'smithi116' 2024-03-12T05:06:05.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.221+0000 7f6e667fc700 1 -- 172.21.15.116:0/1584159454 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6e7002da20 con 0x7f6e5c0a1ac0 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 msgr2=0x7f6e5c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 0x7f6e5c0a3f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e70102d00 tx=0x7f6e68009040 comp rx=0 tx=0).stop 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 msgr2=0x7f6e70074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00b580 tx=0x7f6e6c009580 comp rx=0 tx=0).stop 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 shutdown_connections 2024-03-12T05:06:05.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6e5c0a1ac0 0x7f6e5c0a3f80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e70103840 0x7f6e700745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e70101ca0 0x7f6e70075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 --2- 172.21.15.116:0/1584159454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e70102ea0 0x7f6e70074060 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 >> 172.21.15.116:0/1584159454 conn(0x7f6e700fd280 msgr2=0x7f6e70106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 shutdown_connections 2024-03-12T05:06:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:06:05.223+0000 7f6e77465700 1 -- 172.21.15.116:0/1584159454 wait complete. 2024-03-12T05:06:05.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:05 smithi045 ceph-mon[32174]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:05.459 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:05 smithi116 ceph-mon[35486]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:05.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:05 smithi130 ceph-mon[35398]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:05.932 DEBUG:teuthology.orchestra.run.smithi116:osd.7> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.7.service 2024-03-12T05:06:05.935 INFO:tasks.cephadm:Deploying osd.8 on smithi130 with /dev/vg_nvme/lv_2... 2024-03-12T05:06:05.935 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:06:06.185 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:06:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:06 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:06 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:06.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:06 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:07.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:06:07.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:06:07.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:07.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:07 smithi130 ceph-mon[35398]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-mon[35486]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:07.497 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:06:07.220+0000 7fa4d6cd6700 -1 osd.7 0 waiting for initial osdmap 2024-03-12T05:06:07.497 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:06:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:06:07.229+0000 7fa4d2ac9700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: from='osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:06:07.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:07 smithi045 ceph-mon[32174]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111064 s, 944 MB/s 2024-03-12T05:06:08.109 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-12T05:06:08.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:08 smithi116 ceph-mon[35486]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:06:08.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:08 smithi116 ceph-mon[35486]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:06:08.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.574 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:08 smithi130 ceph-mon[35398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:06:08.574 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:08 smithi130 ceph-mon[35398]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:06:08.574 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.574 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:08 smithi045 ceph-mon[32174]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:06:08.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:08 smithi045 ceph-mon[32174]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:06:08.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:08.737 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2024-03-12T05:06:08.984 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:06:09.385 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:06:09.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:06:09.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] boot 2024-03-12T05:06:09.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:06:09.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:09.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:09 smithi130 ceph-mon[35398]: pgmap v174: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] boot 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:09.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:09 smithi116 ceph-mon[35486]: pgmap v174: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: osd.7 [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] boot 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:06:09.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:09 smithi045 ceph-mon[32174]: pgmap v174: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi116 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.353 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.354 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:06:10.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi116 2024-03-12T05:06:10.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.583 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 -- 172.21.15.130:0/1339681031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102e90 msgr2=0x7f11e81032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:10.583 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/1339681031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102e90 0x7f11e81032f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009990 tx=0x7f11dc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 -- 172.21.15.130:0/1339681031 shutdown_connections 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/1339681031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8103830 0x7f11e810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/1339681031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8101c90 0x7f11e81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/1339681031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8102e90 0x7f11e81032f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.582+0000 7f11ed5ce700 1 -- 172.21.15.130:0/1339681031 >> 172.21.15.130:0/1339681031 conn(0x7f11e80fd250 msgr2=0x7f11e80ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- 172.21.15.130:0/1339681031 shutdown_connections 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- 172.21.15.130:0/1339681031 wait complete. 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 Processor -- start 2024-03-12T05:06:10.584 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- start start 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 0x7f11e8198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8103830 0x7f11e819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e819f760 con 0x7f11e8102e90 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e819f8a0 con 0x7f11e8103830 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11ed5ce700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e819fa30 con 0x7f11e8101c90 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.583+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:58936/0 (socket says 172.21.15.130:58936) 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 -- 172.21.15.130:0/3722124892 learned_addr learned my addr 172.21.15.130:0/3722124892 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e67fc700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 0x7f11e8198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:10.586 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e77fe700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8103830 0x7f11e819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 -- 172.21.15.130:0/3722124892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8103830 msgr2=0x7f11e819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8103830 0x7f11e819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 -- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 msgr2=0x7f11e8198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 0x7f11e8198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 -- 172.21.15.130:0/3722124892 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0095f0 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e67fc700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 0x7f11e8198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11e6ffd700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c9a0 tx=0x7f11d800cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8004220 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f11d8007d10 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e819fd10 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8003d00 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.584+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81a01c0 con 0x7f11e8101c90 2024-03-12T05:06:10.587 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.585+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e8068930 con 0x7f11e8101c90 2024-03-12T05:06:10.588 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.586+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f11d8007860 con 0x7f11e8101c90 2024-03-12T05:06:10.588 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.587+0000 7f11cffff700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 0x7f11d00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:10.588 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.587+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f11d80cc550 con 0x7f11e8101c90 2024-03-12T05:06:10.588 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.587+0000 7f11e67fc700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 0x7f11d00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:10.588 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.588+0000 7f11e67fc700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 0x7f11d00a3f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00b5c0 tx=0x7f11dc003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:06:10.592 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.592+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f11d80962a0 con 0x7f11e8101c90 2024-03-12T05:06:10.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi116 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:10.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:10.790 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:10.789+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f11e802d910 con 0x7f11d00a1ac0 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:11.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:11 smithi116 ceph-mon[35486]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:11.531 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:06:11.531 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:11.531 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:11.531 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:11.531 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:11.532 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:11 smithi130 ceph-mon[35398]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:11.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:06:11.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:11.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:11.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:11.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:11.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:11 smithi045 ceph-mon[32174]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:12.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:12 smithi116 ceph-mon[35486]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:06:12.540 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:12 smithi130 ceph-mon[35398]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:06:12.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:12 smithi045 ceph-mon[32174]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:06:13.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:13 smithi045 ceph-mon[32174]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:06:13.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:13 smithi045 ceph-mon[32174]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:13.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:13 smithi130 ceph-mon[35398]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:06:13.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:13 smithi130 ceph-mon[35398]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:13.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:13 smithi116 ceph-mon[35486]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:06:13.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:13 smithi116 ceph-mon[35486]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:14 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/4137672782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:14 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:14 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]': finished 2024-03-12T05:06:14.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:14 smithi130 ceph-mon[35398]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:06:14.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:14.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:14 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/4137672782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:14 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:14 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]': finished 2024-03-12T05:06:14.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:14 smithi045 ceph-mon[32174]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:06:14.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:14.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:14 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/4137672782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:14 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]: dispatch 2024-03-12T05:06:14.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:14 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae6cce03-2fb7-4fe5-a943-33e47113ef19"}]': finished 2024-03-12T05:06:14.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:14 smithi116 ceph-mon[35486]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:06:14.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:15.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:15 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/529896978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:15.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:15 smithi045 ceph-mon[32174]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:15.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:15 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/529896978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:15.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:15 smithi130 ceph-mon[35398]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:15.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:15 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/529896978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:15.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:15 smithi116 ceph-mon[35486]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:17 smithi130 ceph-mon[35398]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:17.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:17 smithi116 ceph-mon[35486]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:17.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:17 smithi045 ceph-mon[32174]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:19.448 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:19 smithi130 ceph-mon[35398]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:19.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:19 smithi116 ceph-mon[35486]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:19.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:19 smithi045 ceph-mon[32174]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:06:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:06:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:06:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:22.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:21 smithi130 ceph-mon[35398]: Deploying daemon osd.8 on smithi130 2024-03-12T05:06:22.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:21 smithi130 ceph-mon[35398]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:22.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:21 smithi116 ceph-mon[35486]: Deploying daemon osd.8 on smithi130 2024-03-12T05:06:22.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:21 smithi116 ceph-mon[35486]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:22.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:21 smithi045 ceph-mon[32174]: Deploying daemon osd.8 on smithi130 2024-03-12T05:06:22.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:21 smithi045 ceph-mon[32174]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:23.229 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:23 smithi130 ceph-mon[35398]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:23.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:23 smithi116 ceph-mon[35486]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:23.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:23 smithi045 ceph-mon[32174]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:25.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:25 smithi130 ceph-mon[35398]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:25 smithi116 ceph-mon[35486]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:25.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:25 smithi045 ceph-mon[32174]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:27.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:27.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:27 smithi130 ceph-mon[35398]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:27.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:27.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:27 smithi116 ceph-mon[35486]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:27.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:27.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:27.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:27 smithi045 ceph-mon[32174]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:29.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:29 smithi130 ceph-mon[35398]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:29.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:29 smithi116 ceph-mon[35486]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:29.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:29 smithi045 ceph-mon[32174]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:29.860 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 8 on host 'smithi130' 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.860+0000 7f11cffff700 1 -- 172.21.15.130:0/3722124892 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f11e802d910 con 0x7f11d00a1ac0 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 msgr2=0x7f11d00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 0x7f11d00a3f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00b5c0 tx=0x7f11dc003680 comp rx=0 tx=0).stop 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 msgr2=0x7f11e8197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c9a0 tx=0x7f11d800cd60 comp rx=0 tx=0).stop 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 shutdown_connections 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f11d00a1ac0 0x7f11d00a3f80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:29.868 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8103830 0x7f11e819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:29.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e8101c90 0x7f11e8197bd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:29.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 --2- 172.21.15.130:0/3722124892 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11e8102e90 0x7f11e8198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:29.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.863+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 >> 172.21.15.130:0/3722124892 conn(0x7f11e80fd250 msgr2=0x7f11e8106260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:06:29.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.864+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 shutdown_connections 2024-03-12T05:06:29.869 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:06:29.864+0000 7f11ed5ce700 1 -- 172.21.15.130:0/3722124892 wait complete. 2024-03-12T05:06:30.600 DEBUG:teuthology.orchestra.run.smithi130:osd.8> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.8.service 2024-03-12T05:06:30.603 INFO:tasks.cephadm:Deploying osd.9 on smithi045 with /dev/vg_nvme/lv_1... 2024-03-12T05:06:30.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:06:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:06:30.912 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.912 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.912 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.912 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.912 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:30.913 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:30 smithi130 ceph-mon[35398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:30.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:30.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:30.980 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:30 smithi045 ceph-mon[32174]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:30 smithi116 ceph-mon[35486]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:06:32.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:32.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:06:32.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:06:32.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:32.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:31 smithi130 ceph-mon[35398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:32.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:06:32.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:06:32.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:32.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:31 smithi045 ceph-mon[32174]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: from='osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:31 smithi116 ceph-mon[35486]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:06:32.471 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:06:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:06:32.186+0000 7f479d34c700 -1 osd.8 0 waiting for initial osdmap 2024-03-12T05:06:32.471 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:06:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:06:32.199+0000 7f479913f700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00836143 s, 1.3 GB/s 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 2024-03-12T05:06:32.810 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-12T05:06:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:33 smithi130 ceph-mon[35398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:06:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:33 smithi130 ceph-mon[35398]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:06:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:33 smithi130 ceph-mon[35398]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:33 smithi116 ceph-mon[35486]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:06:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:33 smithi116 ceph-mon[35486]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:06:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:33 smithi116 ceph-mon[35486]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:33 smithi045 ceph-mon[32174]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:06:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:33 smithi045 ceph-mon[32174]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:06:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:33.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:33 smithi045 ceph-mon[32174]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:06:33.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-03-12T05:06:33.876 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] boot 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: osdmap e59: 9 total, 9 up, 9 in 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi130 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 5696M 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] boot 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: osdmap e59: 9 total, 9 up, 9 in 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi130 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 5696M 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: osd.8 [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] boot 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: osdmap e59: 9 total, 9 up, 9 in 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi130 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 5696M 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:34.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:35.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:35 smithi045 ceph-mon[32174]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:06:35.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:35 smithi045 ceph-mon[32174]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:35.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:35 smithi130 ceph-mon[35398]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:06:35.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:35 smithi130 ceph-mon[35398]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:35.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:35 smithi116 ceph-mon[35486]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:06:35.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:35 smithi116 ceph-mon[35486]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:35.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- 172.21.15.45:0/1294671283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 msgr2=0x7fbf9c108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:35.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 --2- 172.21.15.45:0/1294671283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c108ff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b0d0 tx=0x7fbf8c00b3e0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- 172.21.15.45:0/1294671283 shutdown_connections 2024-03-12T05:06:35.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 --2- 172.21.15.45:0/1294671283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c108ff0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 --2- 172.21.15.45:0/1294671283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c100ba0 0x7fbf9c100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 --2- 172.21.15.45:0/1294671283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c101da0 0x7fbf9c102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- 172.21.15.45:0/1294671283 >> 172.21.15.45:0/1294671283 conn(0x7fbf9c0fc140 msgr2=0x7fbf9c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- 172.21.15.45:0/1294671283 shutdown_connections 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- 172.21.15.45:0/1294671283 wait complete. 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 Processor -- start 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.567+0000 7fbfa1747700 1 -- start start 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c100ba0 0x7fbf9c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c101da0 0x7fbf9c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1a1a80 con 0x7fbf9c102740 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1a1bc0 con 0x7fbf9c101da0 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1a1d50 con 0x7fbf9c100ba0 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:35.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38884/0 (socket says 172.21.15.45:38884) 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 -- 172.21.15.45:0/2596844206 learned_addr learned my addr 172.21.15.45:0/2596844206 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9affd700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c100ba0 0x7fbf9c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9a7fc700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c101da0 0x7fbf9c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 -- 172.21.15.45:0/2596844206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c100ba0 msgr2=0x7fbf9c199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c100ba0 0x7fbf9c199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 -- 172.21.15.45:0/2596844206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c101da0 msgr2=0x7fbf9c19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c101da0 0x7fbf9c19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 -- 172.21.15.45:0/2596844206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c009d00 con 0x7fbf9c102740 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf9b7fe700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009390 tx=0x7fbf8c0093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c00e040 con 0x7fbf9c102740 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbf8c003e10 con 0x7fbf9c102740 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c1a1fd0 con 0x7fbf9c102740 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.568+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c01ba00 con 0x7fbf9c102740 2024-03-12T05:06:35.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.569+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c1a2380 con 0x7fbf9c102740 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.569+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c068930 con 0x7fbf9c102740 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.571+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbf8c01bb60 con 0x7fbf9c102740 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.572+0000 7fbf93fff700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 0x7fbf880a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.572+0000 7fbf9affd700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 0x7fbf880a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.572+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fbf8c0ce840 con 0x7fbf9c102740 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.572+0000 7fbf9affd700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 0x7fbf880a3f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c101c00 tx=0x7fbf9400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:06:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.576+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbf8c098470 con 0x7fbf9c102740 2024-03-12T05:06:35.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:35.789+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf9c0fe360 con 0x7fbf880a1ac0 2024-03-12T05:06:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:36 smithi130 ceph-mon[35398]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:06:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:36 smithi130 ceph-mon[35398]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:36.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:36 smithi116 ceph-mon[35486]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:06:36.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:36 smithi116 ceph-mon[35486]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:36.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:36.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:36.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:36.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:36 smithi045 ceph-mon[32174]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:06:36.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:36 smithi045 ceph-mon[32174]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:06:36.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:06:36.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:06:36.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:37 smithi045 ceph-mon[32174]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:06:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:37 smithi045 ceph-mon[32174]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:37.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:37 smithi130 ceph-mon[35398]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:06:37.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:37 smithi130 ceph-mon[35398]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:37.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:37 smithi116 ceph-mon[35486]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:06:37.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:37 smithi116 ceph-mon[35486]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:38.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:38 smithi130 ceph-mon[35398]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:06:38.498 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:38 smithi116 ceph-mon[35486]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:06:38.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:38 smithi045 ceph-mon[32174]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]: dispatch 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]': finished 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: osdmap e64: 10 total, 9 up, 10 in 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:39.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/310598050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:39.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]: dispatch 2024-03-12T05:06:39.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]': finished 2024-03-12T05:06:39.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: osdmap e64: 10 total, 9 up, 10 in 2024-03-12T05:06:39.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:39.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:39.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/310598050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:39.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]: dispatch 2024-03-12T05:06:39.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/198414804' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a"}]': finished 2024-03-12T05:06:39.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: osdmap e64: 10 total, 9 up, 10 in 2024-03-12T05:06:39.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:39.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:39.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/310598050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:06:41.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:41 smithi130 ceph-mon[35398]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-12T05:06:41.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:41 smithi116 ceph-mon[35486]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-12T05:06:41.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:41 smithi045 ceph-mon[32174]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-12T05:06:43.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:43 smithi130 ceph-mon[35398]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-12T05:06:43.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:43 smithi045 ceph-mon[32174]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-12T05:06:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:43 smithi116 ceph-mon[35486]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-12T05:06:45.180 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:45 smithi045 ceph-mon[32174]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:06:45.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:45 smithi130 ceph-mon[35398]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:06:45.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:45 smithi116 ceph-mon[35486]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:06:46.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:06:46.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:46.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:46 smithi045 ceph-mon[32174]: Deploying daemon osd.9 on smithi045 2024-03-12T05:06:46.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:06:46.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:46.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:46 smithi130 ceph-mon[35398]: Deploying daemon osd.9 on smithi045 2024-03-12T05:06:46.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:06:46.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:46.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:46 smithi116 ceph-mon[35486]: Deploying daemon osd.9 on smithi045 2024-03-12T05:06:47.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:47 smithi130 ceph-mon[35398]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:06:47.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:47 smithi116 ceph-mon[35486]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:06:47.584 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:47 smithi045 ceph-mon[32174]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:06:49.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:49 smithi045 ceph-mon[32174]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-12T05:06:49.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:49 smithi130 ceph-mon[35398]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-12T05:06:49.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:49 smithi116 ceph-mon[35486]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-12T05:06:51.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:51 smithi045 ceph-mon[32174]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-12T05:06:51.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:51 smithi130 ceph-mon[35398]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-12T05:06:51.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:51 smithi116 ceph-mon[35486]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-12T05:06:52.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:52 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:52 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:52 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:52.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:52 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:52 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:52 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:52.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:52 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:52 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:52.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:52 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:53.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:53 smithi045 ceph-mon[32174]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:53.892 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:53 smithi130 ceph-mon[35398]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:53.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:53 smithi116 ceph-mon[35486]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:55.547 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 9 on host 'smithi045' 2024-03-12T05:06:55.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.546+0000 7fbf93fff700 1 -- 172.21.15.45:0/2596844206 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbf9c0fe360 con 0x7fbf880a1ac0 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 msgr2=0x7fbf880a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 0x7fbf880a3f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c101c00 tx=0x7fbf9400a380 comp rx=0 tx=0).stop 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 msgr2=0x7fbf9c1a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009390 tx=0x7fbf8c0093c0 comp rx=0 tx=0).stop 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 shutdown_connections 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbf880a1ac0 0x7fbf880a3f80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf9c102740 0x7fbf9c1a14b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf9c100ba0 0x7fbf9c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:55.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 --2- 172.21.15.45:0/2596844206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf9c101da0 0x7fbf9c19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:06:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 >> 172.21.15.45:0/2596844206 conn(0x7fbf9c0fc140 msgr2=0x7fbf9c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:06:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 shutdown_connections 2024-03-12T05:06:55.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:06:55.549+0000 7fbfa1747700 1 -- 172.21.15.45:0/2596844206 wait complete. 2024-03-12T05:06:55.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:55.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:55 smithi045 ceph-mon[32174]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:06:55.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:55.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:55 smithi130 ceph-mon[35398]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:06:55.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:06:55.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:55 smithi116 ceph-mon[35486]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:06:56.196 DEBUG:teuthology.orchestra.run.smithi045:osd.9> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.9.service 2024-03-12T05:06:56.199 INFO:tasks.cephadm:Deploying osd.10 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-12T05:06:56.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:06:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:06:57.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:57 smithi045 ceph-mon[32174]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:06:57.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:57 smithi045 ceph-mon[32174]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:06:57.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:57 smithi045 ceph-mon[32174]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:06:57.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:57 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:57.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:57 smithi045 ceph-mon[32174]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:57.848 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:57 smithi116 ceph-mon[35486]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:06:57.849 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:57 smithi116 ceph-mon[35486]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:06:57.849 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:57 smithi116 ceph-mon[35486]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:06:57.849 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:57 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:57.849 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:57 smithi116 ceph-mon[35486]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:57.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:57 smithi130 ceph-mon[35398]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:06:57.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:57 smithi130 ceph-mon[35398]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:06:57.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:57 smithi130 ceph-mon[35398]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-12T05:06:57.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:57 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:57.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:57 smithi130 ceph-mon[35398]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:58.039 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:06:58.022+0000 7f3e7e98b700 -1 osd.9 0 waiting for initial osdmap 2024-03-12T05:06:58.039 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:06:58.038+0000 7f3e76776700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114977 s, 912 MB/s 2024-03-12T05:06:58.431 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-12T05:06:58.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-mon[32174]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:06:58.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-mon[32174]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:06:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:58 smithi045 ceph-mon[32174]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' 2024-03-12T05:06:58.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:58 smithi130 ceph-mon[35398]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:06:58.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:58 smithi130 ceph-mon[35398]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:06:58.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:58 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:58 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:58 smithi130 ceph-mon[35398]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' 2024-03-12T05:06:58.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:58 smithi116 ceph-mon[35486]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-12T05:06:58.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:58 smithi116 ceph-mon[35486]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:06:58.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:58 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:58 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:58.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:58 smithi116 ceph-mon[35486]: from='osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149]' entity='osd.9' 2024-03-12T05:06:59.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-12T05:06:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:06:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:06:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:06:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] boot 2024-03-12T05:06:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: osdmap e67: 10 total, 10 up, 10 in 2024-03-12T05:06:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:06:59 smithi116 ceph-mon[35486]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:59.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:06:59.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:06:59.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] boot 2024-03-12T05:06:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: osdmap e67: 10 total, 10 up, 10 in 2024-03-12T05:06:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:06:59 smithi045 ceph-mon[32174]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: osd.9 [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] boot 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: osdmap e67: 10 total, 10 up, 10 in 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:06:59.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:06:59 smithi130 ceph-mon[35398]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi045 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:00.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:00.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:00 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 -- 172.21.15.116:0/3850037521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 msgr2=0x7f57a4103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:00.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 --2- 172.21.15.116:0/3850037521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a4103300 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f57940099c0 tx=0x7f5794009cd0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 -- 172.21.15.116:0/3850037521 shutdown_connections 2024-03-12T05:07:00.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 --2- 172.21.15.116:0/3850037521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4103840 0x7f57a410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 --2- 172.21.15.116:0/3850037521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4101ca0 0x7f57a41020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 --2- 172.21.15.116:0/3850037521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a4103300 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 -- 172.21.15.116:0/3850037521 >> 172.21.15.116:0/3850037521 conn(0x7f57a40fd280 msgr2=0x7f57a40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 -- 172.21.15.116:0/3850037521 shutdown_connections 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.965+0000 7f57a9d10700 1 -- 172.21.15.116:0/3850037521 wait complete. 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 Processor -- start 2024-03-12T05:07:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 -- start start 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4101ca0 0x7f57a4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4103840 0x7f57a419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a419f710 con 0x7f57a4101ca0 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a419f850 con 0x7f57a4102ea0 2024-03-12T05:07:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a9d10700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a419f9e0 con 0x7f57a4103840 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60440/0 (socket says 172.21.15.116:60440) 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a2ffd700 1 -- 172.21.15.116:0/1576743794 learned_addr learned my addr 172.21.15.116:0/1576743794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a37fe700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4101ca0 0x7f57a4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.966+0000 7f57a3fff700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4103840 0x7f57a419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 -- 172.21.15.116:0/1576743794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4103840 msgr2=0x7f57a419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4103840 0x7f57a419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 -- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4101ca0 msgr2=0x7f57a4197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4101ca0 0x7f57a4197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 -- 172.21.15.116:0/1576743794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940095f0 con 0x7f57a4102ea0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a2ffd700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f579400b580 tx=0x7f5794009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579401d070 con 0x7f57a4102ea0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5794003f50 con 0x7f57a4102ea0 2024-03-12T05:07:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794022c60 con 0x7f57a4102ea0 2024-03-12T05:07:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a419fc60 con 0x7f57a4102ea0 2024-03-12T05:07:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.967+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a41a0150 con 0x7f57a4102ea0 2024-03-12T05:07:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.968+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a4109180 con 0x7f57a4102ea0 2024-03-12T05:07:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.969+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f57940040c0 con 0x7f57a4102ea0 2024-03-12T05:07:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.970+0000 7f57a0ff9700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 0x7f578c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.970+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f57940cdbc0 con 0x7f57a4102ea0 2024-03-12T05:07:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.970+0000 7f57a37fe700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 0x7f578c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.971+0000 7f57a37fe700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 0x7f578c0a4000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57a4102d00 tx=0x7f579800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:00.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi045 2024-03-12T05:07:00.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:00.972 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:00 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:00.974+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f579405dd20 con 0x7f57a4102ea0 2024-03-12T05:07:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi045 2024-03-12T05:07:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 4273M 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:01.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:00 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:01.177+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f57a402d870 con 0x7f578c0a1b40 2024-03-12T05:07:01.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:01 smithi116 ceph-mon[35486]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:07:01.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:01 smithi116 ceph-mon[35486]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:01.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:01.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:01.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:01 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:01.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:01 smithi130 ceph-mon[35398]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:07:01.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:01 smithi130 ceph-mon[35398]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:01.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:01.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:01.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:01 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:02.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:01 smithi045 ceph-mon[32174]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:07:02.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:01 smithi045 ceph-mon[32174]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:02.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:02.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:02.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:01 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:02.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:02 smithi130 ceph-mon[35398]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:02.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:02 smithi116 ceph-mon[35486]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:03.020 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:02 smithi045 ceph-mon[32174]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:03.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:03 smithi130 ceph-mon[35398]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:03.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:03 smithi116 ceph-mon[35486]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:04.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:03 smithi045 ceph-mon[32174]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/819736310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]': finished 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: osdmap e69: 11 total, 10 up, 11 in 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:04.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:04 smithi130 ceph-mon[35398]: from='client.? 172.21.15.116:0/837198296' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:04.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/819736310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:04.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:04.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]': finished 2024-03-12T05:07:04.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: osdmap e69: 11 total, 10 up, 11 in 2024-03-12T05:07:04.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:04.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:04 smithi116 ceph-mon[35486]: from='client.? 172.21.15.116:0/837198296' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:05.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/819736310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:05.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]: dispatch 2024-03-12T05:07:05.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00932bcc-b7f2-4a9c-9421-ce264276091b"}]': finished 2024-03-12T05:07:05.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: osdmap e69: 11 total, 10 up, 11 in 2024-03-12T05:07:05.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:05.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:04 smithi045 ceph-mon[32174]: from='client.? 172.21.15.116:0/837198296' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:05.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:05 smithi130 ceph-mon[35398]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:05.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:05 smithi116 ceph-mon[35486]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:05 smithi045 ceph-mon[32174]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:07.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:07 smithi130 ceph-mon[35398]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:07.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:07 smithi116 ceph-mon[35486]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:07.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:07 smithi045 ceph-mon[32174]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:09.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:09 smithi130 ceph-mon[35398]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:09.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:09 smithi116 ceph-mon[35486]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:09.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:09 smithi045 ceph-mon[32174]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:11.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:11 smithi116 ceph-mon[35486]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:11.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:11 smithi130 ceph-mon[35398]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:11.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:11 smithi045 ceph-mon[32174]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:12.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:07:12.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:12 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:12.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:12 smithi116 ceph-mon[35486]: Deploying daemon osd.10 on smithi116 2024-03-12T05:07:12.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:07:12.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:12 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:12.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:12 smithi130 ceph-mon[35398]: Deploying daemon osd.10 on smithi116 2024-03-12T05:07:12.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:07:12.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:12 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:12.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:12 smithi045 ceph-mon[32174]: Deploying daemon osd.10 on smithi116 2024-03-12T05:07:13.197 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:13 smithi116 ceph-mon[35486]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:13.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:13 smithi130 ceph-mon[35398]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:13.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:13 smithi045 ceph-mon[32174]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:15.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:15 smithi116 ceph-mon[35486]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:15.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:15 smithi130 ceph-mon[35398]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:15.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:15 smithi045 ceph-mon[32174]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:17.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:17.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:17.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:17.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:17 smithi116 ceph-mon[35486]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:18.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:18.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:18.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:18.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:17 smithi130 ceph-mon[35398]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:18.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:18.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:18.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:18.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:17 smithi045 ceph-mon[32174]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:19.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:19 smithi130 ceph-mon[35398]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:19.472 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:19 smithi116 ceph-mon[35486]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:19.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:19 smithi045 ceph-mon[32174]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:20.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.648+0000 7f57a0ff9700 1 -- 172.21.15.116:0/1576743794 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f57a402d870 con 0x7f578c0a1b40 2024-03-12T05:07:20.648 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 10 on host 'smithi116' 2024-03-12T05:07:20.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 msgr2=0x7f578c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 0x7f578c0a4000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57a4102d00 tx=0x7f579800b410 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 msgr2=0x7f57a41980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f579400b580 tx=0x7f5794009580 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 shutdown_connections 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f578c0a1b40 0x7f578c0a4000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57a4103840 0x7f57a419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a4101ca0 0x7f57a4197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 --2- 172.21.15.116:0/1576743794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57a4102ea0 0x7f57a41980c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 >> 172.21.15.116:0/1576743794 conn(0x7f57a40fd280 msgr2=0x7f57a41062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:20.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 shutdown_connections 2024-03-12T05:07:20.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-12T05:07:20.650+0000 7f57a9d10700 1 -- 172.21.15.116:0/1576743794 wait complete. 2024-03-12T05:07:21.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:20 smithi130 ceph-mon[35398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:21.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:20 smithi116 ceph-mon[35486]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:21.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:21.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:21.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:20 smithi045 ceph-mon[32174]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:07:21.429 DEBUG:teuthology.orchestra.run.smithi116:osd.10> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.10.service 2024-03-12T05:07:21.431 INFO:tasks.cephadm:Deploying osd.11 on smithi130 with /dev/vg_nvme/lv_1... 2024-03-12T05:07:21.431 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:07:21.681 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:07:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:07:21.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:21.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:07:21.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:21.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:21 smithi116 ceph-mon[35486]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:22.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:22.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:07:22.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:22.129 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:07:22.129 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:22.129 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:21 smithi130 ceph-mon[35398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:22.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:22.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:07:22.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: from='osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:22.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:07:22.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:22.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:21 smithi045 ceph-mon[32174]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-12T05:07:22.917 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:07:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:07:22.656+0000 7f2e1dcc5700 -1 osd.10 0 waiting for initial osdmap 2024-03-12T05:07:22.918 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:07:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:07:22.672+0000 7f2e19ab8700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111691 s, 939 MB/s 2024-03-12T05:07:23.775 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-03-12T05:07:23.775 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:23 smithi130 ceph-mon[35398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:07:23.776 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:23 smithi130 ceph-mon[35398]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:07:23.776 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:23.776 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:23 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:23.776 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:23 smithi130 ceph-mon[35398]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:23 smithi116 ceph-mon[35486]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:07:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:23 smithi116 ceph-mon[35486]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:07:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:23 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:23 smithi116 ceph-mon[35486]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:24.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:23 smithi045 ceph-mon[32174]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-12T05:07:24.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:23 smithi045 ceph-mon[32174]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:07:24.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:24.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:23 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:24.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:23 smithi045 ceph-mon[32174]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:07:24.535 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2024-03-12T05:07:24.780 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] boot 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: osdmap e72: 11 total, 11 up, 11 in 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.832 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:24.833 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:24 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:07:24.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:07:24.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] boot 2024-03-12T05:07:24.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: osdmap e72: 11 total, 11 up, 11 in 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:24.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:24 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: osd.10 [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] boot 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: osdmap e72: 11 total, 11 up, 11 in 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:07:25.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:25.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:24 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:25.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:25 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi116 2024-03-12T05:07:25.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:25 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-12T05:07:25.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:25 smithi130 ceph-mon[35398]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:25.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:25 smithi130 ceph-mon[35398]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:07:25.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:25 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi116 2024-03-12T05:07:25.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:25 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-12T05:07:25.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:25 smithi116 ceph-mon[35486]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:25.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:25 smithi116 ceph-mon[35486]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:07:26.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:25 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi116 2024-03-12T05:07:26.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:25 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-12T05:07:26.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:25 smithi045 ceph-mon[32174]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:26.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:25 smithi045 ceph-mon[32174]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:07:26.642 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 -- 172.21.15.130:0/3998423411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 msgr2=0x7f44d81036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3998423411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d81036c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009990 tx=0x7f44cc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 -- 172.21.15.130:0/3998423411 shutdown_connections 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3998423411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d81006c0 0x7f44d810a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3998423411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d81036c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3998423411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d80ffd20 0x7f44d8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.641+0000 7f44ddac9700 1 -- 172.21.15.130:0/3998423411 >> 172.21.15.130:0/3998423411 conn(0x7f44d80fb8c0 msgr2=0x7f44d80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.642+0000 7f44ddac9700 1 -- 172.21.15.130:0/3998423411 shutdown_connections 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.642+0000 7f44ddac9700 1 -- 172.21.15.130:0/3998423411 wait complete. 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.642+0000 7f44ddac9700 1 Processor -- start 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.642+0000 7f44ddac9700 1 -- start start 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d80ffd20 0x7f44d8102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d81006c0 0x7f44d8100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:26.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8101cd0 con 0x7f44d81006c0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8101e40 con 0x7f44d80ffd20 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d81a1d70 con 0x7f44d81032a0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:50182/0 (socket says 172.21.15.130:50182) 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 -- 172.21.15.130:0/3593931781 learned_addr learned my addr 172.21.15.130:0/3593931781 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d6ffd700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d81006c0 0x7f44d8100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d77fe700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d80ffd20 0x7f44d8102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 -- 172.21.15.130:0/3593931781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d80ffd20 msgr2=0x7f44d8102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d80ffd20 0x7f44d8102950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 -- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d81006c0 msgr2=0x7f44d8100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d81006c0 0x7f44d8100ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 -- 172.21.15.130:0/3593931781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc0095f0 con 0x7f44d81032a0 2024-03-12T05:07:26.644 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d7fff700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d000b970 tx=0x7f44d000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:26.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000c930 con 0x7f44d81032a0 2024-03-12T05:07:26.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f44d0014440 con 0x7f44d81032a0 2024-03-12T05:07:26.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d81a1fa0 con 0x7f44d81032a0 2024-03-12T05:07:26.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.643+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00075d0 con 0x7f44d81032a0 2024-03-12T05:07:26.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.644+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81a2490 con 0x7f44d81032a0 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.644+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8109370 con 0x7f44d81032a0 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.645+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f44d00145b0 con 0x7f44d81032a0 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.646+0000 7f44d4ff9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 0x7f44c00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.646+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f44d0006e40 con 0x7f44d81032a0 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.646+0000 7f44d77fe700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 0x7f44c00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:26.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.647+0000 7f44d77fe700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 0x7f44c00a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44cc003820 tx=0x7f44cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:26.649 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.650+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44d0095cc0 con 0x7f44d81032a0 2024-03-12T05:07:26.848 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:26.849+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d802d870 con 0x7f44c00a1ac0 2024-03-12T05:07:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:07:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:27.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:27.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:27.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:27 smithi116 ceph-mon[35486]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:27.570 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:27 smithi130 ceph-mon[35398]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:27.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:27 smithi045 ceph-mon[32174]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:28.436 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:28 smithi130 ceph-mon[35398]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:07:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:28 smithi116 ceph-mon[35486]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:07:28.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:28 smithi045 ceph-mon[32174]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:07:29.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:29 smithi045 ceph-mon[32174]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:29.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:29 smithi130 ceph-mon[35398]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:29.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:29 smithi116 ceph-mon[35486]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/4098958159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]': finished 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: osdmap e77: 12 total, 11 up, 12 in 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:30.622 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.130:0/416001064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:30.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/4098958159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]': finished 2024-03-12T05:07:30.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: osdmap e77: 12 total, 11 up, 12 in 2024-03-12T05:07:30.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:30.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.130:0/416001064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:30.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/4098958159' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]: dispatch 2024-03-12T05:07:30.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57499d1f-cdfc-473e-b09c-9c906168826d"}]': finished 2024-03-12T05:07:30.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: osdmap e77: 12 total, 11 up, 12 in 2024-03-12T05:07:30.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:30.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.130:0/416001064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:07:31.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:31 smithi045 ceph-mon[32174]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-12T05:07:31.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:31 smithi130 ceph-mon[35398]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-12T05:07:31.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:31 smithi116 ceph-mon[35486]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-12T05:07:33.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:33 smithi045 ceph-mon[32174]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-12T05:07:33.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:33 smithi130 ceph-mon[35398]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-12T05:07:33.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:33 smithi116 ceph-mon[35486]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-12T05:07:35.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:35 smithi130 ceph-mon[35398]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:07:35.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:35 smithi116 ceph-mon[35486]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:07:35.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:35 smithi045 ceph-mon[32174]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:07:37.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:37 smithi130 ceph-mon[35398]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-12T05:07:37.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:37 smithi116 ceph-mon[35486]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-12T05:07:37.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:37 smithi045 ceph-mon[32174]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-12T05:07:38.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:07:38.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:38.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:38 smithi130 ceph-mon[35398]: Deploying daemon osd.11 on smithi130 2024-03-12T05:07:38.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:07:38.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:38.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:38 smithi116 ceph-mon[35486]: Deploying daemon osd.11 on smithi130 2024-03-12T05:07:38.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:07:38.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:38.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:38 smithi045 ceph-mon[32174]: Deploying daemon osd.11 on smithi130 2024-03-12T05:07:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:39 smithi130 ceph-mon[35398]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:07:39.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:39 smithi116 ceph-mon[35486]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:07:39.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:39 smithi045 ceph-mon[32174]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:07:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:41 smithi130 ceph-mon[35398]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:41.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:41 smithi116 ceph-mon[35486]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:41.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:41 smithi045 ceph-mon[32174]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:44.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:43 smithi130 ceph-mon[35398]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:43 smithi116 ceph-mon[35486]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:44.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:44.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:44.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:43 smithi045 ceph-mon[32174]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:45.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:45 smithi130 ceph-mon[35398]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:45.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:45 smithi116 ceph-mon[35486]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:45 smithi045 ceph-mon[32174]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:46.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.823+0000 7f44d4ff9700 1 -- 172.21.15.130:0/3593931781 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f44d802d870 con 0x7f44c00a1ac0 2024-03-12T05:07:46.822 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 11 on host 'smithi130' 2024-03-12T05:07:46.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 msgr2=0x7f44c00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:46.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 0x7f44c00a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44cc003820 tx=0x7f44cc005fb0 comp rx=0 tx=0).stop 2024-03-12T05:07:46.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 msgr2=0x7f44d8101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:46.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44d000b970 tx=0x7f44d000bc80 comp rx=0 tx=0).stop 2024-03-12T05:07:46.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 shutdown_connections 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f44c00a1ac0 0x7f44c00a3f80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44d81032a0 0x7f44d8101530 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d80ffd20 0x7f44d8102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 --2- 172.21.15.130:0/3593931781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44d81006c0 0x7f44d8100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.825+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 >> 172.21.15.130:0/3593931781 conn(0x7f44d80fb8c0 msgr2=0x7f44d81064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.826+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 shutdown_connections 2024-03-12T05:07:46.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:46.826+0000 7f44ddac9700 1 -- 172.21.15.130:0/3593931781 wait complete. 2024-03-12T05:07:47.127 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.127 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.128 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:46 smithi130 ceph-mon[35398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:47.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:46 smithi116 ceph-mon[35486]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:47.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:47.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:46 smithi045 ceph-mon[32174]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:07:47.547 DEBUG:teuthology.orchestra.run.smithi130:osd.11> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.11.service 2024-03-12T05:07:47.550 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-12T05:07:47.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T05:07:47.805 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:07:48.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:48.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:07:48.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:07:48.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:48.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:48.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:47 smithi130 ceph-mon[35398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:48.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:47 smithi116 ceph-mon[35486]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: from='osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:48.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:47 smithi045 ceph-mon[32174]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-03-12T05:07:49.056 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:07:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:07:48.790+0000 7fe727d16700 -1 osd.11 0 waiting for initial osdmap 2024-03-12T05:07:49.056 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:07:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:07:48.807+0000 7fe723b09700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:07:49.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 -- 172.21.15.45:0/837997117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 msgr2=0x7f0b6810a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:49.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 --2- 172.21.15.45:0/837997117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b6810a0f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000b0d0 tx=0x7f0b6000b3e0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 -- 172.21.15.45:0/837997117 shutdown_connections 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 --2- 172.21.15.45:0/837997117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b6810a0f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 --2- 172.21.15.45:0/837997117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b681020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 --2- 172.21.15.45:0/837997117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 0x7f0b68103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 -- 172.21.15.45:0/837997117 >> 172.21.15.45:0/837997117 conn(0x7f0b680fd280 msgr2=0x7f0b680ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 -- 172.21.15.45:0/837997117 shutdown_connections 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.715+0000 7f0b6f074700 1 -- 172.21.15.45:0/837997117 wait complete. 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 Processor -- start 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 -- start start 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b68075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 0x7f0b68074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b680745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68074d20 con 0x7f0b68103840 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68074e90 con 0x7f0b68102ea0 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6f074700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b681a1cd0 con 0x7f0b68101ca0 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6ce10700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b68075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6d611700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b680745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6ce10700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b68075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47402/0 (socket says 172.21.15.45:47402) 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b6ce10700 1 -- 172.21.15.45:0/167682648 learned_addr learned my addr 172.21.15.45:0/167682648 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.716+0000 7f0b67fff700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 0x7f0b68074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 -- 172.21.15.45:0/167682648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 msgr2=0x7f0b68075a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b68075a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 -- 172.21.15.45:0/167682648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 msgr2=0x7f0b68074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 0x7f0b68074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 -- 172.21.15.45:0/167682648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60009d00 con 0x7f0b68103840 2024-03-12T05:07:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6d611700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b680745a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b60007a90 tx=0x7f0b60009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:49.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6000e040 con 0x7f0b68103840 2024-03-12T05:07:49.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b681a1f50 con 0x7f0b68103840 2024-03-12T05:07:49.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0b60003e10 con 0x7f0b68103840 2024-03-12T05:07:49.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6001d9c0 con 0x7f0b68103840 2024-03-12T05:07:49.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.717+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b681a2360 con 0x7f0b68103840 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.718+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b68109060 con 0x7f0b68103840 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.719+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0b6001db20 con 0x7f0b68103840 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.720+0000 7f0b65ffb700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 0x7f0b580a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.720+0000 7f0b6ce10700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 0x7f0b580a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.720+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f0b600ce400 con 0x7f0b68103840 2024-03-12T05:07:49.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.720+0000 7f0b6ce10700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 0x7f0b580a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00bd70 tx=0x7f0b5c00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:49.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.724+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b60097cf0 con 0x7f0b68103840 2024-03-12T05:07:49.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.928+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0b6810d4e0 con 0x7f0b68103840 2024-03-12T05:07:49.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.928+0000 7f0b65ffb700 1 -- 172.21.15.45:0/167682648 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f0b6009b3b0 con 0x7f0b68103840 2024-03-12T05:07:49.927 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:07:49.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 msgr2=0x7f0b580a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:49.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 0x7f0b580a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00bd70 tx=0x7f0b5c00b4a0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 msgr2=0x7f0b680745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:49.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b680745a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0b60007a90 tx=0x7f0b60009380 comp rx=0 tx=0).stop 2024-03-12T05:07:49.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 shutdown_connections 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0b580a1ac0 0x7f0b580a3f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b68103840 0x7f0b680745a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b68101ca0 0x7f0b68075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 --2- 172.21.15.45:0/167682648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68102ea0 0x7f0b68074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 >> 172.21.15.45:0/167682648 conn(0x7f0b680fd280 msgr2=0x7f0b681061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 shutdown_connections 2024-03-12T05:07:49.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:49.930+0000 7f0b6f074700 1 -- 172.21.15.45:0/167682648 wait complete. 2024-03-12T05:07:50.046 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:49 smithi130 ceph-mon[35398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:07:50.046 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:49 smithi130 ceph-mon[35398]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:07:50.046 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:49 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.046 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:49 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.046 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:49 smithi130 ceph-mon[35398]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:50.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:49 smithi045 ceph-mon[32174]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:07:50.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:49 smithi045 ceph-mon[32174]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:07:50.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:49 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:49 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:49 smithi045 ceph-mon[32174]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:50.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:49 smithi116 ceph-mon[35486]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-03-12T05:07:50.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:49 smithi116 ceph-mon[35486]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:07:50.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:49 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:49 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:49 smithi116 ceph-mon[35486]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:07:50.622 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710220069,"num_in_osds":12,"osd_in_since":1710220049,"num_remapped_pgs":0} 2024-03-12T05:07:50.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:07:50.885 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:07:50.925 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: purged_snaps scrub starts 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: purged_snaps scrub ok 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] boot 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: osdmap e80: 12 total, 12 up, 12 in 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/167682648' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.927 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.927 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.927 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:50.927 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:50.927 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:50 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: purged_snaps scrub starts 2024-03-12T05:07:50.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: purged_snaps scrub ok 2024-03-12T05:07:50.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] boot 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: osdmap e80: 12 total, 12 up, 12 in 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/167682648' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:50.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:50 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: purged_snaps scrub starts 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: purged_snaps scrub ok 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: osd.11 [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] boot 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: osdmap e80: 12 total, 12 up, 12 in 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:07:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/167682648' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:51.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:50 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:52.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:51 smithi045 ceph-mon[32174]: Detected new or changed devices on smithi130 2024-03-12T05:07:52.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:51 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 4272M 2024-03-12T05:07:52.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:51 smithi045 ceph-mon[32174]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:07:52.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:51 smithi045 ceph-mon[32174]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:52.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:51 smithi130 ceph-mon[35398]: Detected new or changed devices on smithi130 2024-03-12T05:07:52.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:51 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 4272M 2024-03-12T05:07:52.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:51 smithi130 ceph-mon[35398]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:07:52.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:51 smithi130 ceph-mon[35398]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:52.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:51 smithi116 ceph-mon[35486]: Detected new or changed devices on smithi130 2024-03-12T05:07:52.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:51 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 4272M 2024-03-12T05:07:52.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:51 smithi116 ceph-mon[35486]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:07:52.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:51 smithi116 ceph-mon[35486]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:52.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.688+0000 7f554bc5c700 1 -- 172.21.15.45:0/4267392393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544101cc0 msgr2=0x7f5544102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:52.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.688+0000 7f554bc5c700 1 --2- 172.21.15.45:0/4267392393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544101cc0 0x7f5544102140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f55380099c0 tx=0x7f5538009cd0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 -- 172.21.15.45:0/4267392393 shutdown_connections 2024-03-12T05:07:52.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- 172.21.15.45:0/4267392393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f554410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- 172.21.15.45:0/4267392393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544107ca0 0x7f5544108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- 172.21.15.45:0/4267392393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544101cc0 0x7f5544102140 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 -- 172.21.15.45:0/4267392393 >> 172.21.15.45:0/4267392393 conn(0x7f5544078da0 msgr2=0x7f55440791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 -- 172.21.15.45:0/4267392393 shutdown_connections 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 -- 172.21.15.45:0/4267392393 wait complete. 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 Processor -- start 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 -- start start 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544101cc0 0x7f5544198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f55441985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.689+0000 7f554bc5c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554bc5c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554419cf60 con 0x7f5544107ca0 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554bc5c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554419d0d0 con 0x7f5544102680 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554bc5c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554419d290 con 0x7f5544101cc0 2024-03-12T05:07:52.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43746/0 (socket says 172.21.15.45:43746) 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 -- 172.21.15.45:0/3771328331 learned_addr learned my addr 172.21.15.45:0/3771328331 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f55499f8700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544101cc0 0x7f5544198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f55491f7700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f55441985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 -- 172.21.15.45:0/3771328331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544101cc0 msgr2=0x7f5544198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544101cc0 0x7f5544198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 -- 172.21.15.45:0/3771328331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 msgr2=0x7f55441985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f55441985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 -- 172.21.15.45:0/3771328331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55380095f0 con 0x7f5544107ca0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f55491f7700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f55441985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554a1f9700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c900 tx=0x7f553c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c007b00 con 0x7f5544107ca0 2024-03-12T05:07:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f553c00ce50 con 0x7f5544107ca0 2024-03-12T05:07:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554419d570 con 0x7f5544107ca0 2024-03-12T05:07:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c018460 con 0x7f5544107ca0 2024-03-12T05:07:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.690+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55441a4dd0 con 0x7f5544107ca0 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.691+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554410a180 con 0x7f5544107ca0 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.692+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f553c00f450 con 0x7f5544107ca0 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.693+0000 7f5542ffd700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 0x7f552c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.693+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f553c0cd040 con 0x7f5544107ca0 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.693+0000 7f55499f8700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 0x7f552c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.693+0000 7f55499f8700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 0x7f552c0a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f553400a400 tx=0x7f553400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:52.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.697+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f553c0966f0 con 0x7f5544107ca0 2024-03-12T05:07:52.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.902+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5544068930 con 0x7f5544107ca0 2024-03-12T05:07:52.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.903+0000 7f5542ffd700 1 -- 172.21.15.45:0/3771328331 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18421 (secure 0 0 0) 0x7f553c099db0 con 0x7f5544107ca0 2024-03-12T05:07:52.902 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:07:52.902 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:07:50.788069+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 msgr2=0x7f552c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 0x7f552c0a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f553400a400 tx=0x7f553400d040 comp rx=0 tx=0).stop 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 msgr2=0x7f554419c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f553c00c900 tx=0x7f553c00cc10 comp rx=0 tx=0).stop 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 shutdown_connections 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f552c0a1ac0 0x7f552c0a3f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5544107ca0 0x7f554419c950 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5544101cc0 0x7f5544198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 --2- 172.21.15.45:0/3771328331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5544102680 0x7f55441985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:52.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 >> 172.21.15.45:0/3771328331 conn(0x7f5544078da0 msgr2=0x7f5544100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:52.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 shutdown_connections 2024-03-12T05:07:52.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:52.905+0000 7f554bc5c700 1 -- 172.21.15.45:0/3771328331 wait complete. 2024-03-12T05:07:53.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:52 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3771328331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:07:53.223 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:52 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3771328331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:07:53.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:52 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3771328331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:07:53.620 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:07:53.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T05:07:53.863 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:07:54.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:53 smithi130 ceph-mon[35398]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:54.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:53 smithi116 ceph-mon[35486]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:54.311 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:53 smithi045 ceph-mon[32174]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:55.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:55 smithi045 ceph-mon[32174]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:55.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:55 smithi130 ceph-mon[35398]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 -- 172.21.15.45:0/2761520873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0102680 msgr2=0x7f64a010eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 --2- 172.21.15.45:0/2761520873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0102680 0x7f64a010eac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f649000b0d0 tx=0x7f649000b3e0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 -- 172.21.15.45:0/2761520873 shutdown_connections 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 --2- 172.21.15.45:0/2761520873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0102680 0x7f64a010eac0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 --2- 172.21.15.45:0/2761520873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0107ca0 0x7f64a0108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 --2- 172.21.15.45:0/2761520873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0101cc0 0x7f64a0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 -- 172.21.15.45:0/2761520873 >> 172.21.15.45:0/2761520873 conn(0x7f64a0078da0 msgr2=0x7f64a00791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 -- 172.21.15.45:0/2761520873 shutdown_connections 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 -- 172.21.15.45:0/2761520873 wait complete. 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.491+0000 7f64a54e6700 1 Processor -- start 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 -- start start 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 0x7f64a019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:55.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 0x7f64a019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019f300 con 0x7f64a0107ca0 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019f470 con 0x7f64a0102680 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f64a54e6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019f630 con 0x7f64a0101cc0 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43766/0 (socket says 172.21.15.45:43766) 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 -- 172.21.15.45:0/4132210730 learned_addr learned my addr 172.21.15.45:0/4132210730 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649effd700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 0x7f64a019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:55.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649e7fc700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 0x7f64a019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:55.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 -- 172.21.15.45:0/4132210730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 msgr2=0x7f64a019a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:55.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 0x7f64a019a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 -- 172.21.15.45:0/4132210730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 msgr2=0x7f64a019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:55.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 0x7f64a019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.492+0000 7f649f7fe700 1 -- 172.21.15.45:0/4132210730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490009d00 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f649f7fe700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f649000d010 tx=0x7f64900079e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f649effd700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 0x7f64a019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f649e7fc700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 0x7f64a019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649000e040 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a019f8b0 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6490003c00 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649001b690 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.493+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a01a7170 con 0x7f64a0107ca0 2024-03-12T05:07:55.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.495+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0068930 con 0x7f64a0107ca0 2024-03-12T05:07:55.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:55 smithi116 ceph-mon[35486]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:55.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.495+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6490005070 con 0x7f64a0107ca0 2024-03-12T05:07:55.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.496+0000 7f6497fff700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 0x7f648c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:55.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.496+0000 7f649effd700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 0x7f648c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:55.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.496+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64900ce1d0 con 0x7f64a0107ca0 2024-03-12T05:07:55.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.496+0000 7f649effd700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 0x7f648c0a4000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6498005e10 tx=0x7f6498005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:55.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.501+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6490097910 con 0x7f64a0107ca0 2024-03-12T05:07:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.702+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64a0111bb0 con 0x7f64a0107ca0 2024-03-12T05:07:55.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.702+0000 7f6497fff700 1 -- 172.21.15.45:0/4132210730 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f649009afd0 con 0x7f64a0107ca0 2024-03-12T05:07:55.701 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 msgr2=0x7f648c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 0x7f648c0a4000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6498005e10 tx=0x7f6498005d60 comp rx=0 tx=0).stop 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 msgr2=0x7f64a019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f649000d010 tx=0x7f64900079e0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 shutdown_connections 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f648c0a1b40 0x7f648c0a4000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64a0107ca0 0x7f64a019ecf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64a0101cc0 0x7f64a019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 --2- 172.21.15.45:0/4132210730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64a0102680 0x7f64a019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:55.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.704+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 >> 172.21.15.45:0/4132210730 conn(0x7f64a0078da0 msgr2=0x7f64a0100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:55.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.705+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 shutdown_connections 2024-03-12T05:07:55.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:55.705+0000 7f64a54e6700 1 -- 172.21.15.45:0/4132210730 wait complete. 2024-03-12T05:07:56.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:56 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4132210730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.a on smithi045 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.d on smithi045 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.g on smithi045 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.j on smithi045 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.b on smithi116 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.e on smithi116 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.h on smithi116 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.k on smithi116 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.c on smithi130 2024-03-12T05:07:56.409 INFO:tasks.cephadm:Adding mds.f on smithi130 2024-03-12T05:07:56.410 INFO:tasks.cephadm:Adding mds.i on smithi130 2024-03-12T05:07:56.410 INFO:tasks.cephadm:Adding mds.l on smithi130 2024-03-12T05:07:56.410 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph orch apply mds all '12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l' 2024-03-12T05:07:56.441 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:56 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4132210730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:07:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:56 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4132210730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:07:56.661 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:07:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:57 smithi130 ceph-mon[35398]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:57 smithi116 ceph-mon[35486]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:57 smithi045 ceph-mon[32174]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:58.428 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.426+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/2433544172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 msgr2=0x7fe7ec103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:58.428 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.426+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/2433544172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec103300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009990 tx=0x7fe7dc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.428 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/2433544172 shutdown_connections 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/2433544172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/2433544172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 0x7fe7ec1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/2433544172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec103300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/2433544172 >> 172.21.15.130:0/2433544172 conn(0x7fe7ec0fd280 msgr2=0x7fe7ec0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/2433544172 shutdown_connections 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/2433544172 wait complete. 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 Processor -- start 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.427+0000 7fe7f3ad1700 1 -- start start 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 0x7fe7ec197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec19f700 con 0x7fe7ec103840 2024-03-12T05:07:58.429 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec19f840 con 0x7fe7ec101ca0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec19f9d0 con 0x7fe7ec102ea0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:46496/0 (socket says 172.21.15.130:46496) 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 -- 172.21.15.130:0/1773446820 learned_addr learned my addr 172.21.15.130:0/1773446820 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f186d700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 0x7fe7ec197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f206e700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 -- 172.21.15.130:0/1773446820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 msgr2=0x7fe7ec197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 0x7fe7ec197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 -- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 msgr2=0x7fe7ec19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 -- 172.21.15.130:0/1773446820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0095f0 con 0x7fe7ec102ea0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f206e700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f106c700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000c00 tx=0x7fe7dc009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:58.430 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc01d070 con 0x7fe7ec102ea0 2024-03-12T05:07:58.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe7dc0047e0 con 0x7fe7ec102ea0 2024-03-12T05:07:58.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec073f60 con 0x7fe7ec102ea0 2024-03-12T05:07:58.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0053b0 con 0x7fe7ec102ea0 2024-03-12T05:07:58.431 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.428+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec074400 con 0x7fe7ec102ea0 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.429+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec109180 con 0x7fe7ec102ea0 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.430+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe7dc004950 con 0x7fe7ec102ea0 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.431+0000 7fe7e2ffd700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 0x7fe7d80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.431+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe7dc0ce4e0 con 0x7fe7ec102ea0 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.431+0000 7fe7f186d700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 0x7fe7d80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:58.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.432+0000 7fe7f186d700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 0x7fe7d80a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8005fd0 tx=0x7fe7e8005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:58.434 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.434+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe7dc09b750 con 0x7fe7ec102ea0 2024-03-12T05:07:58.635 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.635+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7ec02d8d0 con 0x7fe7d80a1ac0 2024-03-12T05:07:58.643 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.643+0000 7fe7e2ffd700 1 -- 172.21.15.130:0/1773446820 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe7ec02d8d0 con 0x7fe7d80a1ac0 2024-03-12T05:07:58.643 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mds.all update... 2024-03-12T05:07:58.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.645+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 msgr2=0x7fe7d80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:58.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.645+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 0x7fe7d80a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8005fd0 tx=0x7fe7e8005dd0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.645 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 msgr2=0x7fe7ec1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc000c00 tx=0x7fe7dc009ee0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 shutdown_connections 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe7d80a1ac0 0x7fe7d80a3f80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec103840 0x7fe7ec19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7ec101ca0 0x7fe7ec197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 --2- 172.21.15.130:0/1773446820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7ec102ea0 0x7fe7ec1980b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 >> 172.21.15.130:0/1773446820 conn(0x7fe7ec0fd280 msgr2=0x7fe7ec1062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 shutdown_connections 2024-03-12T05:07:58.646 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-12T05:07:58.646+0000 7fe7f3ad1700 1 -- 172.21.15.130:0/1773446820 wait complete. 2024-03-12T05:07:59.324 DEBUG:teuthology.orchestra.run.smithi045:mds.a> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.a.service 2024-03-12T05:07:59.327 DEBUG:teuthology.orchestra.run.smithi045:mds.d> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.d.service 2024-03-12T05:07:59.369 DEBUG:teuthology.orchestra.run.smithi045:mds.g> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.g.service 2024-03-12T05:07:59.414 DEBUG:teuthology.orchestra.run.smithi045:mds.j> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.j.service 2024-03-12T05:07:59.457 DEBUG:teuthology.orchestra.run.smithi116:mds.b> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.b.service 2024-03-12T05:07:59.460 DEBUG:teuthology.orchestra.run.smithi116:mds.e> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.e.service 2024-03-12T05:07:59.503 DEBUG:teuthology.orchestra.run.smithi116:mds.h> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.h.service 2024-03-12T05:07:59.545 DEBUG:teuthology.orchestra.run.smithi116:mds.k> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.k.service 2024-03-12T05:07:59.588 DEBUG:teuthology.orchestra.run.smithi130:mds.c> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.c.service 2024-03-12T05:07:59.591 DEBUG:teuthology.orchestra.run.smithi130:mds.f> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.f.service 2024-03-12T05:07:59.633 DEBUG:teuthology.orchestra.run.smithi130:mds.i> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.i.service 2024-03-12T05:07:59.676 DEBUG:teuthology.orchestra.run.smithi130:mds.l> sudo journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.l.service 2024-03-12T05:07:59.720 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-12T05:07:59.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:59.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:07:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:07:59.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:07:59 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.833+0000 7fa777f93640 1 -- 172.21.15.45:0/1622000448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa770101ca0 msgr2=0x7fa770104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.833+0000 7fa777f93640 1 --2- 172.21.15.45:0/1622000448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa770101ca0 0x7fa770104090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009900 tx=0x7fa76c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.833+0000 7fa777f93640 1 -- 172.21.15.45:0/1622000448 shutdown_connections 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.833+0000 7fa777f93640 1 --2- 172.21.15.45:0/1622000448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 0x7fa770109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.833+0000 7fa777f93640 1 --2- 172.21.15.45:0/1622000448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7701045d0 0x7fa7701069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 --2- 172.21.15.45:0/1622000448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa770101ca0 0x7fa770104090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- 172.21.15.45:0/1622000448 >> 172.21.15.45:0/1622000448 conn(0x7fa7700fb6b0 msgr2=0x7fa7700fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- 172.21.15.45:0/1622000448 shutdown_connections 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- 172.21.15.45:0/1622000448 wait complete. 2024-03-12T05:07:59.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 Processor -- start 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- start start 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 0x7fa770101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 0x7fa770100190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701008f0 con 0x7fa7701045d0 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa770100a60 con 0x7fa770101ca0 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa777f93640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7701a11e0 con 0x7fa770106f00 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.834+0000 7fa77552e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52316/0 (socket says 172.21.15.45:52316) 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 -- 172.21.15.45:0/3961489451 learned_addr learned my addr 172.21.15.45:0/3961489451 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa776530640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 0x7fa770100190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa775d2f640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 0x7fa770101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:59.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 -- 172.21.15.45:0/3961489451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 msgr2=0x7fa770100190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 0x7fa770100190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 -- 172.21.15.45:0/3961489451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 msgr2=0x7fa770101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 0x7fa770101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 -- 172.21.15.45:0/3961489451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009560 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa775d2f640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 0x7fa770101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa77552e640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa76000e9f0 tx=0x7fa76000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760009700 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa76000cd80 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760010430 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7701a1490 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.835+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7701a19f0 con 0x7fa7701045d0 2024-03-12T05:07:59.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.836+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7700fd1a0 con 0x7fa7701045d0 2024-03-12T05:07:59.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.837+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa760005000 con 0x7fa7701045d0 2024-03-12T05:07:59.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.838+0000 7fa766ffd640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 0x7fa75c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:07:59.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.838+0000 7fa775d2f640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 0x7fa75c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:07:59.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.838+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa760014070 con 0x7fa7701045d0 2024-03-12T05:07:59.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.838+0000 7fa775d2f640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 0x7fa75c0a3f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa76c036040 tx=0x7fa76c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:07:59.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:07:59.842+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa760096950 con 0x7fa7701045d0 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:07:59.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:07:59 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:00.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.022+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa77010ca70 con 0x7fa7701045d0 2024-03-12T05:08:00.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.023+0000 7fa766ffd640 1 -- 172.21.15.45:0/3961489451 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18421 (secure 0 0 0) 0x7fa76009a010 con 0x7fa7701045d0 2024-03-12T05:08:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:00.022 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:07:50.788069+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 msgr2=0x7fa75c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 0x7fa75c0a3f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa76c036040 tx=0x7fa76c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 msgr2=0x7fa7700ffc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa76000e9f0 tx=0x7fa76000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 shutdown_connections 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa75c0a1ab0 0x7fa75c0a3f70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7701045d0 0x7fa7700ffc50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa770106f00 0x7fa770100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 --2- 172.21.15.45:0/3961489451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa770101ca0 0x7fa770101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 >> 172.21.15.45:0/3961489451 conn(0x7fa7700fb6b0 msgr2=0x7fa770102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 shutdown_connections 2024-03-12T05:08:00.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.025+0000 7fa777f93640 1 -- 172.21.15.45:0/3961489451 wait complete. 2024-03-12T05:08:00.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:00.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: Saving service mds.all spec with placement smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-03-12T05:08:00.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:00.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:07:59 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:00.035 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:08:00.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 -- 172.21.15.45:0/1652302843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06501000a0 msgr2=0x7f06501004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 --2- 172.21.15.45:0/1652302843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06501000a0 0x7f06501004c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0640009900 tx=0x7f064002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 -- 172.21.15.45:0/1652302843 shutdown_connections 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 --2- 172.21.15.45:0/1652302843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0650101c50 0x7f06501084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 --2- 172.21.15.45:0/1652302843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101290 0x7f0650101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 --2- 172.21.15.45:0/1652302843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06501000a0 0x7f06501004c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 -- 172.21.15.45:0/1652302843 >> 172.21.15.45:0/1652302843 conn(0x7f06500fb630 msgr2=0x7f06500fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 -- 172.21.15.45:0/1652302843 shutdown_connections 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.142+0000 7f0655b77640 1 -- 172.21.15.45:0/1652302843 wait complete. 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 Processor -- start 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 -- start start 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06501000a0 0x7f06501971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101c50 0x7f065019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065019ec90 con 0x7f0650101290 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065019ee00 con 0x7f06501000a0 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f0655b77640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065019efc0 con 0x7f0650101c50 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f064effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f064effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52326/0 (socket says 172.21.15.45:52326) 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f064effd640 1 -- 172.21.15.45:0/1582965738 learned_addr learned my addr 172.21.15.45:0/1582965738 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.143+0000 7f064f7fe640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06501000a0 0x7f06501971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064ffff640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101c50 0x7f065019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 -- 172.21.15.45:0/1582965738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101c50 msgr2=0x7f065019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101c50 0x7f065019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 -- 172.21.15.45:0/1582965738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06501000a0 msgr2=0x7f06501971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06501000a0 0x7f06501971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 -- 172.21.15.45:0/1582965738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640009560 con 0x7f0650101290 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064effd640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f064400e9f0 tx=0x7f064400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644009720 con 0x7f0650101290 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065019f2a0 con 0x7f0650101290 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f064400cd80 con 0x7f0650101290 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644010430 con 0x7f0650101290 2024-03-12T05:08:00.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.144+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065019f7b0 con 0x7f0650101290 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.145+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065010b960 con 0x7f0650101290 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.146+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0644005020 con 0x7f0650101290 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.146+0000 7f064cff9640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 0x7f06380a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.147+0000 7f064f7fe640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 0x7f06380a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.147+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0644014070 con 0x7f0650101290 2024-03-12T05:08:00.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.147+0000 7f064f7fe640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 0x7f06380a3ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f064002c7d0 tx=0x7f0640005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:00.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.151+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0644096970 con 0x7f0650101290 2024-03-12T05:08:00.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.331+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f065019fc40 con 0x7f0650101290 2024-03-12T05:08:00.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.331+0000 7f064cff9640 1 -- 172.21.15.45:0/1582965738 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f064409a030 con 0x7f0650101290 2024-03-12T05:08:00.330 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 msgr2=0x7f06380a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 0x7f06380a3ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f064002c7d0 tx=0x7f0640005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 msgr2=0x7f0650197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f064400e9f0 tx=0x7f064400edf0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 shutdown_connections 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f06380a1a30 0x7f06380a3ef0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0650101290 0x7f0650197730 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0650101c50 0x7f065019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 --2- 172.21.15.45:0/1582965738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06501000a0 0x7f06501971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 >> 172.21.15.45:0/1582965738 conn(0x7f06500fb630 msgr2=0x7f0650069780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 shutdown_connections 2024-03-12T05:08:00.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.333+0000 7f0655b77640 1 -- 172.21.15.45:0/1582965738 wait complete. 2024-03-12T05:08:00.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:00.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 -- 172.21.15.45:0/3391115022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741011b0 msgr2=0x7fe5741015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 --2- 172.21.15.45:0/3391115022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741011b0 0x7fe5741015d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe564009900 tx=0x7fe56402bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:00.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 -- 172.21.15.45:0/3391115022 shutdown_connections 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 --2- 172.21.15.45:0/3391115022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 0x7fe5741095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 --2- 172.21.15.45:0/3391115022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741023a0 0x7fe574102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 --2- 172.21.15.45:0/3391115022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741011b0 0x7fe5741015d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 -- 172.21.15.45:0/3391115022 >> 172.21.15.45:0/3391115022 conn(0x7fe5740fc760 msgr2=0x7fe5740feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 -- 172.21.15.45:0/3391115022 shutdown_connections 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.446+0000 7fe578993640 1 -- 172.21.15.45:0/3391115022 wait complete. 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 Processor -- start 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 -- start start 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741011b0 0x7fe574197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 0x7fe57419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57419ec70 con 0x7fe5741023a0 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57419ede0 con 0x7fe574102d60 2024-03-12T05:08:00.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe578993640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57419efa0 con 0x7fe5741011b0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe56bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52346/0 (socket says 172.21.15.45:52346) 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe56bfff640 1 -- 172.21.15.45:0/1094121311 learned_addr learned my addr 172.21.15.45:0/1094121311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.447+0000 7fe57259c640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741011b0 0x7fe574197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe572d9d640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 0x7fe57419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 -- 172.21.15.45:0/1094121311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741011b0 msgr2=0x7fe574197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741011b0 0x7fe574197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 -- 172.21.15.45:0/1094121311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 msgr2=0x7fe57419e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 0x7fe57419e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 -- 172.21.15.45:0/1094121311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009560 con 0x7fe5741023a0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56bfff640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e9f0 tx=0x7fe55c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c009720 con 0x7fe5741023a0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe55c00cd80 con 0x7fe5741023a0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c010430 con 0x7fe5741023a0 2024-03-12T05:08:00.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57419f280 con 0x7fe5741023a0 2024-03-12T05:08:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.448+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57419f740 con 0x7fe5741023a0 2024-03-12T05:08:00.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.449+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574108540 con 0x7fe5741023a0 2024-03-12T05:08:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.450+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe55c005000 con 0x7fe5741023a0 2024-03-12T05:08:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.451+0000 7fe56b7fe640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 0x7fe5600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.451+0000 7fe57259c640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 0x7fe5600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.451+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe55c014070 con 0x7fe5741023a0 2024-03-12T05:08:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.451+0000 7fe57259c640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 0x7fe5600a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe56402c7d0 tx=0x7fe564005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:00.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.454+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe55c096c50 con 0x7fe5741023a0 2024-03-12T05:08:00.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.674+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57410ccc0 con 0x7fe5741023a0 2024-03-12T05:08:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.674+0000 7fe56b7fe640 1 -- 172.21.15.45:0/1094121311 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fe55c09a310 con 0x7fe5741023a0 2024-03-12T05:08:00.673 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:00.673 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-12T05:08:00.673 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 1 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 msgr2=0x7fe5600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 0x7fe5600a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe56402c7d0 tx=0x7fe564005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 msgr2=0x7fe5741976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e9f0 tx=0x7fe55c00edf0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 shutdown_connections 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5600a1a30 0x7fe5600a3ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5741023a0 0x7fe5741976c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102d60 0x7fe57419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 --2- 172.21.15.45:0/1094121311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5741011b0 0x7fe574197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 >> 172.21.15.45:0/1094121311 conn(0x7fe5740fc760 msgr2=0x7fe574105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 shutdown_connections 2024-03-12T05:08:00.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:00.676+0000 7fe578993640 1 -- 172.21.15.45:0/1094121311 wait complete. 2024-03-12T05:08:00.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:00 smithi130 ceph-mon[35398]: Deploying daemon mds.l on smithi130 2024-03-12T05:08:00.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:00 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3961489451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:00.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:00 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1582965738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:00.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:00 smithi116 ceph-mon[35486]: Deploying daemon mds.l on smithi130 2024-03-12T05:08:00.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:00 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3961489451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:00.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:00 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1582965738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:01.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:00 smithi045 ceph-mon[32174]: Deploying daemon mds.l on smithi130 2024-03-12T05:08:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:00 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3961489451' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:01.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:00 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1582965738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:01.470 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 systemd[1]: Starting Ceph mds.l for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:01.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:01 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1094121311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:01.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:01 smithi130 ceph-mon[35398]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:01.853 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 podman[65442]: 2024-03-12 05:08:01.485000919 +0000 UTC m=+0.017691142 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:01.854 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 podman[65442]: 2024-03-12 05:08:01.593769962 +0000 UTC m=+0.126460223 container create e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:08:01.854 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 podman[65442]: 2024-03-12 05:08:01.855304279 +0000 UTC m=+0.387994494 container init e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:08:01.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:01 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1094121311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:01.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:01 smithi045 ceph-mon[32174]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:01.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:01 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1094121311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:01.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:01 smithi116 ceph-mon[35486]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:02.220 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 podman[65442]: 2024-03-12 05:08:01.858268108 +0000 UTC m=+0.390958331 container start e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-12T05:08:02.220 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 ceph-mds[65461]: -- 172.21.15.130:0/3077071836 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee54a661a0 con 0x55ee54a3cc00 2024-03-12T05:08:02.220 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: starting mds.l at 2024-03-12T05:08:02.221 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 bash[65442]: e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e 2024-03-12T05:08:02.221 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:08:01 smithi130 systemd[1]: Started Ceph mds.l for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:02.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:02.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 -- 172.21.15.45:0/435916235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102d50 msgr2=0x7f02801095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:02.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 --2- 172.21.15.45:0/435916235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102d50 0x7f02801095e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0270009ca0 tx=0x7f027002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 -- 172.21.15.45:0/435916235 shutdown_connections 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 --2- 172.21.15.45:0/435916235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102d50 0x7f02801095e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 --2- 172.21.15.45:0/435916235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0280102390 0x7f0280102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 --2- 172.21.15.45:0/435916235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02801011a0 0x7f02801015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 -- 172.21.15.45:0/435916235 >> 172.21.15.45:0/435916235 conn(0x7f02800fc730 msgr2=0x7f02800feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 -- 172.21.15.45:0/435916235 shutdown_connections 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 -- 172.21.15.45:0/435916235 wait complete. 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.802+0000 7f0287a55640 1 Processor -- start 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 -- start start 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02801011a0 0x7f02801970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0280102d50 0x7f028019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:02.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028019ebd0 con 0x7f0280102390 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028019ed40 con 0x7f02801011a0 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028019ef00 con 0x7f0280102d50 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52372/0 (socket says 172.21.15.45:52372) 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 -- 172.21.15.45:0/2951397660 learned_addr learned my addr 172.21.15.45:0/2951397660 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0285ff2640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0280102d50 0x7f028019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f02857f1640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02801011a0 0x7f02801970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 -- 172.21.15.45:0/2951397660 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0280102d50 msgr2=0x7f028019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0280102d50 0x7f028019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 -- 172.21.15.45:0/2951397660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02801011a0 msgr2=0x7f02801970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02801011a0 0x7f02801970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 -- 172.21.15.45:0/2951397660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270009900 con 0x7f0280102390 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0284ff0640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f027400c890 tx=0x7f027400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:02.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274004220 con 0x7f0280102390 2024-03-12T05:08:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0274007e60 con 0x7f0280102390 2024-03-12T05:08:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.803+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028019f1e0 con 0x7f0280102390 2024-03-12T05:08:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.804+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274004a00 con 0x7f0280102390 2024-03-12T05:08:02.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.804+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028019f6f0 con 0x7f0280102390 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.804+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0280108540 con 0x7f0280102390 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.806+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f02740079b0 con 0x7f0280102390 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.806+0000 7f027e7fc640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 0x7f02680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.806+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f02740cd040 con 0x7f0280102390 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.806+0000 7f02857f1640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 0x7f02680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:02.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.807+0000 7f02857f1640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 0x7f02680a3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0278005fd0 tx=0x7f0278005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:02.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:02.810+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02740966f0 con 0x7f0280102390 2024-03-12T05:08:02.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6832/1602847227,v1:172.21.15.130:6833/1602847227] up:boot 2024-03-12T05:08:02.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: fsmap 1 up:standby 2024-03-12T05:08:02.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:08:02.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:02.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:02.965 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:02.965 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:02.965 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:02.965 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:02.965 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:02 smithi116 ceph-mon[35486]: Deploying daemon mds.k on smithi116 2024-03-12T05:08:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.029+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f028010ca60 con 0x7f0280102390 2024-03-12T05:08:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.030+0000 7f027e7fc640 1 -- 172.21.15.45:0/2951397660 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f0274099db0 con 0x7f0280102390 2024-03-12T05:08:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:03.030 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:08:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 2 2024-03-12T05:08:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.031+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 msgr2=0x7f02680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.031+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 0x7f02680a3ef0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0278005fd0 tx=0x7f0278005850 comp rx=0 tx=0).stop 2024-03-12T05:08:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.031+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 msgr2=0x7f0280197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.031+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f027400c890 tx=0x7f027400cd50 comp rx=0 tx=0).stop 2024-03-12T05:08:03.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 shutdown_connections 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f02680a1a30 0x7f02680a3ef0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0280102390 0x7f0280197620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0280102d50 0x7f028019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 --2- 172.21.15.45:0/2951397660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02801011a0 0x7f02801970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 >> 172.21.15.45:0/2951397660 conn(0x7f02800fc730 msgr2=0x7f0280105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 shutdown_connections 2024-03-12T05:08:03.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:03.032+0000 7f0287a55640 1 -- 172.21.15.45:0/2951397660 wait complete. 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6832/1602847227,v1:172.21.15.130:6833/1602847227] up:boot 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: fsmap 1 up:standby 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:03.041 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:02 smithi045 ceph-mon[32174]: Deploying daemon mds.k on smithi116 2024-03-12T05:08:03.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6832/1602847227,v1:172.21.15.130:6833/1602847227] up:boot 2024-03-12T05:08:03.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: fsmap 1 up:standby 2024-03-12T05:08:03.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:08:03.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:03.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:03.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:03.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:03.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:02 smithi130 ceph-mon[35398]: Deploying daemon mds.k on smithi116 2024-03-12T05:08:04.063 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:03 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2951397660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:04.063 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:03 smithi116 ceph-mon[35486]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:04.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:03 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2951397660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:04.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:03 smithi130 ceph-mon[35398]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:04.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:03 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2951397660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:04.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:03 smithi045 ceph-mon[32174]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:04.459 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 systemd[1]: Starting Ceph mds.k for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:04.725 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 podman[65090]: 2024-03-12 05:08:04.363195573 +0000 UTC m=+0.018335467 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:04.725 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 podman[65090]: 2024-03-12 05:08:04.473655834 +0000 UTC m=+0.128795722 container create 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:08:04.725 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 podman[65090]: 2024-03-12 05:08:04.701504268 +0000 UTC m=+0.356644157 container init 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-12T05:08:04.725 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 podman[65090]: 2024-03-12 05:08:04.704058185 +0000 UTC m=+0.359198083 container start 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-12T05:08:04.725 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 ceph-mds[65109]: -- 172.21.15.116:0/3994460758 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff6638c1a0 con 0x55ff66362800 2024-03-12T05:08:04.995 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: starting mds.k at 2024-03-12T05:08:04.995 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 bash[65090]: 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 2024-03-12T05:08:04.995 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:08:04 smithi116 systemd[1]: Started Ceph mds.k for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:05.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 -- 172.21.15.45:0/132485372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101c50 msgr2=0x7fad4c1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:05.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 --2- 172.21.15.45:0/132485372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101c50 0x7fad4c1084e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fad44009ca0 tx=0x7fad4402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 -- 172.21.15.45:0/132485372 shutdown_connections 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 --2- 172.21.15.45:0/132485372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101c50 0x7fad4c1084e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 --2- 172.21.15.45:0/132485372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c101290 0x7fad4c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 --2- 172.21.15.45:0/132485372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c1000a0 0x7fad4c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 -- 172.21.15.45:0/132485372 >> 172.21.15.45:0/132485372 conn(0x7fad4c0fb610 msgr2=0x7fad4c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 -- 172.21.15.45:0/132485372 shutdown_connections 2024-03-12T05:08:05.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.149+0000 7fad51c15640 1 -- 172.21.15.45:0/132485372 wait complete. 2024-03-12T05:08:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 Processor -- start 2024-03-12T05:08:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 -- start start 2024-03-12T05:08:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 0x7fad4c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c101c50 0x7fad4c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c19ec50 con 0x7fad4c101290 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c19edc0 con 0x7fad4c101c50 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad51c15640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c19ef80 con 0x7fad4c1000a0 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52394/0 (socket says 172.21.15.45:52394) 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 -- 172.21.15.45:0/897996536 learned_addr learned my addr 172.21.15.45:0/897996536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad4bfff640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c101c50 0x7fad4c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad4b7fe640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 0x7fad4c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 -- 172.21.15.45:0/897996536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 msgr2=0x7fad4c197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 0x7fad4c197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 -- 172.21.15.45:0/897996536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c101c50 msgr2=0x7fad4c19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:05.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c101c50 0x7fad4c19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad42ffd640 1 -- 172.21.15.45:0/897996536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009900 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.150+0000 7fad4b7fe640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 0x7fad4c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad42ffd640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad3400d770 tx=0x7fad3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34009c30 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c19f260 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fad3400de80 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34004e00 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.151+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c19f770 con 0x7fad4c101290 2024-03-12T05:08:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.152+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c10b960 con 0x7fad4c101290 2024-03-12T05:08:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.153+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fad34004f60 con 0x7fad4c101290 2024-03-12T05:08:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.153+0000 7fad497fa640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 0x7fad380a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.154+0000 7fad4b7fe640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 0x7fad380a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.154+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fad340cca90 con 0x7fad4c101290 2024-03-12T05:08:05.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.154+0000 7fad4b7fe640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 0x7fad380a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fad4c1010f0 tx=0x7fad3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:05.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.158+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad340960a0 con 0x7fad4c101290 2024-03-12T05:08:05.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.376+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4c19fbe0 con 0x7fad4c101290 2024-03-12T05:08:05.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.377+0000 7fad497fa640 1 -- 172.21.15.45:0/897996536 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7fad34017860 con 0x7fad4c101290 2024-03-12T05:08:05.377 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:05.377 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:08:05.377 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 msgr2=0x7fad380a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 0x7fad380a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fad4c1010f0 tx=0x7fad3c005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 msgr2=0x7fad4c1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fad3400d770 tx=0x7fad3400dc30 comp rx=0 tx=0).stop 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 shutdown_connections 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fad380a1a30 0x7fad380a3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c101290 0x7fad4c1976a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad4c101c50 0x7fad4c19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 --2- 172.21.15.45:0/897996536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad4c1000a0 0x7fad4c197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:05.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 >> 172.21.15.45:0/897996536 conn(0x7fad4c0fb610 msgr2=0x7fad4c0fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:05.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.379+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 shutdown_connections 2024-03-12T05:08:05.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:05.380+0000 7fad51c15640 1 -- 172.21.15.45:0/897996536 wait complete. 2024-03-12T05:08:06.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: Deploying daemon mds.j on smithi045 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6832/2049389812,v1:172.21.15.116:6833/2049389812] up:boot 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: fsmap 2 up:standby 2024-03-12T05:08:06.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:08:06.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:06.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:05 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/897996536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: Deploying daemon mds.j on smithi045 2024-03-12T05:08:06.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6832/2049389812,v1:172.21.15.116:6833/2049389812] up:boot 2024-03-12T05:08:06.199 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: fsmap 2 up:standby 2024-03-12T05:08:06.199 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:08:06.199 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:06.199 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:05 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/897996536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:06.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:06.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: Deploying daemon mds.j on smithi045 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6832/2049389812,v1:172.21.15.116:6833/2049389812] up:boot 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: fsmap 2 up:standby 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:06.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:05 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/897996536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:07.119 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:06 smithi045 systemd[1]: Starting Ceph mds.j for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:07.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:07 smithi116 ceph-mon[35486]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:07.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:07.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:07 smithi045 ceph-mon[32174]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:07.421 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 podman[73271]: 2024-03-12 05:08:07.103334936 +0000 UTC m=+0.017703861 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:07.421 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 podman[73271]: 2024-03-12 05:08:07.203833382 +0000 UTC m=+0.118202341 container create 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-12T05:08:07.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:07 smithi130 ceph-mon[35398]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 -- 172.21.15.45:0/2768860542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c101c50 msgr2=0x7f8b4c1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2768860542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c101c50 0x7f8b4c1084e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009ca0 tx=0x7f8b4402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 -- 172.21.15.45:0/2768860542 shutdown_connections 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2768860542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c101c50 0x7f8b4c1084e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2768860542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101290 0x7f8b4c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2768860542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 -- 172.21.15.45:0/2768860542 >> 172.21.15.45:0/2768860542 conn(0x7f8b4c0fb610 msgr2=0x7f8b4c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 -- 172.21.15.45:0/2768860542 shutdown_connections 2024-03-12T05:08:07.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.504+0000 7f8b50c55640 1 -- 172.21.15.45:0/2768860542 wait complete. 2024-03-12T05:08:07.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 Processor -- start 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 -- start start 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 0x7f8b4c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 0x7f8b4c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c19da70 con 0x7f8b4c1000a0 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c19dbe0 con 0x7f8b4c101c50 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b50c55640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c19dda0 con 0x7f8b4c101290 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:07.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47288/0 (socket says 172.21.15.45:47288) 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4a59c640 1 -- 172.21.15.45:0/2995677657 learned_addr learned my addr 172.21.15.45:0/2995677657 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4ad9d640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 0x7f8b4c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4a59c640 1 -- 172.21.15.45:0/2995677657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 msgr2=0x7f8b4c1964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b49d9b640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 0x7f8b4c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.505+0000 7f8b4a59c640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 0x7f8b4c1964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b4a59c640 1 -- 172.21.15.45:0/2995677657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 msgr2=0x7f8b4c19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b4a59c640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 0x7f8b4c19d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b4a59c640 1 -- 172.21.15.45:0/2995677657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44009900 con 0x7f8b4c1000a0 2024-03-12T05:08:07.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b4ad9d640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 0x7f8b4c19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b49d9b640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 0x7f8b4c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b4a59c640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00d950 tx=0x7f8b3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c009c30 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c19e080 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b3c009d90 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00f6b0 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.506+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c19e590 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.507+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4c1073c0 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.508+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8b3c010a30 con 0x7f8b4c1000a0 2024-03-12T05:08:07.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.508+0000 7f8b3b7fe640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 0x7f8b340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:07.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.509+0000 7f8b49d9b640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 0x7f8b340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:07.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.509+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b3c0ccda0 con 0x7f8b4c1000a0 2024-03-12T05:08:07.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.509+0000 7f8b49d9b640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 0x7f8b340a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c197490 tx=0x7f8b40005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:07.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.513+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8b3c096450 con 0x7f8b4c1000a0 2024-03-12T05:08:07.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.736+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4c02cdd0 con 0x7f8b4c1000a0 2024-03-12T05:08:07.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.737+0000 7f8b3b7fe640 1 -- 172.21.15.45:0/2995677657 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f8b3c017860 con 0x7f8b4c1000a0 2024-03-12T05:08:07.737 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:07.737 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:08:07.737 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 3 2024-03-12T05:08:07.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 msgr2=0x7f8b340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:07.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 0x7f8b340a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c197490 tx=0x7f8b40005d90 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 msgr2=0x7f8b4c195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00d950 tx=0x7f8b3c00de10 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 shutdown_connections 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8b340a1a30 0x7f8b340a3ef0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b4c101c50 0x7f8b4c19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b4c101290 0x7f8b4c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 --2- 172.21.15.45:0/2995677657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b4c1000a0 0x7f8b4c195f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 >> 172.21.15.45:0/2995677657 conn(0x7f8b4c0fb610 msgr2=0x7f8b4c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 shutdown_connections 2024-03-12T05:08:07.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:07.739+0000 7f8b50c55640 1 -- 172.21.15.45:0/2995677657 wait complete. 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 podman[73271]: 2024-03-12 05:08:07.473675503 +0000 UTC m=+0.388044427 container init 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 podman[73271]: 2024-03-12 05:08:07.476578565 +0000 UTC m=+0.390947494 container start 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 ceph-mds[73310]: -- 172.21.15.45:0/581925964 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a42b041a0 con 0x560a42adac00 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j[73286]: starting mds.j at 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 bash[73271]: 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 2024-03-12T05:08:07.749 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:08:07 smithi045 systemd[1]: Started Ceph mds.j for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:08.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:08.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: Deploying daemon mds.d on smithi045 2024-03-12T05:08:08.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2995677657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:08.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6834/1645552275,v1:172.21.15.45:6835/1645552275] up:boot 2024-03-12T05:08:08.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: fsmap 3 up:standby 2024-03-12T05:08:08.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:08 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:08:08.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: Deploying daemon mds.d on smithi045 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2995677657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6834/1645552275,v1:172.21.15.45:6835/1645552275] up:boot 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: fsmap 3 up:standby 2024-03-12T05:08:08.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:08 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:08:08.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:08.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:08.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: Deploying daemon mds.d on smithi045 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2995677657' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6834/1645552275,v1:172.21.15.45:6835/1645552275] up:boot 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: fsmap 3 up:standby 2024-03-12T05:08:08.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:08 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:08:09.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:09.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:09 smithi045 ceph-mon[32174]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:09.869 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:09 smithi045 systemd[1]: Starting Ceph mds.d for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 -- 172.21.15.45:0/3322614734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf141000a0 msgr2=0x7faf141004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 --2- 172.21.15.45:0/3322614734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf141000a0 0x7faf141004c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf10009900 tx=0x7faf1002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 -- 172.21.15.45:0/3322614734 shutdown_connections 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 --2- 172.21.15.45:0/3322614734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf14101c50 0x7faf141084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 --2- 172.21.15.45:0/3322614734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 0x7faf14101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 --2- 172.21.15.45:0/3322614734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf141000a0 0x7faf141004c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 -- 172.21.15.45:0/3322614734 >> 172.21.15.45:0/3322614734 conn(0x7faf140fb610 msgr2=0x7faf140fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:09.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 -- 172.21.15.45:0/3322614734 shutdown_connections 2024-03-12T05:08:09.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.881+0000 7faf1aaae640 1 -- 172.21.15.45:0/3322614734 wait complete. 2024-03-12T05:08:09.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 Processor -- start 2024-03-12T05:08:09.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 -- start start 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 0x7faf14068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 0x7faf1419bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1419ecc0 con 0x7faf14101c50 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1419ee30 con 0x7faf14101290 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1aaae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1419eff0 con 0x7faf141000a0 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:09.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47308/0 (socket says 172.21.15.45:47308) 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 -- 172.21.15.45:0/2796831334 learned_addr learned my addr 172.21.15.45:0/2796831334 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1884a640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 0x7faf14068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf0bfff640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 0x7faf1419bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 -- 172.21.15.45:0/2796831334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 msgr2=0x7faf14068e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 0x7faf14068e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 -- 172.21.15.45:0/2796831334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 msgr2=0x7faf1419bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.882+0000 7faf1904b640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 0x7faf1419bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf1904b640 1 -- 172.21.15.45:0/2796831334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf10009560 con 0x7faf14101c50 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf1884a640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 0x7faf14068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf1904b640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00eaa0 tx=0x7faf0c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c00cdb0 con 0x7faf14101c50 2024-03-12T05:08:09.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1419f270 con 0x7faf14101c50 2024-03-12T05:08:09.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf0c005020 con 0x7faf14101c50 2024-03-12T05:08:09.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0116d0 con 0x7faf14101c50 2024-03-12T05:08:09.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.883+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1419f780 con 0x7faf14101c50 2024-03-12T05:08:09.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.884+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf141073e0 con 0x7faf14101c50 2024-03-12T05:08:09.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.886+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faf0c005190 con 0x7faf14101c50 2024-03-12T05:08:09.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.886+0000 7faf09ffb640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 0x7faefc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:09.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.886+0000 7faf1884a640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 0x7faefc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:09.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.886+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faf0c0cd1a0 con 0x7faf14101c50 2024-03-12T05:08:09.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.887+0000 7faf1884a640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 0x7faefc0a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf10036040 tx=0x7faf10000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:09.892+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf0c096850 con 0x7faf14101c50 2024-03-12T05:08:09.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:09 smithi130 ceph-mon[35398]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:09.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:09 smithi116 ceph-mon[35486]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:10.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.117+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf1402ce00 con 0x7faf14101c50 2024-03-12T05:08:10.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.117+0000 7faf09ffb640 1 -- 172.21.15.45:0/2796831334 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7faf0c099f10 con 0x7faf14101c50 2024-03-12T05:08:10.116 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:10.116 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:08:10.116 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 4 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 msgr2=0x7faefc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 0x7faefc0a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf10036040 tx=0x7faf10000bc0 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 msgr2=0x7faf1419e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00eaa0 tx=0x7faf0c00ef60 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 shutdown_connections 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7faefc0a1a30 0x7faefc0a3ef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf14101c50 0x7faf1419e4e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf14101290 0x7faf1419bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 --2- 172.21.15.45:0/2796831334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf141000a0 0x7faf14068e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:10.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 >> 172.21.15.45:0/2796831334 conn(0x7faf140fb610 msgr2=0x7faf141045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:10.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 shutdown_connections 2024-03-12T05:08:10.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:10.119+0000 7faf1aaae640 1 -- 172.21.15.45:0/2796831334 wait complete. 2024-03-12T05:08:10.288 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 podman[73655]: 2024-03-12 05:08:09.940487711 +0000 UTC m=+0.018530027 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:10.288 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 podman[73655]: 2024-03-12 05:08:10.046094232 +0000 UTC m=+0.124136552 container create 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-12T05:08:10.288 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 podman[73655]: 2024-03-12 05:08:10.290711998 +0000 UTC m=+0.368754316 container init 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T05:08:10.614 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 podman[73655]: 2024-03-12 05:08:10.293475097 +0000 UTC m=+0.371517420 container start 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:08:10.615 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mds[73676]: -- 172.21.15.45:0/3424406857 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640a8cac1a0 con 0x5640a8c82c00 2024-03-12T05:08:10.615 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d[73672]: starting mds.d at 2024-03-12T05:08:10.615 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 bash[73655]: 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 2024-03-12T05:08:10.615 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:08:10 smithi045 systemd[1]: Started Ceph mds.d for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2796831334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:10.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:10 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2796831334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:10.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:10 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2796831334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:10.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:10 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:11.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:11 smithi116 ceph-mon[35486]: Deploying daemon mds.e on smithi116 2024-03-12T05:08:11.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:11 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6836/378571831,v1:172.21.15.45:6837/378571831] up:boot 2024-03-12T05:08:11.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:11 smithi116 ceph-mon[35486]: fsmap 4 up:standby 2024-03-12T05:08:11.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:11 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:08:11.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:11 smithi116 ceph-mon[35486]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:11.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:11 smithi130 ceph-mon[35398]: Deploying daemon mds.e on smithi116 2024-03-12T05:08:11.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:11 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6836/378571831,v1:172.21.15.45:6837/378571831] up:boot 2024-03-12T05:08:11.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:11 smithi130 ceph-mon[35398]: fsmap 4 up:standby 2024-03-12T05:08:11.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:11 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:08:11.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:11 smithi130 ceph-mon[35398]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:12.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:11 smithi045 ceph-mon[32174]: Deploying daemon mds.e on smithi116 2024-03-12T05:08:12.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:11 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6836/378571831,v1:172.21.15.45:6837/378571831] up:boot 2024-03-12T05:08:12.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:11 smithi045 ceph-mon[32174]: fsmap 4 up:standby 2024-03-12T05:08:12.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:11 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:08:12.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:11 smithi045 ceph-mon[32174]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:12.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 -- 172.21.15.45:0/1123836206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 msgr2=0x7fdad4102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/1123836206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4102800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009900 tx=0x7fdac402bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 -- 172.21.15.45:0/1123836206 shutdown_connections 2024-03-12T05:08:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/1123836206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4102800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/1123836206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 0x7fdad41095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/1123836206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 0x7fdad41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 -- 172.21.15.45:0/1123836206 >> 172.21.15.45:0/1123836206 conn(0x7fdad40fc760 msgr2=0x7fdad40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.246+0000 7fdadbf2e640 1 -- 172.21.15.45:0/1123836206 shutdown_connections 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 -- 172.21.15.45:0/1123836206 wait complete. 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 Processor -- start 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 -- start start 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 0x7fdad4074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 0x7fdad4073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4074210 con 0x7fdad4102380 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4074380 con 0x7fdad4102d40 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdadbf2e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad41a11c0 con 0x7fdad4101190 2024-03-12T05:08:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdad94c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdad94c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47350/0 (socket says 172.21.15.45:47350) 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdad94c9640 1 -- 172.21.15.45:0/4250918933 learned_addr learned my addr 172.21.15.45:0/4250918933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdad9cca640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 0x7fdad4074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.247+0000 7fdada4cb640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 0x7fdad4073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 -- 172.21.15.45:0/4250918933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 msgr2=0x7fdad4074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 0x7fdad4074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 -- 172.21.15.45:0/4250918933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 msgr2=0x7fdad4073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 0x7fdad4073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 -- 172.21.15.45:0/4250918933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009560 con 0x7fdad4102380 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdad94c9640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdac4000c00 tx=0x7fdac40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004ac0 con 0x7fdad4102380 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdac40044e0 con 0x7fdad4102380 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40055c0 con 0x7fdad4102380 2024-03-12T05:08:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad41a1410 con 0x7fdad4102380 2024-03-12T05:08:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.248+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad41a18d0 con 0x7fdad4102380 2024-03-12T05:08:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.249+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad4108530 con 0x7fdad4102380 2024-03-12T05:08:12.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.250+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdac4004dc0 con 0x7fdad4102380 2024-03-12T05:08:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.251+0000 7fdacaffd640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 0x7fdac00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.251+0000 7fdad9cca640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 0x7fdac00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.251+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdac402de40 con 0x7fdad4102380 2024-03-12T05:08:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.251+0000 7fdad9cca640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 0x7fdac00a3f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdad0005fd0 tx=0x7fdad000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:12.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.255+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdac40b8800 con 0x7fdad4102380 2024-03-12T05:08:12.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.474+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdad410caa0 con 0x7fdad4102380 2024-03-12T05:08:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.474+0000 7fdacaffd640 1 -- 172.21.15.45:0/4250918933 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3701 (secure 0 0 0) 0x7fdac4043e90 con 0x7fdad4102380 2024-03-12T05:08:12.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:12.473 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:08:12.473 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 5 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 msgr2=0x7fdac00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 0x7fdac00a3f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdad0005fd0 tx=0x7fdad000b390 comp rx=0 tx=0).stop 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 msgr2=0x7fdad4073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdac4000c00 tx=0x7fdac40042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 shutdown_connections 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdac00a1ab0 0x7fdac00a3f70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdad4102380 0x7fdad4073570 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad4102d40 0x7fdad4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 --2- 172.21.15.45:0/4250918933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdad4101190 0x7fdad4074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 >> 172.21.15.45:0/4250918933 conn(0x7fdad40fc760 msgr2=0x7fdad4105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 shutdown_connections 2024-03-12T05:08:12.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:12.476+0000 7fdadbf2e640 1 -- 172.21.15.45:0/4250918933 wait complete. 2024-03-12T05:08:12.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:12 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4250918933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:12.746 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 podman[65397]: 2024-03-12 05:08:12.653338024 +0000 UTC m=+0.115641343 container create b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:08:12.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:12 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4250918933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:12.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:12 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4250918933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 podman[65397]: 2024-03-12 05:08:12.88977626 +0000 UTC m=+0.352079580 container init b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 podman[65397]: 2024-03-12 05:08:12.892280417 +0000 UTC m=+0.354583738 container start b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 ceph-mds[65416]: -- 172.21.15.116:0/3058925694 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a7d1fba1a0 con 0x55a7d1f90800 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: starting mds.e at 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 bash[65397]: b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce 2024-03-12T05:08:13.245 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:08:12 smithi116 systemd[1]: Started Ceph mds.e for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: Deploying daemon mds.f on smithi130 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6834/4238162743,v1:172.21.15.116:6835/4238162743] up:boot 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: fsmap 5 up:standby 2024-03-12T05:08:14.342 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:14 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:08:14.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: Deploying daemon mds.f on smithi130 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6834/4238162743,v1:172.21.15.116:6835/4238162743] up:boot 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: fsmap 5 up:standby 2024-03-12T05:08:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:14 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:08:14.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: Deploying daemon mds.f on smithi130 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:14.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6834/4238162743,v1:172.21.15.116:6835/4238162743] up:boot 2024-03-12T05:08:14.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: fsmap 5 up:standby 2024-03-12T05:08:14.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:14 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:08:14.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.597+0000 7fbd32e15640 1 -- 172.21.15.45:0/622606618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101c50 msgr2=0x7fbd2c1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.597+0000 7fbd32e15640 1 --2- 172.21.15.45:0/622606618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101c50 0x7fbd2c1084e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009900 tx=0x7fbd1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- 172.21.15.45:0/622606618 shutdown_connections 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- 172.21.15.45:0/622606618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101c50 0x7fbd2c1084e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- 172.21.15.45:0/622606618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101290 0x7fbd2c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- 172.21.15.45:0/622606618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- 172.21.15.45:0/622606618 >> 172.21.15.45:0/622606618 conn(0x7fbd2c0fb610 msgr2=0x7fbd2c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- 172.21.15.45:0/622606618 shutdown_connections 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- 172.21.15.45:0/622606618 wait complete. 2024-03-12T05:08:14.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 Processor -- start 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- start start 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 0x7fbd2c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c19dbb0 con 0x7fbd2c101290 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c19dd20 con 0x7fbd2c101c50 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd32e15640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c19dee0 con 0x7fbd2c1000a0 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47370/0 (socket says 172.21.15.45:47370) 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.598+0000 7fbd2bfff640 1 -- 172.21.15.45:0/4043693306 learned_addr learned my addr 172.21.15.45:0/4043693306 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd30bb1640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd313b2640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 0x7fbd2c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:14.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 -- 172.21.15.45:0/4043693306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 msgr2=0x7fbd2c1960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 -- 172.21.15.45:0/4043693306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 msgr2=0x7fbd2c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 0x7fbd2c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 -- 172.21.15.45:0/4043693306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009560 con 0x7fbd2c101290 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd30bb1640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd313b2640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 0x7fbd2c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd2bfff640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000d990 tx=0x7fbd2000de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000edf0 con 0x7fbd2c101290 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd200168f0 con 0x7fbd2c101290 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c068340 con 0x7fbd2c101290 2024-03-12T05:08:14.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd200148d0 con 0x7fbd2c101290 2024-03-12T05:08:14.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.599+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c0688d0 con 0x7fbd2c101290 2024-03-12T05:08:14.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.600+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c067f70 con 0x7fbd2c101290 2024-03-12T05:08:14.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.601+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbd20016a60 con 0x7fbd2c101290 2024-03-12T05:08:14.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.602+0000 7fbd29ffb640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 0x7fbd140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:14.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.602+0000 7fbd30bb1640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 0x7fbd140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:14.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.602+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbd200ceb50 con 0x7fbd2c101290 2024-03-12T05:08:14.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.602+0000 7fbd30bb1640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 0x7fbd140a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd24005fd0 tx=0x7fbd24008310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:14.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.606+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbd20051d20 con 0x7fbd2c101290 2024-03-12T05:08:14.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.826+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd2c106fb0 con 0x7fbd2c101290 2024-03-12T05:08:14.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.827+0000 7fbd29ffb640 1 -- 172.21.15.45:0/4043693306 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4495 (secure 0 0 0) 0x7fbd20098160 con 0x7fbd2c101290 2024-03-12T05:08:14.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:14.826 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-12T05:08:14.826 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 6 2024-03-12T05:08:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 msgr2=0x7fbd140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 0x7fbd140a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd24005fd0 tx=0x7fbd24008310 comp rx=0 tx=0).stop 2024-03-12T05:08:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 msgr2=0x7fbd2c196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000d990 tx=0x7fbd2000de50 comp rx=0 tx=0).stop 2024-03-12T05:08:14.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 shutdown_connections 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbd140a1a30 0x7fbd140a3ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2c101290 0x7fbd2c196600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd2c101c50 0x7fbd2c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 --2- 172.21.15.45:0/4043693306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd2c1000a0 0x7fbd2c1960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 >> 172.21.15.45:0/4043693306 conn(0x7fbd2c0fb610 msgr2=0x7fbd2c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 shutdown_connections 2024-03-12T05:08:14.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:14.829+0000 7fbd32e15640 1 -- 172.21.15.45:0/4043693306 wait complete. 2024-03-12T05:08:15.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:15 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4043693306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:15.422 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:15 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4043693306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:15.422 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 systemd[1]: Starting Ceph mds.f for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:15.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:15 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4043693306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:15.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 podman[65750]: 2024-03-12 05:08:15.423852042 +0000 UTC m=+0.098510375 container create 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-12T05:08:15.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 podman[65750]: 2024-03-12 05:08:15.344092903 +0000 UTC m=+0.018751237 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:15.717 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 podman[65750]: 2024-03-12 05:08:15.618300362 +0000 UTC m=+0.292958696 container init 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-12T05:08:15.717 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 podman[65750]: 2024-03-12 05:08:15.620811925 +0000 UTC m=+0.295470259 container start 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T05:08:15.717 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 ceph-mds[65769]: -- 172.21.15.130:0/4135974752 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5598aa1381a0 con 0x5598aa10e800 2024-03-12T05:08:15.717 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: starting mds.f at 2024-03-12T05:08:15.717 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 bash[65750]: 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 2024-03-12T05:08:15.970 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:08:15 smithi130 systemd[1]: Started Ceph mds.f for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:16.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:16.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:16.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:16.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:16 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:16.417 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:16.417 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.417 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.418 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.418 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:16.418 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:16.418 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:16 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:16.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:16.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:16.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:16.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:16.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:16 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:16.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:16.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 -- 172.21.15.45:0/1878843152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5170104b60 msgr2=0x7f5170104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 --2- 172.21.15.45:0/1878843152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5170104b60 0x7f5170104fe0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5160009ca0 tx=0x7f516002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 -- 172.21.15.45:0/1878843152 shutdown_connections 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 --2- 172.21.15.45:0/1878843152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5170104b60 0x7f5170104fe0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 --2- 172.21.15.45:0/1878843152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 0x7f5170104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 --2- 172.21.15.45:0/1878843152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51700ff840 0x7f5170101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 -- 172.21.15.45:0/1878843152 >> 172.21.15.45:0/1878843152 conn(0x7f51700fb460 msgr2=0x7f51700fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 -- 172.21.15.45:0/1878843152 shutdown_connections 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 -- 172.21.15.45:0/1878843152 wait complete. 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.957+0000 7f517679f640 1 Processor -- start 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 -- start start 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 0x7f517019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 0x7f51701a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51701a5010 con 0x7f51700ff840 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51701a5180 con 0x7f51701021b0 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517679f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51701a5340 con 0x7f5170104b60 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47394/0 (socket says 172.21.15.45:47394) 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 -- 172.21.15.45:0/1427729957 learned_addr learned my addr 172.21.15.45:0/1427729957 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f5175f9e640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 0x7f51701a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 -- 172.21.15.45:0/1427729957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 msgr2=0x7f51701a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f5174f9c640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 0x7f517019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 0x7f51701a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 -- 172.21.15.45:0/1427729957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 msgr2=0x7f517019dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 0x7f517019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 -- 172.21.15.45:0/1427729957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5160009900 con 0x7f51700ff840 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f5175f9e640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 0x7f51701a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f517579d640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f516800d950 tx=0x7f516800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:16.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.958+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5168009c30 con 0x7f51700ff840 2024-03-12T05:08:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.959+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51701a5620 con 0x7f51700ff840 2024-03-12T05:08:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.959+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5168009d90 con 0x7f51700ff840 2024-03-12T05:08:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.959+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516800f6b0 con 0x7f51700ff840 2024-03-12T05:08:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.959+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51701a5b80 con 0x7f51700ff840 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.960+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51700fcf50 con 0x7f51700ff840 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.961+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5168010aa0 con 0x7f51700ff840 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.961+0000 7f516e7fc640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 0x7f51580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.961+0000 7f5174f9c640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 0x7f51580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.961+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f51680ccd30 con 0x7f51700ff840 2024-03-12T05:08:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.962+0000 7f5174f9c640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 0x7f51580a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5164005fd0 tx=0x7f5164009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:16.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:16.966+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51680963e0 con 0x7f51700ff840 2024-03-12T05:08:17.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:17 smithi045 ceph-mon[32174]: Deploying daemon mds.c on smithi130 2024-03-12T05:08:17.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:17 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:boot 2024-03-12T05:08:17.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:17 smithi045 ceph-mon[32174]: fsmap 6 up:standby 2024-03-12T05:08:17.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:17 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:08:17.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.188+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f517002cdd0 con 0x7f51700ff840 2024-03-12T05:08:17.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.188+0000 7f516e7fc640 1 -- 172.21.15.45:0/1427729957 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7f5168099aa0 con 0x7f51700ff840 2024-03-12T05:08:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:17.187 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-12T05:08:17.187 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 7 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 msgr2=0x7f51580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 0x7f51580a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5164005fd0 tx=0x7f5164009470 comp rx=0 tx=0).stop 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 msgr2=0x7f517019d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f516800d950 tx=0x7f516800de10 comp rx=0 tx=0).stop 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 shutdown_connections 2024-03-12T05:08:17.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f51580a1a30 0x7f51580a3ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5170104b60 0x7f51701a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51701021b0 0x7f517019dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 --2- 172.21.15.45:0/1427729957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f51700ff840 0x7f517019d590 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 >> 172.21.15.45:0/1427729957 conn(0x7f51700fb460 msgr2=0x7f51701005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 shutdown_connections 2024-03-12T05:08:17.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:17.190+0000 7f517679f640 1 -- 172.21.15.45:0/1427729957 wait complete. 2024-03-12T05:08:17.462 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:17 smithi130 ceph-mon[35398]: Deploying daemon mds.c on smithi130 2024-03-12T05:08:17.462 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:17 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:boot 2024-03-12T05:08:17.462 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:17 smithi130 ceph-mon[35398]: fsmap 6 up:standby 2024-03-12T05:08:17.462 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:17 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:08:17.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:17 smithi116 ceph-mon[35486]: Deploying daemon mds.c on smithi130 2024-03-12T05:08:17.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:17 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:boot 2024-03-12T05:08:17.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:17 smithi116 ceph-mon[35486]: fsmap 6 up:standby 2024-03-12T05:08:17.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:17 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:08:18.092 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:17 smithi130 systemd[1]: Starting Ceph mds.c for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:18.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:18 smithi045 ceph-mon[32174]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:18.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:18 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1427729957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:18.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:18 smithi130 ceph-mon[35398]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:18.372 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:18 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1427729957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:18.372 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 podman[66056]: 2024-03-12 05:08:18.05318763 +0000 UTC m=+0.018429233 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:18.372 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 podman[66056]: 2024-03-12 05:08:18.163193825 +0000 UTC m=+0.128435453 container create 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T05:08:18.372 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 podman[66056]: 2024-03-12 05:08:18.374186912 +0000 UTC m=+0.339428501 container init 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T05:08:18.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:18 smithi116 ceph-mon[35486]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:18.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:18 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1427729957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:18.720 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 podman[66056]: 2024-03-12 05:08:18.376918543 +0000 UTC m=+0.342160133 container start 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-12T05:08:18.720 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 ceph-mds[66075]: -- 172.21.15.130:0/1156515394 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0dfdec1a0 con 0x55b0dfdc2c00 2024-03-12T05:08:18.720 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: starting mds.c at 2024-03-12T05:08:18.720 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 bash[66056]: 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf 2024-03-12T05:08:18.720 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:08:18 smithi130 systemd[1]: Started Ceph mds.c for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:19.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 -- 172.21.15.45:0/1375335546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a40ff490 msgr2=0x7f46a40ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 --2- 172.21.15.45:0/1375335546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a40ff490 0x7f46a40ff8b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4694009ca0 tx=0x7f469402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 -- 172.21.15.45:0/1375335546 shutdown_connections 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 --2- 172.21.15.45:0/1375335546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102e20 0x7f46a41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 --2- 172.21.15.45:0/1375335546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a4102460 0x7f46a41028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 --2- 172.21.15.45:0/1375335546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a40ff490 0x7f46a40ff8b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 -- 172.21.15.45:0/1375335546 >> 172.21.15.45:0/1375335546 conn(0x7f46a40fadc0 msgr2=0x7f46a40fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 -- 172.21.15.45:0/1375335546 shutdown_connections 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.317+0000 7f46ab713640 1 -- 172.21.15.45:0/1375335546 wait complete. 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 Processor -- start 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 -- start start 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a40ff490 0x7f46a4197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102460 0x7f46a4197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:19.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a419ed30 con 0x7f46a4102e20 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a419eea0 con 0x7f46a4102460 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46ab713640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a419f060 con 0x7f46a40ff490 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59334/0 (socket says 172.21.15.45:59334) 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 -- 172.21.15.45:0/1301389311 learned_addr learned my addr 172.21.15.45:0/1301389311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a94af640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a40ff490 0x7f46a4197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a8cae640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102460 0x7f46a4197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 -- 172.21.15.45:0/1301389311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a40ff490 msgr2=0x7f46a4197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a40ff490 0x7f46a4197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 -- 172.21.15.45:0/1301389311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102460 msgr2=0x7f46a4197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102460 0x7f46a4197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 -- 172.21.15.45:0/1301389311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4694009900 con 0x7f46a4102e20 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.318+0000 7f46a9cb0640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f469c00d950 tx=0x7f469c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.319+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c009a30 con 0x7f46a4102e20 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.319+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f469c0044d0 con 0x7f46a4102e20 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.319+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a419f340 con 0x7f46a4102e20 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.319+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c003c30 con 0x7f46a4102e20 2024-03-12T05:08:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.319+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a419f800 con 0x7f46a4102e20 2024-03-12T05:08:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.320+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a4108630 con 0x7f46a4102e20 2024-03-12T05:08:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.321+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f469c003d90 con 0x7f46a4102e20 2024-03-12T05:08:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.321+0000 7f46a27fc640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 0x7f468c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.321+0000 7f46a94af640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 0x7f468c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.321+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f469c0ccef0 con 0x7f46a4102e20 2024-03-12T05:08:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.322+0000 7f46a94af640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 0x7f468c0a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f469402c700 tx=0x7f4694009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.325+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f469c0965a0 con 0x7f46a4102e20 2024-03-12T05:08:19.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.546+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46a410cae0 con 0x7f46a4102e20 2024-03-12T05:08:19.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.547+0000 7f46a27fc640 1 -- 172.21.15.45:0/1301389311 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6083 (secure 0 0 0) 0x7f469c099c60 con 0x7f46a4102e20 2024-03-12T05:08:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:19.546 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-12T05:08:19.546 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 msgr2=0x7f468c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 0x7f468c0a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f469402c700 tx=0x7f4694009800 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 msgr2=0x7f46a419e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f469c00d950 tx=0x7f469c00de10 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 shutdown_connections 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f468c0a1a30 0x7f468c0a3ef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46a4102e20 0x7f46a419e7f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a4102460 0x7f46a4197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 --2- 172.21.15.45:0/1301389311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46a40ff490 0x7f46a4197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 >> 172.21.15.45:0/1301389311 conn(0x7f46a40fadc0 msgr2=0x7f46a4105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 shutdown_connections 2024-03-12T05:08:19.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:19.549+0000 7f46ab713640 1 -- 172.21.15.45:0/1301389311 wait complete. 2024-03-12T05:08:19.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: Deploying daemon mds.h on smithi116 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:boot 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: fsmap 7 up:standby 2024-03-12T05:08:19.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:19 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: Deploying daemon mds.h on smithi116 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:19.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:boot 2024-03-12T05:08:19.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: fsmap 7 up:standby 2024-03-12T05:08:19.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:19 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:08:19.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: Deploying daemon mds.h on smithi116 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:boot 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: fsmap 7 up:standby 2024-03-12T05:08:19.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:19 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:08:20.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:20 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1301389311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:20.954 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:20 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1301389311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:20.954 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:20 smithi116 systemd[1]: Starting Ceph mds.h for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:20.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:20 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1301389311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:21.227 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:20 smithi116 podman[65705]: 2024-03-12 05:08:20.858761913 +0000 UTC m=+0.017426814 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:21.227 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:20 smithi116 podman[65705]: 2024-03-12 05:08:20.967531704 +0000 UTC m=+0.126196606 container create 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 podman[65705]: 2024-03-12 05:08:21.229422532 +0000 UTC m=+0.388087435 container init 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 podman[65705]: 2024-03-12 05:08:21.232724787 +0000 UTC m=+0.391389696 container start 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mds[65724]: -- 172.21.15.116:0/2573583058 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560dfa3c01a0 con 0x560dfa396800 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: starting mds.h at 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 bash[65705]: 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 2024-03-12T05:08:21.495 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:08:21 smithi116 systemd[1]: Started Ceph mds.h for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:21.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- 172.21.15.45:0/1907453316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c101290 msgr2=0x7fc48c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- 172.21.15.45:0/1907453316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c101290 0x7fc48c101710 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc480009900 tx=0x7fc48002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- 172.21.15.45:0/1907453316 shutdown_connections 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- 172.21.15.45:0/1907453316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c101290 0x7fc48c101710 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- 172.21.15.45:0/1907453316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 0x7fc48c1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- 172.21.15.45:0/1907453316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c1000a0 0x7fc48c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- 172.21.15.45:0/1907453316 >> 172.21.15.45:0/1907453316 conn(0x7fc48c0fb610 msgr2=0x7fc48c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- 172.21.15.45:0/1907453316 shutdown_connections 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- 172.21.15.45:0/1907453316 wait complete. 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 Processor -- start 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- start start 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:21.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 0x7fc48c102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 0x7fc48c102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c103310 con 0x7fc48c1000a0 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c103480 con 0x7fc48c101290 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.674+0000 7fc493356640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1a11d0 con 0x7fc48c101c50 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59350/0 (socket says 172.21.15.45:59350) 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 -- 172.21.15.45:0/1878044456 learned_addr learned my addr 172.21.15.45:0/1878044456 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4918f3640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 0x7fc48c102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4908f1640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 0x7fc48c102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 -- 172.21.15.45:0/1878044456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 msgr2=0x7fc48c102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 0x7fc48c102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 -- 172.21.15.45:0/1878044456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 msgr2=0x7fc48c102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 0x7fc48c102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 -- 172.21.15.45:0/1878044456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480009560 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4908f1640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 0x7fc48c102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc4910f2640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc48400d7a0 tx=0x7fc48400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484009c30 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c1a1480 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4840044d0 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc484003dd0 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.675+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c1a19e0 con 0x7fc48c1000a0 2024-03-12T05:08:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.676+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c10b960 con 0x7fc48c1000a0 2024-03-12T05:08:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.677+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc484004000 con 0x7fc48c1000a0 2024-03-12T05:08:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.678+0000 7fc48a7fc640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 0x7fc4740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.678+0000 7fc4908f1640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 0x7fc4740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.678+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc4840ccdf0 con 0x7fc48c1000a0 2024-03-12T05:08:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.678+0000 7fc4908f1640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 0x7fc4740a3ef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc480036040 tx=0x7fc480000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.681+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc484096400 con 0x7fc48c1000a0 2024-03-12T05:08:21.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:21.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:21 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.902+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc48c10baf0 con 0x7fc48c1000a0 2024-03-12T05:08:21.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.902+0000 7fc48a7fc640 1 -- 172.21.15.45:0/1878044456 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6877 (secure 0 0 0) 0x7fc484020090 con 0x7fc48c1000a0 2024-03-12T05:08:21.901 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:21.901 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-12T05:08:21.901 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 msgr2=0x7fc4740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 0x7fc4740a3ef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc480036040 tx=0x7fc480000bc0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 msgr2=0x7fc48c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc48400d7a0 tx=0x7fc48400dc60 comp rx=0 tx=0).stop 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 shutdown_connections 2024-03-12T05:08:21.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc4740a1a30 0x7fc4740a3ef0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc48c101c50 0x7fc48c102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc48c101290 0x7fc48c102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 --2- 172.21.15.45:0/1878044456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc48c1000a0 0x7fc48c104030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 >> 172.21.15.45:0/1878044456 conn(0x7fc48c0fb610 msgr2=0x7fc48c0fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 shutdown_connections 2024-03-12T05:08:21.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:21.904+0000 7fc493356640 1 -- 172.21.15.45:0/1878044456 wait complete. 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:21.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:21.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:21 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:21.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:21 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:22.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:22 smithi130 ceph-mon[35398]: Deploying daemon mds.i on smithi130 2024-03-12T05:08:22.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:22 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:boot 2024-03-12T05:08:22.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:22 smithi130 ceph-mon[35398]: fsmap 8 up:standby 2024-03-12T05:08:22.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:22 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:08:22.721 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:22 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1878044456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:22.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:22 smithi045 ceph-mon[32174]: Deploying daemon mds.i on smithi130 2024-03-12T05:08:22.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:22 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:boot 2024-03-12T05:08:22.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:22 smithi045 ceph-mon[32174]: fsmap 8 up:standby 2024-03-12T05:08:22.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:22 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:08:22.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:22 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1878044456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:22.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:22 smithi116 ceph-mon[35486]: Deploying daemon mds.i on smithi130 2024-03-12T05:08:22.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:22 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:boot 2024-03-12T05:08:22.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:22 smithi116 ceph-mon[35486]: fsmap 8 up:standby 2024-03-12T05:08:22.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:22 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:08:22.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:22 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1878044456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:23.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:23 smithi130 ceph-mon[35398]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:23.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:23 smithi045 ceph-mon[32174]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:23.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:23.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:23 smithi116 ceph-mon[35486]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:24.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 -- 172.21.15.45:0/2578337995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 msgr2=0x7f8654101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:24.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2578337995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f8654101210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8650009ca0 tx=0x7f865002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 -- 172.21.15.45:0/2578337995 shutdown_connections 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2578337995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f8654101210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2578337995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f8654100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2578337995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 0x7f86540ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.031+0000 7f865b3a7640 1 -- 172.21.15.45:0/2578337995 >> 172.21.15.45:0/2578337995 conn(0x7f86540fb6b0 msgr2=0x7f86540fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- 172.21.15.45:0/2578337995 shutdown_connections 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- 172.21.15.45:0/2578337995 wait complete. 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 Processor -- start 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- start start 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 0x7f8654199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:24.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f86541998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86541a0e80 con 0x7f8654100d90 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86541a0ff0 con 0x7f86541003f0 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f865b3a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86541a11b0 con 0x7f86540ffa90 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f8659944640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f8659944640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59364/0 (socket says 172.21.15.45:59364) 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f8659944640 1 -- 172.21.15.45:0/2689165046 learned_addr learned my addr 172.21.15.45:0/2689165046 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.032+0000 7f8659143640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 0x7f8654199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8658942640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f86541998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 -- 172.21.15.45:0/2689165046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 msgr2=0x7f8654199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:24.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 0x7f8654199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 -- 172.21.15.45:0/2689165046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 msgr2=0x7f86541998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f86541998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 -- 172.21.15.45:0/2689165046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650009900 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8658942640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f86541998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f8659944640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8650037040 tx=0x7f8650032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650034e70 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86541a1430 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8650003e20 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650004800 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.033+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86541a1960 con 0x7f8654100d90 2024-03-12T05:08:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.034+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865410a7e0 con 0x7f8654100d90 2024-03-12T05:08:24.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.035+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f865003b040 con 0x7f8654100d90 2024-03-12T05:08:24.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.036+0000 7f864a7fc640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 0x7f86400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:24.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.036+0000 7f8659143640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 0x7f86400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:24.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.036+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f86500ef760 con 0x7f8654100d90 2024-03-12T05:08:24.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.036+0000 7f8659143640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 0x7f86400a3ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f864c009bd0 tx=0x7f864c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:24.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.040+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86500b8d70 con 0x7f8654100d90 2024-03-12T05:08:24.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.261+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865402ce40 con 0x7f8654100d90 2024-03-12T05:08:24.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.262+0000 7f864a7fc640 1 -- 172.21.15.45:0/2689165046 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6877 (secure 0 0 0) 0x7f86500bc430 con 0x7f8654100d90 2024-03-12T05:08:24.260 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:24.260 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-12T05:08:24.261 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2024-03-12T05:08:24.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 msgr2=0x7f86400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:24.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 0x7f86400a3ef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f864c009bd0 tx=0x7f864c005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:24.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 msgr2=0x7f86541a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8650037040 tx=0x7f8650032980 comp rx=0 tx=0).stop 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 shutdown_connections 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f86400a1a30 0x7f86400a3ef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8654100d90 0x7f86541a0940 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86541003f0 0x7f86541998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 --2- 172.21.15.45:0/2689165046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86540ffa90 0x7f8654199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 >> 172.21.15.45:0/2689165046 conn(0x7f86540fb6b0 msgr2=0x7f86540fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 shutdown_connections 2024-03-12T05:08:24.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:24.264+0000 7f865b3a7640 1 -- 172.21.15.45:0/2689165046 wait complete. 2024-03-12T05:08:24.470 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:08:24 smithi130 systemd[1]: Started Ceph mds.i for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2689165046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:boot 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: fsmap 9 up:standby 2024-03-12T05:08:25.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:25 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2689165046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:boot 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: fsmap 9 up:standby 2024-03-12T05:08:25.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:25 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:25.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:25.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:25.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2689165046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:25.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:boot 2024-03-12T05:08:25.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: fsmap 9 up:standby 2024-03-12T05:08:25.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:25 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:08:26.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:26 smithi045 ceph-mon[32174]: Deploying daemon mds.g on smithi045 2024-03-12T05:08:26.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:26 smithi045 ceph-mon[32174]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:26.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.429+0000 7f2cf55a7640 1 -- 172.21.15.45:0/1626297021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 msgr2=0x7f2cf01069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.429+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/1626297021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf01069e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009900 tx=0x7f2ce402bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- 172.21.15.45:0/1626297021 shutdown_connections 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/1626297021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf01069e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/1626297021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0106f20 0x7f2cf0109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/1626297021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- 172.21.15.45:0/1626297021 >> 172.21.15.45:0/1626297021 conn(0x7f2cf00fb610 msgr2=0x7f2cf00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- 172.21.15.45:0/1626297021 shutdown_connections 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- 172.21.15.45:0/1626297021 wait complete. 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 Processor -- start 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- start start 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0106f20 0x7f2cf019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf019ecd0 con 0x7f2cf01045f0 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf019ee40 con 0x7f2cf0101cc0 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.430+0000 7f2cf55a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf019f000 con 0x7f2cf0106f20 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59378/0 (socket says 172.21.15.45:59378) 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 -- 172.21.15.45:0/2107700445 learned_addr learned my addr 172.21.15.45:0/2107700445 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2ceeffd640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cef7fe640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0106f20 0x7f2cf019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 -- 172.21.15.45:0/2107700445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0106f20 msgr2=0x7f2cf019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0106f20 0x7f2cf019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 -- 172.21.15.45:0/2107700445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 msgr2=0x7f2cf01971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 -- 172.21.15.45:0/2107700445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce4009560 con 0x7f2cf01045f0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2ceeffd640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cee7fc640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce402c7d0 tx=0x7f2ce40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4004aa0 con 0x7f2cf01045f0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf019f280 con 0x7f2cf01045f0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2ce402de90 con 0x7f2cf01045f0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4005440 con 0x7f2cf01045f0 2024-03-12T05:08:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.431+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf019f790 con 0x7f2cf01045f0 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.433+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf00fd100 con 0x7f2cf01045f0 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.433+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2ce40055a0 con 0x7f2cf01045f0 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.434+0000 7f2cd7fff640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 0x7f2cd80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.434+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2ce40ef180 con 0x7f2cf01045f0 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.434+0000 7f2ceeffd640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 0x7f2cd80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.435+0000 7f2ceeffd640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 0x7f2cd80a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009c20 tx=0x7f2ce0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.439+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ce40b8830 con 0x7f2cf01045f0 2024-03-12T05:08:26.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:26 smithi130 ceph-mon[35398]: Deploying daemon mds.g on smithi045 2024-03-12T05:08:26.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:26 smithi130 ceph-mon[35398]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:26.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:26 smithi116 ceph-mon[35486]: Deploying daemon mds.g on smithi045 2024-03-12T05:08:26.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:26 smithi116 ceph-mon[35486]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:26.591 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 systemd[1]: Starting Ceph mds.g for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:26.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.674+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf010cc30 con 0x7f2cf01045f0 2024-03-12T05:08:26.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.674+0000 7f2cd7fff640 1 -- 172.21.15.45:0/2107700445 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7673 (secure 0 0 0) 0x7f2ce40bbef0 con 0x7f2cf01045f0 2024-03-12T05:08:26.673 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:26.673 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:08:26.673 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 10 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 msgr2=0x7f2cd80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 0x7f2cd80a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009c20 tx=0x7f2ce0005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 msgr2=0x7f2cf0197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce402c7d0 tx=0x7f2ce40042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 shutdown_connections 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2cd80a1a30 0x7f2cd80a3ef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cf01045f0 0x7f2cf0197720 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2cf0106f20 0x7f2cf019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 --2- 172.21.15.45:0/2107700445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2cf0101cc0 0x7f2cf01971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 >> 172.21.15.45:0/2107700445 conn(0x7f2cf00fb610 msgr2=0x7f2cf0103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 shutdown_connections 2024-03-12T05:08:26.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:26.676+0000 7f2cf55a7640 1 -- 172.21.15.45:0/2107700445 wait complete. 2024-03-12T05:08:26.868 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 podman[74249]: 2024-03-12 05:08:26.495940874 +0000 UTC m=+0.017973207 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:26.869 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 podman[74249]: 2024-03-12 05:08:26.628708665 +0000 UTC m=+0.150740982 container create 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-12T05:08:27.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2107700445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:27 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 podman[74249]: 2024-03-12 05:08:26.894753187 +0000 UTC m=+0.416785509 container init 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 podman[74249]: 2024-03-12 05:08:26.897498934 +0000 UTC m=+0.419531248 container start 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 ceph-mds[74270]: -- 172.21.15.45:0/3183942641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5606e84561a0 con 0x5606e842c800 2024-03-12T05:08:27.369 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: starting mds.g at 2024-03-12T05:08:27.370 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 bash[74249]: 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 2024-03-12T05:08:27.370 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:08:26 smithi045 systemd[1]: Started Ceph mds.g for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2107700445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:27.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:27 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2107700445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:27.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:27 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:28.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:28 smithi045 ceph-mon[32174]: Deploying daemon mds.a on smithi045 2024-03-12T05:08:28.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:28 smithi045 ceph-mon[32174]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:28.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:28 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6838/1651909238,v1:172.21.15.45:6839/1651909238] up:boot 2024-03-12T05:08:28.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:28 smithi045 ceph-mon[32174]: fsmap 10 up:standby 2024-03-12T05:08:28.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:28 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:08:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:28 smithi130 ceph-mon[35398]: Deploying daemon mds.a on smithi045 2024-03-12T05:08:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:28 smithi130 ceph-mon[35398]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:28 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6838/1651909238,v1:172.21.15.45:6839/1651909238] up:boot 2024-03-12T05:08:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:28 smithi130 ceph-mon[35398]: fsmap 10 up:standby 2024-03-12T05:08:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:28 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:08:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:28 smithi116 ceph-mon[35486]: Deploying daemon mds.a on smithi045 2024-03-12T05:08:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:28 smithi116 ceph-mon[35486]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:28 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6838/1651909238,v1:172.21.15.45:6839/1651909238] up:boot 2024-03-12T05:08:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:28 smithi116 ceph-mon[35486]: fsmap 10 up:standby 2024-03-12T05:08:28.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:28 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:08:28.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:28.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 -- 172.21.15.45:0/3838255514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ff1e0 msgr2=0x7f3bc00ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:28.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 --2- 172.21.15.45:0/3838255514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc00ff660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009900 tx=0x7f3bb002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 -- 172.21.15.45:0/3838255514 shutdown_connections 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 --2- 172.21.15.45:0/3838255514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc00ff660 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 --2- 172.21.15.45:0/3838255514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 --2- 172.21.15.45:0/3838255514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc0102770 0x7f3bc0102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 -- 172.21.15.45:0/3838255514 >> 172.21.15.45:0/3838255514 conn(0x7f3bc00fad80 msgr2=0x7f3bc00fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 -- 172.21.15.45:0/3838255514 shutdown_connections 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.803+0000 7f3bc75d0640 1 -- 172.21.15.45:0/3838255514 wait complete. 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 Processor -- start 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 -- start start 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc0197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0102770 0x7f3bc019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc019ec40 con 0x7f3bc00ffba0 2024-03-12T05:08:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc019edb0 con 0x7f3bc0102770 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc75d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc019ef70 con 0x7f3bc00ff1e0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48244/0 (socket says 172.21.15.45:48244) 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 -- 172.21.15.45:0/329235827 learned_addr learned my addr 172.21.15.45:0/329235827 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc536c640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc0197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc5b6d640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0102770 0x7f3bc019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 -- 172.21.15.45:0/329235827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc00ff1e0 msgr2=0x7f3bc0197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc0197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 -- 172.21.15.45:0/329235827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0102770 msgr2=0x7f3bc019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0102770 0x7f3bc019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 -- 172.21.15.45:0/329235827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb0009560 con 0x7f3bc00ffba0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.804+0000 7f3bc4b6b640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0036040 tx=0x7f3bb00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.805+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00044e0 con 0x7f3bc00ffba0 2024-03-12T05:08:28.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.805+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3bb0003cb0 con 0x7f3bc00ffba0 2024-03-12T05:08:28.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.805+0000 7f3bc75d0640 1 -- 172.21.15.45:0/329235827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc019f1f0 con 0x7f3bc00ffba0 2024-03-12T05:08:28.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.805+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb003e450 con 0x7f3bc00ffba0 2024-03-12T05:08:28.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.805+0000 7f3bc75d0640 1 -- 172.21.15.45:0/329235827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc019f700 con 0x7f3bc00ffba0 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.806+0000 7f3bc75d0640 1 -- 172.21.15.45:0/329235827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc01086a0 con 0x7f3bc00ffba0 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.807+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3bb0004e50 con 0x7f3bc00ffba0 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.807+0000 7f3bb67fc640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 0x7f3bac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.808+0000 7f3bc536c640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 0x7f3bac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.808+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3bb00ef300 con 0x7f3bc00ffba0 2024-03-12T05:08:28.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.808+0000 7f3bc536c640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 0x7f3bac0a3ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005fd0 tx=0x7f3bbc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:28.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:28.812+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bb00b89b0 con 0x7f3bc00ffba0 2024-03-12T05:08:29.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.053+0000 7f3bc75d0640 1 -- 172.21.15.45:0/329235827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bc010ccd0 con 0x7f3bc00ffba0 2024-03-12T05:08:29.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.057+0000 7f3bb67fc640 1 -- 172.21.15.45:0/329235827 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8465 (secure 0 0 0) 0x7f3bb00bc070 con 0x7f3bc00ffba0 2024-03-12T05:08:29.056 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:29.056 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:08:29.056 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 11 2024-03-12T05:08:29.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 msgr2=0x7f3bac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:29.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 0x7f3bac0a3ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc005fd0 tx=0x7f3bbc00b410 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 msgr2=0x7f3bc0197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0036040 tx=0x7f3bb00042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 shutdown_connections 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3bac0a1a30 0x7f3bac0a3ef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bc00ffba0 0x7f3bc0197690 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0036040 tx=0x7f3bb00042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bc0102770 0x7f3bc019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 --2- 172.21.15.45:0/329235827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bc00ff1e0 0x7f3bc0197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:29.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.061+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 >> 172.21.15.45:0/329235827 conn(0x7f3bc00fad80 msgr2=0x7f3bc0105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:29.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.062+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 shutdown_connections 2024-03-12T05:08:29.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:29.062+0000 7f3babfff640 1 -- 172.21.15.45:0/329235827 wait complete. 2024-03-12T05:08:29.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/329235827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:29.369 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 systemd[1]: Starting Ceph mds.a for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:08:29.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/329235827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:29.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/329235827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:29.722 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 podman[74610]: 2024-03-12 05:08:29.328590709 +0000 UTC m=+0.019965621 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 podman[74610]: 2024-03-12 05:08:29.429509041 +0000 UTC m=+0.120883934 container create 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 podman[74610]: 2024-03-12 05:08:29.632710643 +0000 UTC m=+0.324085549 container init 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 podman[74610]: 2024-03-12 05:08:29.635367256 +0000 UTC m=+0.326742143 container start 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 ceph-mds[74631]: -- 172.21.15.45:0/1234732712 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f64b9841a0 con 0x55f64b95ac00 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: starting mds.a at 2024-03-12T05:08:29.723 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 bash[74610]: 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 2024-03-12T05:08:30.012 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:08:29 smithi045 systemd[1]: Started Ceph mds.a for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:30.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:30.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:30.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:30.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:30 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:30.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:30 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:08:30.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:08:30.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:30 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:31.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 -- 172.21.15.45:0/3101164662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00101ca0 msgr2=0x7f1e00104090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/3101164662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00101ca0 0x7f1e00104090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009900 tx=0x7f1df002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 -- 172.21.15.45:0/3101164662 shutdown_connections 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/3101164662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00106f00 0x7f1e00109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/3101164662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e001069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/3101164662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00101ca0 0x7f1e00104090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 -- 172.21.15.45:0/3101164662 >> 172.21.15.45:0/3101164662 conn(0x7f1e000fb6b0 msgr2=0x7f1e000fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 -- 172.21.15.45:0/3101164662 shutdown_connections 2024-03-12T05:08:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.195+0000 7f1e04dd0640 1 -- 172.21.15.45:0/3101164662 wait complete. 2024-03-12T05:08:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.196+0000 7f1e04dd0640 1 Processor -- start 2024-03-12T05:08:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.196+0000 7f1e04dd0640 1 -- start start 2024-03-12T05:08:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00101ca0 0x7f1e00197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e00197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019ecf0 con 0x7f1e00106f00 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019ee60 con 0x7f1e00101ca0 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019f020 con 0x7f1e001045d0 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48286/0 (socket says 172.21.15.45:48286) 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 -- 172.21.15.45:0/2264109398 learned_addr learned my addr 172.21.15.45:0/2264109398 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfe59c640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00101ca0 0x7f1e00197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1df5d9b640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e00197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 -- 172.21.15.45:0/2264109398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 msgr2=0x7f1e00197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e00197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 -- 172.21.15.45:0/2264109398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00101ca0 msgr2=0x7f1e00197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00101ca0 0x7f1e00197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 -- 172.21.15.45:0/2264109398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0009560 con 0x7f1e00106f00 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1df5d9b640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e00197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1dfed9d640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cad0 tx=0x7f1df800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df800ae20 con 0x7f1e00106f00 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0019f300 con 0x7f1e00106f00 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1df8005020 con 0x7f1e00106f00 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df800f690 con 0x7f1e00106f00 2024-03-12T05:08:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.197+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0019f840 con 0x7f1e00106f00 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.199+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e000fd1a0 con 0x7f1e00106f00 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.200+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1df8005190 con 0x7f1e00106f00 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.200+0000 7f1df7fff640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 0x7f1dec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.200+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1df80cd390 con 0x7f1e00106f00 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.200+0000 7f1dfe59c640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 0x7f1dec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.201+0000 7f1dfe59c640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 0x7f1dec0a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009230 tx=0x7f1df0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:31.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.204+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1df8096a40 con 0x7f1e00106f00 2024-03-12T05:08:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:31 smithi045 ceph-mon[32174]: Deploying daemon mds.b on smithi116 2024-03-12T05:08:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:31 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6840/3025141330,v1:172.21.15.45:6841/3025141330] up:boot 2024-03-12T05:08:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:31 smithi045 ceph-mon[32174]: fsmap 11 up:standby 2024-03-12T05:08:31.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:31 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:08:31.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.422+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e0010cad0 con 0x7f1e00106f00 2024-03-12T05:08:31.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.423+0000 7f1df7fff640 1 -- 172.21.15.45:0/2264109398 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9257 (secure 0 0 0) 0x7f1df809a100 con 0x7f1e00106f00 2024-03-12T05:08:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:31.421 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:08:31.422 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 12 2024-03-12T05:08:31.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 msgr2=0x7f1dec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:31.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 0x7f1dec0a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009230 tx=0x7f1df0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 msgr2=0x7f1e0019e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cad0 tx=0x7f1df800cf90 comp rx=0 tx=0).stop 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 shutdown_connections 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1dec0a1a30 0x7f1dec0a3ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e00106f00 0x7f1e0019e7b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e001045d0 0x7f1e00197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 --2- 172.21.15.45:0/2264109398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e00101ca0 0x7f1e00197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 >> 172.21.15.45:0/2264109398 conn(0x7f1e000fb6b0 msgr2=0x7f1e001025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 shutdown_connections 2024-03-12T05:08:31.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:31.425+0000 7f1e04dd0640 1 -- 172.21.15.45:0/2264109398 wait complete. 2024-03-12T05:08:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:31 smithi116 ceph-mon[35486]: Deploying daemon mds.b on smithi116 2024-03-12T05:08:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:31 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6840/3025141330,v1:172.21.15.45:6841/3025141330] up:boot 2024-03-12T05:08:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:31 smithi116 ceph-mon[35486]: fsmap 11 up:standby 2024-03-12T05:08:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:31 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:08:31.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:31 smithi130 ceph-mon[35398]: Deploying daemon mds.b on smithi116 2024-03-12T05:08:31.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:31 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6840/3025141330,v1:172.21.15.45:6841/3025141330] up:boot 2024-03-12T05:08:31.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:31 smithi130 ceph-mon[35398]: fsmap 11 up:standby 2024-03-12T05:08:31.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:31 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:08:32.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:32 smithi116 ceph-mon[35486]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:32.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:32 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2264109398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:32.322 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 podman[66025]: 2024-03-12 05:08:32.189596215 +0000 UTC m=+0.017530785 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:08:32.322 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 podman[66025]: 2024-03-12 05:08:32.323644277 +0000 UTC m=+0.151578888 container create 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-12T05:08:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:32 smithi045 ceph-mon[32174]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:32.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:32 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2264109398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:32.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:32 smithi130 ceph-mon[35398]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:32.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:32 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2264109398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:32.632 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 podman[66025]: 2024-03-12 05:08:32.54321084 +0000 UTC m=+0.371145407 container init 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T05:08:32.633 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 podman[66025]: 2024-03-12 05:08:32.545780686 +0000 UTC m=+0.373715253 container start 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:08:32.633 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 ceph-mds[66044]: -- 172.21.15.116:0/3844010742 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bd0d9521a0 con 0x55bd0d928800 2024-03-12T05:08:32.633 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: starting mds.b at 2024-03-12T05:08:32.633 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 bash[66025]: 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 2024-03-12T05:08:32.994 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:32 smithi116 systemd[1]: Started Ceph mds.b for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:08:33.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 -- 172.21.15.45:0/2264983618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 msgr2=0x7f55701004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2264983618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f55701004e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5560009900 tx=0x7f556002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 -- 172.21.15.45:0/2264983618 shutdown_connections 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2264983618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f5570108500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2264983618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2264983618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f55701004e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 -- 172.21.15.45:0/2264983618 >> 172.21.15.45:0/2264983618 conn(0x7f55700fb630 msgr2=0x7f55700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 -- 172.21.15.45:0/2264983618 shutdown_connections 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 -- 172.21.15.45:0/2264983618 wait complete. 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.552+0000 7f5577d4b640 1 Processor -- start 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 -- start start 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f557019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557019ed20 con 0x7f55701000c0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557019ee90 con 0x7f55701012b0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5577d4b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557019f050 con 0x7f5570101c70 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48300/0 (socket says 172.21.15.45:48300) 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 -- 172.21.15.45:0/2006074900 learned_addr learned my addr 172.21.15.45:0/2006074900 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f55752e6640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f55762e8640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f557019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 -- 172.21.15.45:0/2006074900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 msgr2=0x7f557019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f557019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 -- 172.21.15.45:0/2006074900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 msgr2=0x7f5570197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 -- 172.21.15.45:0/2006074900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560009560 con 0x7f55701000c0 2024-03-12T05:08:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f55762e8640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f557019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f55752e6640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5575ae7640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f556002c7d0 tx=0x7f5560004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.553+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560003800 con 0x7f55701000c0 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.554+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557019f2d0 con 0x7f55701000c0 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.554+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f556002de90 con 0x7f55701000c0 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.554+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600055f0 con 0x7f55701000c0 2024-03-12T05:08:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.554+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557019f8b0 con 0x7f55701000c0 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.555+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557010b9c0 con 0x7f55701000c0 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.556+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5560004dc0 con 0x7f55701000c0 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.557+0000 7f5566ffd640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 0x7f555c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.557+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f55600ef430 con 0x7f55701000c0 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.557+0000 7f55752e6640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 0x7f555c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.557+0000 7f55752e6640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 0x7f555c0a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5570198740 tx=0x7f556c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.561+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f55600b8a40 con 0x7f55701000c0 2024-03-12T05:08:33.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:boot 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: fsmap 12 up:standby 2024-03-12T05:08:33.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:33 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:08:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.788+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557010bb50 con 0x7f55701000c0 2024-03-12T05:08:33.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.788+0000 7f5566ffd640 1 -- 172.21.15.45:0/2006074900 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10049 (secure 0 0 0) 0x7f55600bc100 con 0x7f55701000c0 2024-03-12T05:08:33.787 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:33.787 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14619,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-12T05:08:33.787 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 13 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 msgr2=0x7f555c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 0x7f555c0a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5570198740 tx=0x7f556c008040 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 msgr2=0x7f5570197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f556002c7d0 tx=0x7f5560004360 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 shutdown_connections 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f555c0a1a30 0x7f555c0a3ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5570101c70 0x7f557019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55701012b0 0x7f5570197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 --2- 172.21.15.45:0/2006074900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55701000c0 0x7f5570197230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 >> 172.21.15.45:0/2006074900 conn(0x7f55700fb630 msgr2=0x7f55700697b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 shutdown_connections 2024-03-12T05:08:33.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.790+0000 7f5577d4b640 1 -- 172.21.15.45:0/2006074900 wait complete. 2024-03-12T05:08:33.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/1642190240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd41000a0 msgr2=0x7f7bd41004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/1642190240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd41000a0 0x7f7bd41004c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009900 tx=0x7f7bd002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/1642190240 shutdown_connections 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/1642190240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 0x7f7bd41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/1642190240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd4101290 0x7f7bd4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/1642190240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd41000a0 0x7f7bd41004c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/1642190240 >> 172.21.15.45:0/1642190240 conn(0x7f7bd40fb610 msgr2=0x7f7bd40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/1642190240 shutdown_connections 2024-03-12T05:08:33.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.908+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/1642190240 wait complete. 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 Processor -- start 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 -- start start 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd41000a0 0x7f7bd41041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 0x7f7bd4102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd41034b0 con 0x7f7bd4101290 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4103620 con 0x7f7bd41000a0 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd41a11b0 con 0x7f7bd4101c50 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48318/0 (socket says 172.21.15.45:48318) 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 -- 172.21.15.45:0/587172989 learned_addr learned my addr 172.21.15.45:0/587172989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd9490640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd41000a0 0x7f7bd41041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd9c91640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 0x7f7bd4102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 -- 172.21.15.45:0/587172989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 msgr2=0x7f7bd4102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 0x7f7bd4102d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 -- 172.21.15.45:0/587172989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd41000a0 msgr2=0x7f7bd41041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd41000a0 0x7f7bd41041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 -- 172.21.15.45:0/587172989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd0009560 con 0x7f7bd4101290 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bd8c8f640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400d980 tx=0x7f7bc400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4009a30 con 0x7f7bd4101290 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7bc40044d0 con 0x7f7bd4101290 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd41a1460 con 0x7f7bd4101290 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.909+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc40047a0 con 0x7f7bd4101290 2024-03-12T05:08:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.910+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd41a1970 con 0x7f7bd4101290 2024-03-12T05:08:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.910+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd410b960 con 0x7f7bd4101290 2024-03-12T05:08:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.912+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7bc4004020 con 0x7f7bd4101290 2024-03-12T05:08:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.912+0000 7f7bca7fc640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 0x7f7bc00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:33.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.912+0000 7f7bd9490640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 0x7f7bc00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:33.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.912+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7bc40cd070 con 0x7f7bd4101290 2024-03-12T05:08:33.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.913+0000 7f7bd9490640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 0x7f7bc00a3ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7bd002c7d0 tx=0x7f7bd0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:33.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:33.916+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7bc4096720 con 0x7f7bd4101290 2024-03-12T05:08:33.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:boot 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: fsmap 12 up:standby 2024-03-12T05:08:33.971 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:33 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:08:33.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:boot 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: fsmap 12 up:standby 2024-03-12T05:08:33.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:33 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:08:34.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.098+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bd410baf0 con 0x7f7bd4101290 2024-03-12T05:08:34.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.099+0000 7f7bca7fc640 1 -- 172.21.15.45:0/587172989 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18421 (secure 0 0 0) 0x7f7bc4099de0 con 0x7f7bd4101290 2024-03-12T05:08:34.098 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:34.098 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":81,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:07:50.788069+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 msgr2=0x7f7bc00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 0x7f7bc00a3ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7bd002c7d0 tx=0x7f7bd0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 msgr2=0x7f7bd4102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7bc400d980 tx=0x7f7bc400de40 comp rx=0 tx=0).stop 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 shutdown_connections 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f7bc00a1a30 0x7f7bc00a3ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bd4101290 0x7f7bd4102810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7bd4101c50 0x7f7bd4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 --2- 172.21.15.45:0/587172989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bd41000a0 0x7f7bd41041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.101+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 >> 172.21.15.45:0/587172989 conn(0x7f7bd40fb610 msgr2=0x7f7bd40697f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.102+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 shutdown_connections 2024-03-12T05:08:34.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.102+0000 7f7bdb6f4640 1 -- 172.21.15.45:0/587172989 wait complete. 2024-03-12T05:08:34.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:08:34.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 -- 172.21.15.45:0/1081070673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 msgr2=0x7fc1c01097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1081070673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c01097a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009ca0 tx=0x7fc1b802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 -- 172.21.15.45:0/1081070673 shutdown_connections 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1081070673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c01097a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1081070673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c00ff230 0x7fc1c00ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1081070673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c01027c0 0x7fc1c0102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.219+0000 7fc1c870c640 1 -- 172.21.15.45:0/1081070673 >> 172.21.15.45:0/1081070673 conn(0x7fc1c00fadb0 msgr2=0x7fc1c00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- 172.21.15.45:0/1081070673 shutdown_connections 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- 172.21.15.45:0/1081070673 wait complete. 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 Processor -- start 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- start start 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c00ff230 0x7fc1c0101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 0x7fc1c0100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01011a0 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0101310 con 0x7fc1c00ff230 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c870c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01a1100 con 0x7fc1c01027c0 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c5ca7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c5ca7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48324/0 (socket says 172.21.15.45:48324) 2024-03-12T05:08:34.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c5ca7640 1 -- 172.21.15.45:0/1404743095 learned_addr learned my addr 172.21.15.45:0/1404743095 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c6ca9640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 0x7fc1c0100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.220+0000 7fc1c64a8640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c00ff230 0x7fc1c0101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 -- 172.21.15.45:0/1404743095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 msgr2=0x7fc1c0100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 0x7fc1c0100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 -- 172.21.15.45:0/1404743095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c00ff230 msgr2=0x7fc1c0101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c00ff230 0x7fc1c0101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 -- 172.21.15.45:0/1404743095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009900 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c6ca9640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 0x7fc1c0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c5ca7640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d980 tx=0x7fc1b000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0009a30 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c01a13b0 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1b00044d0 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00047a0 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.221+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c01a1910 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.222+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0108700 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.223+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc1b0004020 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.224+0000 7fc1b77fe640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 0x7fc1ac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.224+0000 7fc1c64a8640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 0x7fc1ac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.224+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc1b00cca50 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.225+0000 7fc1c64a8640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 0x7fc1ac0a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00694b0 tx=0x7fc1bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.229+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1b0096100 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.433+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1c010ca80 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.434+0000 7fc1b77fe640 1 -- 172.21.15.45:0/1404743095 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1b00997c0 con 0x7fc1c00ffbf0 2024-03-12T05:08:34.432 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 msgr2=0x7fc1ac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 0x7fc1ac0a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00694b0 tx=0x7fc1bc005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 msgr2=0x7fc1c0100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d980 tx=0x7fc1b000de40 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 shutdown_connections 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc1ac0a1a30 0x7fc1ac0a3ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c00ffbf0 0x7fc1c0100500 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c01027c0 0x7fc1c0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 --2- 172.21.15.45:0/1404743095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c00ff230 0x7fc1c0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 >> 172.21.15.45:0/1404743095 conn(0x7fc1c00fadb0 msgr2=0x7fc1c01058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 shutdown_connections 2024-03-12T05:08:34.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.436+0000 7fc1c870c640 1 -- 172.21.15.45:0/1404743095 wait complete. 2024-03-12T05:08:34.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 -- 172.21.15.45:0/2083817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb348101c70 msgr2=0x7fb348108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/2083817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb348101c70 0x7fb348108500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb340009ca0 tx=0x7fb34002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 -- 172.21.15.45:0/2083817886 shutdown_connections 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/2083817886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb348101c70 0x7fb348108500 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/2083817886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481012b0 0x7fb348101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/2083817886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3481000c0 0x7fb3481004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 -- 172.21.15.45:0/2083817886 >> 172.21.15.45:0/2083817886 conn(0x7fb3480fb630 msgr2=0x7fb3480fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 -- 172.21.15.45:0/2083817886 shutdown_connections 2024-03-12T05:08:34.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.557+0000 7fb34eb2c640 1 -- 172.21.15.45:0/2083817886 wait complete. 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 Processor -- start 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 -- start start 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481000c0 0x7fb3481971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb348101c70 0x7fb34819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819ecb0 con 0x7fb3481012b0 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819ee20 con 0x7fb348101c70 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34eb2c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819efe0 con 0x7fb3481000c0 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48348/0 (socket says 172.21.15.45:48348) 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 -- 172.21.15.45:0/1268625544 learned_addr learned my addr 172.21.15.45:0/1268625544 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34d0c9640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb348101c70 0x7fb34819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb34c8c8640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481000c0 0x7fb3481971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 -- 172.21.15.45:0/1268625544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481000c0 msgr2=0x7fb3481971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481000c0 0x7fb3481971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 -- 172.21.15.45:0/1268625544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb348101c70 msgr2=0x7fb34819e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb348101c70 0x7fb34819e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.558+0000 7fb347fff640 1 -- 172.21.15.45:0/1268625544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340009900 con 0x7fb3481012b0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb347fff640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb33400e9f0 tx=0x7fb33400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334009720 con 0x7fb3481012b0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34819f2c0 con 0x7fb3481012b0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb33400cd80 con 0x7fb3481012b0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334010430 con 0x7fb3481012b0 2024-03-12T05:08:34.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.559+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34819f7d0 con 0x7fb3481012b0 2024-03-12T05:08:34.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.560+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34810b980 con 0x7fb3481012b0 2024-03-12T05:08:34.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.561+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb334005020 con 0x7fb3481012b0 2024-03-12T05:08:34.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.561+0000 7fb345ffb640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 0x7fb3380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.562+0000 7fb34c8c8640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 0x7fb3380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.562+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb334014070 con 0x7fb3481012b0 2024-03-12T05:08:34.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.562+0000 7fb34c8c8640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 0x7fb3380a3f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb348101110 tx=0x7fb33c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.568+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb334096920 con 0x7fb3481012b0 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.762+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb34819fc30 con 0x7fb3481012b0 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.762+0000 7fb345ffb640 1 -- 172.21.15.45:0/1268625544 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb334099fe0 con 0x7fb3481012b0 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout: "poolnum": 1, 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout: "poolname": ".mgr" 2024-03-12T05:08:34.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:08:34.762 INFO:teuthology.orchestra.run.smithi045.stdout:] 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 msgr2=0x7fb3380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 0x7fb3380a3f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb348101110 tx=0x7fb33c005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 msgr2=0x7fb348197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb33400e9f0 tx=0x7fb33400edf0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 shutdown_connections 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb3380a1ab0 0x7fb3380a3f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3481012b0 0x7fb348197700 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb348101c70 0x7fb34819e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 --2- 172.21.15.45:0/1268625544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3481000c0 0x7fb3481971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 >> 172.21.15.45:0/1268625544 conn(0x7fb3480fb630 msgr2=0x7fb3480fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 shutdown_connections 2024-03-12T05:08:34.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.765+0000 7fb34eb2c640 1 -- 172.21.15.45:0/1268625544 wait complete. 2024-03-12T05:08:34.774 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-12T05:08:34.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 -- 172.21.15.45:0/1500064332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101c30 msgr2=0x7fdf181084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/1500064332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101c30 0x7fdf181084c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009ca0 tx=0x7fdf1002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 -- 172.21.15.45:0/1500064332 shutdown_connections 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/1500064332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101c30 0x7fdf181084c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/1500064332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18101270 0x7fdf181016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/1500064332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18100080 0x7fdf181004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 -- 172.21.15.45:0/1500064332 >> 172.21.15.45:0/1500064332 conn(0x7fdf180fb610 msgr2=0x7fdf180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 -- 172.21.15.45:0/1500064332 shutdown_connections 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.891+0000 7fdf1c97a640 1 -- 172.21.15.45:0/1500064332 wait complete. 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 Processor -- start 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 -- start start 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18100080 0x7fdf18196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18101c30 0x7fdf1819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1819db20 con 0x7fdf18101270 2024-03-12T05:08:34.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1819dc90 con 0x7fdf18101c30 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1c97a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1819de50 con 0x7fdf18100080 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48360/0 (socket says 172.21.15.45:48360) 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 -- 172.21.15.45:0/3350197887 learned_addr learned my addr 172.21.15.45:0/3350197887 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf16d9d640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18101c30 0x7fdf1819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf1659c640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18100080 0x7fdf18196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 -- 172.21.15.45:0/3350197887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18100080 msgr2=0x7fdf18196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18100080 0x7fdf18196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 -- 172.21.15.45:0/3350197887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18101c30 msgr2=0x7fdf1819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18101c30 0x7fdf1819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 -- 172.21.15.45:0/3350197887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10009900 con 0x7fdf18101270 2024-03-12T05:08:34.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.892+0000 7fdf0ffff640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000d950 tx=0x7fdf0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.893+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00009a30 con 0x7fdf18101270 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.893+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18068340 con 0x7fdf18101270 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.893+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdf000044d0 con 0x7fdf18101270 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.893+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf000047a0 con 0x7fdf18101270 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.893+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf180688a0 con 0x7fdf18101270 2024-03-12T05:08:34.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.894+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf18067f70 con 0x7fdf18101270 2024-03-12T05:08:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.895+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdf00009b90 con 0x7fdf18101270 2024-03-12T05:08:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.895+0000 7fdf0f7fe640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 0x7fdf040a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.895+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdf000cc830 con 0x7fdf18101270 2024-03-12T05:08:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.895+0000 7fdf1659c640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 0x7fdf040a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:34.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.896+0000 7fdf1659c640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 0x7fdf040a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf08005fd0 tx=0x7fdf0800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:34.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:34.900+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf00095ee0 con 0x7fdf18101270 2024-03-12T05:08:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2006074900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/587172989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1404743095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:34 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.085+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdf18106ee0 con 0x7fdf18101270 2024-03-12T05:08:35.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2006074900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/587172989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1404743095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:35.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:35.120 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:34 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2006074900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/587172989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1404743095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:08:35.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:35.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:34 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.744+0000 7fdf0f7fe640 1 -- 172.21.15.45:0/3350197887 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdf000995a0 con 0x7fdf18101270 2024-03-12T05:08:35.743 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' created 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 msgr2=0x7fdf040a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 0x7fdf040a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf08005fd0 tx=0x7fdf0800b410 comp rx=0 tx=0).stop 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 msgr2=0x7fdf18196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000d950 tx=0x7fdf0000de10 comp rx=0 tx=0).stop 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 shutdown_connections 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdf040a1a30 0x7fdf040a3ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf18101270 0x7fdf18196570 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf18101c30 0x7fdf1819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.747+0000 7fdf1c97a640 1 --2- 172.21.15.45:0/3350197887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf18100080 0x7fdf18196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.748+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 >> 172.21.15.45:0/3350197887 conn(0x7fdf180fb610 msgr2=0x7fdf18104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.748+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 shutdown_connections 2024-03-12T05:08:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.748+0000 7fdf1c97a640 1 -- 172.21.15.45:0/3350197887 wait complete. 2024-03-12T05:08:35.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-12T05:08:35.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 -- 172.21.15.45:0/3427861834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10100080 msgr2=0x7f8d101004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 --2- 172.21.15.45:0/3427861834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10100080 0x7f8d101004a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009900 tx=0x7f8d0802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 -- 172.21.15.45:0/3427861834 shutdown_connections 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 --2- 172.21.15.45:0/3427861834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101c30 0x7f8d101084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 --2- 172.21.15.45:0/3427861834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10101270 0x7f8d101016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 --2- 172.21.15.45:0/3427861834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10100080 0x7f8d101004a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 -- 172.21.15.45:0/3427861834 >> 172.21.15.45:0/3427861834 conn(0x7f8d100fb5f0 msgr2=0x7f8d100fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 -- 172.21.15.45:0/3427861834 shutdown_connections 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.871+0000 7f8d15449640 1 -- 172.21.15.45:0/3427861834 wait complete. 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 Processor -- start 2024-03-12T05:08:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 -- start start 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10100080 0x7f8d10197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 0x7f8d10197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1019ecf0 con 0x7f8d10101c30 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1019ee60 con 0x7f8d10100080 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d15449640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1019f020 con 0x7f8d10101270 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48376/0 (socket says 172.21.15.45:48376) 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 -- 172.21.15.45:0/4216743671 learned_addr learned my addr 172.21.15.45:0/4216743671 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0effd640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10100080 0x7f8d10197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0e7fc640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 0x7f8d10197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 -- 172.21.15.45:0/4216743671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 msgr2=0x7f8d10197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 0x7f8d10197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 -- 172.21.15.45:0/4216743671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10100080 msgr2=0x7f8d10197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.872+0000 7f8d0f7fe640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10100080 0x7f8d10197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8d0f7fe640 1 -- 172.21.15.45:0/4216743671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009560 con 0x7f8d10101c30 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8d0e7fc640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 0x7f8d10197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8d0f7fe640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000b970 tx=0x7f8d0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000cb40 con 0x7f8d10101c30 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d0000cca0 con 0x7f8d10101c30 2024-03-12T05:08:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1019f300 con 0x7f8d10101c30 2024-03-12T05:08:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d000076e0 con 0x7f8d10101c30 2024-03-12T05:08:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.873+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1019f840 con 0x7f8d10101c30 2024-03-12T05:08:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.874+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1010b980 con 0x7f8d10101c30 2024-03-12T05:08:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.876+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8d00014440 con 0x7f8d10101c30 2024-03-12T05:08:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.876+0000 7f8cf7fff640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 0x7f8cf80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.876+0000 7f8d0effd640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 0x7f8cf80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.876+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f8d000cceb0 con 0x7f8d10101c30 2024-03-12T05:08:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.877+0000 7f8d0effd640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 0x7f8cf80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8d08006010 tx=0x7f8d08000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:35.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:35.880+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d000fad20 con 0x7f8d10101c30 2024-03-12T05:08:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:35 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1268625544' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:08:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:35 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:35 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:08:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:35 smithi130 ceph-mon[35398]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:35.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:35 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1268625544' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:08:35.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:35 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:35.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:35 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:08:35.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:35 smithi116 ceph-mon[35486]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:36.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.064+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8d1010bb10 con 0x7f8d10101c30 2024-03-12T05:08:36.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1268625544' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:08:36.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:35 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:08:36.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:08:36.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:35 smithi045 ceph-mon[32174]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.752+0000 7f8cf7fff640 1 -- 172.21.15.45:0/4216743671 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8d000963f0 con 0x7f8d10101c30 2024-03-12T05:08:36.750 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' created 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 msgr2=0x7f8cf80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 0x7f8cf80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8d08006010 tx=0x7f8d08000bc0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 msgr2=0x7f8d1019e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000b970 tx=0x7f8d0000be30 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 shutdown_connections 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8cf80a1a30 0x7f8cf80a3ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d10101c30 0x7f8d1019e7b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d10101270 0x7f8d10197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 --2- 172.21.15.45:0/4216743671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10100080 0x7f8d10197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.754+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 >> 172.21.15.45:0/4216743671 conn(0x7f8d100fb5f0 msgr2=0x7f8d10068350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.755+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 shutdown_connections 2024-03-12T05:08:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.755+0000 7f8d15449640 1 -- 172.21.15.45:0/4216743671 wait complete. 2024-03-12T05:08:36.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-12T05:08:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 -- 172.21.15.45:0/428633158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 msgr2=0x7f21981084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:36.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 --2- 172.21.15.45:0/428633158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f21981084c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2194009ca0 tx=0x7f219402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 -- 172.21.15.45:0/428633158 shutdown_connections 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 --2- 172.21.15.45:0/428633158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f21981084c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 --2- 172.21.15.45:0/428633158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2198101270 0x7f21981016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 --2- 172.21.15.45:0/428633158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198100080 0x7f21981004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 -- 172.21.15.45:0/428633158 >> 172.21.15.45:0/428633158 conn(0x7f21980fb610 msgr2=0x7f21980fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 -- 172.21.15.45:0/428633158 shutdown_connections 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.879+0000 7f219f58a640 1 -- 172.21.15.45:0/428633158 wait complete. 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 Processor -- start 2024-03-12T05:08:36.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 -- start start 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2198100080 0x7f2198104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 0x7f2198102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981032f0 con 0x7f2198101c30 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2198103460 con 0x7f2198101270 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219f58a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981a1150 con 0x7f2198100080 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48398/0 (socket says 172.21.15.45:48398) 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 -- 172.21.15.45:0/1403631144 learned_addr learned my addr 172.21.15.45:0/1403631144 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:36.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219cb25640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 0x7f2198102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 -- 172.21.15.45:0/1403631144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2198100080 msgr2=0x7f2198104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2198100080 0x7f2198104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 -- 172.21.15.45:0/1403631144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 msgr2=0x7f2198102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 0x7f2198102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219db27640 1 -- 172.21.15.45:0/1403631144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194009900 con 0x7f2198101c30 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.880+0000 7f219cb25640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 0x7f2198102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f219db27640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2194033000 tx=0x7f2194032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194034e70 con 0x7f2198101c30 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21981a13a0 con 0x7f2198101c30 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2194003e20 con 0x7f2198101c30 2024-03-12T05:08:36.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194004880 con 0x7f2198101c30 2024-03-12T05:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.881+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21981a1850 con 0x7f2198101c30 2024-03-12T05:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.882+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219810b940 con 0x7f2198101c30 2024-03-12T05:08:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.883+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f219403b040 con 0x7f2198101c30 2024-03-12T05:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.884+0000 7f218e7fc640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 0x7f21840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.884+0000 7f219d326640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 0x7f21840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.884+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f21940efa50 con 0x7f2198101c30 2024-03-12T05:08:36.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.884+0000 7f219d326640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 0x7f21840a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21981010d0 tx=0x7f2190009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:36.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:36.888+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21940b8d70 con 0x7f2198101c30 2024-03-12T05:08:37.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.111+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f21981a1a70 con 0x7f2198101c30 2024-03-12T05:08:37.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:08:37.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:08:37.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:08:37.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:37.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:37.208 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:37.209 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3350197887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:08:37.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:08:37.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:08:37.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.766+0000 7f218e7fc640 1 -- 172.21.15.45:0/1403631144 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f21940bc430 con 0x7f2198101c30 2024-03-12T05:08:37.765 INFO:teuthology.orchestra.run.smithi045.stderr:new fs with metadata pool 2 and data pool 3 2024-03-12T05:08:37.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 msgr2=0x7f21840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 0x7f21840a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21981010d0 tx=0x7f2190009630 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 msgr2=0x7f2198102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2194033000 tx=0x7f2194032980 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 shutdown_connections 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21840a1a30 0x7f21840a3ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2198101c30 0x7f2198102b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2198101270 0x7f2198102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 --2- 172.21.15.45:0/1403631144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2198100080 0x7f2198104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.770+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 >> 172.21.15.45:0/1403631144 conn(0x7f21980fb610 msgr2=0x7f21980fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.771+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 shutdown_connections 2024-03-12T05:08:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.771+0000 7f219f58a640 1 -- 172.21.15.45:0/1403631144 wait complete. 2024-03-12T05:08:37.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 -- 172.21.15.45:0/3410775285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901011d0 msgr2=0x7fae901015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 --2- 172.21.15.45:0/3410775285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901011d0 0x7fae901015f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fae88009900 tx=0x7fae8802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 -- 172.21.15.45:0/3410775285 shutdown_connections 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 --2- 172.21.15.45:0/3410775285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae90109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 --2- 172.21.15.45:0/3410775285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901023c0 0x7fae90102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 --2- 172.21.15.45:0/3410775285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901011d0 0x7fae901015f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 -- 172.21.15.45:0/3410775285 >> 172.21.15.45:0/3410775285 conn(0x7fae900fc720 msgr2=0x7fae900feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:37.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 -- 172.21.15.45:0/3410775285 shutdown_connections 2024-03-12T05:08:37.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.900+0000 7fae9513a640 1 -- 172.21.15.45:0/3410775285 wait complete. 2024-03-12T05:08:37.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 Processor -- start 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 -- start start 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901011d0 0x7fae901970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae9019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019ebe0 con 0x7fae901023c0 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019ed50 con 0x7fae90102d80 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae9513a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019ef10 con 0x7fae901011d0 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae8e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae8e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43136/0 (socket says 172.21.15.45:43136) 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae8e59c640 1 -- 172.21.15.45:0/4059743352 learned_addr learned my addr 172.21.15.45:0/4059743352 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:37.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.901+0000 7fae8f59e640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae9019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8ed9d640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901011d0 0x7fae901970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 -- 172.21.15.45:0/4059743352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901011d0 msgr2=0x7fae901970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901011d0 0x7fae901970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 -- 172.21.15.45:0/4059743352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 msgr2=0x7fae9019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae9019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 -- 172.21.15.45:0/4059743352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88009560 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8f59e640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae9019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae8e59c640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c920 tx=0x7fae8400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84007c70 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9019f1f0 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae84007dd0 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840187c0 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.902+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9019f700 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.903+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae901084f0 con 0x7fae901023c0 2024-03-12T05:08:37.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.904+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fae8400f450 con 0x7fae901023c0 2024-03-12T05:08:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.904+0000 7fae77fff640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 0x7fae780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.905+0000 7fae8ed9d640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 0x7fae780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.905+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fae840cd800 con 0x7fae901023c0 2024-03-12T05:08:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.905+0000 7fae8ed9d640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 0x7fae780a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fae88036040 tx=0x7fae88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:37.908+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae84096b50 con 0x7fae901023c0 2024-03-12T05:08:38.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.089+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae9010ca90 con 0x7fae901023c0 2024-03-12T05:08:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.090+0000 7fae77fff640 1 -- 172.21.15.45:0/4059743352 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22197 (secure 0 0 0) 0x7fae8409a210 con 0x7fae901023c0 2024-03-12T05:08:38.089 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:38.089 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:08:37.112615+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 msgr2=0x7fae780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 0x7fae780a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fae88036040 tx=0x7fae88005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 msgr2=0x7fae90197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c920 tx=0x7fae8400cde0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 shutdown_connections 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fae780a1a30 0x7fae780a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae901023c0 0x7fae90197630 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae90102d80 0x7fae9019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 --2- 172.21.15.45:0/4059743352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae901011d0 0x7fae901970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 >> 172.21.15.45:0/4059743352 conn(0x7fae900fc720 msgr2=0x7fae901056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 shutdown_connections 2024-03-12T05:08:38.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.092+0000 7fae9513a640 1 -- 172.21.15.45:0/4059743352 wait complete. 2024-03-12T05:08:38.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:37 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:08:38.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:37 smithi045 ceph-mon[32174]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:08:38.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:37 smithi045 ceph-mon[32174]: pgmap v280: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:38.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:37 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:08:38.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:37 smithi045 ceph-mon[32174]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:08:38.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 -- 172.21.15.45:0/1322298592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780104650 msgr2=0x7f4780106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 --2- 172.21.15.45:0/1322298592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780104650 0x7f4780106a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4774009900 tx=0x7f477402bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 -- 172.21.15.45:0/1322298592 shutdown_connections 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 --2- 172.21.15.45:0/1322298592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780104650 0x7f4780106a40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 --2- 172.21.15.45:0/1322298592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4780106f80 0x7f4780109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 --2- 172.21.15.45:0/1322298592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.206+0000 7f47858eb640 1 -- 172.21.15.45:0/1322298592 >> 172.21.15.45:0/1322298592 conn(0x7f47800fb6b0 msgr2=0x7f47800fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- 172.21.15.45:0/1322298592 shutdown_connections 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- 172.21.15.45:0/1322298592 wait complete. 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 Processor -- start 2024-03-12T05:08:38.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- start start 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4780104650 0x7f4780197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478019ec20 con 0x7f4780106f80 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478019ed90 con 0x7f4780101d20 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f47858eb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478019ef50 con 0x7f4780104650 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f477f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f477f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43158/0 (socket says 172.21.15.45:43158) 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.207+0000 7f477f7fe640 1 -- 172.21.15.45:0/289884839 learned_addr learned my addr 172.21.15.45:0/289884839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477effd640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 -- 172.21.15.45:0/289884839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4780104650 msgr2=0x7f4780197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4780104650 0x7f4780197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 -- 172.21.15.45:0/289884839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 msgr2=0x7f4780197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 -- 172.21.15.45:0/289884839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774009560 con 0x7f4780106f80 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477effd640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f477f7fe640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f477800cce0 tx=0x7f477800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800f450 con 0x7f4780106f80 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478019f230 con 0x7f4780106f80 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f477800fa60 con 0x7f4780106f80 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778018b10 con 0x7f4780106f80 2024-03-12T05:08:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.208+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478019f6f0 con 0x7f4780106f80 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.209+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47800fd1a0 con 0x7f4780106f80 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.210+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f477800fbd0 con 0x7f4780106f80 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.211+0000 7f47848e9640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 0x7f47680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.211+0000 7f477effd640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 0x7f47680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.211+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f47780cd790 con 0x7f4780106f80 2024-03-12T05:08:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.211+0000 7f477effd640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 0x7f47680a3ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4770009fc0 tx=0x7f4770005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.215+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4778096a50 con 0x7f4780106f80 2024-03-12T05:08:38.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:37 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:08:38.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:37 smithi130 ceph-mon[35398]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:08:38.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:37 smithi130 ceph-mon[35398]: pgmap v280: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:38.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:37 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:08:38.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:37 smithi130 ceph-mon[35398]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:08:38.244 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:37 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4216743671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:08:38.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:37 smithi116 ceph-mon[35486]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:08:38.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:37 smithi116 ceph-mon[35486]: pgmap v280: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:08:38.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:37 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:08:38.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:37 smithi116 ceph-mon[35486]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:08:38.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.399+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47800fcc10 con 0x7f4780106f80 2024-03-12T05:08:38.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.401+0000 7f47848e9640 1 -- 172.21.15.45:0/289884839 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+22197 (secure 0 0 0) 0x7f477809a110 con 0x7f4780106f80 2024-03-12T05:08:38.399 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:38.399 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:08:37.112615+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 msgr2=0x7f47680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 0x7f47680a3ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4770009fc0 tx=0x7f4770005f90 comp rx=0 tx=0).stop 2024-03-12T05:08:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 msgr2=0x7f478019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f477800cce0 tx=0x7f477800a620 comp rx=0 tx=0).stop 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 shutdown_connections 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f47680a1a30 0x7f47680a3ef0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4780106f80 0x7f478019e6e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4780104650 0x7f4780197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 --2- 172.21.15.45:0/289884839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4780101d20 0x7f4780197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 >> 172.21.15.45:0/289884839 conn(0x7f47800fb6b0 msgr2=0x7f47801027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 shutdown_connections 2024-03-12T05:08:38.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.403+0000 7f47858eb640 1 -- 172.21.15.45:0/289884839 wait complete. 2024-03-12T05:08:38.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 -- 172.21.15.45:0/4101095065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 msgr2=0x7f534c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 --2- 172.21.15.45:0/4101095065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c100e90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5344009ca0 tx=0x7f534402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 -- 172.21.15.45:0/4101095065 shutdown_connections 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 --2- 172.21.15.45:0/4101095065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c100e90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 --2- 172.21.15.45:0/4101095065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c0ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 --2- 172.21.15.45:0/4101095065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 0x7f534c0ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 -- 172.21.15.45:0/4101095065 >> 172.21.15.45:0/4101095065 conn(0x7f534c0fad70 msgr2=0x7f534c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 -- 172.21.15.45:0/4101095065 shutdown_connections 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.520+0000 7f53537c7640 1 -- 172.21.15.45:0/4101095065 wait complete. 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 Processor -- start 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 -- start start 2024-03-12T05:08:38.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 0x7f534c197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c19eb80 con 0x7f534c100a10 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c19ecf0 con 0x7f534c069370 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f53537c7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c19eeb0 con 0x7f534c068ec0 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43178/0 (socket says 172.21.15.45:43178) 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 -- 172.21.15.45:0/1401337201 learned_addr learned my addr 172.21.15.45:0/1401337201 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5350d62640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351563640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 0x7f534c197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 -- 172.21.15.45:0/1401337201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 msgr2=0x7f534c197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 0x7f534c197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 -- 172.21.15.45:0/1401337201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 msgr2=0x7f534c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5351d64640 1 -- 172.21.15.45:0/1401337201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5344009900 con 0x7f534c100a10 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.521+0000 7f5350d62640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f5351d64640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5344037040 tx=0x7f5344032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344034e70 con 0x7f534c100a10 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5344003e20 con 0x7f534c100a10 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c19f130 con 0x7f534c100a10 2024-03-12T05:08:38.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534403d3f0 con 0x7f534c100a10 2024-03-12T05:08:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.522+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c19f630 con 0x7f534c100a10 2024-03-12T05:08:38.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.523+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534c108670 con 0x7f534c100a10 2024-03-12T05:08:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.524+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f534403b040 con 0x7f534c100a10 2024-03-12T05:08:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.525+0000 7f53427fc640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 0x7f53380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.525+0000 7f5351563640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 0x7f53380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.525+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f53440efec0 con 0x7f534c100a10 2024-03-12T05:08:38.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.525+0000 7f5351563640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 0x7f53380a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5348005fd0 tx=0x7f5348005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.529+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f53440b9220 con 0x7f534c100a10 2024-03-12T05:08:38.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.749+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f534c10b970 con 0x7f534c100a10 2024-03-12T05:08:38.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.767+0000 7f53427fc640 1 -- 172.21.15.45:0/1401337201 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f53440390b0 con 0x7f534c100a10 2024-03-12T05:08:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 msgr2=0x7f53380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 0x7f53380a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5348005fd0 tx=0x7f5348005850 comp rx=0 tx=0).stop 2024-03-12T05:08:38.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 msgr2=0x7f534c19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5344037040 tx=0x7f5344032980 comp rx=0 tx=0).stop 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 shutdown_connections 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f53380a1ab0 0x7f53380a3f70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f534c100a10 0x7f534c19e640 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f534c069370 0x7f534c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 --2- 172.21.15.45:0/1401337201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f534c068ec0 0x7f534c197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 >> 172.21.15.45:0/1401337201 conn(0x7f534c0fad70 msgr2=0x7f534c1024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 shutdown_connections 2024-03-12T05:08:38.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.769+0000 7f53537c7640 1 -- 172.21.15.45:0/1401337201 wait complete. 2024-03-12T05:08:38.778 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-12T05:08:38.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- 172.21.15.45:0/785482076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20101c00 msgr2=0x7efd20103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 --2- 172.21.15.45:0/785482076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20101c00 0x7efd20103ff0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efd18009ca0 tx=0x7efd1802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- 172.21.15.45:0/785482076 shutdown_connections 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 --2- 172.21.15.45:0/785482076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20106e60 0x7efd201092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 --2- 172.21.15.45:0/785482076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd20104530 0x7efd20106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 --2- 172.21.15.45:0/785482076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20101c00 0x7efd20103ff0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- 172.21.15.45:0/785482076 >> 172.21.15.45:0/785482076 conn(0x7efd200fb610 msgr2=0x7efd200fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:38.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- 172.21.15.45:0/785482076 shutdown_connections 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- 172.21.15.45:0/785482076 wait complete. 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 Processor -- start 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.886+0000 7efd2861c640 1 -- start start 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 0x7efd20197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd20106e60 0x7efd2019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2019ed00 con 0x7efd20104530 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2019ee70 con 0x7efd20106e60 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2019f030 con 0x7efd20101c00 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43190/0 (socket says 172.21.15.45:43190) 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 -- 172.21.15.45:0/626065339 learned_addr learned my addr 172.21.15.45:0/626065339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd263b8640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 0x7efd20197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 -- 172.21.15.45:0/626065339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 msgr2=0x7efd20197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 0x7efd20197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 -- 172.21.15.45:0/626065339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd20106e60 msgr2=0x7efd2019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd20106e60 0x7efd2019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 -- 172.21.15.45:0/626065339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18009900 con 0x7efd20104530 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd263b8640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 0x7efd20197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd25bb7640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c970 tx=0x7efd1400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14007bb0 con 0x7efd20104530 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd14007d10 con 0x7efd20104530 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2019f310 con 0x7efd20104530 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1400f450 con 0x7efd20104530 2024-03-12T05:08:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.887+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2019f8f0 con 0x7efd20104530 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.888+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd200fd100 con 0x7efd20104530 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.889+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efd14004eb0 con 0x7efd20104530 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.890+0000 7efd1f7fe640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 0x7efd080a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.890+0000 7efd263b8640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 0x7efd080a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.890+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efd140cd3c0 con 0x7efd20104530 2024-03-12T05:08:38.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.891+0000 7efd263b8640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 0x7efd080a3f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efd18006010 tx=0x7efd180325f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:38.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:38.895+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd14096680 con 0x7efd20104530 2024-03-12T05:08:39.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.115+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7efd2010caa0 con 0x7efd20104530 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:08:39.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4059743352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/289884839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:08:39.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:38 smithi045 ceph-mon[32174]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4059743352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/289884839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:08:39.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:38 smithi130 ceph-mon[35398]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1403631144' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4059743352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/289884839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:08:39.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:38 smithi116 ceph-mon[35486]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:08:39.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.319+0000 7efd1f7fe640 1 -- 172.21.15.45:0/626065339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) v1 ==== 105+0+0 (secure 0 0 0) 0x7efd2010caa0 con 0x7efd20104530 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 msgr2=0x7efd080a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 0x7efd080a3f70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efd18006010 tx=0x7efd180325f0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 msgr2=0x7efd201977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c970 tx=0x7efd1400ce30 comp rx=0 tx=0).stop 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 shutdown_connections 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efd080a1ab0 0x7efd080a3f70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd20104530 0x7efd201977a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd20106e60 0x7efd2019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 --2- 172.21.15.45:0/626065339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd20101c00 0x7efd20197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 >> 172.21.15.45:0/626065339 conn(0x7efd200fb610 msgr2=0x7efd201026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:39.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 shutdown_connections 2024-03-12T05:08:39.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.322+0000 7efd2861c640 1 -- 172.21.15.45:0/626065339 wait complete. 2024-03-12T05:08:39.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-12T05:08:39.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- 172.21.15.45:0/4284108949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 msgr2=0x7f3628101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 --2- 172.21.15.45:0/4284108949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f3628101700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3618009900 tx=0x7f361802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- 172.21.15.45:0/4284108949 shutdown_connections 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 --2- 172.21.15.45:0/4284108949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f3628101700 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 --2- 172.21.15.45:0/4284108949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f36281084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 --2- 172.21.15.45:0/4284108949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 0x7f36281004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- 172.21.15.45:0/4284108949 >> 172.21.15.45:0/4284108949 conn(0x7f36280fb620 msgr2=0x7f36280fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- 172.21.15.45:0/4284108949 shutdown_connections 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- 172.21.15.45:0/4284108949 wait complete. 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 Processor -- start 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.444+0000 7f362fd3a640 1 -- start start 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 0x7f362806a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:39.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f362819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819ece0 con 0x7f3628101280 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819ee50 con 0x7f3628101c40 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362819f010 con 0x7f3628100090 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43210/0 (socket says 172.21.15.45:43210) 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 -- 172.21.15.45:0/941742282 learned_addr learned my addr 172.21.15.45:0/941742282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362dad6640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 0x7f362806a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362e2d7640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f362819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 -- 172.21.15.45:0/941742282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 msgr2=0x7f362806a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 0x7f362806a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 -- 172.21.15.45:0/941742282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 msgr2=0x7f362819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f362819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 -- 172.21.15.45:0/941742282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618009560 con 0x7f3628101280 2024-03-12T05:08:39.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362e2d7640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f362819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362d2d5640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3618036040 tx=0x7f36180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36180044e0 con 0x7f3628101280 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f361802de90 con 0x7f3628101280 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362819f230 con 0x7f3628101280 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618005470 con 0x7f3628101280 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.445+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362819f790 con 0x7f3628101280 2024-03-12T05:08:39.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.446+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3628107470 con 0x7f3628101280 2024-03-12T05:08:39.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.447+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3618003800 con 0x7f3628101280 2024-03-12T05:08:39.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.448+0000 7f361effd640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 0x7f36140a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:39.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.448+0000 7f362dad6640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 0x7f36140a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:39.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.448+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f36180ef490 con 0x7f3628101280 2024-03-12T05:08:39.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.448+0000 7f362dad6640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 0x7f36140a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f36200058c0 tx=0x7f362000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:39.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.452+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f36180b87f0 con 0x7f3628101280 2024-03-12T05:08:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:39.674+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f362810bb00 con 0x7f3628101280 2024-03-12T05:08:40.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:active 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: pgmap v283: 97 pgs: 6 creating+peering, 37 active+clean, 54 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:08:40.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:08:40.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:08:40.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:active 2024-03-12T05:08:40.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:08:40.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: pgmap v283: 97 pgs: 6 creating+peering, 37 active+clean, 54 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:08:40.221 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1401337201' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] up:active 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: pgmap v283: 97 pgs: 6 creating+peering, 37 active+clean, 54 unknown; 578 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 7 op/s 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/626065339' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:08:40.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:08:40.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.326+0000 7f361effd640 1 -- 172.21.15.45:0/941742282 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) v1 ==== 121+0+0 (secure 0 0 0) 0x7f36180bbeb0 con 0x7f3628101280 2024-03-12T05:08:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 msgr2=0x7f36140a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:40.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 0x7f36140a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f36200058c0 tx=0x7f362000a2d0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 msgr2=0x7f362819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3618036040 tx=0x7f36180042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 shutdown_connections 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36140a1ab0 0x7f36140a3f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3628101280 0x7f362819bee0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3628101c40 0x7f362819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 --2- 172.21.15.45:0/941742282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3628100090 0x7f362806a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 >> 172.21.15.45:0/941742282 conn(0x7f36280fb620 msgr2=0x7f3628104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 shutdown_connections 2024-03-12T05:08:40.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.329+0000 7f362fd3a640 1 -- 172.21.15.45:0/941742282 wait complete. 2024-03-12T05:08:40.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- 172.21.15.45:0/3796227194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101c50 msgr2=0x7f79cc1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 --2- 172.21.15.45:0/3796227194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101c50 0x7f79cc1084e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f79bc009ca0 tx=0x7f79bc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- 172.21.15.45:0/3796227194 shutdown_connections 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 --2- 172.21.15.45:0/3796227194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101c50 0x7f79cc1084e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 --2- 172.21.15.45:0/3796227194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79cc101290 0x7f79cc101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 --2- 172.21.15.45:0/3796227194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc1000a0 0x7f79cc1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- 172.21.15.45:0/3796227194 >> 172.21.15.45:0/3796227194 conn(0x7f79cc0fb630 msgr2=0x7f79cc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- 172.21.15.45:0/3796227194 shutdown_connections 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- 172.21.15.45:0/3796227194 wait complete. 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 Processor -- start 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.458+0000 7f79d3688640 1 -- start start 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79cc1000a0 0x7f79cc1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc101c50 0x7f79cc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc19eba0 con 0x7f79cc101290 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc19ed10 con 0x7f79cc1000a0 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc19eed0 con 0x7f79cc101c50 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43236/0 (socket says 172.21.15.45:43236) 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 -- 172.21.15.45:0/1629252151 learned_addr learned my addr 172.21.15.45:0/1629252151 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d1c25640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc101c50 0x7f79cc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:40.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 -- 172.21.15.45:0/1629252151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc101c50 msgr2=0x7f79cc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc101c50 0x7f79cc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 -- 172.21.15.45:0/1629252151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79cc1000a0 msgr2=0x7f79cc1970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79cc1000a0 0x7f79cc1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 -- 172.21.15.45:0/1629252151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc009900 con 0x7f79cc101290 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d0c23640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d950 tx=0x7f79c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c0009c30 con 0x7f79cc101290 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f79c0009d90 con 0x7f79cc101290 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79cc19f1b0 con 0x7f79cc101290 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c000f6b0 con 0x7f79cc101290 2024-03-12T05:08:40.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.459+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79cc19f710 con 0x7f79cc101290 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.460+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79cc10b960 con 0x7f79cc101290 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.461+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f79c0010aa0 con 0x7f79cc101290 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.462+0000 7f79ca7fc640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 0x7f79b40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.462+0000 7f79d1424640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 0x7f79b40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.462+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f79c00ccf30 con 0x7f79cc101290 2024-03-12T05:08:40.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.462+0000 7f79d1424640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 0x7f79b40a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:40.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.465+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f79c00fad20 con 0x7f79cc101290 2024-03-12T05:08:40.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:40.692+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f79cc19f930 con 0x7f79cc101290 2024-03-12T05:08:41.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-12T05:08:41.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:08:41.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:active 2024-03-12T05:08:41.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:08:41.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-12T05:08:41.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-12T05:08:41.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-12T05:08:41.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:active 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-12T05:08:41.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/941742282' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] up:active 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-12T05:08:41.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:08:41.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.333+0000 7f79ca7fc640 1 -- 172.21.15.45:0/1629252151 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f79c0017020 con 0x7f79cc101290 2024-03-12T05:08:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 msgr2=0x7f79b40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 0x7f79b40a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005fd0 tx=0x7f79c400b410 comp rx=0 tx=0).stop 2024-03-12T05:08:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 msgr2=0x7f79cc1975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f79c000d950 tx=0x7f79c000de10 comp rx=0 tx=0).stop 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 shutdown_connections 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f79b40a1ab0 0x7f79b40a3f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79cc101290 0x7f79cc1975f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f79cc101c50 0x7f79cc19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 --2- 172.21.15.45:0/1629252151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79cc1000a0 0x7f79cc1970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 >> 172.21.15.45:0/1629252151 conn(0x7f79cc0fb630 msgr2=0x7f79cc069750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 shutdown_connections 2024-03-12T05:08:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.336+0000 7f79d3688640 1 -- 172.21.15.45:0/1629252151 wait complete. 2024-03-12T05:08:41.350 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-12T05:08:41.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-12T05:08:41.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 -- 172.21.15.45:0/2219278379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b701023a0 msgr2=0x7f2b70102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:41.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 --2- 172.21.15.45:0/2219278379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b701023a0 0x7f2b70102820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009900 tx=0x7f2b6402bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 -- 172.21.15.45:0/2219278379 shutdown_connections 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 --2- 172.21.15.45:0/2219278379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b701023a0 0x7f2b70102820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 --2- 172.21.15.45:0/2219278379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b70102d60 0x7f2b701095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 --2- 172.21.15.45:0/2219278379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701011b0 0x7f2b701015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 -- 172.21.15.45:0/2219278379 >> 172.21.15.45:0/2219278379 conn(0x7f2b700fc720 msgr2=0x7f2b700feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 -- 172.21.15.45:0/2219278379 shutdown_connections 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.464+0000 7f2b76548640 1 -- 172.21.15.45:0/2219278379 wait complete. 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 Processor -- start 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 -- start start 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b701011b0 0x7f2b70197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 0x7f2b70197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:41.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019ec40 con 0x7f2b70102d60 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019edb0 con 0x7f2b701023a0 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b76548640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7019ef70 con 0x7f2b701011b0 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.465+0000 7f2b74ae5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43244/0 (socket says 172.21.15.45:43244) 2024-03-12T05:08:41.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 -- 172.21.15.45:0/56460682 learned_addr learned my addr 172.21.15.45:0/56460682 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6ffff640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b701011b0 0x7f2b70197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6f7fe640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 0x7f2b70197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 -- 172.21.15.45:0/56460682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b701011b0 msgr2=0x7f2b70197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b701011b0 0x7f2b70197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 -- 172.21.15.45:0/56460682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 msgr2=0x7f2b70197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 0x7f2b70197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 -- 172.21.15.45:0/56460682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009560 con 0x7f2b70102d60 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6f7fe640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 0x7f2b70197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b74ae5640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800eaa0 tx=0x7f2b6800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6800cdb0 con 0x7f2b70102d60 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7019f250 con 0x7f2b70102d60 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2b68005020 con 0x7f2b70102d60 2024-03-12T05:08:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b680116d0 con 0x7f2b70102d60 2024-03-12T05:08:41.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.466+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7019f760 con 0x7f2b70102d60 2024-03-12T05:08:41.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.467+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70108570 con 0x7f2b70102d60 2024-03-12T05:08:41.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.469+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2b68005190 con 0x7f2b70102d60 2024-03-12T05:08:41.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.469+0000 7f2b6d7fa640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 0x7f2b580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:41.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.469+0000 7f2b6ffff640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 0x7f2b580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:41.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.469+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2b680cd4f0 con 0x7f2b70102d60 2024-03-12T05:08:41.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.470+0000 7f2b6ffff640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 0x7f2b580a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b60005db0 tx=0x7f2b60005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:41.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.473+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b68096850 con 0x7f2b70102d60 2024-03-12T05:08:41.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:41.681+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b7002ceb0 con 0x7f2b580a1a30 2024-03-12T05:08:42.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: pgmap v284: 97 pgs: 6 creating+peering, 91 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:active 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-mon[32174]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:08:41.686+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:08:41.686+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:08:41.686+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:08:41.686+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:08:42.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:08:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:08:41.686+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: pgmap v284: 97 pgs: 6 creating+peering, 91 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:active 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-12T05:08:42.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:41 smithi130 ceph-mon[35398]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: pgmap v284: 97 pgs: 6 creating+peering, 91 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1629252151' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] up:active 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-12T05:08:42.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:41 smithi116 ceph-mon[35486]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-12T05:08:42.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.370+0000 7f2b6d7fa640 1 -- 172.21.15.45:0/56460682 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2b7002ceb0 con 0x7f2b580a1a30 2024-03-12T05:08:42.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 msgr2=0x7f2b580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 0x7f2b580a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b60005db0 tx=0x7f2b60005d40 comp rx=0 tx=0).stop 2024-03-12T05:08:42.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 msgr2=0x7f2b7019e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800eaa0 tx=0x7f2b6800ef60 comp rx=0 tx=0).stop 2024-03-12T05:08:42.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 shutdown_connections 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2b580a1a30 0x7f2b580a3ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b70102d60 0x7f2b7019e700 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b701023a0 0x7f2b70197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 --2- 172.21.15.45:0/56460682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b701011b0 0x7f2b70197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 >> 172.21.15.45:0/56460682 conn(0x7f2b700fc720 msgr2=0x7f2b70105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 shutdown_connections 2024-03-12T05:08:42.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.373+0000 7f2b76548640 1 -- 172.21.15.45:0/56460682 wait complete. 2024-03-12T05:08:42.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-12T05:08:42.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2992775268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 msgr2=0x7fcb581047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2992775268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb581047f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009900 tx=0x7fcb4802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2992775268 shutdown_connections 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2992775268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb581047f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2992775268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2992775268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb58101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2992775268 >> 172.21.15.45:0/2992775268 conn(0x7fcb580fb630 msgr2=0x7fcb580fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2992775268 shutdown_connections 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.500+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2992775268 wait complete. 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 Processor -- start 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 -- start start 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb5819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581a52a0 con 0x7fcb58102380 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581a5410 con 0x7fcb580ffa10 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5f8a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb581a55d0 con 0x7fcb58104d30 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43280/0 (socket says 172.21.15.45:43280) 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 -- 172.21.15.45:0/2267726429 learned_addr learned my addr 172.21.15.45:0/2267726429 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5d643640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb5819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5de44640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 -- 172.21.15.45:0/2267726429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 msgr2=0x7fcb581a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 -- 172.21.15.45:0/2267726429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 msgr2=0x7fcb5819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb5819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 -- 172.21.15.45:0/2267726429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009560 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5de44640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5d643640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb5819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb5ce42640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcb48036040 tx=0x7fcb480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.501+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480044e0 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.502+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb581a5850 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.502+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcb4802de90 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.502+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48005470 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.502+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581a5e30 con 0x7fcb58102380 2024-03-12T05:08:42.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.502+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580fd120 con 0x7fcb58102380 2024-03-12T05:08:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.503+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcb48003800 con 0x7fcb58102380 2024-03-12T05:08:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.504+0000 7fcb4e7fc640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 0x7fcb440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.504+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcb480ef5e0 con 0x7fcb58102380 2024-03-12T05:08:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.504+0000 7fcb5d643640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 0x7fcb440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.505+0000 7fcb5d643640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 0x7fcb440a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb540058c0 tx=0x7fcb54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:42.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.508+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb480b8940 con 0x7fcb58102380 2024-03-12T05:08:42.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.719+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb5802ceb0 con 0x7fcb440a1a30 2024-03-12T05:08:42.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.740+0000 7fcb4e7fc640 1 -- 172.21.15.45:0/2267726429 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcb5802ceb0 con 0x7fcb440a1a30 2024-03-12T05:08:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.742+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 msgr2=0x7fcb440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.742+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 0x7fcb440a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb540058c0 tx=0x7fcb54005850 comp rx=0 tx=0).stop 2024-03-12T05:08:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 msgr2=0x7fcb5819dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcb48036040 tx=0x7fcb480042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 shutdown_connections 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fcb440a1a30 0x7fcb440a3ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb58102380 0x7fcb5819dcf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb58104d30 0x7fcb581a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 --2- 172.21.15.45:0/2267726429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb580ffa10 0x7fcb5819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 >> 172.21.15.45:0/2267726429 conn(0x7fcb580fb630 msgr2=0x7fcb58100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 shutdown_connections 2024-03-12T05:08:42.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.743+0000 7fcb5f8a7640 1 -- 172.21.15.45:0/2267726429 wait complete. 2024-03-12T05:08:42.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 -- 172.21.15.45:0/10410266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130100080 msgr2=0x7f11301004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 --2- 172.21.15.45:0/10410266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130100080 0x7f11301004a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f112c009900 tx=0x7f112c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 -- 172.21.15.45:0/10410266 shutdown_connections 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 --2- 172.21.15.45:0/10410266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 0x7f11301084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 --2- 172.21.15.45:0/10410266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130101270 0x7f11301016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 --2- 172.21.15.45:0/10410266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130100080 0x7f11301004a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 -- 172.21.15.45:0/10410266 >> 172.21.15.45:0/10410266 conn(0x7f11300fb610 msgr2=0x7f11300fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 -- 172.21.15.45:0/10410266 shutdown_connections 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.861+0000 7f1137df4640 1 -- 172.21.15.45:0/10410266 wait complete. 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 Processor -- start 2024-03-12T05:08:42.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 -- start start 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 0x7f1130197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 0x7f113019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113019ec80 con 0x7f1130101270 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113019edf0 con 0x7f1130101c30 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1137df4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113019efb0 con 0x7f1130100080 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43298/0 (socket says 172.21.15.45:43298) 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 -- 172.21.15.45:0/3424416296 learned_addr learned my addr 172.21.15.45:0/3424416296 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1136391640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 0x7f113019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1135b90640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 0x7f1130197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 -- 172.21.15.45:0/3424416296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 msgr2=0x7f1130197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 0x7f1130197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 -- 172.21.15.45:0/3424416296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 msgr2=0x7f113019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:42.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 0x7f113019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f113538f640 1 -- 172.21.15.45:0/3424416296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009560 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.862+0000 7f1135b90640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 0x7f1130197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f113538f640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f112000d770 tx=0x7f112000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120009c30 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f112000de80 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113019f290 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120004e00 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.863+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113019f750 con 0x7f1130101270 2024-03-12T05:08:42.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.864+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11301074c0 con 0x7f1130101270 2024-03-12T05:08:42.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.865+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1120004f60 con 0x7f1130101270 2024-03-12T05:08:42.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.865+0000 7f1126ffd640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 0x7f111c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:42.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.866+0000 7f1135b90640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 0x7f111c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:42.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.866+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f11200ccde0 con 0x7f1130101270 2024-03-12T05:08:42.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.866+0000 7f1135b90640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 0x7f111c0a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:42.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:42.870+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f11200960a0 con 0x7f1130101270 2024-03-12T05:08:43.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.080+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f113002ce40 con 0x7f111c0a1a30 2024-03-12T05:08:43.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.084+0000 7f1126ffd640 1 -- 172.21.15.45:0/3424416296 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f113002ce40 con 0x7f111c0a1a30 2024-03-12T05:08:43.082 INFO:teuthology.orchestra.run.smithi045.stdout:/volumes/qa/sv_0/aa62c577-3bcc-4fae-b8a4-44471f267da8 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.085+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 msgr2=0x7f111c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.085+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 0x7f111c0a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c005c00 comp rx=0 tx=0).stop 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 msgr2=0x7f11301976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f112000d770 tx=0x7f112000dc30 comp rx=0 tx=0).stop 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 shutdown_connections 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f111c0a1a30 0x7f111c0a3ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1130101270 0x7f11301976d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1130101c30 0x7f113019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 --2- 172.21.15.45:0/3424416296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1130100080 0x7f1130197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 >> 172.21.15.45:0/3424416296 conn(0x7f11300fb610 msgr2=0x7f1130104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 shutdown_connections 2024-03-12T05:08:43.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.086+0000 7f1137df4640 1 -- 172.21.15.45:0/3424416296 wait complete. 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:active 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: Cluster is now healthy 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-12T05:08:43.094 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:42 smithi045 ceph-mon[32174]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-12T05:08:43.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:active 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: Cluster is now healthy 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-12T05:08:43.129 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-12T05:08:43.130 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:42 smithi116 ceph-mon[35486]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-12T05:08:43.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 -- 172.21.15.45:0/4247022269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 msgr2=0x7f2138101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 --2- 172.21.15.45:0/4247022269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138101700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2128009900 tx=0x7f212802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 -- 172.21.15.45:0/4247022269 shutdown_connections 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 --2- 172.21.15.45:0/4247022269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138101700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 --2- 172.21.15.45:0/4247022269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 0x7f21381084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 --2- 172.21.15.45:0/4247022269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 0x7f21381004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 -- 172.21.15.45:0/4247022269 >> 172.21.15.45:0/4247022269 conn(0x7f21380fb620 msgr2=0x7f21380fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 -- 172.21.15.45:0/4247022269 shutdown_connections 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.204+0000 7f214022a640 1 -- 172.21.15.45:0/4247022269 wait complete. 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 Processor -- start 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 -- start start 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 0x7f2138195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 0x7f213819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213819dab0 con 0x7f2138101280 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213819dc20 con 0x7f2138100090 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f214022a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213819dde0 con 0x7f2138101c40 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43314/0 (socket says 172.21.15.45:43314) 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 -- 172.21.15.45:0/3587178403 learned_addr learned my addr 172.21.15.45:0/3587178403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213e7c7640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 0x7f213819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213dfc6640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 0x7f2138195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 -- 172.21.15.45:0/3587178403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 msgr2=0x7f213819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 0x7f213819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 -- 172.21.15.45:0/3587178403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 msgr2=0x7f2138195fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 0x7f2138195fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 -- 172.21.15.45:0/3587178403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128009560 con 0x7f2138101280 2024-03-12T05:08:43.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.205+0000 7f213d7c5640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2128036040 tx=0x7f21280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.206+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21280044e0 con 0x7f2138101280 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.206+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213819e060 con 0x7f2138101280 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.206+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f212802de90 con 0x7f2138101280 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.206+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128005470 con 0x7f2138101280 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.206+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213819e570 con 0x7f2138101280 2024-03-12T05:08:43.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.207+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21381074a0 con 0x7f2138101280 2024-03-12T05:08:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.208+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2128003800 con 0x7f2138101280 2024-03-12T05:08:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.208+0000 7f212effd640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 0x7f21240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.208+0000 7f213dfc6640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 0x7f21240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.208+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f21280efd30 con 0x7f2138101280 2024-03-12T05:08:43.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.209+0000 7f213dfc6640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 0x7f21240a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f21381010e0 tx=0x7f2130009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.212+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21280b9090 con 0x7f2138101280 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] up:active 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: Cluster is now healthy 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-12T05:08:43.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:42 smithi130 ceph-mon[35398]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-12T05:08:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.423+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f213802ce40 con 0x7f21240a1a30 2024-03-12T05:08:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.445+0000 7f212effd640 1 -- 172.21.15.45:0/3587178403 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f213802ce40 con 0x7f21240a1a30 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 msgr2=0x7f21240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 0x7f21240a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f21381010e0 tx=0x7f2130009870 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 msgr2=0x7f2138196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2128036040 tx=0x7f21280042f0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 shutdown_connections 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f21240a1a30 0x7f21240a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2138101280 0x7f2138196500 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2138101c40 0x7f213819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 --2- 172.21.15.45:0/3587178403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2138100090 0x7f2138195fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 >> 172.21.15.45:0/3587178403 conn(0x7f21380fb620 msgr2=0x7f2138104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 shutdown_connections 2024-03-12T05:08:43.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.447+0000 7f214022a640 1 -- 172.21.15.45:0/3587178403 wait complete. 2024-03-12T05:08:43.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-12T05:08:43.494 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:08:43.130+0000 7f1bc0bbd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-12T05:08:43.495 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:08:43.130+0000 7f1bc0bbd700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-12T05:08:43.495 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:08:43.130+0000 7f1bc0bbd700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-12T05:08:43.495 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:08:43.130+0000 7f1bc0bbd700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.565+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/2017443779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101c30 msgr2=0x7f3b281084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.565+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/2017443779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101c30 0x7f3b281084c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009ca0 tx=0x7f3b2002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.565+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/2017443779 shutdown_connections 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.565+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/2017443779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101c30 0x7f3b281084c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/2017443779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101270 0x7f3b281016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/2017443779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b281004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/2017443779 >> 172.21.15.45:0/2017443779 conn(0x7f3b280fb610 msgr2=0x7f3b280fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/2017443779 shutdown_connections 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/2017443779 wait complete. 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 Processor -- start 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- start start 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b2810a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101c30 0x7f3b281094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28109c20 con 0x7f3b28101270 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28109d90 con 0x7f3b28100080 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2fd0c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281a1190 con 0x7f3b28101c30 2024-03-12T05:08:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2d2a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.566+0000 7f3b2d2a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43324/0 (socket says 172.21.15.45:43324) 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 -- 172.21.15.45:0/91968551 learned_addr learned my addr 172.21.15.45:0/91968551 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2daa8640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b2810a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2e2a9640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101c30 0x7f3b281094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 -- 172.21.15.45:0/91968551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101c30 msgr2=0x7f3b281094c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101c30 0x7f3b281094c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 -- 172.21.15.45:0/91968551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 msgr2=0x7f3b2810a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b2810a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 -- 172.21.15.45:0/91968551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20009900 con 0x7f3b28101270 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2daa8640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b2810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2d2a7640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800d7a0 tx=0x7f3b1800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18009c30 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b1800deb0 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18004dd0 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b281a1440 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.567+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281a1950 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.568+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b28107470 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.569+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3b18004f30 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.569+0000 7f3b1effd640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 0x7f3b140a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.570+0000 7f3b2daa8640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 0x7f3b140a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.570+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3b180ccd40 con 0x7f3b28101270 2024-03-12T05:08:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.570+0000 7f3b2daa8640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 0x7f3b140a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3b240058c0 tx=0x7f3b24005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.574+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3b18099bd0 con 0x7f3b28101270 2024-03-12T05:08:43.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.785+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b2802ceb0 con 0x7f3b140a1ab0 2024-03-12T05:08:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.789+0000 7f3b1effd640 1 -- 172.21.15.45:0/91968551 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f3b2802ceb0 con 0x7f3b140a1ab0 2024-03-12T05:08:43.787 INFO:teuthology.orchestra.run.smithi045.stdout:/volumes/qa/sv_1/e7219533-9ea7-4b7b-a1d8-6384c5247364 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 msgr2=0x7f3b140a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 0x7f3b140a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3b240058c0 tx=0x7f3b24005850 comp rx=0 tx=0).stop 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 msgr2=0x7f3b28108f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800d7a0 tx=0x7f3b1800dc60 comp rx=0 tx=0).stop 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 shutdown_connections 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3b140a1ab0 0x7f3b140a3f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b28101270 0x7f3b28108f80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28101c30 0x7f3b281094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 --2- 172.21.15.45:0/91968551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b28100080 0x7f3b2810a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 >> 172.21.15.45:0/91968551 conn(0x7f3b280fb610 msgr2=0x7f3b28104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 shutdown_connections 2024-03-12T05:08:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.792+0000 7f3b2fd0c640 1 -- 172.21.15.45:0/91968551 wait complete. 2024-03-12T05:08:43.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 -- 172.21.15.45:0/1294436180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc101c80 msgr2=0x7f62dc104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/1294436180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc101c80 0x7f62dc104070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f62d0009900 tx=0x7f62d002bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 -- 172.21.15.45:0/1294436180 shutdown_connections 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/1294436180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc106ee0 0x7f62dc1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/1294436180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc1045b0 0x7f62dc1069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/1294436180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc101c80 0x7f62dc104070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.910+0000 7f62e3ef4640 1 -- 172.21.15.45:0/1294436180 >> 172.21.15.45:0/1294436180 conn(0x7f62dc0fb610 msgr2=0x7f62dc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- 172.21.15.45:0/1294436180 shutdown_connections 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- 172.21.15.45:0/1294436180 wait complete. 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 Processor -- start 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- start start 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc101c80 0x7f62dc197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 0x7f62dc197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc19ee10 con 0x7f62dc106ee0 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc19ef80 con 0x7f62dc1045b0 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e3ef4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc19f140 con 0x7f62dc101c80 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e2491640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e2491640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43342/0 (socket says 172.21.15.45:43342) 2024-03-12T05:08:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.911+0000 7f62e2491640 1 -- 172.21.15.45:0/2826466645 learned_addr learned my addr 172.21.15.45:0/2826466645 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e1c90640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc101c80 0x7f62dc197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 -- 172.21.15.45:0/2826466645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc101c80 msgr2=0x7f62dc197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e148f640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 0x7f62dc197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc101c80 0x7f62dc197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 -- 172.21.15.45:0/2826466645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 msgr2=0x7f62dc197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 0x7f62dc197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 -- 172.21.15.45:0/2826466645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d0009560 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e148f640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 0x7f62dc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e2491640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b970 tx=0x7f62d400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400cb40 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f62d400cca0 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40076e0 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc19f420 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.912+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc19f960 con 0x7f62dc106ee0 2024-03-12T05:08:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.913+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc0fd100 con 0x7f62dc106ee0 2024-03-12T05:08:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.914+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f62d4014440 con 0x7f62dc106ee0 2024-03-12T05:08:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.914+0000 7f62ceffd640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 0x7f62c80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.915+0000 7f62e1c90640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 0x7f62c80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.915+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f62d40cccc0 con 0x7f62dc106ee0 2024-03-12T05:08:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.915+0000 7f62e1c90640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 0x7f62c80a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005ec0 tx=0x7f62d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:43.919+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f62d40d3080 con 0x7f62dc106ee0 2024-03-12T05:08:44.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: pgmap v285: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-12T05:08:44.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:active 2024-03-12T05:08:44.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-12T05:08:44.119 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:43 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:08:44.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.137+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc10cb90 con 0x7f62dc106ee0 2024-03-12T05:08:44.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.138+0000 7f62ceffd640 1 -- 172.21.15.45:0/2826466645 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f62d4019090 con 0x7f62dc106ee0 2024-03-12T05:08:44.136 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:44.136 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:08:44.137 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:08:44.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 msgr2=0x7f62c80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 0x7f62c80a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62d0005ec0 tx=0x7f62d0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 msgr2=0x7f62dc19e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62d400b970 tx=0x7f62d400be30 comp rx=0 tx=0).stop 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 shutdown_connections 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f62c80a1a30 0x7f62c80a3ef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62dc106ee0 0x7f62dc19e8d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62dc1045b0 0x7f62dc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 --2- 172.21.15.45:0/2826466645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62dc101c80 0x7f62dc197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 >> 172.21.15.45:0/2826466645 conn(0x7f62dc0fb610 msgr2=0x7f62dc1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 shutdown_connections 2024-03-12T05:08:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.140+0000 7f62e3ef4640 1 -- 172.21.15.45:0/2826466645 wait complete. 2024-03-12T05:08:44.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: pgmap v285: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:active 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-12T05:08:44.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:43 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: pgmap v285: 97 pgs: 97 active+clean; 580 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] up:active 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-12T05:08:44.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:43 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:08:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 -- 172.21.15.45:0/108166920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2304106eb0 msgr2=0x7f23041092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 --2- 172.21.15.45:0/108166920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2304106eb0 0x7f23041092a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009ca0 tx=0x7f22fc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 -- 172.21.15.45:0/108166920 shutdown_connections 2024-03-12T05:08:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 --2- 172.21.15.45:0/108166920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2304106eb0 0x7f23041092a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 --2- 172.21.15.45:0/108166920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 0x7f2304106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 --2- 172.21.15.45:0/108166920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23040feda0 0x7f23040ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 -- 172.21.15.45:0/108166920 >> 172.21.15.45:0/108166920 conn(0x7f23040fa980 msgr2=0x7f23040fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 -- 172.21.15.45:0/108166920 shutdown_connections 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 -- 172.21.15.45:0/108166920 wait complete. 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.254+0000 7f2309455640 1 Processor -- start 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- start start 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 0x7f2304197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2304106eb0 0x7f230419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230419ebe0 con 0x7f23040feda0 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230419ed50 con 0x7f23040ff790 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230419ef10 con 0x7f2304106eb0 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43366/0 (socket says 172.21.15.45:43366) 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 -- 172.21.15.45:0/4136555458 learned_addr learned my addr 172.21.15.45:0/4136555458 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f23037fe640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2304106eb0 0x7f230419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 -- 172.21.15.45:0/4136555458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2304106eb0 msgr2=0x7f230419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2304106eb0 0x7f230419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 -- 172.21.15.45:0/4136555458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 msgr2=0x7f2304197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f22fa7fc640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 0x7f2304197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 0x7f2304197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 -- 172.21.15.45:0/4136555458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc009900 con 0x7f23040feda0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2302ffd640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22f400d770 tx=0x7f22f400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4009c30 con 0x7f23040feda0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230419f1f0 con 0x7f23040feda0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f22f400de80 con 0x7f23040feda0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004e00 con 0x7f23040feda0 2024-03-12T05:08:44.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.255+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230419f6b0 con 0x7f23040feda0 2024-03-12T05:08:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.256+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2304105780 con 0x7f23040feda0 2024-03-12T05:08:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.257+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f22f4004f60 con 0x7f23040feda0 2024-03-12T05:08:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.258+0000 7f2300ff9640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 0x7f22f00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.258+0000 7f22fa7fc640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 0x7f22f00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.258+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f22f40cd270 con 0x7f23040feda0 2024-03-12T05:08:44.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.259+0000 7f22fa7fc640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 0x7f22f00a3ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f22ec005fd0 tx=0x7f22ec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.262+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f22f4096530 con 0x7f23040feda0 2024-03-12T05:08:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.365+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f22f4096900 con 0x7f23040feda0 2024-03-12T05:08:44.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.450+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f230410cbf0 con 0x7f23040feda0 2024-03-12T05:08:44.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.452+0000 7f2300ff9640 1 -- 172.21.15.45:0/4136555458 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22184 (secure 0 0 0) 0x7f230410cbf0 con 0x7f23040feda0 2024-03-12T05:08:44.450 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:44.450 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:08:38.754759+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 msgr2=0x7f22f00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 0x7f22f00a3ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f22ec005fd0 tx=0x7f22ec005850 comp rx=0 tx=0).stop 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 msgr2=0x7f2304197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22f400d770 tx=0x7f22f400dc30 comp rx=0 tx=0).stop 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 shutdown_connections 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f22f00a1a30 0x7f22f00a3ef0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2304106eb0 0x7f230419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23040ff790 0x7f2304197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 --2- 172.21.15.45:0/4136555458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23040feda0 0x7f2304197140 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 >> 172.21.15.45:0/4136555458 conn(0x7f23040fa980 msgr2=0x7f2304102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 shutdown_connections 2024-03-12T05:08:44.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.454+0000 7f2309455640 1 -- 172.21.15.45:0/4136555458 wait complete. 2024-03-12T05:08:44.463 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-12T05:08:44.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:08:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 -- 172.21.15.45:0/3527233523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14101cd0 msgr2=0x7fee14108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/3527233523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14101cd0 0x7fee14108560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009ca0 tx=0x7fee0c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 -- 172.21.15.45:0/3527233523 shutdown_connections 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/3527233523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14101cd0 0x7fee14108560 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/3527233523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 0x7fee14101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/3527233523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14100120 0x7fee14100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 -- 172.21.15.45:0/3527233523 >> 172.21.15.45:0/3527233523 conn(0x7fee140fb6b0 msgr2=0x7fee140fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 -- 172.21.15.45:0/3527233523 shutdown_connections 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.568+0000 7fee1a8f4640 1 -- 172.21.15.45:0/3527233523 wait complete. 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 Processor -- start 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 -- start start 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 0x7fee1419bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 0x7fee1419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1419ebd0 con 0x7fee14100120 2024-03-12T05:08:44.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1419ed40 con 0x7fee14101310 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee1a8f4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1419ef00 con 0x7fee14101cd0 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43398/0 (socket says 172.21.15.45:43398) 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 -- 172.21.15.45:0/2774288173 learned_addr learned my addr 172.21.15.45:0/2774288173 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee137fe640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 0x7fee1419bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee18e91640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 0x7fee1419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 -- 172.21.15.45:0/2774288173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 msgr2=0x7fee1419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 0x7fee1419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 -- 172.21.15.45:0/2774288173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 msgr2=0x7fee1419bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 0x7fee1419bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 -- 172.21.15.45:0/2774288173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c009900 con 0x7fee14100120 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee18e91640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 0x7fee1419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.569+0000 7fee13fff640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee0400e9c0 tx=0x7fee0400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.570+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04009700 con 0x7fee14100120 2024-03-12T05:08:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.570+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fee040044d0 con 0x7fee14100120 2024-03-12T05:08:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.570+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1419f1e0 con 0x7fee14100120 2024-03-12T05:08:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.570+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee04010430 con 0x7fee14100120 2024-03-12T05:08:44.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.570+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1419f740 con 0x7fee14100120 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.571+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14107440 con 0x7fee14100120 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.572+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fee04003f30 con 0x7fee14100120 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.572+0000 7fee117fa640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 0x7fedfc0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.572+0000 7fee137fe640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 0x7fedfc0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.573+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fee04014070 con 0x7fee14100120 2024-03-12T05:08:44.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.573+0000 7fee137fe640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 0x7fedfc0a4000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee1406a0f0 tx=0x7fee0800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.577+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fee04096a50 con 0x7fee14100120 2024-03-12T05:08:44.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.755+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee1402ceb0 con 0x7fedfc0a1b40 2024-03-12T05:08:44.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.763+0000 7fee117fa640 1 -- 172.21.15.45:0/2774288173 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258854 (secure 0 0 0) 0x7fee1402ceb0 con 0x7fedfc0a1b40 2024-03-12T05:08:44.765 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:44.767 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:08:44.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 msgr2=0x7fedfc0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 0x7fedfc0a4000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee1406a0f0 tx=0x7fee0800b390 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 msgr2=0x7fee1406a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee0400e9c0 tx=0x7fee0400ee80 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 shutdown_connections 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fedfc0a1b40 0x7fedfc0a4000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fee14101cd0 0x7fee1419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee14101310 0x7fee1419bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 --2- 172.21.15.45:0/2774288173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee14100120 0x7fee1406a380 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 >> 172.21.15.45:0/2774288173 conn(0x7fee140fb6b0 msgr2=0x7fee14104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 shutdown_connections 2024-03-12T05:08:44.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.766+0000 7fee1a8f4640 1 -- 172.21.15.45:0/2774288173 wait complete. 2024-03-12T05:08:44.776 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-03-12T05:08:43.099115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593450,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":249,"num_write_kb":2960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368404,"kb_used_data":10856,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573420,"statfs":{"total":1151940427776,"available":1151563182080,"internally_reserved":0,"allocated":11116544,"data_stored":6959701,"data_compressed":108348,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3082,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.345544"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.339026+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:08:39.339026+0000","last_peered":"2024-03-12T05:08:39.339026+0000","last_clean":"2024-03-12T05:08:39.339026+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:08:39.339026+0000","last_undegraded":"2024-03-12T05:08:39.339026+0000","last_fullsized":"2024-03-12T05:08:39.339026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.785674+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:08:37.785674+0000","last_peered":"2024-03-12T05:08:37.785674+0000","last_clean":"2024-03-12T05:08:37.785674+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:08:37.785674+0000","last_undegraded":"2024-03-12T05:08:37.785674+0000","last_fullsized":"2024-03-12T05:08:37.785674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:19.901166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.790144+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:08:37.790144+0000","last_peered":"2024-03-12T05:08:37.790144+0000","last_clean":"2024-03-12T05:08:37.790144+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:08:37.790144+0000","last_undegraded":"2024-03-12T05:08:37.790144+0000","last_fullsized":"2024-03-12T05:08:37.790144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:07:50.192999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333019+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:08:39.333019+0000","last_peered":"2024-03-12T05:08:39.333019+0000","last_clean":"2024-03-12T05:08:39.333019+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:08:39.333019+0000","last_undegraded":"2024-03-12T05:08:39.333019+0000","last_fullsized":"2024-03-12T05:08:39.333019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.779361+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:08:37.779361+0000","last_peered":"2024-03-12T05:08:37.779361+0000","last_clean":"2024-03-12T05:08:37.779361+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:08:37.779361+0000","last_undegraded":"2024-03-12T05:08:37.779361+0000","last_fullsized":"2024-03-12T05:08:37.779361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:12:33.424323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778323+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:08:37.778323+0000","last_peered":"2024-03-12T05:08:37.778323+0000","last_clean":"2024-03-12T05:08:37.778323+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:08:37.778323+0000","last_undegraded":"2024-03-12T05:08:37.778323+0000","last_fullsized":"2024-03-12T05:08:37.778323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:36:57.986242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.771566+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:08:37.771566+0000","last_peered":"2024-03-12T05:08:37.771566+0000","last_clean":"2024-03-12T05:08:37.771566+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:08:37.771566+0000","last_undegraded":"2024-03-12T05:08:37.771566+0000","last_fullsized":"2024-03-12T05:08:37.771566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:22.878852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783072+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:08:38.783072+0000","last_peered":"2024-03-12T05:08:38.783072+0000","last_clean":"2024-03-12T05:08:38.783072+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:08:38.783072+0000","last_undegraded":"2024-03-12T05:08:38.783072+0000","last_fullsized":"2024-03-12T05:08:38.783072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.759068+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:08:38.759068+0000","last_peered":"2024-03-12T05:08:38.759068+0000","last_clean":"2024-03-12T05:08:38.759068+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:08:38.759068+0000","last_undegraded":"2024-03-12T05:08:38.759068+0000","last_fullsized":"2024-03-12T05:08:38.759068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778078+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:08:37.778078+0000","last_peered":"2024-03-12T05:08:37.778078+0000","last_clean":"2024-03-12T05:08:37.778078+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:08:37.778078+0000","last_undegraded":"2024-03-12T05:08:37.778078+0000","last_fullsized":"2024-03-12T05:08:37.778078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:48:21.736854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.783682+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:08:37.783682+0000","last_peered":"2024-03-12T05:08:37.783682+0000","last_clean":"2024-03-12T05:08:37.783682+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:08:37.783682+0000","last_undegraded":"2024-03-12T05:08:37.783682+0000","last_fullsized":"2024-03-12T05:08:37.783682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:45:26.748627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333182+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:08:39.333182+0000","last_peered":"2024-03-12T05:08:39.333182+0000","last_clean":"2024-03-12T05:08:39.333182+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:08:39.333182+0000","last_undegraded":"2024-03-12T05:08:39.333182+0000","last_fullsized":"2024-03-12T05:08:39.333182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757985+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:08:38.757985+0000","last_peered":"2024-03-12T05:08:38.757985+0000","last_clean":"2024-03-12T05:08:38.757985+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:08:38.757985+0000","last_undegraded":"2024-03-12T05:08:38.757985+0000","last_fullsized":"2024-03-12T05:08:38.757985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.784968+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:08:37.784968+0000","last_peered":"2024-03-12T05:08:37.784968+0000","last_clean":"2024-03-12T05:08:37.784968+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:08:37.784968+0000","last_undegraded":"2024-03-12T05:08:37.784968+0000","last_fullsized":"2024-03-12T05:08:37.784968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:26:12.643909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.782745+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:08:37.782745+0000","last_peered":"2024-03-12T05:08:37.782745+0000","last_clean":"2024-03-12T05:08:37.782745+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:08:37.782745+0000","last_undegraded":"2024-03-12T05:08:37.782745+0000","last_fullsized":"2024-03-12T05:08:37.782745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:59.593000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758463+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:08:38.758463+0000","last_peered":"2024-03-12T05:08:38.758463+0000","last_clean":"2024-03-12T05:08:38.758463+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:08:38.758463+0000","last_undegraded":"2024-03-12T05:08:38.758463+0000","last_fullsized":"2024-03-12T05:08:38.758463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.339103+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:08:39.339103+0000","last_peered":"2024-03-12T05:08:39.339103+0000","last_clean":"2024-03-12T05:08:39.339103+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:08:39.339103+0000","last_undegraded":"2024-03-12T05:08:39.339103+0000","last_fullsized":"2024-03-12T05:08:39.339103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781635+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:08:38.781635+0000","last_peered":"2024-03-12T05:08:38.781635+0000","last_clean":"2024-03-12T05:08:38.781635+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:08:38.781635+0000","last_undegraded":"2024-03-12T05:08:38.781635+0000","last_fullsized":"2024-03-12T05:08:38.781635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758569+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:08:38.758569+0000","last_peered":"2024-03-12T05:08:38.758569+0000","last_clean":"2024-03-12T05:08:38.758569+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:08:38.758569+0000","last_undegraded":"2024-03-12T05:08:38.758569+0000","last_fullsized":"2024-03-12T05:08:38.758569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781469+0000","last_change":"2024-03-12T05:08:36.774328+0000","last_active":"2024-03-12T05:08:38.781469+0000","last_peered":"2024-03-12T05:08:38.781469+0000","last_clean":"2024-03-12T05:08:38.781469+0000","last_became_active":"2024-03-12T05:08:36.774000+0000","last_became_peered":"2024-03-12T05:08:36.774000+0000","last_unstale":"2024-03-12T05:08:38.781469+0000","last_undegraded":"2024-03-12T05:08:38.781469+0000","last_fullsized":"2024-03-12T05:08:38.781469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:59.134654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783400+0000","last_change":"2024-03-12T05:08:37.779921+0000","last_active":"2024-03-12T05:08:38.783400+0000","last_peered":"2024-03-12T05:08:38.783400+0000","last_clean":"2024-03-12T05:08:38.783400+0000","last_became_active":"2024-03-12T05:08:37.779542+0000","last_became_peered":"2024-03-12T05:08:37.779542+0000","last_unstale":"2024-03-12T05:08:38.783400+0000","last_undegraded":"2024-03-12T05:08:38.783400+0000","last_fullsized":"2024-03-12T05:08:38.783400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:00.663534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783354+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:08:38.783354+0000","last_peered":"2024-03-12T05:08:38.783354+0000","last_clean":"2024-03-12T05:08:38.783354+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:08:38.783354+0000","last_undegraded":"2024-03-12T05:08:38.783354+0000","last_fullsized":"2024-03-12T05:08:38.783354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.781817+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:08:37.781817+0000","last_peered":"2024-03-12T05:08:37.781817+0000","last_clean":"2024-03-12T05:08:37.781817+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:08:37.781817+0000","last_undegraded":"2024-03-12T05:08:37.781817+0000","last_fullsized":"2024-03-12T05:08:37.781817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:42.322149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.791918+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:08:37.791918+0000","last_peered":"2024-03-12T05:08:37.791918+0000","last_clean":"2024-03-12T05:08:37.791918+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:08:37.791918+0000","last_undegraded":"2024-03-12T05:08:37.791918+0000","last_fullsized":"2024-03-12T05:08:37.791918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758715+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:08:38.758715+0000","last_peered":"2024-03-12T05:08:38.758715+0000","last_clean":"2024-03-12T05:08:38.758715+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:08:38.758715+0000","last_undegraded":"2024-03-12T05:08:38.758715+0000","last_fullsized":"2024-03-12T05:08:38.758715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.792379+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:08:37.792379+0000","last_peered":"2024-03-12T05:08:37.792379+0000","last_clean":"2024-03-12T05:08:37.792379+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:08:37.792379+0000","last_undegraded":"2024-03-12T05:08:37.792379+0000","last_fullsized":"2024-03-12T05:08:37.792379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778817+0000","last_change":"2024-03-12T05:08:37.777137+0000","last_active":"2024-03-12T05:08:37.778817+0000","last_peered":"2024-03-12T05:08:37.778817+0000","last_clean":"2024-03-12T05:08:37.778817+0000","last_became_active":"2024-03-12T05:08:37.776751+0000","last_became_peered":"2024-03-12T05:08:37.776751+0000","last_unstale":"2024-03-12T05:08:37.778817+0000","last_undegraded":"2024-03-12T05:08:37.778817+0000","last_fullsized":"2024-03-12T05:08:37.778817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:16:31.860472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.787753+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:08:37.787753+0000","last_peered":"2024-03-12T05:08:37.787753+0000","last_clean":"2024-03-12T05:08:37.787753+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:08:37.787753+0000","last_undegraded":"2024-03-12T05:08:37.787753+0000","last_fullsized":"2024-03-12T05:08:37.787753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.773532+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:08:37.773532+0000","last_peered":"2024-03-12T05:08:37.773532+0000","last_clean":"2024-03-12T05:08:37.773532+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:08:37.773532+0000","last_undegraded":"2024-03-12T05:08:37.773532+0000","last_fullsized":"2024-03-12T05:08:37.773532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:10:31.118524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.788228+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:08:37.788228+0000","last_peered":"2024-03-12T05:08:37.788228+0000","last_clean":"2024-03-12T05:08:37.788228+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:08:37.788228+0000","last_undegraded":"2024-03-12T05:08:37.788228+0000","last_fullsized":"2024-03-12T05:08:37.788228+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778816+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:08:37.778816+0000","last_peered":"2024-03-12T05:08:37.778816+0000","last_clean":"2024-03-12T05:08:37.778816+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:08:37.778816+0000","last_undegraded":"2024-03-12T05:08:37.778816+0000","last_fullsized":"2024-03-12T05:08:37.778816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:55:45.602709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757896+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:08:38.757896+0000","last_peered":"2024-03-12T05:08:38.757896+0000","last_clean":"2024-03-12T05:08:38.757896+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:08:38.757896+0000","last_undegraded":"2024-03-12T05:08:38.757896+0000","last_fullsized":"2024-03-12T05:08:38.757896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.782940+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:08:38.782940+0000","last_peered":"2024-03-12T05:08:38.782940+0000","last_clean":"2024-03-12T05:08:38.782940+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:08:38.782940+0000","last_undegraded":"2024-03-12T05:08:38.782940+0000","last_fullsized":"2024-03-12T05:08:38.782940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.757938+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:08:37.757938+0000","last_peered":"2024-03-12T05:08:37.757938+0000","last_clean":"2024-03-12T05:08:37.757938+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:08:37.757938+0000","last_undegraded":"2024-03-12T05:08:37.757938+0000","last_fullsized":"2024-03-12T05:08:37.757938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781500+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:08:38.781500+0000","last_peered":"2024-03-12T05:08:38.781500+0000","last_clean":"2024-03-12T05:08:38.781500+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:08:38.781500+0000","last_undegraded":"2024-03-12T05:08:38.781500+0000","last_fullsized":"2024-03-12T05:08:38.781500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.757470+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:08:37.757470+0000","last_peered":"2024-03-12T05:08:37.757470+0000","last_clean":"2024-03-12T05:08:37.757470+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:08:37.757470+0000","last_undegraded":"2024-03-12T05:08:37.757470+0000","last_fullsized":"2024-03-12T05:08:37.757470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758966+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:08:38.758966+0000","last_peered":"2024-03-12T05:08:38.758966+0000","last_clean":"2024-03-12T05:08:38.758966+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:08:38.758966+0000","last_undegraded":"2024-03-12T05:08:38.758966+0000","last_fullsized":"2024-03-12T05:08:38.758966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.761061+0000","last_change":"2024-03-12T05:08:36.770968+0000","last_active":"2024-03-12T05:08:37.761061+0000","last_peered":"2024-03-12T05:08:37.761061+0000","last_clean":"2024-03-12T05:08:37.761061+0000","last_became_active":"2024-03-12T05:08:36.770646+0000","last_became_peered":"2024-03-12T05:08:36.770646+0000","last_unstale":"2024-03-12T05:08:37.761061+0000","last_undegraded":"2024-03-12T05:08:37.761061+0000","last_fullsized":"2024-03-12T05:08:37.761061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:45:26.921443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.783216+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:08:37.783216+0000","last_peered":"2024-03-12T05:08:37.783216+0000","last_clean":"2024-03-12T05:08:37.783216+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:08:37.783216+0000","last_undegraded":"2024-03-12T05:08:37.783216+0000","last_fullsized":"2024-03-12T05:08:37.783216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:40.265235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.793895+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:08:37.793895+0000","last_peered":"2024-03-12T05:08:37.793895+0000","last_clean":"2024-03-12T05:08:37.793895+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:08:37.793895+0000","last_undegraded":"2024-03-12T05:08:37.793895+0000","last_fullsized":"2024-03-12T05:08:37.793895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758750+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:08:38.758750+0000","last_peered":"2024-03-12T05:08:38.758750+0000","last_clean":"2024-03-12T05:08:38.758750+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:08:38.758750+0000","last_undegraded":"2024-03-12T05:08:38.758750+0000","last_fullsized":"2024-03-12T05:08:38.758750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"84'6","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.794152+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:08:37.794152+0000","last_peered":"2024-03-12T05:08:37.794152+0000","last_clean":"2024-03-12T05:08:37.794152+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:08:37.794152+0000","last_undegraded":"2024-03-12T05:08:37.794152+0000","last_fullsized":"2024-03-12T05:08:37.794152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.779020+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:08:37.779020+0000","last_peered":"2024-03-12T05:08:37.779020+0000","last_clean":"2024-03-12T05:08:37.779020+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:08:37.779020+0000","last_undegraded":"2024-03-12T05:08:37.779020+0000","last_fullsized":"2024-03-12T05:08:37.779020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:00:14.264549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.343239+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:08:40.343239+0000","last_peered":"2024-03-12T05:08:40.343239+0000","last_clean":"2024-03-12T05:08:40.343239+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:08:40.343239+0000","last_undegraded":"2024-03-12T05:08:40.343239+0000","last_fullsized":"2024-03-12T05:08:40.343239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758050+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:08:38.758050+0000","last_peered":"2024-03-12T05:08:38.758050+0000","last_clean":"2024-03-12T05:08:38.758050+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:08:38.758050+0000","last_undegraded":"2024-03-12T05:08:38.758050+0000","last_fullsized":"2024-03-12T05:08:38.758050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.757352+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:08:37.757352+0000","last_peered":"2024-03-12T05:08:37.757352+0000","last_clean":"2024-03-12T05:08:37.757352+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:08:37.757352+0000","last_undegraded":"2024-03-12T05:08:37.757352+0000","last_fullsized":"2024-03-12T05:08:37.757352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778433+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:08:37.778433+0000","last_peered":"2024-03-12T05:08:37.778433+0000","last_clean":"2024-03-12T05:08:37.778433+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:08:37.778433+0000","last_undegraded":"2024-03-12T05:08:37.778433+0000","last_fullsized":"2024-03-12T05:08:37.778433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:49:51.551325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.332491+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:08:39.332491+0000","last_peered":"2024-03-12T05:08:39.332491+0000","last_clean":"2024-03-12T05:08:39.332491+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:08:39.332491+0000","last_undegraded":"2024-03-12T05:08:39.332491+0000","last_fullsized":"2024-03-12T05:08:39.332491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778606+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:08:37.778606+0000","last_peered":"2024-03-12T05:08:37.778606+0000","last_clean":"2024-03-12T05:08:37.778606+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:08:37.778606+0000","last_undegraded":"2024-03-12T05:08:37.778606+0000","last_fullsized":"2024-03-12T05:08:37.778606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:02:13.217878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.794216+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:08:37.794216+0000","last_peered":"2024-03-12T05:08:37.794216+0000","last_clean":"2024-03-12T05:08:37.794216+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:08:37.794216+0000","last_undegraded":"2024-03-12T05:08:37.794216+0000","last_fullsized":"2024-03-12T05:08:37.794216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.777312+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:08:37.777312+0000","last_peered":"2024-03-12T05:08:37.777312+0000","last_clean":"2024-03-12T05:08:37.777312+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:08:37.777312+0000","last_undegraded":"2024-03-12T05:08:37.777312+0000","last_fullsized":"2024-03-12T05:08:37.777312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:04.981962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.789554+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:08:37.789554+0000","last_peered":"2024-03-12T05:08:37.789554+0000","last_clean":"2024-03-12T05:08:37.789554+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:08:37.789554+0000","last_undegraded":"2024-03-12T05:08:37.789554+0000","last_fullsized":"2024-03-12T05:08:37.789554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338619+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:08:39.338619+0000","last_peered":"2024-03-12T05:08:39.338619+0000","last_clean":"2024-03-12T05:08:39.338619+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:08:39.338619+0000","last_undegraded":"2024-03-12T05:08:39.338619+0000","last_fullsized":"2024-03-12T05:08:39.338619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781537+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:08:38.781537+0000","last_peered":"2024-03-12T05:08:38.781537+0000","last_clean":"2024-03-12T05:08:38.781537+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:08:38.781537+0000","last_undegraded":"2024-03-12T05:08:38.781537+0000","last_fullsized":"2024-03-12T05:08:38.781537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.773569+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:08:37.773569+0000","last_peered":"2024-03-12T05:08:37.773569+0000","last_clean":"2024-03-12T05:08:37.773569+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:08:37.773569+0000","last_undegraded":"2024-03-12T05:08:37.773569+0000","last_fullsized":"2024-03-12T05:08:37.773569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:57:39.169274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.766121+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:08:37.766121+0000","last_peered":"2024-03-12T05:08:37.766121+0000","last_clean":"2024-03-12T05:08:37.766121+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:08:37.766121+0000","last_undegraded":"2024-03-12T05:08:37.766121+0000","last_fullsized":"2024-03-12T05:08:37.766121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778282+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:08:37.778282+0000","last_peered":"2024-03-12T05:08:37.778282+0000","last_clean":"2024-03-12T05:08:37.778282+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:08:37.778282+0000","last_undegraded":"2024-03-12T05:08:37.778282+0000","last_fullsized":"2024-03-12T05:08:37.778282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:31:51.899286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.779275+0000","last_change":"2024-03-12T05:08:37.777792+0000","last_active":"2024-03-12T05:08:37.779275+0000","last_peered":"2024-03-12T05:08:37.779275+0000","last_clean":"2024-03-12T05:08:37.779275+0000","last_became_active":"2024-03-12T05:08:37.777220+0000","last_became_peered":"2024-03-12T05:08:37.777220+0000","last_unstale":"2024-03-12T05:08:37.779275+0000","last_undegraded":"2024-03-12T05:08:37.779275+0000","last_fullsized":"2024-03-12T05:08:37.779275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:30.197600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331719+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:08:39.331719+0000","last_peered":"2024-03-12T05:08:39.331719+0000","last_clean":"2024-03-12T05:08:39.331719+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:08:39.331719+0000","last_undegraded":"2024-03-12T05:08:39.331719+0000","last_fullsized":"2024-03-12T05:08:39.331719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783427+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:08:38.783427+0000","last_peered":"2024-03-12T05:08:38.783427+0000","last_clean":"2024-03-12T05:08:38.783427+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:08:38.783427+0000","last_undegraded":"2024-03-12T05:08:38.783427+0000","last_fullsized":"2024-03-12T05:08:38.783427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.337143+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:08:39.337143+0000","last_peered":"2024-03-12T05:08:39.337143+0000","last_clean":"2024-03-12T05:08:39.337143+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:08:39.337143+0000","last_undegraded":"2024-03-12T05:08:39.337143+0000","last_fullsized":"2024-03-12T05:08:39.337143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.761658+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:08:37.761658+0000","last_peered":"2024-03-12T05:08:37.761658+0000","last_clean":"2024-03-12T05:08:37.761658+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:08:37.761658+0000","last_undegraded":"2024-03-12T05:08:37.761658+0000","last_fullsized":"2024-03-12T05:08:37.761658+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757747+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:08:38.757747+0000","last_peered":"2024-03-12T05:08:38.757747+0000","last_clean":"2024-03-12T05:08:38.757747+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:08:38.757747+0000","last_undegraded":"2024-03-12T05:08:38.757747+0000","last_fullsized":"2024-03-12T05:08:38.757747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.761863+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:08:37.761863+0000","last_peered":"2024-03-12T05:08:37.761863+0000","last_clean":"2024-03-12T05:08:37.761863+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:08:37.761863+0000","last_undegraded":"2024-03-12T05:08:37.761863+0000","last_fullsized":"2024-03-12T05:08:37.761863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.762764+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:08:37.762764+0000","last_peered":"2024-03-12T05:08:37.762764+0000","last_clean":"2024-03-12T05:08:37.762764+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:08:37.762764+0000","last_undegraded":"2024-03-12T05:08:37.762764+0000","last_fullsized":"2024-03-12T05:08:37.762764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758042+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:08:38.758042+0000","last_peered":"2024-03-12T05:08:38.758042+0000","last_clean":"2024-03-12T05:08:38.758042+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:08:38.758042+0000","last_undegraded":"2024-03-12T05:08:38.758042+0000","last_fullsized":"2024-03-12T05:08:38.758042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783170+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:08:38.783170+0000","last_peered":"2024-03-12T05:08:38.783170+0000","last_clean":"2024-03-12T05:08:38.783170+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:08:38.783170+0000","last_undegraded":"2024-03-12T05:08:38.783170+0000","last_fullsized":"2024-03-12T05:08:38.783170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783473+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:08:38.783473+0000","last_peered":"2024-03-12T05:08:38.783473+0000","last_clean":"2024-03-12T05:08:38.783473+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:08:38.783473+0000","last_undegraded":"2024-03-12T05:08:38.783473+0000","last_fullsized":"2024-03-12T05:08:38.783473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.337955+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:08:39.337955+0000","last_peered":"2024-03-12T05:08:39.337955+0000","last_clean":"2024-03-12T05:08:39.337955+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:08:39.337955+0000","last_undegraded":"2024-03-12T05:08:39.337955+0000","last_fullsized":"2024-03-12T05:08:39.337955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.777601+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:08:37.777601+0000","last_peered":"2024-03-12T05:08:37.777601+0000","last_clean":"2024-03-12T05:08:37.777601+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:08:37.777601+0000","last_undegraded":"2024-03-12T05:08:37.777601+0000","last_fullsized":"2024-03-12T05:08:37.777601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:53:00.316718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330509+0000","last_change":"2024-03-12T05:08:36.768605+0000","last_active":"2024-03-12T05:08:39.330509+0000","last_peered":"2024-03-12T05:08:39.330509+0000","last_clean":"2024-03-12T05:08:39.330509+0000","last_became_active":"2024-03-12T05:08:36.768154+0000","last_became_peered":"2024-03-12T05:08:36.768154+0000","last_unstale":"2024-03-12T05:08:39.330509+0000","last_undegraded":"2024-03-12T05:08:39.330509+0000","last_fullsized":"2024-03-12T05:08:39.330509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:32:53.952973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.778738+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:08:37.778738+0000","last_peered":"2024-03-12T05:08:37.778738+0000","last_clean":"2024-03-12T05:08:37.778738+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:08:37.778738+0000","last_undegraded":"2024-03-12T05:08:37.778738+0000","last_fullsized":"2024-03-12T05:08:37.778738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:19.144923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333776+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:08:39.333776+0000","last_peered":"2024-03-12T05:08:39.333776+0000","last_clean":"2024-03-12T05:08:39.333776+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:08:39.333776+0000","last_undegraded":"2024-03-12T05:08:39.333776+0000","last_fullsized":"2024-03-12T05:08:39.333776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338212+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:08:39.338212+0000","last_peered":"2024-03-12T05:08:39.338212+0000","last_clean":"2024-03-12T05:08:39.338212+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:08:39.338212+0000","last_undegraded":"2024-03-12T05:08:39.338212+0000","last_fullsized":"2024-03-12T05:08:39.338212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758254+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:08:38.758254+0000","last_peered":"2024-03-12T05:08:38.758254+0000","last_clean":"2024-03-12T05:08:38.758254+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:08:38.758254+0000","last_undegraded":"2024-03-12T05:08:38.758254+0000","last_fullsized":"2024-03-12T05:08:38.758254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783371+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:08:38.783371+0000","last_peered":"2024-03-12T05:08:38.783371+0000","last_clean":"2024-03-12T05:08:38.783371+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:08:38.783371+0000","last_undegraded":"2024-03-12T05:08:38.783371+0000","last_fullsized":"2024-03-12T05:08:38.783371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.791917+0000","last_change":"2024-03-12T05:08:36.773558+0000","last_active":"2024-03-12T05:08:37.791917+0000","last_peered":"2024-03-12T05:08:37.791917+0000","last_clean":"2024-03-12T05:08:37.791917+0000","last_became_active":"2024-03-12T05:08:36.773272+0000","last_became_peered":"2024-03-12T05:08:36.773272+0000","last_unstale":"2024-03-12T05:08:37.791917+0000","last_undegraded":"2024-03-12T05:08:37.791917+0000","last_fullsized":"2024-03-12T05:08:37.791917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:05.069202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.784063+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:08:37.784063+0000","last_peered":"2024-03-12T05:08:37.784063+0000","last_clean":"2024-03-12T05:08:37.784063+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:08:37.784063+0000","last_undegraded":"2024-03-12T05:08:37.784063+0000","last_fullsized":"2024-03-12T05:08:37.784063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:25:55.286591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.343210+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:08:40.343210+0000","last_peered":"2024-03-12T05:08:40.343210+0000","last_clean":"2024-03-12T05:08:40.343210+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:08:40.343210+0000","last_undegraded":"2024-03-12T05:08:40.343210+0000","last_fullsized":"2024-03-12T05:08:40.343210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.779805+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:08:37.779805+0000","last_peered":"2024-03-12T05:08:37.779805+0000","last_clean":"2024-03-12T05:08:37.779805+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:08:37.779805+0000","last_undegraded":"2024-03-12T05:08:37.779805+0000","last_fullsized":"2024-03-12T05:08:37.779805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:32:06.628464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.788872+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:08:37.788872+0000","last_peered":"2024-03-12T05:08:37.788872+0000","last_clean":"2024-03-12T05:08:37.788872+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:08:37.788872+0000","last_undegraded":"2024-03-12T05:08:37.788872+0000","last_fullsized":"2024-03-12T05:08:37.788872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.779777+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:08:37.779777+0000","last_peered":"2024-03-12T05:08:37.779777+0000","last_clean":"2024-03-12T05:08:37.779777+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:08:37.779777+0000","last_undegraded":"2024-03-12T05:08:37.779777+0000","last_fullsized":"2024-03-12T05:08:37.779777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:25:39.839043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757907+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:08:38.757907+0000","last_peered":"2024-03-12T05:08:38.757907+0000","last_clean":"2024-03-12T05:08:38.757907+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:08:38.757907+0000","last_undegraded":"2024-03-12T05:08:38.757907+0000","last_fullsized":"2024-03-12T05:08:38.757907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.771596+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:08:37.771596+0000","last_peered":"2024-03-12T05:08:37.771596+0000","last_clean":"2024-03-12T05:08:37.771596+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:08:37.771596+0000","last_undegraded":"2024-03-12T05:08:37.771596+0000","last_fullsized":"2024-03-12T05:08:37.771596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:55.438968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.337860+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:08:39.337860+0000","last_peered":"2024-03-12T05:08:39.337860+0000","last_clean":"2024-03-12T05:08:39.337860+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:08:39.337860+0000","last_undegraded":"2024-03-12T05:08:39.337860+0000","last_fullsized":"2024-03-12T05:08:39.337860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758440+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:08:38.758440+0000","last_peered":"2024-03-12T05:08:38.758440+0000","last_clean":"2024-03-12T05:08:38.758440+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:08:38.758440+0000","last_undegraded":"2024-03-12T05:08:38.758440+0000","last_fullsized":"2024-03-12T05:08:38.758440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758253+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:08:38.758253+0000","last_peered":"2024-03-12T05:08:38.758253+0000","last_clean":"2024-03-12T05:08:38.758253+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:08:38.758253+0000","last_undegraded":"2024-03-12T05:08:38.758253+0000","last_fullsized":"2024-03-12T05:08:38.758253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.790753+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:08:37.790753+0000","last_peered":"2024-03-12T05:08:37.790753+0000","last_clean":"2024-03-12T05:08:37.790753+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:08:37.790753+0000","last_undegraded":"2024-03-12T05:08:37.790753+0000","last_fullsized":"2024-03-12T05:08:37.790753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:15:45.254628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758046+0000","last_change":"2024-03-12T05:08:37.774717+0000","last_active":"2024-03-12T05:08:38.758046+0000","last_peered":"2024-03-12T05:08:38.758046+0000","last_clean":"2024-03-12T05:08:38.758046+0000","last_became_active":"2024-03-12T05:08:37.774289+0000","last_became_peered":"2024-03-12T05:08:37.774289+0000","last_unstale":"2024-03-12T05:08:38.758046+0000","last_undegraded":"2024-03-12T05:08:38.758046+0000","last_fullsized":"2024-03-12T05:08:38.758046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:21:43.200061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338754+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:08:39.338754+0000","last_peered":"2024-03-12T05:08:39.338754+0000","last_clean":"2024-03-12T05:08:39.338754+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:08:39.338754+0000","last_undegraded":"2024-03-12T05:08:39.338754+0000","last_fullsized":"2024-03-12T05:08:39.338754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758877+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:08:38.758877+0000","last_peered":"2024-03-12T05:08:38.758877+0000","last_clean":"2024-03-12T05:08:38.758877+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:08:38.758877+0000","last_undegraded":"2024-03-12T05:08:38.758877+0000","last_fullsized":"2024-03-12T05:08:38.758877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783273+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:08:38.783273+0000","last_peered":"2024-03-12T05:08:38.783273+0000","last_clean":"2024-03-12T05:08:38.783273+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:08:38.783273+0000","last_undegraded":"2024-03-12T05:08:38.783273+0000","last_fullsized":"2024-03-12T05:08:38.783273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-12T05:08:37.773050+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:08:37.773050+0000","last_peered":"2024-03-12T05:08:37.773050+0000","last_clean":"2024-03-12T05:08:37.773050+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:08:37.773050+0000","last_undegraded":"2024-03-12T05:08:37.773050+0000","last_fullsized":"2024-03-12T05:08:37.773050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.378976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783364+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:08:38.783364+0000","last_peered":"2024-03-12T05:08:38.783364+0000","last_clean":"2024-03-12T05:08:38.783364+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:08:38.783364+0000","last_undegraded":"2024-03-12T05:08:38.783364+0000","last_fullsized":"2024-03-12T05:08:38.783364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758669+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:08:38.758669+0000","last_peered":"2024-03-12T05:08:38.758669+0000","last_clean":"2024-03-12T05:08:38.758669+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:08:38.758669+0000","last_undegraded":"2024-03-12T05:08:38.758669+0000","last_fullsized":"2024-03-12T05:08:38.758669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758000+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:08:38.758000+0000","last_peered":"2024-03-12T05:08:38.758000+0000","last_clean":"2024-03-12T05:08:38.758000+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:08:38.758000+0000","last_undegraded":"2024-03-12T05:08:38.758000+0000","last_fullsized":"2024-03-12T05:08:38.758000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338167+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:08:39.338167+0000","last_peered":"2024-03-12T05:08:39.338167+0000","last_clean":"2024-03-12T05:08:39.338167+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:08:39.338167+0000","last_undegraded":"2024-03-12T05:08:39.338167+0000","last_fullsized":"2024-03-12T05:08:39.338167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3082,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":9420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383692,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":901120,"data_stored":429565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645329,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":1021739,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":430147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":420488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":1020515,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":913408,"data_stored":438135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":440329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":905216,"data_stored":429655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":441097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":438329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":905216,"data_stored":429587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":1020115,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:08:44.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 -- 172.21.15.45:0/1489975128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101290 msgr2=0x7feadc101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 --2- 172.21.15.45:0/1489975128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101290 0x7feadc101710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feac8009900 tx=0x7feac802bc30 comp rx=0 tx=0).stop 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 -- 172.21.15.45:0/1489975128 shutdown_connections 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 --2- 172.21.15.45:0/1489975128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101290 0x7feadc101710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 --2- 172.21.15.45:0/1489975128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101c50 0x7feadc1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 --2- 172.21.15.45:0/1489975128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 -- 172.21.15.45:0/1489975128 >> 172.21.15.45:0/1489975128 conn(0x7feadc0fb610 msgr2=0x7feadc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 -- 172.21.15.45:0/1489975128 shutdown_connections 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.920+0000 7feae30d4640 1 -- 172.21.15.45:0/1489975128 wait complete. 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 Processor -- start 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 -- start start 2024-03-12T05:08:44.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101290 0x7feadc102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc103310 con 0x7feadc101c50 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc103480 con 0x7feadc1000a0 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae30d4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1a11f0 con 0x7feadc101290 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae1671640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae1671640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43428/0 (socket says 172.21.15.45:43428) 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feae1671640 1 -- 172.21.15.45:0/3931326937 learned_addr learned my addr 172.21.15.45:0/3931326937 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.921+0000 7feadbfff640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101290 0x7feadc102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae0e70640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 -- 172.21.15.45:0/3931326937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101290 msgr2=0x7feadc102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101290 0x7feadc102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 -- 172.21.15.45:0/3931326937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 msgr2=0x7feadc104030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc104030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 -- 172.21.15.45:0/3931326937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8009560 con 0x7feadc101c50 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae0e70640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae1671640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fead400ead0 tx=0x7fead400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead400cde0 con 0x7feadc101c50 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feadc1a14a0 con 0x7feadc101c50 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fead4005020 con 0x7feadc101c50 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead40116d0 con 0x7feadc101c50 2024-03-12T05:08:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.922+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feadc1a19d0 con 0x7feadc101c50 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.923+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc10b960 con 0x7feadc101c50 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.924+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fead4005190 con 0x7feadc101c50 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.924+0000 7fead9ffb640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 0x7feacc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.925+0000 7feae0e70640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 0x7feacc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.925+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fead40cd650 con 0x7feadc101c50 2024-03-12T05:08:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.925+0000 7feae0e70640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 0x7feacc0a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feadc1010f0 tx=0x7fead0005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:44.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:44.929+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fead40967b0 con 0x7feadc101c50 2024-03-12T05:08:45.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.106+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feadc068ae0 con 0x7feacc0a1ac0 2024-03-12T05:08:45.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.113+0000 7fead9ffb640 1 -- 172.21.15.45:0/3931326937 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258947 (secure 0 0 0) 0x7feadc068ae0 con 0x7feacc0a1ac0 2024-03-12T05:08:45.114 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 msgr2=0x7feacc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 0x7feacc0a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feadc1010f0 tx=0x7fead0005e70 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 msgr2=0x7feadc102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fead400ead0 tx=0x7fead400ef90 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 shutdown_connections 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7feacc0a1ac0 0x7feacc0a3f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feadc101c50 0x7feadc102bb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feadc101290 0x7feadc102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 --2- 172.21.15.45:0/3931326937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feadc1000a0 0x7feadc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 >> 172.21.15.45:0/3931326937 conn(0x7feadc0fb610 msgr2=0x7feadc0683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:45.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 shutdown_connections 2024-03-12T05:08:45.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:45.116+0000 7feae30d4640 1 -- 172.21.15.45:0/3931326937 wait complete. 2024-03-12T05:08:45.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:44 smithi045 ceph-mon[32174]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:44 smithi045 ceph-mon[32174]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2826466645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:44 smithi045 ceph-mon[32174]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:08:45.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4136555458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.125 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-03-12T05:08:45.099482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635377,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":168,"num_read_kb":165,"num_write":397,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":324,"ondisk_log_size":324,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368724,"kb_used_data":11252,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573100,"statfs":{"total":1151940427776,"available":1151562854400,"internally_reserved":0,"allocated":11522048,"data_stored":7073639,"data_compressed":108348,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45009,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":32,"num_read_kb":46,"num_write":216,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001569"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.339026+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:08:39.339026+0000","last_peered":"2024-03-12T05:08:39.339026+0000","last_clean":"2024-03-12T05:08:39.339026+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:08:39.339026+0000","last_undegraded":"2024-03-12T05:08:39.339026+0000","last_fullsized":"2024-03-12T05:08:39.339026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331324+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:08:39.331324+0000","last_peered":"2024-03-12T05:08:39.331324+0000","last_clean":"2024-03-12T05:08:39.331324+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:08:39.331324+0000","last_undegraded":"2024-03-12T05:08:39.331324+0000","last_fullsized":"2024-03-12T05:08:39.331324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331903+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:08:39.331903+0000","last_peered":"2024-03-12T05:08:39.331903+0000","last_clean":"2024-03-12T05:08:39.331903+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:08:39.331903+0000","last_undegraded":"2024-03-12T05:08:39.331903+0000","last_fullsized":"2024-03-12T05:08:39.331903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333019+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:08:39.333019+0000","last_peered":"2024-03-12T05:08:39.333019+0000","last_clean":"2024-03-12T05:08:39.333019+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:08:39.333019+0000","last_undegraded":"2024-03-12T05:08:39.333019+0000","last_fullsized":"2024-03-12T05:08:39.333019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783361+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:08:38.783361+0000","last_peered":"2024-03-12T05:08:38.783361+0000","last_clean":"2024-03-12T05:08:38.783361+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:08:38.783361+0000","last_undegraded":"2024-03-12T05:08:38.783361+0000","last_fullsized":"2024-03-12T05:08:38.783361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331370+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:08:39.331370+0000","last_peered":"2024-03-12T05:08:39.331370+0000","last_clean":"2024-03-12T05:08:39.331370+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:08:39.331370+0000","last_undegraded":"2024-03-12T05:08:39.331370+0000","last_fullsized":"2024-03-12T05:08:39.331370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330039+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:08:39.330039+0000","last_peered":"2024-03-12T05:08:39.330039+0000","last_clean":"2024-03-12T05:08:39.330039+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:08:39.330039+0000","last_undegraded":"2024-03-12T05:08:39.330039+0000","last_fullsized":"2024-03-12T05:08:39.330039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783072+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:08:38.783072+0000","last_peered":"2024-03-12T05:08:38.783072+0000","last_clean":"2024-03-12T05:08:38.783072+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:08:38.783072+0000","last_undegraded":"2024-03-12T05:08:38.783072+0000","last_fullsized":"2024-03-12T05:08:38.783072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.759068+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:08:38.759068+0000","last_peered":"2024-03-12T05:08:38.759068+0000","last_clean":"2024-03-12T05:08:38.759068+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:08:38.759068+0000","last_undegraded":"2024-03-12T05:08:38.759068+0000","last_fullsized":"2024-03-12T05:08:38.759068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330643+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:08:39.330643+0000","last_peered":"2024-03-12T05:08:39.330643+0000","last_clean":"2024-03-12T05:08:39.330643+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:08:39.330643+0000","last_undegraded":"2024-03-12T05:08:39.330643+0000","last_fullsized":"2024-03-12T05:08:39.330643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330422+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:08:39.330422+0000","last_peered":"2024-03-12T05:08:39.330422+0000","last_clean":"2024-03-12T05:08:39.330422+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:08:39.330422+0000","last_undegraded":"2024-03-12T05:08:39.330422+0000","last_fullsized":"2024-03-12T05:08:39.330422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333182+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:08:39.333182+0000","last_peered":"2024-03-12T05:08:39.333182+0000","last_clean":"2024-03-12T05:08:39.333182+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:08:39.333182+0000","last_undegraded":"2024-03-12T05:08:39.333182+0000","last_fullsized":"2024-03-12T05:08:39.333182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757985+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:08:38.757985+0000","last_peered":"2024-03-12T05:08:38.757985+0000","last_clean":"2024-03-12T05:08:38.757985+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:08:38.757985+0000","last_undegraded":"2024-03-12T05:08:38.757985+0000","last_fullsized":"2024-03-12T05:08:38.757985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331484+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:08:39.331484+0000","last_peered":"2024-03-12T05:08:39.331484+0000","last_clean":"2024-03-12T05:08:39.331484+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:08:39.331484+0000","last_undegraded":"2024-03-12T05:08:39.331484+0000","last_fullsized":"2024-03-12T05:08:39.331484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331718+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:08:39.331718+0000","last_peered":"2024-03-12T05:08:39.331718+0000","last_clean":"2024-03-12T05:08:39.331718+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:08:39.331718+0000","last_undegraded":"2024-03-12T05:08:39.331718+0000","last_fullsized":"2024-03-12T05:08:39.331718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758463+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:08:38.758463+0000","last_peered":"2024-03-12T05:08:38.758463+0000","last_clean":"2024-03-12T05:08:38.758463+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:08:38.758463+0000","last_undegraded":"2024-03-12T05:08:38.758463+0000","last_fullsized":"2024-03-12T05:08:38.758463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.339103+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:08:39.339103+0000","last_peered":"2024-03-12T05:08:39.339103+0000","last_clean":"2024-03-12T05:08:39.339103+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:08:39.339103+0000","last_undegraded":"2024-03-12T05:08:39.339103+0000","last_fullsized":"2024-03-12T05:08:39.339103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781635+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:08:38.781635+0000","last_peered":"2024-03-12T05:08:38.781635+0000","last_clean":"2024-03-12T05:08:38.781635+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:08:38.781635+0000","last_undegraded":"2024-03-12T05:08:38.781635+0000","last_fullsized":"2024-03-12T05:08:38.781635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758569+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:08:38.758569+0000","last_peered":"2024-03-12T05:08:38.758569+0000","last_clean":"2024-03-12T05:08:38.758569+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:08:38.758569+0000","last_undegraded":"2024-03-12T05:08:38.758569+0000","last_fullsized":"2024-03-12T05:08:38.758569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.362295+0000","last_change":"2024-03-12T05:08:36.774328+0000","last_active":"2024-03-12T05:08:43.362295+0000","last_peered":"2024-03-12T05:08:43.362295+0000","last_clean":"2024-03-12T05:08:43.362295+0000","last_became_active":"2024-03-12T05:08:36.774000+0000","last_became_peered":"2024-03-12T05:08:36.774000+0000","last_unstale":"2024-03-12T05:08:43.362295+0000","last_undegraded":"2024-03-12T05:08:43.362295+0000","last_fullsized":"2024-03-12T05:08:43.362295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:59.134654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783400+0000","last_change":"2024-03-12T05:08:37.779921+0000","last_active":"2024-03-12T05:08:38.783400+0000","last_peered":"2024-03-12T05:08:38.783400+0000","last_clean":"2024-03-12T05:08:38.783400+0000","last_became_active":"2024-03-12T05:08:37.779542+0000","last_became_peered":"2024-03-12T05:08:37.779542+0000","last_unstale":"2024-03-12T05:08:38.783400+0000","last_undegraded":"2024-03-12T05:08:38.783400+0000","last_fullsized":"2024-03-12T05:08:38.783400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:00.663534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.029459+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:08:40.029459+0000","last_peered":"2024-03-12T05:08:40.029459+0000","last_clean":"2024-03-12T05:08:40.029459+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:08:40.029459+0000","last_undegraded":"2024-03-12T05:08:40.029459+0000","last_fullsized":"2024-03-12T05:08:40.029459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331258+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:08:39.331258+0000","last_peered":"2024-03-12T05:08:39.331258+0000","last_clean":"2024-03-12T05:08:39.331258+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:08:39.331258+0000","last_undegraded":"2024-03-12T05:08:39.331258+0000","last_fullsized":"2024-03-12T05:08:39.331258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'11","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.356474+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:08:42.356474+0000","last_peered":"2024-03-12T05:08:42.356474+0000","last_clean":"2024-03-12T05:08:42.356474+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:08:42.356474+0000","last_undegraded":"2024-03-12T05:08:42.356474+0000","last_fullsized":"2024-03-12T05:08:42.356474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758715+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:08:38.758715+0000","last_peered":"2024-03-12T05:08:38.758715+0000","last_clean":"2024-03-12T05:08:38.758715+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:08:38.758715+0000","last_undegraded":"2024-03-12T05:08:38.758715+0000","last_fullsized":"2024-03-12T05:08:38.758715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.356526+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:08:42.356526+0000","last_peered":"2024-03-12T05:08:42.356526+0000","last_clean":"2024-03-12T05:08:42.356526+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:08:42.356526+0000","last_undegraded":"2024-03-12T05:08:42.356526+0000","last_fullsized":"2024-03-12T05:08:42.356526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.334424+0000","last_change":"2024-03-12T05:08:37.777137+0000","last_active":"2024-03-12T05:08:39.334424+0000","last_peered":"2024-03-12T05:08:39.334424+0000","last_clean":"2024-03-12T05:08:39.334424+0000","last_became_active":"2024-03-12T05:08:37.776751+0000","last_became_peered":"2024-03-12T05:08:37.776751+0000","last_unstale":"2024-03-12T05:08:39.334424+0000","last_undegraded":"2024-03-12T05:08:39.334424+0000","last_fullsized":"2024-03-12T05:08:39.334424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:03:58.003069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.358714+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:08:43.358714+0000","last_peered":"2024-03-12T05:08:43.358714+0000","last_clean":"2024-03-12T05:08:43.358714+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:08:43.358714+0000","last_undegraded":"2024-03-12T05:08:43.358714+0000","last_fullsized":"2024-03-12T05:08:43.358714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783542+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:08:38.783542+0000","last_peered":"2024-03-12T05:08:38.783542+0000","last_clean":"2024-03-12T05:08:38.783542+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:08:38.783542+0000","last_undegraded":"2024-03-12T05:08:38.783542+0000","last_fullsized":"2024-03-12T05:08:38.783542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:41.349711+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:08:41.349711+0000","last_peered":"2024-03-12T05:08:41.349711+0000","last_clean":"2024-03-12T05:08:41.349711+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:08:41.349711+0000","last_undegraded":"2024-03-12T05:08:41.349711+0000","last_fullsized":"2024-03-12T05:08:41.349711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333519+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:08:39.333519+0000","last_peered":"2024-03-12T05:08:39.333519+0000","last_clean":"2024-03-12T05:08:39.333519+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:08:39.333519+0000","last_undegraded":"2024-03-12T05:08:39.333519+0000","last_fullsized":"2024-03-12T05:08:39.333519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.343687+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:08:40.343687+0000","last_peered":"2024-03-12T05:08:40.343687+0000","last_clean":"2024-03-12T05:08:40.343687+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:08:40.343687+0000","last_undegraded":"2024-03-12T05:08:40.343687+0000","last_fullsized":"2024-03-12T05:08:40.343687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.782940+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:08:38.782940+0000","last_peered":"2024-03-12T05:08:38.782940+0000","last_clean":"2024-03-12T05:08:38.782940+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:08:38.782940+0000","last_undegraded":"2024-03-12T05:08:38.782940+0000","last_fullsized":"2024-03-12T05:08:38.782940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.356851+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:08:42.356851+0000","last_peered":"2024-03-12T05:08:42.356851+0000","last_clean":"2024-03-12T05:08:42.356851+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:08:42.356851+0000","last_undegraded":"2024-03-12T05:08:42.356851+0000","last_fullsized":"2024-03-12T05:08:42.356851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.781500+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:08:38.781500+0000","last_peered":"2024-03-12T05:08:38.781500+0000","last_clean":"2024-03-12T05:08:38.781500+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:08:38.781500+0000","last_undegraded":"2024-03-12T05:08:38.781500+0000","last_fullsized":"2024-03-12T05:08:38.781500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.357695+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:08:43.357695+0000","last_peered":"2024-03-12T05:08:43.357695+0000","last_clean":"2024-03-12T05:08:43.357695+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:08:43.357695+0000","last_undegraded":"2024-03-12T05:08:43.357695+0000","last_fullsized":"2024-03-12T05:08:43.357695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758966+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:08:38.758966+0000","last_peered":"2024-03-12T05:08:38.758966+0000","last_clean":"2024-03-12T05:08:38.758966+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:08:38.758966+0000","last_undegraded":"2024-03-12T05:08:38.758966+0000","last_fullsized":"2024-03-12T05:08:38.758966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.355229+0000","last_change":"2024-03-12T05:08:36.770968+0000","last_active":"2024-03-12T05:08:42.355229+0000","last_peered":"2024-03-12T05:08:42.355229+0000","last_clean":"2024-03-12T05:08:42.355229+0000","last_became_active":"2024-03-12T05:08:36.770646+0000","last_became_peered":"2024-03-12T05:08:36.770646+0000","last_unstale":"2024-03-12T05:08:42.355229+0000","last_undegraded":"2024-03-12T05:08:42.355229+0000","last_fullsized":"2024-03-12T05:08:42.355229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:45:26.921443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331258+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:08:39.331258+0000","last_peered":"2024-03-12T05:08:39.331258+0000","last_clean":"2024-03-12T05:08:39.331258+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:08:39.331258+0000","last_undegraded":"2024-03-12T05:08:39.331258+0000","last_fullsized":"2024-03-12T05:08:39.331258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:41.350483+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:08:41.350483+0000","last_peered":"2024-03-12T05:08:41.350483+0000","last_clean":"2024-03-12T05:08:41.350483+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:08:41.350483+0000","last_undegraded":"2024-03-12T05:08:41.350483+0000","last_fullsized":"2024-03-12T05:08:41.350483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758750+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:08:38.758750+0000","last_peered":"2024-03-12T05:08:38.758750+0000","last_clean":"2024-03-12T05:08:38.758750+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:08:38.758750+0000","last_undegraded":"2024-03-12T05:08:38.758750+0000","last_fullsized":"2024-03-12T05:08:38.758750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'21","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.439268+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:08:43.439268+0000","last_peered":"2024-03-12T05:08:43.439268+0000","last_clean":"2024-03-12T05:08:43.439268+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:08:43.439268+0000","last_undegraded":"2024-03-12T05:08:43.439268+0000","last_fullsized":"2024-03-12T05:08:43.439268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":23,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783565+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:08:38.783565+0000","last_peered":"2024-03-12T05:08:38.783565+0000","last_clean":"2024-03-12T05:08:38.783565+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:08:38.783565+0000","last_undegraded":"2024-03-12T05:08:38.783565+0000","last_fullsized":"2024-03-12T05:08:38.783565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.343239+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:08:40.343239+0000","last_peered":"2024-03-12T05:08:40.343239+0000","last_clean":"2024-03-12T05:08:40.343239+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:08:40.343239+0000","last_undegraded":"2024-03-12T05:08:40.343239+0000","last_fullsized":"2024-03-12T05:08:40.343239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758050+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:08:38.758050+0000","last_peered":"2024-03-12T05:08:38.758050+0000","last_clean":"2024-03-12T05:08:38.758050+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:08:38.758050+0000","last_undegraded":"2024-03-12T05:08:38.758050+0000","last_fullsized":"2024-03-12T05:08:38.758050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.357113+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:08:42.357113+0000","last_peered":"2024-03-12T05:08:42.357113+0000","last_clean":"2024-03-12T05:08:42.357113+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:08:42.357113+0000","last_undegraded":"2024-03-12T05:08:42.357113+0000","last_fullsized":"2024-03-12T05:08:42.357113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333962+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:08:39.333962+0000","last_peered":"2024-03-12T05:08:39.333962+0000","last_clean":"2024-03-12T05:08:39.333962+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:08:39.333962+0000","last_undegraded":"2024-03-12T05:08:39.333962+0000","last_fullsized":"2024-03-12T05:08:39.333962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.362132+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:08:43.362132+0000","last_peered":"2024-03-12T05:08:43.362132+0000","last_clean":"2024-03-12T05:08:43.362132+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:08:43.362132+0000","last_undegraded":"2024-03-12T05:08:43.362132+0000","last_fullsized":"2024-03-12T05:08:43.362132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331891+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:08:39.331891+0000","last_peered":"2024-03-12T05:08:39.331891+0000","last_clean":"2024-03-12T05:08:39.331891+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:08:39.331891+0000","last_undegraded":"2024-03-12T05:08:39.331891+0000","last_fullsized":"2024-03-12T05:08:39.331891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:41.350535+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:08:41.350535+0000","last_peered":"2024-03-12T05:08:41.350535+0000","last_clean":"2024-03-12T05:08:41.350535+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:08:41.350535+0000","last_undegraded":"2024-03-12T05:08:41.350535+0000","last_fullsized":"2024-03-12T05:08:41.350535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.332925+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:08:39.332925+0000","last_peered":"2024-03-12T05:08:39.332925+0000","last_clean":"2024-03-12T05:08:39.332925+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:08:39.332925+0000","last_undegraded":"2024-03-12T05:08:39.332925+0000","last_fullsized":"2024-03-12T05:08:39.332925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783773+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:08:38.783773+0000","last_peered":"2024-03-12T05:08:38.783773+0000","last_clean":"2024-03-12T05:08:38.783773+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:08:38.783773+0000","last_undegraded":"2024-03-12T05:08:38.783773+0000","last_fullsized":"2024-03-12T05:08:38.783773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338619+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:08:39.338619+0000","last_peered":"2024-03-12T05:08:39.338619+0000","last_clean":"2024-03-12T05:08:39.338619+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:08:39.338619+0000","last_undegraded":"2024-03-12T05:08:39.338619+0000","last_fullsized":"2024-03-12T05:08:39.338619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.546615+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:08:39.546615+0000","last_peered":"2024-03-12T05:08:39.546615+0000","last_clean":"2024-03-12T05:08:39.546615+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:08:39.546615+0000","last_undegraded":"2024-03-12T05:08:39.546615+0000","last_fullsized":"2024-03-12T05:08:39.546615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330581+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:08:39.330581+0000","last_peered":"2024-03-12T05:08:39.330581+0000","last_clean":"2024-03-12T05:08:39.330581+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:08:39.330581+0000","last_undegraded":"2024-03-12T05:08:39.330581+0000","last_fullsized":"2024-03-12T05:08:39.330581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.357712+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:08:43.357712+0000","last_peered":"2024-03-12T05:08:43.357712+0000","last_clean":"2024-03-12T05:08:43.357712+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:08:43.357712+0000","last_undegraded":"2024-03-12T05:08:43.357712+0000","last_fullsized":"2024-03-12T05:08:43.357712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330702+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:08:39.330702+0000","last_peered":"2024-03-12T05:08:39.330702+0000","last_clean":"2024-03-12T05:08:39.330702+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:08:39.330702+0000","last_undegraded":"2024-03-12T05:08:39.330702+0000","last_fullsized":"2024-03-12T05:08:39.330702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.333133+0000","last_change":"2024-03-12T05:08:37.777792+0000","last_active":"2024-03-12T05:08:39.333133+0000","last_peered":"2024-03-12T05:08:39.333133+0000","last_clean":"2024-03-12T05:08:39.333133+0000","last_became_active":"2024-03-12T05:08:37.777220+0000","last_became_peered":"2024-03-12T05:08:37.777220+0000","last_unstale":"2024-03-12T05:08:39.333133+0000","last_undegraded":"2024-03-12T05:08:39.333133+0000","last_fullsized":"2024-03-12T05:08:39.333133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:37:40.783234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.360313+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:08:43.360313+0000","last_peered":"2024-03-12T05:08:43.360313+0000","last_clean":"2024-03-12T05:08:43.360313+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:08:43.360313+0000","last_undegraded":"2024-03-12T05:08:43.360313+0000","last_fullsized":"2024-03-12T05:08:43.360313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783427+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:08:38.783427+0000","last_peered":"2024-03-12T05:08:38.783427+0000","last_clean":"2024-03-12T05:08:38.783427+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:08:38.783427+0000","last_undegraded":"2024-03-12T05:08:38.783427+0000","last_fullsized":"2024-03-12T05:08:38.783427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:43.356260+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:08:43.356260+0000","last_peered":"2024-03-12T05:08:43.356260+0000","last_clean":"2024-03-12T05:08:43.356260+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:08:43.356260+0000","last_undegraded":"2024-03-12T05:08:43.356260+0000","last_fullsized":"2024-03-12T05:08:43.356260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.332071+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:08:39.332071+0000","last_peered":"2024-03-12T05:08:39.332071+0000","last_clean":"2024-03-12T05:08:39.332071+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:08:39.332071+0000","last_undegraded":"2024-03-12T05:08:39.332071+0000","last_fullsized":"2024-03-12T05:08:39.332071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.757747+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:08:38.757747+0000","last_peered":"2024-03-12T05:08:38.757747+0000","last_clean":"2024-03-12T05:08:38.757747+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:08:38.757747+0000","last_undegraded":"2024-03-12T05:08:38.757747+0000","last_fullsized":"2024-03-12T05:08:38.757747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.341379+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:08:40.341379+0000","last_peered":"2024-03-12T05:08:40.341379+0000","last_clean":"2024-03-12T05:08:40.341379+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:08:40.341379+0000","last_undegraded":"2024-03-12T05:08:40.341379+0000","last_fullsized":"2024-03-12T05:08:40.341379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:41.350814+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:08:41.350814+0000","last_peered":"2024-03-12T05:08:41.350814+0000","last_clean":"2024-03-12T05:08:41.350814+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:08:41.350814+0000","last_undegraded":"2024-03-12T05:08:41.350814+0000","last_fullsized":"2024-03-12T05:08:41.350814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758042+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:08:38.758042+0000","last_peered":"2024-03-12T05:08:38.758042+0000","last_clean":"2024-03-12T05:08:38.758042+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:08:38.758042+0000","last_undegraded":"2024-03-12T05:08:38.758042+0000","last_fullsized":"2024-03-12T05:08:38.758042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.782172+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:08:42.782172+0000","last_peered":"2024-03-12T05:08:42.782172+0000","last_clean":"2024-03-12T05:08:42.782172+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:08:42.782172+0000","last_undegraded":"2024-03-12T05:08:42.782172+0000","last_fullsized":"2024-03-12T05:08:42.782172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783473+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:08:38.783473+0000","last_peered":"2024-03-12T05:08:38.783473+0000","last_clean":"2024-03-12T05:08:38.783473+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:08:38.783473+0000","last_undegraded":"2024-03-12T05:08:38.783473+0000","last_fullsized":"2024-03-12T05:08:38.783473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.355232+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:08:42.355232+0000","last_peered":"2024-03-12T05:08:42.355232+0000","last_clean":"2024-03-12T05:08:42.355232+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:08:42.355232+0000","last_undegraded":"2024-03-12T05:08:42.355232+0000","last_fullsized":"2024-03-12T05:08:42.355232+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331140+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:08:39.331140+0000","last_peered":"2024-03-12T05:08:39.331140+0000","last_clean":"2024-03-12T05:08:39.331140+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:08:39.331140+0000","last_undegraded":"2024-03-12T05:08:39.331140+0000","last_fullsized":"2024-03-12T05:08:39.331140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330509+0000","last_change":"2024-03-12T05:08:36.768605+0000","last_active":"2024-03-12T05:08:39.330509+0000","last_peered":"2024-03-12T05:08:39.330509+0000","last_clean":"2024-03-12T05:08:39.330509+0000","last_became_active":"2024-03-12T05:08:36.768154+0000","last_became_peered":"2024-03-12T05:08:36.768154+0000","last_unstale":"2024-03-12T05:08:39.330509+0000","last_undegraded":"2024-03-12T05:08:39.330509+0000","last_fullsized":"2024-03-12T05:08:39.330509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:32:53.952973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783274+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:08:38.783274+0000","last_peered":"2024-03-12T05:08:38.783274+0000","last_clean":"2024-03-12T05:08:38.783274+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:08:38.783274+0000","last_undegraded":"2024-03-12T05:08:38.783274+0000","last_fullsized":"2024-03-12T05:08:38.783274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.354978+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:08:42.354978+0000","last_peered":"2024-03-12T05:08:42.354978+0000","last_clean":"2024-03-12T05:08:42.354978+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:08:42.354978+0000","last_undegraded":"2024-03-12T05:08:42.354978+0000","last_fullsized":"2024-03-12T05:08:42.354978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338212+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:08:39.338212+0000","last_peered":"2024-03-12T05:08:39.338212+0000","last_clean":"2024-03-12T05:08:39.338212+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:08:39.338212+0000","last_undegraded":"2024-03-12T05:08:39.338212+0000","last_fullsized":"2024-03-12T05:08:39.338212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758254+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:08:38.758254+0000","last_peered":"2024-03-12T05:08:38.758254+0000","last_clean":"2024-03-12T05:08:38.758254+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:08:38.758254+0000","last_undegraded":"2024-03-12T05:08:38.758254+0000","last_fullsized":"2024-03-12T05:08:38.758254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783371+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:08:38.783371+0000","last_peered":"2024-03-12T05:08:38.783371+0000","last_clean":"2024-03-12T05:08:38.783371+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:08:38.783371+0000","last_undegraded":"2024-03-12T05:08:38.783371+0000","last_fullsized":"2024-03-12T05:08:38.783371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.334864+0000","last_change":"2024-03-12T05:08:36.773558+0000","last_active":"2024-03-12T05:08:39.334864+0000","last_peered":"2024-03-12T05:08:39.334864+0000","last_clean":"2024-03-12T05:08:39.334864+0000","last_became_active":"2024-03-12T05:08:36.773272+0000","last_became_peered":"2024-03-12T05:08:36.773272+0000","last_unstale":"2024-03-12T05:08:39.334864+0000","last_undegraded":"2024-03-12T05:08:39.334864+0000","last_fullsized":"2024-03-12T05:08:39.334864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:05.069202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331443+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:08:39.331443+0000","last_peered":"2024-03-12T05:08:39.331443+0000","last_clean":"2024-03-12T05:08:39.331443+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:08:39.331443+0000","last_undegraded":"2024-03-12T05:08:39.331443+0000","last_fullsized":"2024-03-12T05:08:39.331443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:40.343210+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:08:40.343210+0000","last_peered":"2024-03-12T05:08:40.343210+0000","last_clean":"2024-03-12T05:08:40.343210+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:08:40.343210+0000","last_undegraded":"2024-03-12T05:08:40.343210+0000","last_fullsized":"2024-03-12T05:08:40.343210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330669+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:08:39.330669+0000","last_peered":"2024-03-12T05:08:39.330669+0000","last_clean":"2024-03-12T05:08:39.330669+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:08:39.330669+0000","last_undegraded":"2024-03-12T05:08:39.330669+0000","last_fullsized":"2024-03-12T05:08:39.330669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.356322+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:08:42.356322+0000","last_peered":"2024-03-12T05:08:42.356322+0000","last_clean":"2024-03-12T05:08:42.356322+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:08:42.356322+0000","last_undegraded":"2024-03-12T05:08:42.356322+0000","last_fullsized":"2024-03-12T05:08:42.356322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330042+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:08:39.330042+0000","last_peered":"2024-03-12T05:08:39.330042+0000","last_clean":"2024-03-12T05:08:39.330042+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:08:39.330042+0000","last_undegraded":"2024-03-12T05:08:39.330042+0000","last_fullsized":"2024-03-12T05:08:39.330042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:42.355283+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:08:42.355283+0000","last_peered":"2024-03-12T05:08:42.355283+0000","last_clean":"2024-03-12T05:08:42.355283+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:08:42.355283+0000","last_undegraded":"2024-03-12T05:08:42.355283+0000","last_fullsized":"2024-03-12T05:08:42.355283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.330986+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:08:39.330986+0000","last_peered":"2024-03-12T05:08:39.330986+0000","last_clean":"2024-03-12T05:08:39.330986+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:08:39.330986+0000","last_undegraded":"2024-03-12T05:08:39.330986+0000","last_fullsized":"2024-03-12T05:08:39.330986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.337860+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:08:39.337860+0000","last_peered":"2024-03-12T05:08:39.337860+0000","last_clean":"2024-03-12T05:08:39.337860+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:08:39.337860+0000","last_undegraded":"2024-03-12T05:08:39.337860+0000","last_fullsized":"2024-03-12T05:08:39.337860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758440+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:08:38.758440+0000","last_peered":"2024-03-12T05:08:38.758440+0000","last_clean":"2024-03-12T05:08:38.758440+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:08:38.758440+0000","last_undegraded":"2024-03-12T05:08:38.758440+0000","last_fullsized":"2024-03-12T05:08:38.758440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758253+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:08:38.758253+0000","last_peered":"2024-03-12T05:08:38.758253+0000","last_clean":"2024-03-12T05:08:38.758253+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:08:38.758253+0000","last_undegraded":"2024-03-12T05:08:38.758253+0000","last_fullsized":"2024-03-12T05:08:38.758253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.331547+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:08:39.331547+0000","last_peered":"2024-03-12T05:08:39.331547+0000","last_clean":"2024-03-12T05:08:39.331547+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:08:39.331547+0000","last_undegraded":"2024-03-12T05:08:39.331547+0000","last_fullsized":"2024-03-12T05:08:39.331547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758046+0000","last_change":"2024-03-12T05:08:37.774717+0000","last_active":"2024-03-12T05:08:38.758046+0000","last_peered":"2024-03-12T05:08:38.758046+0000","last_clean":"2024-03-12T05:08:38.758046+0000","last_became_active":"2024-03-12T05:08:37.774289+0000","last_became_peered":"2024-03-12T05:08:37.774289+0000","last_unstale":"2024-03-12T05:08:38.758046+0000","last_undegraded":"2024-03-12T05:08:38.758046+0000","last_fullsized":"2024-03-12T05:08:38.758046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:21:43.200061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338754+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:08:39.338754+0000","last_peered":"2024-03-12T05:08:39.338754+0000","last_clean":"2024-03-12T05:08:39.338754+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:08:39.338754+0000","last_undegraded":"2024-03-12T05:08:39.338754+0000","last_fullsized":"2024-03-12T05:08:39.338754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758877+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:08:38.758877+0000","last_peered":"2024-03-12T05:08:38.758877+0000","last_clean":"2024-03-12T05:08:38.758877+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:08:38.758877+0000","last_undegraded":"2024-03-12T05:08:38.758877+0000","last_fullsized":"2024-03-12T05:08:38.758877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783273+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:08:38.783273+0000","last_peered":"2024-03-12T05:08:38.783273+0000","last_clean":"2024-03-12T05:08:38.783273+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:08:38.783273+0000","last_undegraded":"2024-03-12T05:08:38.783273+0000","last_fullsized":"2024-03-12T05:08:38.783273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.332907+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:08:39.332907+0000","last_peered":"2024-03-12T05:08:39.332907+0000","last_clean":"2024-03-12T05:08:39.332907+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:08:39.332907+0000","last_undegraded":"2024-03-12T05:08:39.332907+0000","last_fullsized":"2024-03-12T05:08:39.332907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.783364+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:08:38.783364+0000","last_peered":"2024-03-12T05:08:38.783364+0000","last_clean":"2024-03-12T05:08:38.783364+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:08:38.783364+0000","last_undegraded":"2024-03-12T05:08:38.783364+0000","last_fullsized":"2024-03-12T05:08:38.783364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758669+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:08:38.758669+0000","last_peered":"2024-03-12T05:08:38.758669+0000","last_clean":"2024-03-12T05:08:38.758669+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:08:38.758669+0000","last_undegraded":"2024-03-12T05:08:38.758669+0000","last_fullsized":"2024-03-12T05:08:38.758669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:38.758000+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:08:38.758000+0000","last_peered":"2024-03-12T05:08:38.758000+0000","last_clean":"2024-03-12T05:08:38.758000+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:08:38.758000+0000","last_undegraded":"2024-03-12T05:08:38.758000+0000","last_fullsized":"2024-03-12T05:08:38.758000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:08:39.338167+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:08:39.338167+0000","last_peered":"2024-03-12T05:08:39.338167+0000","last_clean":"2024-03-12T05:08:39.338167+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:08:39.338167+0000","last_undegraded":"2024-03-12T05:08:39.338167+0000","last_fullsized":"2024-03-12T05:08:39.338167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":44775,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":32,"num_read_kb":46,"num_write":214,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":61628,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":439718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645330,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":1051642,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":439257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":439635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1032544,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":933888,"data_stored":438974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":440329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36268,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708884,"statfs":{"total":95995035648,"available":95957897216,"internally_reserved":0,"allocated":954368,"data_stored":440704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":441097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":438799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":439226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":1031714,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":40320,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:08:45.129 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-12T05:08:45.130 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T05:08:45.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T05:08:45.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:44 smithi130 ceph-mon[35398]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:44 smithi130 ceph-mon[35398]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2826466645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:44 smithi130 ceph-mon[35398]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:08:45.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4136555458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:44 smithi116 ceph-mon[35486]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:44 smithi116 ceph-mon[35486]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:08:45.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2826466645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:44 smithi116 ceph-mon[35486]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:08:45.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4136555458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:45.404 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:46.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:45 smithi045 ceph-mon[32174]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:46.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:45 smithi045 ceph-mon[32174]: pgmap v286: 97 pgs: 97 active+clean; 620 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 30 op/s 2024-03-12T05:08:46.118 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:45 smithi045 ceph-mon[32174]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:46.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:45 smithi130 ceph-mon[35398]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:46.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:45 smithi130 ceph-mon[35398]: pgmap v286: 97 pgs: 97 active+clean; 620 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 30 op/s 2024-03-12T05:08:46.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:45 smithi130 ceph-mon[35398]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:46.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:45 smithi116 ceph-mon[35486]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:46.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:45 smithi116 ceph-mon[35486]: pgmap v286: 97 pgs: 97 active+clean; 620 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 13 KiB/s wr, 30 op/s 2024-03-12T05:08:46.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:45 smithi116 ceph-mon[35486]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:08:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.159+0000 7f49af548700 1 -- 172.21.15.45:0/843812852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 msgr2=0x7f49a81001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.159+0000 7f49af548700 1 --2- 172.21.15.45:0/843812852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a81001f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4998009990 tx=0x7f4998009ca0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.159+0000 7f49af548700 1 -- 172.21.15.45:0/843812852 shutdown_connections 2024-03-12T05:08:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.159+0000 7f49af548700 1 --2- 172.21.15.45:0/843812852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8100730 0x7f49a8100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 --2- 172.21.15.45:0/843812852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8102270 0x7f49a8102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 --2- 172.21.15.45:0/843812852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a81001f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 -- 172.21.15.45:0/843812852 >> 172.21.15.45:0/843812852 conn(0x7f49a80fb910 msgr2=0x7f49a80fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 -- 172.21.15.45:0/843812852 shutdown_connections 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 -- 172.21.15.45:0/843812852 wait complete. 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 Processor -- start 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 -- start start 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100730 0x7f49a819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.160+0000 7f49af548700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 0x7f49a81a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49af548700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81a1830 con 0x7f49a80ffd70 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49af548700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81a1970 con 0x7f49a8100730 2024-03-12T05:08:47.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49af548700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a81a1b30 con 0x7f49a8102270 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43440/0 (socket says 172.21.15.45:43440) 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 -- 172.21.15.45:0/2950938789 learned_addr learned my addr 172.21.15.45:0/2950938789 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49acae3700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100730 0x7f49a819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49adae5700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 0x7f49a81a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 -- 172.21.15.45:0/2950938789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 msgr2=0x7f49a81a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 0x7f49a81a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 -- 172.21.15.45:0/2950938789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100730 msgr2=0x7f49a819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100730 0x7f49a819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 -- 172.21.15.45:0/2950938789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980095f0 con 0x7f49a80ffd70 2024-03-12T05:08:47.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49adae5700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 0x7f49a81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49ad2e4700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c920 tx=0x7f49a400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4007b00 con 0x7f49a80ffd70 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49a400ce70 con 0x7f49a80ffd70 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.161+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a81a1e10 con 0x7f49a80ffd70 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.162+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40054e0 con 0x7f49a80ffd70 2024-03-12T05:08:47.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.162+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a81a2230 con 0x7f49a80ffd70 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.163+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a8050180 con 0x7f49a80ffd70 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.164+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f49a4004750 con 0x7f49a80ffd70 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.164+0000 7f499e7fc700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 0x7f49940a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.164+0000 7f49acae3700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 0x7f49940a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.164+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f49a40ccee0 con 0x7f49a80ffd70 2024-03-12T05:08:47.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.165+0000 7f49acae3700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 0x7f49940a4140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f499800b5c0 tx=0x7f4998005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:47.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.169+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f49a4096140 con 0x7f49a80ffd70 2024-03-12T05:08:47.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.455+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f49a810bc20 con 0x7f49a80ffd70 2024-03-12T05:08:47.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:47 smithi045 ceph-mon[32174]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-03-12T05:08:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.457+0000 7f499e7fc700 1 -- 172.21.15.45:0/2950938789 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f49a401d0f0 con 0x7f49a80ffd70 2024-03-12T05:08:47.455 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-03-12T05:08:47.456 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBf4+9lPvc8GxAAOBd1tbGw8s6zG5Wx2nO+Rw== 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.459+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 msgr2=0x7f49940a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.459+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 0x7f49940a4140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f499800b5c0 tx=0x7f4998005fd0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.459+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 msgr2=0x7f49a8199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.459+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c920 tx=0x7f49a400cc30 comp rx=0 tx=0).stop 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 shutdown_connections 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f49940a1c80 0x7f49940a4140 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49a8102270 0x7f49a81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49a80ffd70 0x7f49a8199ce0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 --2- 172.21.15.45:0/2950938789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100730 0x7f49a819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 >> 172.21.15.45:0/2950938789 conn(0x7f49a80fb910 msgr2=0x7f49a80fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 shutdown_connections 2024-03-12T05:08:47.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:47.460+0000 7f49af548700 1 -- 172.21.15.45:0/2950938789 wait complete. 2024-03-12T05:08:47.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:47 smithi130 ceph-mon[35398]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-03-12T05:08:47.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:47 smithi116 ceph-mon[35486]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-03-12T05:08:48.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:48 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:08:48.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:48 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:08:48.363 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:08:48.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T05:08:48.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T05:08:48.409 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T05:08:48.409 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T05:08:48.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T05:08:48.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:48 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:08:48.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:48 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:08:48.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:48 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:08:48.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:48 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2950938789' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:08:48.694 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:49.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:49 smithi045 ceph-mon[32174]: pgmap v288: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-03-12T05:08:49.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:49 smithi130 ceph-mon[35398]: pgmap v288: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-03-12T05:08:49.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:49 smithi116 ceph-mon[35486]: pgmap v288: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 KiB/s wr, 26 op/s 2024-03-12T05:08:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.457+0000 7fb8c1135700 1 -- 172.21.15.45:0/836961564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 msgr2=0x7fb8bc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.457+0000 7fb8c1135700 1 --2- 172.21.15.45:0/836961564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc06a7d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40099c0 tx=0x7fb8a4009cd0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 -- 172.21.15.45:0/836961564 shutdown_connections 2024-03-12T05:08:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 --2- 172.21.15.45:0/836961564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 0x7fb8bc10eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 --2- 172.21.15.45:0/836961564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 0x7fb8bc1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 --2- 172.21.15.45:0/836961564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc06a7d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 -- 172.21.15.45:0/836961564 >> 172.21.15.45:0/836961564 conn(0x7fb8bc076b90 msgr2=0x7fb8bc076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 -- 172.21.15.45:0/836961564 shutdown_connections 2024-03-12T05:08:50.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 -- 172.21.15.45:0/836961564 wait complete. 2024-03-12T05:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.458+0000 7fb8c1135700 1 Processor -- start 2024-03-12T05:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 -- start start 2024-03-12T05:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 0x7fb8bc198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:50.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 0x7fb8bc19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc19d0f0 con 0x7fb8bc06a370 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc19d260 con 0x7fb8bc107cf0 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8c1135700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc19d420 con 0x7fb8bc06ad10 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60576/0 (socket says 172.21.15.45:60576) 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 -- 172.21.15.45:0/2591828061 learned_addr learned my addr 172.21.15.45:0/2591828061 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bb59e700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 0x7fb8bc19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8ba59c700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 0x7fb8bc198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 -- 172.21.15.45:0/2591828061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 msgr2=0x7fb8bc198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 0x7fb8bc198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 -- 172.21.15.45:0/2591828061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 msgr2=0x7fb8bc19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 0x7fb8bc19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.459+0000 7fb8bad9d700 1 -- 172.21.15.45:0/2591828061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40095f0 con 0x7fb8bc06a370 2024-03-12T05:08:50.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8bad9d700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b530 tx=0x7fb8ac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac011830 con 0x7fb8bc06a370 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc19d700 con 0x7fb8bc06a370 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb8ac011e40 con 0x7fb8bc06a370 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00f2e0 con 0x7fb8bc06a370 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.460+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc1a4f60 con 0x7fb8bc06a370 2024-03-12T05:08:50.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.461+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc10a210 con 0x7fb8bc06a370 2024-03-12T05:08:50.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.462+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb8ac011990 con 0x7fb8bc06a370 2024-03-12T05:08:50.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.462+0000 7fb8b3fff700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 0x7fb8a80a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:50.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.462+0000 7fb8ba59c700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 0x7fb8a80a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:50.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.462+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb8ac0cd1b0 con 0x7fb8bc06a370 2024-03-12T05:08:50.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.463+0000 7fb8ba59c700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 0x7fb8a80a4010 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009990 tx=0x7fb8a4005120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.466+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8ac096420 con 0x7fb8bc06a370 2024-03-12T05:08:50.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.729+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb8bc068930 con 0x7fb8bc06a370 2024-03-12T05:08:50.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.733+0000 7fb8b3fff700 1 -- 172.21.15.45:0/2591828061 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260661 (secure 0 0 0) 0x7fb8ac099ae0 con 0x7fb8bc06a370 2024-03-12T05:08:50.735 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:50.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 msgr2=0x7fb8a80a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 0x7fb8a80a4010 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009990 tx=0x7fb8a4005120 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 msgr2=0x7fb8bc198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b530 tx=0x7fb8ac00b840 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 shutdown_connections 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8a80a1b50 0x7fb8a80a4010 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8bc107cf0 0x7fb8bc19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8bc06a370 0x7fb8bc198210 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 --2- 172.21.15.45:0/2591828061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8bc06ad10 0x7fb8bc198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.736+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 >> 172.21.15.45:0/2591828061 conn(0x7fb8bc076b90 msgr2=0x7fb8bc0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.737+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 shutdown_connections 2024-03-12T05:08:50.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:50.737+0000 7fb8c1135700 1 -- 172.21.15.45:0/2591828061 wait complete. 2024-03-12T05:08:51.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:50 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2591828061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:08:51.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:50 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2591828061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:08:51.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:50 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2591828061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:08:51.388 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":1134390880},{"type":"v1","addr":"172.21.15.45:6801","nonce":1134390880}]},"active_addr":"172.21.15.45:6801/1134390880","active_change":"2024-03-12T05:01:43.021131+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":4188215761}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2345425176}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1231700554}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":759469548}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":254760803}]}]} 2024-03-12T05:08:51.394 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T05:08:51.394 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T05:08:51.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:08:51.641 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:52.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:51 smithi045 ceph-mon[32174]: pgmap v289: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-03-12T05:08:52.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:51 smithi130 ceph-mon[35398]: pgmap v289: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-03-12T05:08:52.245 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:51 smithi116 ceph-mon[35486]: pgmap v289: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.2 KiB/s wr, 24 op/s 2024-03-12T05:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 -- 172.21.15.45:0/757015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb406ad80 msgr2=0x7f1bb410eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 --2- 172.21.15.45:0/757015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb406ad80 0x7f1bb410eac0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1bac0099c0 tx=0x7f1bac009cd0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 -- 172.21.15.45:0/757015625 shutdown_connections 2024-03-12T05:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 --2- 172.21.15.45:0/757015625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb406ad80 0x7f1bb410eac0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 --2- 172.21.15.45:0/757015625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb4107ca0 0x7f1bb4108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 --2- 172.21.15.45:0/757015625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406a3c0 0x7f1bb406a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 -- 172.21.15.45:0/757015625 >> 172.21.15.45:0/757015625 conn(0x7f1bb4077b80 msgr2=0x7f1bb4077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:53.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:53 smithi045 ceph-mon[32174]: pgmap v290: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 -- 172.21.15.45:0/757015625 shutdown_connections 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.385+0000 7f1bbc547700 1 -- 172.21.15.45:0/757015625 wait complete. 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 Processor -- start 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 -- start start 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 0x7f1bb4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406ad80 0x7f1bb4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb419cf30 con 0x7f1bb4107ca0 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb419d0a0 con 0x7f1bb406ad80 2024-03-12T05:08:53.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbc547700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb419d260 con 0x7f1bb406a3c0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60590/0 (socket says 172.21.15.45:60590) 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 -- 172.21.15.45:0/1982430006 learned_addr learned my addr 172.21.15.45:0/1982430006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bb9ae2700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406ad80 0x7f1bb4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bba2e3700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 0x7f1bb4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 -- 172.21.15.45:0/1982430006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 msgr2=0x7f1bb4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 0x7f1bb4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 -- 172.21.15.45:0/1982430006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406ad80 msgr2=0x7f1bb4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406ad80 0x7f1bb4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bbaae4700 1 -- 172.21.15.45:0/1982430006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac0095f0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.386+0000 7f1bba2e3700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 0x7f1bb4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bbaae4700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1bac004730 tx=0x7f1bac0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac01d070 con 0x7f1bb4107ca0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb419d4e0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1bac004cc0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac00f8d0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.387+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41a4da0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.388+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb410a180 con 0x7f1bb4107ca0 2024-03-12T05:08:53.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.389+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1bac005150 con 0x7f1bb4107ca0 2024-03-12T05:08:53.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.390+0000 7f1bb37fe700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 0x7f1b9c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:53.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.390+0000 7f1bba2e3700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 0x7f1b9c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:53.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.390+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1bac0ce690 con 0x7f1bb4107ca0 2024-03-12T05:08:53.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.390+0000 7f1bba2e3700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 0x7f1b9c0a4010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009fa0 tx=0x7f1ba4009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:53.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.394+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1bac097900 con 0x7f1bb4107ca0 2024-03-12T05:08:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:53 smithi130 ceph-mon[35398]: pgmap v290: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-12T05:08:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:53 smithi116 ceph-mon[35486]: pgmap v290: 97 pgs: 97 active+clean; 628 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-12T05:08:53.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.597+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb41993b0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.598+0000 7f1bb37fe700 1 -- 172.21.15.45:0/1982430006 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22184 (secure 0 0 0) 0x7f1bac09afc0 con 0x7f1bb4107ca0 2024-03-12T05:08:53.596 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:53.597 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:08:38.754759+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 msgr2=0x7f1b9c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 0x7f1b9c0a4010 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009fa0 tx=0x7f1ba4009570 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 msgr2=0x7f1bb419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1bac004730 tx=0x7f1bac0036f0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 shutdown_connections 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1b9c0a1b50 0x7f1b9c0a4010 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bb4107ca0 0x7f1bb419c920 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bb406a3c0 0x7f1bb4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 --2- 172.21.15.45:0/1982430006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bb406ad80 0x7f1bb4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.600+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 >> 172.21.15.45:0/1982430006 conn(0x7f1bb4077b80 msgr2=0x7f1bb40ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.601+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 shutdown_connections 2024-03-12T05:08:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:53.601+0000 7f1bbc547700 1 -- 172.21.15.45:0/1982430006 wait complete. 2024-03-12T05:08:54.333 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T05:08:54.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:08:54.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:54 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1982430006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:54 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1982430006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:54.582 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:54.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:54 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1982430006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:55.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:55 smithi045 ceph-mon[32174]: pgmap v291: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-03-12T05:08:55.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:55 smithi130 ceph-mon[35398]: pgmap v291: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-03-12T05:08:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:55 smithi116 ceph-mon[35486]: pgmap v291: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 26 op/s 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.253+0000 7f1cac9a8700 1 -- 172.21.15.45:0/2454832479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca8100810 msgr2=0x7f1ca8100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.253+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/2454832479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca8100810 0x7f1ca8100bf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009990 tx=0x7f1c98009ca0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 -- 172.21.15.45:0/2454832479 shutdown_connections 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/2454832479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 0x7f1ca810ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/2454832479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca8100810 0x7f1ca8100bf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/2454832479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca81011c0 0x7f1ca8106900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 -- 172.21.15.45:0/2454832479 >> 172.21.15.45:0/2454832479 conn(0x7f1ca8075ed0 msgr2=0x7f1ca80762e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 -- 172.21.15.45:0/2454832479 shutdown_connections 2024-03-12T05:08:56.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 -- 172.21.15.45:0/2454832479 wait complete. 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.254+0000 7f1cac9a8700 1 Processor -- start 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 -- start start 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 0x7f1ca819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 0x7f1ca819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 0x7f1ca819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca819f220 con 0x7f1ca81011c0 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca819f390 con 0x7f1ca8100810 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1cac9a8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca819f550 con 0x7f1ca8107030 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca5d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 0x7f1ca819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca5d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 0x7f1ca819a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60594/0 (socket says 172.21.15.45:60594) 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca5d9b700 1 -- 172.21.15.45:0/1971322552 learned_addr learned my addr 172.21.15.45:0/1971322552 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:08:56.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 0x7f1ca819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca6d9d700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 0x7f1ca819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 -- 172.21.15.45:0/1971322552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 msgr2=0x7f1ca819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 0x7f1ca819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 -- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 msgr2=0x7f1ca819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 0x7f1ca819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 -- 172.21.15.45:0/1971322552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980095f0 con 0x7f1ca8100810 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.255+0000 7f1ca659c700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 0x7f1ca819a340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c980059d0 tx=0x7f1c98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.256+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9801d070 con 0x7f1ca8100810 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.256+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1c98022470 con 0x7f1ca8100810 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.256+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98017650 con 0x7f1ca8100810 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.256+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca819f7d0 con 0x7f1ca8100810 2024-03-12T05:08:56.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.256+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca819fa70 con 0x7f1ca8100810 2024-03-12T05:08:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.257+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca8068930 con 0x7f1ca8100810 2024-03-12T05:08:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.258+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1c980225e0 con 0x7f1ca8100810 2024-03-12T05:08:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.258+0000 7f1c977fe700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 0x7f1c900a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:08:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.258+0000 7f1ca5d9b700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 0x7f1c900a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:08:56.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.258+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1c980ce590 con 0x7f1ca8100810 2024-03-12T05:08:56.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.259+0000 7f1ca5d9b700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 0x7f1c900a4090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ca81004b0 tx=0x7f1c9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:08:56.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.262+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1c98097780 con 0x7f1ca8100810 2024-03-12T05:08:56.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.466+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca8111c50 con 0x7f1ca8100810 2024-03-12T05:08:56.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.468+0000 7f1c977fe700 1 -- 172.21.15.45:0/1971322552 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22184 (secure 0 0 0) 0x7f1c9809ae40 con 0x7f1ca8100810 2024-03-12T05:08:56.467 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:08:56.467 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:08:38.754759+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:08:56.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 msgr2=0x7f1c900a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:56.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 0x7f1c900a4090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ca81004b0 tx=0x7f1c9c009260 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 msgr2=0x7f1ca819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 0x7f1ca819a340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c980059d0 tx=0x7f1c98004730 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 shutdown_connections 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1c900a1bd0 0x7f1c900a4090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ca8107030 0x7f1ca819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8100810 0x7f1ca819a340 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 --2- 172.21.15.45:0/1971322552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ca81011c0 0x7f1ca819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 >> 172.21.15.45:0/1971322552 conn(0x7f1ca8075ed0 msgr2=0x7f1ca80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 shutdown_connections 2024-03-12T05:08:56.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:08:56.470+0000 7f1cac9a8700 1 -- 172.21.15.45:0/1971322552 wait complete. 2024-03-12T05:08:56.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:56 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1971322552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:56.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:56 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1971322552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:56.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:56 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1971322552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:08:57.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T05:08:57.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T05:08:57.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T05:08:57.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T05:08:57.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T05:08:57.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T05:08:57.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T05:08:57.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T05:08:57.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-12T05:08:57.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-12T05:08:57.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-12T05:08:57.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-12T05:08:57.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:57 smithi045 ceph-mon[32174]: pgmap v292: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-12T05:08:57.874 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:57.935 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:57.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:57 smithi130 ceph-mon[35398]: pgmap v292: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-12T05:08:57.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:57 smithi116 ceph-mon[35486]: pgmap v292: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-12T05:08:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.308 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.310 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.311 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.317 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:08:59.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:08:59 smithi045 ceph-mon[32174]: pgmap v293: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-12T05:08:59.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:08:59 smithi130 ceph-mon[35398]: pgmap v293: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-12T05:08:59.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:08:59 smithi116 ceph-mon[35486]: pgmap v293: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-12T05:09:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 -- 172.21.15.45:0/3240251894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304100ba0 msgr2=0x7f8304101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 --2- 172.21.15.45:0/3240251894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304100ba0 0x7f8304101020 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009990 tx=0x7f82f8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 -- 172.21.15.45:0/3240251894 shutdown_connections 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 --2- 172.21.15.45:0/3240251894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8304101560 0x7f830410d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 --2- 172.21.15.45:0/3240251894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8304106bd0 0x7f8304106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 --2- 172.21.15.45:0/3240251894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304100ba0 0x7f8304101020 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 -- 172.21.15.45:0/3240251894 >> 172.21.15.45:0/3240251894 conn(0x7f8304076bc0 msgr2=0x7f8304076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 -- 172.21.15.45:0/3240251894 shutdown_connections 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.621+0000 7f830bf35700 1 -- 172.21.15.45:0/3240251894 wait complete. 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 Processor -- start 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 -- start start 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8304100ba0 0x7f83041980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 0x7f83041985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:00.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8304106bd0 0x7f830419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419cf30 con 0x7f8304101560 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419d0a0 con 0x7f8304106bd0 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830bf35700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419d260 con 0x7f8304100ba0 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 0x7f83041985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 0x7f83041985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60988/0 (socket says 172.21.15.45:60988) 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 -- 172.21.15.45:0/101528964 learned_addr learned my addr 172.21.15.45:0/101528964 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f8309cd1700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8304100ba0 0x7f83041980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f830a4d2700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8304106bd0 0x7f830419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 -- 172.21.15.45:0/101528964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8304100ba0 msgr2=0x7f83041980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8304100ba0 0x7f83041980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 -- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8304106bd0 msgr2=0x7f830419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8304106bd0 0x7f830419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80095f0 con 0x7f8304101560 2024-03-12T05:09:00.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.622+0000 7f83094d0700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 0x7f83041985e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82f8000c00 tx=0x7f82f80047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f801d070 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f82f800fce0 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830419d4e0 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8022e80 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304102890 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.623+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f830419d7b0 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.625+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f82f80225e0 con 0x7f8304101560 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.625+0000 7f8302ffd700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82ec0a1b50 0x7f82ec0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.625+0000 7f8309cd1700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82ec0a1b50 0x7f82ec0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:00.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.625+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f82f80cedf0 con 0x7f8304101560 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f8302ffd700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] conn(0x7f82ec0ab330 0x7f82ec0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82ec0ade00 con 0x7f82ec0ab330 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82f80cf290 con 0x7f8304101560 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f8309cd1700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82ec0a1b50 0x7f82ec0a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82f4007fd0 tx=0x7f82f400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f830a4d2700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] conn(0x7f82ec0ab330 0x7f82ec0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:00.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.626+0000 7f830a4d2700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] conn(0x7f82ec0ab330 0x7f82ec0ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:00.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.628+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== osd.10 v2:172.21.15.116:6824/2890731133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f82ec0ade00 con 0x7f82ec0ab330 2024-03-12T05:09:00.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.645+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 --> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8304110d70 con 0x7f82ec0ab330 2024-03-12T05:09:00.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f8302ffd700 1 -- 172.21.15.45:0/101528964 <== osd.10 v2:172.21.15.116:6824/2890731133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8304110d70 con 0x7f82ec0ab330 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] conn(0x7f82ec0ab330 msgr2=0x7f82ec0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] conn(0x7f82ec0ab330 0x7f82ec0ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82ec0a1b50 msgr2=0x7f82ec0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82ec0a1b50 0x7f82ec0a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82f4007fd0 tx=0x7f82f400c040 comp rx=0 tx=0).stop 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 msgr2=0x7f83041985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 --2- 172.21.15.45:0/101528964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8304101560 0x7f83041985e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82f8000c00 tx=0x7f82f80047a0 comp rx=0 tx=0).stop 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830a4d2700 1 -- 172.21.15.45:0/101528964 reap_dead start 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 shutdown_connections 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 >> 172.21.15.45:0/101528964 conn(0x7f8304076bc0 msgr2=0x7f83040fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 shutdown_connections 2024-03-12T05:09:00.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:00.646+0000 7f830bf35700 1 -- 172.21.15.45:0/101528964 wait complete. 2024-03-12T05:09:01.459 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:01 smithi045 ceph-mon[32174]: pgmap v294: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-12T05:09:01.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:01 smithi130 ceph-mon[35398]: pgmap v294: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-12T05:09:01.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:01 smithi116 ceph-mon[35486]: pgmap v294: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-12T05:09:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 -- 172.21.15.45:0/2673149487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0107ca0 msgr2=0x7fdde0108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 --2- 172.21.15.45:0/2673149487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0107ca0 0x7fdde0108080 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdddc0099c0 tx=0x7fdddc009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 -- 172.21.15.45:0/2673149487 shutdown_connections 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 --2- 172.21.15.45:0/2673149487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdde0102680 0x7fdde010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 --2- 172.21.15.45:0/2673149487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0107ca0 0x7fdde0108080 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 --2- 172.21.15.45:0/2673149487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0101cc0 0x7fdde0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 -- 172.21.15.45:0/2673149487 >> 172.21.15.45:0/2673149487 conn(0x7fdde0078da0 msgr2=0x7fdde00791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 -- 172.21.15.45:0/2673149487 shutdown_connections 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.399+0000 7fdde79a6700 1 -- 172.21.15.45:0/2673149487 wait complete. 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 Processor -- start 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 -- start start 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 0x7fdde0073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0102680 0x7fdde00744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdde0107ca0 0x7fdde00749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01a6530 con 0x7fdde0101cc0 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01a66a0 con 0x7fdde0107ca0 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde79a6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde01a6860 con 0x7fdde0102680 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde5742700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 0x7fdde0073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde5742700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 0x7fdde0073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32768/0 (socket says 172.21.15.45:32768) 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde5742700 1 -- 172.21.15.45:0/238124872 learned_addr learned my addr 172.21.15.45:0/238124872 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.400+0000 7fdde4f41700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0102680 0x7fdde00744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 -- 172.21.15.45:0/238124872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0102680 msgr2=0x7fdde00744b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0102680 0x7fdde00744b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 -- 172.21.15.45:0/238124872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdde0107ca0 msgr2=0x7fdde00749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdde0107ca0 0x7fdde00749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0095f0 con 0x7fdde0101cc0 2024-03-12T05:09:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde4f41700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdde0102680 0x7fdde00744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde5742700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 0x7fdde0073f70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00b580 tx=0x7fdddc009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc01d070 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdddc00fca0 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde01a6ae0 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc022e00 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.401+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde01a6ef0 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.402+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdde0111eb0 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.403+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdddc022600 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fddd67fc700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fddcc0a1b50 0x7fddcc0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fdde4f41700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fddcc0a1b50 0x7fddcc0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdddc0ce980 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fddd67fc700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] conn(0x7fddcc0ab330 0x7fddcc0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddcc0ade00 con 0x7fddcc0ab330 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdddc0cecf0 con 0x7fdde0101cc0 2024-03-12T05:09:02.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fdde5f43700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] conn(0x7fddcc0ab330 0x7fddcc0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fdde4f41700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fddcc0a1b50 0x7fddcc0a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fddd0005950 tx=0x7fddd00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.404+0000 7fdde5f43700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] conn(0x7fddcc0ab330 0x7fddcc0ad750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:02.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.405+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== osd.0 v2:172.21.15.45:6802/55762477 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddcc0ade00 con 0x7fddcc0ab330 2024-03-12T05:09:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.423+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 --> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdde0062380 con 0x7fddcc0ab330 2024-03-12T05:09:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fddd67fc700 1 -- 172.21.15.45:0/238124872 <== osd.0 v2:172.21.15.45:6802/55762477 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdde0062380 con 0x7fddcc0ab330 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] conn(0x7fddcc0ab330 msgr2=0x7fddcc0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] conn(0x7fddcc0ab330 0x7fddcc0ad750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fddcc0a1b50 msgr2=0x7fddcc0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fddcc0a1b50 0x7fddcc0a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fddd0005950 tx=0x7fddd00058e0 comp rx=0 tx=0).stop 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 msgr2=0x7fdde0073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 --2- 172.21.15.45:0/238124872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdde0101cc0 0x7fdde0073f70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00b580 tx=0x7fdddc009510 comp rx=0 tx=0).stop 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde5f43700 1 -- 172.21.15.45:0/238124872 reap_dead start 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 shutdown_connections 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 >> 172.21.15.45:0/238124872 conn(0x7fdde0078da0 msgr2=0x7fdde0100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 shutdown_connections 2024-03-12T05:09:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:02.424+0000 7fdde79a6700 1 -- 172.21.15.45:0/238124872 wait complete. 2024-03-12T05:09:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:309237645334 2024-03-12T05:09:03.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-12T05:09:03.372 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:03 smithi045 ceph-mon[32174]: pgmap v295: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-12T05:09:03.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:03 smithi130 ceph-mon[35398]: pgmap v295: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-12T05:09:03.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:03 smithi116 ceph-mon[35486]: pgmap v295: 97 pgs: 97 active+clean; 639 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-12T05:09:03.884 INFO:teuthology.orchestra.run.smithi045.stdout:34359738440 2024-03-12T05:09:03.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T05:09:05.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:05 smithi045 ceph-mon[32174]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-12T05:09:05.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:05 smithi130 ceph-mon[35398]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-12T05:09:05.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:05 smithi116 ceph-mon[35486]: pgmap v296: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-12T05:09:07.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:07 smithi045 ceph-mon[32174]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:07.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:07 smithi130 ceph-mon[35398]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:07.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:07 smithi116 ceph-mon[35486]: pgmap v297: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:09.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:09 smithi045 ceph-mon[32174]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:09.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:09 smithi130 ceph-mon[35398]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:09.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:09 smithi116 ceph-mon[35486]: pgmap v298: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 -- 172.21.15.45:0/152899014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6810c140 msgr2=0x7f2d6810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 --2- 172.21.15.45:0/152899014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6810c140 0x7f2d6810e600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009990 tx=0x7f2d60009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 -- 172.21.15.45:0/152899014 shutdown_connections 2024-03-12T05:09:11.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 --2- 172.21.15.45:0/152899014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d6810c140 0x7f2d6810e600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 --2- 172.21.15.45:0/152899014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d68102d10 0x7f2d681030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 --2- 172.21.15.45:0/152899014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d68103630 0x7f2d6810bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 -- 172.21.15.45:0/152899014 >> 172.21.15.45:0/152899014 conn(0x7f2d680fc540 msgr2=0x7f2d680fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:11.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 -- 172.21.15.45:0/152899014 shutdown_connections 2024-03-12T05:09:11.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.140+0000 7f2d6759e700 1 -- 172.21.15.45:0/152899014 wait complete. 2024-03-12T05:09:11.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 Processor -- start 2024-03-12T05:09:11.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 -- start start 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 0x7f2d68197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d68103630 0x7f2d68198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6810c140 0x7f2d6819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6819ce00 con 0x7f2d68102d10 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6819cf70 con 0x7f2d6810c140 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6759e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6819d130 con 0x7f2d68103630 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 0x7f2d68197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 0x7f2d68197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44618/0 (socket says 172.21.15.45:44618) 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d6659c700 1 -- 172.21.15.45:0/785657965 learned_addr learned my addr 172.21.15.45:0/785657965 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d65d9b700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d68103630 0x7f2d68198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.141+0000 7f2d66d9d700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6810c140 0x7f2d6819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 -- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d68103630 msgr2=0x7f2d68198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d68103630 0x7f2d68198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 -- 172.21.15.45:0/785657965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6810c140 msgr2=0x7f2d6819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d6810c140 0x7f2d6819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600095f0 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6659c700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 0x7f2d68197f20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ada0 tx=0x7f2d5800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800f070 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2d5800ae90 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6819d410 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d580052c0 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.142+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d681a4c70 con 0x7f2d68102d10 2024-03-12T05:09:11.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.143+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d6806aa90 con 0x7f2d68102d10 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.144+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2d5800c8d0 con 0x7f2d68102d10 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d5f7fe700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2d540a1bd0 0x7f2d540a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2d580cd7d0 con 0x7f2d68102d10 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d65d9b700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2d540a1bd0 0x7f2d540a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d5f7fe700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] conn(0x7f2d540ab3b0 0x7f2d540ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d540ade80 con 0x7f2d540ab3b0 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.145+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d580cdb80 con 0x7f2d68102d10 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.146+0000 7f2d65d9b700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2d540a1bd0 0x7f2d540a4090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2d6810d480 tx=0x7f2d5000b360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.146+0000 7f2d66d9d700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] conn(0x7f2d540ab3b0 0x7f2d540ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.147+0000 7f2d66d9d700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] conn(0x7f2d540ab3b0 0x7f2d540ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.148+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== osd.2 v2:172.21.15.130:6800/845680146 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2d540ade80 con 0x7f2d540ab3b0 2024-03-12T05:09:11.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.176+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 --> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2d68111dc0 con 0x7f2d540ab3b0 2024-03-12T05:09:11.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d5f7fe700 1 -- 172.21.15.45:0/785657965 <== osd.2 v2:172.21.15.130:6800/845680146 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2d68111dc0 con 0x7f2d540ab3b0 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] conn(0x7f2d540ab3b0 msgr2=0x7f2d540ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] conn(0x7f2d540ab3b0 0x7f2d540ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2d540a1bd0 msgr2=0x7f2d540a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2d540a1bd0 0x7f2d540a4090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2d6810d480 tx=0x7f2d5000b360 comp rx=0 tx=0).stop 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 msgr2=0x7f2d68197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 --2- 172.21.15.45:0/785657965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d68102d10 0x7f2d68197f20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800ada0 tx=0x7f2d5800c330 comp rx=0 tx=0).stop 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d66d9d700 1 -- 172.21.15.45:0/785657965 reap_dead start 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 shutdown_connections 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 >> 172.21.15.45:0/785657965 conn(0x7f2d680fc540 msgr2=0x7f2d680fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 shutdown_connections 2024-03-12T05:09:11.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.177+0000 7f2d6759e700 1 -- 172.21.15.45:0/785657965 wait complete. 2024-03-12T05:09:11.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:11 smithi130 ceph-mon[35398]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:11.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:11 smithi116 ceph-mon[35486]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:11.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:11 smithi045 ceph-mon[32174]: pgmap v299: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-12T05:09:11.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 -- 172.21.15.45:0/1880311999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f297006a2c0 msgr2=0x7f297010eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 --2- 172.21.15.45:0/1880311999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f297006a2c0 0x7f297010eb00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29600099c0 tx=0x7f2960009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 -- 172.21.15.45:0/1880311999 shutdown_connections 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 --2- 172.21.15.45:0/1880311999 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f297006a2c0 0x7f297010eb00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 --2- 172.21.15.45:0/1880311999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 0x7f29701080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 --2- 172.21.15.45:0/1880311999 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2970069920 0x7f2970069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 -- 172.21.15.45:0/1880311999 >> 172.21.15.45:0/1880311999 conn(0x7f2970077aa0 msgr2=0x7f2970077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 -- 172.21.15.45:0/1880311999 shutdown_connections 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.723+0000 7f297492d700 1 -- 172.21.15.45:0/1880311999 wait complete. 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 Processor -- start 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 -- start start 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 0x7f29701981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f297006a2c0 0x7f2970198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 0x7f297019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297019d060 con 0x7f2970069920 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297019d1d0 con 0x7f2970107cc0 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f297492d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297019d390 con 0x7f297006a2c0 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 0x7f29701981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 0x7f29701981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44634/0 (socket says 172.21.15.45:44634) 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 -- 172.21.15.45:0/2691027839 learned_addr learned my addr 172.21.15.45:0/2691027839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296dd9b700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f297006a2c0 0x7f2970198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296ed9d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 0x7f297019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 -- 172.21.15.45:0/2691027839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f297006a2c0 msgr2=0x7f2970198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f297006a2c0 0x7f2970198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 -- 172.21.15.45:0/2691027839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 msgr2=0x7f297019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 0x7f297019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29600095f0 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296ed9d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2970107cc0 0x7f297019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.724+0000 7f296e59c700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 0x7f29701981d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f29680060b0 tx=0x7f2968007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296800de20 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297019d670 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f296800f040 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968014720 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29701a4ed0 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.725+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2970111e90 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2968015400 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f295c0a1b50 0x7f295c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f296dd9b700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f295c0a1b50 0x7f295c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f29680cd630 con 0x7f2970069920 2024-03-12T05:09:11.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] conn(0x7f295c0ab330 0x7f295c0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f295c0ade00 con 0x7f295c0ab330 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.727+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29680cdad0 con 0x7f2970069920 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.728+0000 7f296ed9d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] conn(0x7f295c0ab330 0x7f295c0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.728+0000 7f296dd9b700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f295c0a1b50 0x7f295c0a4010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2958006fd0 tx=0x7f2958008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.728+0000 7f296ed9d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] conn(0x7f295c0ab330 0x7f295c0ad750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:11.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.729+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== osd.9 v2:172.21.15.45:6826/15901149 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f295c0ade00 con 0x7f295c0ab330 2024-03-12T05:09:11.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.745+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 --> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2970062300 con 0x7f295c0ab330 2024-03-12T05:09:11.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.745+0000 7f29677fe700 1 -- 172.21.15.45:0/2691027839 <== osd.9 v2:172.21.15.45:6826/15901149 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2970062300 con 0x7f295c0ab330 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] conn(0x7f295c0ab330 msgr2=0x7f295c0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] conn(0x7f295c0ab330 0x7f295c0ad750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f295c0a1b50 msgr2=0x7f295c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f295c0a1b50 0x7f295c0a4010 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2958006fd0 tx=0x7f2958008040 comp rx=0 tx=0).stop 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 msgr2=0x7f29701981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 --2- 172.21.15.45:0/2691027839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2970069920 0x7f29701981d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f29680060b0 tx=0x7f2968007f90 comp rx=0 tx=0).stop 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f296ed9d700 1 -- 172.21.15.45:0/2691027839 reap_dead start 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 shutdown_connections 2024-03-12T05:09:11.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 >> 172.21.15.45:0/2691027839 conn(0x7f2970077aa0 msgr2=0x7f29700ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 shutdown_connections 2024-03-12T05:09:11.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:11.746+0000 7f297492d700 1 -- 172.21.15.45:0/2691027839 wait complete. 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.155+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/2308088877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 msgr2=0x7f5b88104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.155+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/2308088877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b88104fa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0099c0 tx=0x7f5b7c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/2308088877 shutdown_connections 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/2308088877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b881056d0 0x7f5b8810d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/2308088877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88100800 0x7f5b88100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/2308088877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b88104fa0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/2308088877 >> 172.21.15.45:0/2308088877 conn(0x7f5b88075ea0 msgr2=0x7f5b880762b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/2308088877 shutdown_connections 2024-03-12T05:09:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/2308088877 wait complete. 2024-03-12T05:09:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 Processor -- start 2024-03-12T05:09:13.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.156+0000 7f5b8e8e1700 1 -- start start 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88100800 0x7f5b88198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b881994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b881056d0 0x7f5b88199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a4210 con 0x7f5b881011b0 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a4380 con 0x7f5b881056d0 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8e8e1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a4540 con 0x7f5b88100800 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b881994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b881994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44648/0 (socket says 172.21.15.45:44648) 2024-03-12T05:09:13.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b87fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88100800 0x7f5b88198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 -- 172.21.15.45:0/801813854 learned_addr learned my addr 172.21.15.45:0/801813854 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b8ce7e700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b881056d0 0x7f5b88199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 -- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88100800 msgr2=0x7f5b88198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b88100800 0x7f5b88198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 -- 172.21.15.45:0/801813854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b881056d0 msgr2=0x7f5b88199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b881056d0 0x7f5b88199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0095f0 con 0x7f5b881011b0 2024-03-12T05:09:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.157+0000 7f5b877fe700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b881994e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c01d070 con 0x7f5b881011b0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5b7c00fd90 con 0x7f5b881011b0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0173f0 con 0x7f5b881011b0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881a47c0 con 0x7f5b881011b0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881a4c30 con 0x7f5b881011b0 2024-03-12T05:09:13.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.160+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b88101a10 con 0x7f5b881011b0 2024-03-12T05:09:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.162+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5b7c022670 con 0x7f5b881011b0 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.162+0000 7f5b857fa700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b700a1bd0 0x7f5b700a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.162+0000 7f5b87fff700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b700a1bd0 0x7f5b700a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.162+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5b7c0cf500 con 0x7f5b881011b0 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.162+0000 7f5b857fa700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] conn(0x7f5b700ab3b0 0x7f5b700ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.163+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b700ade80 con 0x7f5b700ab3b0 2024-03-12T05:09:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.163+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b7c0cf9a0 con 0x7f5b881011b0 2024-03-12T05:09:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.163+0000 7f5b87fff700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b700a1bd0 0x7f5b700a4090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b881116b0 tx=0x7f5b78009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.163+0000 7f5b8ce7e700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] conn(0x7f5b700ab3b0 0x7f5b700ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.163+0000 7f5b8ce7e700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] conn(0x7f5b700ab3b0 0x7f5b700ad7d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.164+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== osd.11 v2:172.21.15.130:6824/627874902 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5b700ade80 con 0x7f5b700ab3b0 2024-03-12T05:09:13.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.181+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 --> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b881105a0 con 0x7f5b700ab3b0 2024-03-12T05:09:13.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.181+0000 7f5b857fa700 1 -- 172.21.15.45:0/801813854 <== osd.11 v2:172.21.15.130:6824/627874902 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5b881105a0 con 0x7f5b700ab3b0 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.181+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] conn(0x7f5b700ab3b0 msgr2=0x7f5b700ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.181+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] conn(0x7f5b700ab3b0 0x7f5b700ad7d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b700a1bd0 msgr2=0x7f5b700a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b700a1bd0 0x7f5b700a4090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b881116b0 tx=0x7f5b78009190 comp rx=0 tx=0).stop 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 msgr2=0x7f5b881994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 --2- 172.21.15.45:0/801813854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b881011b0 0x7f5b881994e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c004750 comp rx=0 tx=0).stop 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8ce7e700 1 -- 172.21.15.45:0/801813854 reap_dead start 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 shutdown_connections 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 >> 172.21.15.45:0/801813854 conn(0x7f5b88075ea0 msgr2=0x7f5b880fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 shutdown_connections 2024-03-12T05:09:13.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:13.182+0000 7f5b8e8e1700 1 -- 172.21.15.45:0/801813854 wait complete. 2024-03-12T05:09:13.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:13 smithi130 ceph-mon[35398]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:13.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:13 smithi116 ceph-mon[35486]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:13.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:13 smithi045 ceph-mon[32174]: pgmap v300: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-12T05:09:15.374 INFO:teuthology.orchestra.run.smithi045.stdout:287762808862 2024-03-12T05:09:15.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-12T05:09:15.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:15 smithi130 ceph-mon[35398]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-12T05:09:15.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:15 smithi116 ceph-mon[35486]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-12T05:09:15.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:15 smithi045 ceph-mon[32174]: pgmap v301: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-12T05:09:16.100 INFO:teuthology.orchestra.run.smithi045.stdout:343597383699 2024-03-12T05:09:16.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-12T05:09:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- 172.21.15.45:0/2631336024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410107000 msgr2=0x7f741010eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 --2- 172.21.15.45:0/2631336024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410107000 0x7f741010eb80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7408009990 tx=0x7f7408009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- 172.21.15.45:0/2631336024 shutdown_connections 2024-03-12T05:09:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 --2- 172.21.15.45:0/2631336024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410107000 0x7f741010eb80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 --2- 172.21.15.45:0/2631336024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410101f50 0x7f7410102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 --2- 172.21.15.45:0/2631336024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7410102900 0x7f74101068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- 172.21.15.45:0/2631336024 >> 172.21.15.45:0/2631336024 conn(0x7f7410078d90 msgr2=0x7f74100791a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- 172.21.15.45:0/2631336024 shutdown_connections 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- 172.21.15.45:0/2631336024 wait complete. 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 Processor -- start 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.951+0000 7f7417632700 1 -- start start 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7410101f50 0x7f7410198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 0x7f74101985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410107000 0x7f741019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741019cf60 con 0x7f7410102900 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741019d0d0 con 0x7f7410107000 2024-03-12T05:09:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741019d290 con 0x7f7410101f50 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 0x7f74101985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 0x7f74101985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44680/0 (socket says 172.21.15.45:44680) 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 -- 172.21.15.45:0/4045510992 learned_addr learned my addr 172.21.15.45:0/4045510992 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7415bcf700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410107000 0x7f741019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f74153ce700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7410101f50 0x7f7410198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 -- 172.21.15.45:0/4045510992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7410101f50 msgr2=0x7f7410198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7410101f50 0x7f7410198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 -- 172.21.15.45:0/4045510992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410107000 msgr2=0x7f741019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7410107000 0x7f741019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080095f0 con 0x7f7410102900 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7414bcd700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 0x7f74101985c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f740400b4d0 tx=0x7f740400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:16.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404011830 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741019d570 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.952+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7404011e40 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.953+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404010dd0 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.953+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7410074070 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.953+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74101033b0 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.954+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7404011990 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f740e7fc700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f73f80a1b50 0x7f73f80a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f74153ce700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f73f80a1b50 0x7f73f80a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f74040cd280 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f740e7fc700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] conn(0x7f73f80ab330 0x7f73f80ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f80ade00 con 0x7f73f80ab330 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74040cd720 con 0x7f7410102900 2024-03-12T05:09:16.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.955+0000 7f7415bcf700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] conn(0x7f73f80ab330 0x7f73f80ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.956+0000 7f74153ce700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f73f80a1b50 0x7f73f80a4010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7400009550 tx=0x7f74000091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.956+0000 7f7415bcf700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] conn(0x7f73f80ab330 0x7f73f80ad750 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.957+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== osd.6 v2:172.21.15.45:6818/3221804389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f73f80ade00 con 0x7f73f80ab330 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 --> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7410111f20 con 0x7f73f80ab330 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f740e7fc700 1 -- 172.21.15.45:0/4045510992 <== osd.6 v2:172.21.15.45:6818/3221804389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7410111f20 con 0x7f73f80ab330 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] conn(0x7f73f80ab330 msgr2=0x7f73f80ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] conn(0x7f73f80ab330 0x7f73f80ad750 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f73f80a1b50 msgr2=0x7f73f80a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f73f80a1b50 0x7f73f80a4010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7400009550 tx=0x7f74000091b0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 msgr2=0x7f74101985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 --2- 172.21.15.45:0/4045510992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7410102900 0x7f74101985c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f740400b4d0 tx=0x7f740400b7e0 comp rx=0 tx=0).stop 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7415bcf700 1 -- 172.21.15.45:0/4045510992 reap_dead start 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 shutdown_connections 2024-03-12T05:09:16.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.973+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 >> 172.21.15.45:0/4045510992 conn(0x7f7410078d90 msgr2=0x7f74100fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.974+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 shutdown_connections 2024-03-12T05:09:16.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:16.974+0000 7f7417632700 1 -- 172.21.15.45:0/4045510992 wait complete. 2024-03-12T05:09:17.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:17 smithi045 ceph-mon[32174]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-12T05:09:17.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:17 smithi130 ceph-mon[35398]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-12T05:09:17.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:17 smithi116 ceph-mon[35486]: pgmap v302: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 -- 172.21.15.45:0/2921435702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 msgr2=0x7f058410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 --2- 172.21.15.45:0/2921435702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f058410d9a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f057c00b0a0 tx=0x7f057c00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 -- 172.21.15.45:0/2921435702 shutdown_connections 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 --2- 172.21.15.45:0/2921435702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f058410d9a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 --2- 172.21.15.45:0/2921435702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584106bd0 0x7f0584106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 --2- 172.21.15.45:0/2921435702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0584100ba0 0x7f0584101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 -- 172.21.15.45:0/2921435702 >> 172.21.15.45:0/2921435702 conn(0x7f0584076bc0 msgr2=0x7f0584076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 -- 172.21.15.45:0/2921435702 shutdown_connections 2024-03-12T05:09:17.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.515+0000 7f0589f37700 1 -- 172.21.15.45:0/2921435702 wait complete. 2024-03-12T05:09:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 Processor -- start 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 -- start start 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584100ba0 0x7f0584196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f0584197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0584106bd0 0x7f0584197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841a6430 con 0x7f0584101560 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841a65a0 con 0x7f0584106bd0 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0589f37700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841a6760 con 0x7f0584100ba0 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0582ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f0584197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0582ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f0584197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48290/0 (socket says 172.21.15.45:48290) 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.516+0000 7f0582ffd700 1 -- 172.21.15.45:0/2945019650 learned_addr learned my addr 172.21.15.45:0/2945019650 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0583fff700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0584106bd0 0x7f0584197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f05837fe700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584100ba0 0x7f0584196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 -- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584100ba0 msgr2=0x7f0584196cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584100ba0 0x7f0584196cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 -- 172.21.15.45:0/2945019650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0584106bd0 msgr2=0x7f0584197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0584106bd0 0x7f0584197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c009d00 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f05837fe700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0584100ba0 0x7f0584196cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0582ffd700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f0584197230 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c9b0 tx=0x7f056c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c007b00 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f056c007c60 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05841a6a40 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c018660 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.517+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05841a6f40 con 0x7f0584101560 2024-03-12T05:09:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.518+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0584110d70 con 0x7f0584101560 2024-03-12T05:09:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.519+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f056c00f450 con 0x7f0584101560 2024-03-12T05:09:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f0580ff9700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05700a1b50 0x7f05700a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f05837fe700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05700a1b50 0x7f05700a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:17.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f056c0cd910 con 0x7f0584101560 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f0580ff9700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] conn(0x7f05700ab330 0x7f05700ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05700ade00 con 0x7f05700ab330 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f056c0cdc80 con 0x7f0584101560 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.520+0000 7f05837fe700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05700a1b50 0x7f05700a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0574005fd0 tx=0x7f057400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.521+0000 7f0583fff700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] conn(0x7f05700ab330 0x7f05700ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:17.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.521+0000 7f0583fff700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] conn(0x7f05700ab330 0x7f05700ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:17.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.522+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== osd.5 v2:172.21.15.130:6808/2846635408 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f05700ade00 con 0x7f05700ab330 2024-03-12T05:09:17.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.538+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 --> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0584062380 con 0x7f05700ab330 2024-03-12T05:09:17.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0580ff9700 1 -- 172.21.15.45:0/2945019650 <== osd.5 v2:172.21.15.130:6808/2846635408 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0584062380 con 0x7f05700ab330 2024-03-12T05:09:17.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] conn(0x7f05700ab330 msgr2=0x7f05700ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] conn(0x7f05700ab330 0x7f05700ad750 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05700a1b50 msgr2=0x7f05700a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05700a1b50 0x7f05700a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0574005fd0 tx=0x7f057400a5e0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 msgr2=0x7f0584197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 --2- 172.21.15.45:0/2945019650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0584101560 0x7f0584197230 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c9b0 tx=0x7f056c00ccc0 comp rx=0 tx=0).stop 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0583fff700 1 -- 172.21.15.45:0/2945019650 reap_dead start 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 shutdown_connections 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 >> 172.21.15.45:0/2945019650 conn(0x7f0584076bc0 msgr2=0x7f05840ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 shutdown_connections 2024-03-12T05:09:17.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:17.539+0000 7f0589f37700 1 -- 172.21.15.45:0/2945019650 wait complete. 2024-03-12T05:09:19.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 -- 172.21.15.45:0/2687698125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 msgr2=0x7f1ea0102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/2687698125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0102330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009990 tx=0x7f1e9c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 -- 172.21.15.45:0/2687698125 shutdown_connections 2024-03-12T05:09:19.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/2687698125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ea0106f80 0x7f1ea010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/2687698125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0102330 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/2687698125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea0102900 0x7f1ea01068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 -- 172.21.15.45:0/2687698125 >> 172.21.15.45:0/2687698125 conn(0x7f1ea0078d00 msgr2=0x7f1ea0079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 -- 172.21.15.45:0/2687698125 shutdown_connections 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.254+0000 7f1ea7b76700 1 -- 172.21.15.45:0/2687698125 wait complete. 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 Processor -- start 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 -- start start 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ea0102900 0x7f1ea00744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea0106f80 0x7f1ea00749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea01a6490 con 0x7f1ea0101f50 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea01a6600 con 0x7f1ea0106f80 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea7b76700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea01a67c0 con 0x7f1ea0102900 2024-03-12T05:09:19.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48312/0 (socket says 172.21.15.45:48312) 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 -- 172.21.15.45:0/134450746 learned_addr learned my addr 172.21.15.45:0/134450746 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea6113700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea0106f80 0x7f1ea00749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5111700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ea0102900 0x7f1ea00744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 -- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ea0102900 msgr2=0x7f1ea00744b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ea0102900 0x7f1ea00744b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 -- 172.21.15.45:0/134450746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea0106f80 msgr2=0x7f1ea00749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ea0106f80 0x7f1ea00749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.255+0000 7f1ea5912700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c0095f0 con 0x7f1ea0101f50 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1ea5912700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0073f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c000c00 tx=0x7f1e9c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c01d070 con 0x7f1ea0101f50 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea01a6a40 con 0x7f1ea0101f50 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e9c00bdc0 con 0x7f1ea0101f50 2024-03-12T05:09:19.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c022ca0 con 0x7f1ea0101f50 2024-03-12T05:09:19.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.256+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea01a6e50 con 0x7f1ea0101f50 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.257+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ea0111e00 con 0x7f1ea0101f50 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.258+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1e9c022470 con 0x7f1ea0101f50 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.258+0000 7f1e96ffd700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e8c0a1b50 0x7f1e8c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.258+0000 7f1ea5111700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e8c0a1b50 0x7f1e8c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1e9c0ceee0 con 0x7f1ea0101f50 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1e96ffd700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] conn(0x7f1e8c0ab330 0x7f1e8c0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e8c0ade00 con 0x7f1e8c0ab330 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e9c0cf250 con 0x7f1ea0101f50 2024-03-12T05:09:19.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1ea5111700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e8c0a1b50 0x7f1e8c0a4010 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0101820 tx=0x7f1e9000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:19.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1ea6113700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] conn(0x7f1e8c0ab330 0x7f1e8c0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:19.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.259+0000 7f1ea6113700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] conn(0x7f1e8c0ab330 0x7f1e8c0ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:19.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.260+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== osd.8 v2:172.21.15.130:6816/1919686935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1e8c0ade00 con 0x7f1e8c0ab330 2024-03-12T05:09:19.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.277+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 --> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ea0062380 con 0x7f1e8c0ab330 2024-03-12T05:09:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.278+0000 7f1e96ffd700 1 -- 172.21.15.45:0/134450746 <== osd.8 v2:172.21.15.130:6816/1919686935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1ea0062380 con 0x7f1e8c0ab330 2024-03-12T05:09:19.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.278+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] conn(0x7f1e8c0ab330 msgr2=0x7f1e8c0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.278+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] conn(0x7f1e8c0ab330 0x7f1e8c0ad750 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.278+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e8c0a1b50 msgr2=0x7f1e8c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.278+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e8c0a1b50 0x7f1e8c0a4010 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0101820 tx=0x7f1e9000b410 comp rx=0 tx=0).stop 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 msgr2=0x7f1ea0073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 --2- 172.21.15.45:0/134450746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ea0101f50 0x7f1ea0073f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c000c00 tx=0x7f1e9c00f690 comp rx=0 tx=0).stop 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea6113700 1 -- 172.21.15.45:0/134450746 reap_dead start 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 shutdown_connections 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 >> 172.21.15.45:0/134450746 conn(0x7f1ea0078d00 msgr2=0x7f1ea0108100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 shutdown_connections 2024-03-12T05:09:19.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:19.279+0000 7f1ea7b76700 1 -- 172.21.15.45:0/134450746 wait complete. 2024-03-12T05:09:19.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:19 smithi130 ceph-mon[35398]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-12T05:09:19.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:19 smithi116 ceph-mon[35486]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-12T05:09:19.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:19 smithi045 ceph-mon[32174]: pgmap v303: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-12T05:09:19.840 INFO:teuthology.orchestra.run.smithi045.stdout:197568495662 2024-03-12T05:09:19.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T05:09:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 -- 172.21.15.45:0/33668581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 msgr2=0x7f348c115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 --2- 172.21.15.45:0/33668581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c115230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f348400b0a0 tx=0x7f348400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 -- 172.21.15.45:0/33668581 shutdown_connections 2024-03-12T05:09:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 --2- 172.21.15.45:0/33668581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c115230 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 --2- 172.21.15.45:0/33668581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f348c073ea0 0x7f348c074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 --2- 172.21.15.45:0/33668581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c0747c0 0x7f348c112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.229+0000 7f34941f6700 1 -- 172.21.15.45:0/33668581 >> 172.21.15.45:0/33668581 conn(0x7f348c078dd0 msgr2=0x7f348c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.230+0000 7f34941f6700 1 -- 172.21.15.45:0/33668581 shutdown_connections 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.230+0000 7f34941f6700 1 -- 172.21.15.45:0/33668581 wait complete. 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 Processor -- start 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 -- start start 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c073ea0 0x7f348c1a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f348c0747c0 0x7f348c1a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c19ebf0 con 0x7f348c112e40 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c19ed30 con 0x7f348c073ea0 2024-03-12T05:09:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f34941f6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c19eef0 con 0x7f348c0747c0 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48334/0 (socket says 172.21.15.45:48334) 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 -- 172.21.15.45:0/3185027341 learned_addr learned my addr 172.21.15.45:0/3185027341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3491f92700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c073ea0 0x7f348c1a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3491791700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f348c0747c0 0x7f348c1a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 -- 172.21.15.45:0/3185027341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f348c0747c0 msgr2=0x7f348c1a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f348c0747c0 0x7f348c1a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 -- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c073ea0 msgr2=0x7f348c1a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c073ea0 0x7f348c1a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.231+0000 7f3492793700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484009d00 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f3492793700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c19e590 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f348400b830 tx=0x7f3484009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348400e040 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c19f170 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c1ab470 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3484004e60 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.232+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34840053c0 con 0x7f348c112e40 2024-03-12T05:09:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.233+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3470001030 con 0x7f348c112e40 2024-03-12T05:09:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.234+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3484005520 con 0x7f348c112e40 2024-03-12T05:09:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3482ffd700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f34780a1b50 0x7f34780a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3491f92700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f34780a1b50 0x7f34780a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f34840cee80 con 0x7f348c112e40 2024-03-12T05:09:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3482ffd700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] conn(0x7f34780ab330 0x7f34780ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34780ade00 con 0x7f34780ab330 2024-03-12T05:09:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34840cf320 con 0x7f348c112e40 2024-03-12T05:09:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.235+0000 7f3491791700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] conn(0x7f34780ab330 0x7f34780ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.236+0000 7f3491f92700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f34780a1b50 0x7f34780a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3488009a10 tx=0x7f3488009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.236+0000 7f3491791700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] conn(0x7f34780ab330 0x7f34780ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.237+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== osd.7 v2:172.21.15.116:6816/1067697558 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34780ade00 con 0x7f34780ab330 2024-03-12T05:09:20.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.259+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 --> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3470002dc0 con 0x7f34780ab330 2024-03-12T05:09:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.259+0000 7f3482ffd700 1 -- 172.21.15.45:0/3185027341 <== osd.7 v2:172.21.15.116:6816/1067697558 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3470002dc0 con 0x7f34780ab330 2024-03-12T05:09:20.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.259+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] conn(0x7f34780ab330 msgr2=0x7f34780ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.259+0000 7f34941f6700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] conn(0x7f34780ab330 0x7f34780ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f34780a1b50 msgr2=0x7f34780a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f34780a1b50 0x7f34780a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3488009a10 tx=0x7f3488009190 comp rx=0 tx=0).stop 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 msgr2=0x7f348c19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 --2- 172.21.15.45:0/3185027341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f348c112e40 0x7f348c19e590 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f348400b830 tx=0x7f3484009380 comp rx=0 tx=0).stop 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f3492793700 1 -- 172.21.15.45:0/3185027341 reap_dead start 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 shutdown_connections 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 >> 172.21.15.45:0/3185027341 conn(0x7f348c078dd0 msgr2=0x7f348c111b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 shutdown_connections 2024-03-12T05:09:20.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.260+0000 7f34941f6700 1 -- 172.21.15.45:0/3185027341 wait complete. 2024-03-12T05:09:20.690 INFO:teuthology.orchestra.run.smithi045.stdout:77309411393 2024-03-12T05:09:20.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 -- 172.21.15.45:0/2618208464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c106f80 msgr2=0x7f106c10eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 --2- 172.21.15.45:0/2618208464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c106f80 0x7f106c10eb30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1064009990 tx=0x7f1064009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 -- 172.21.15.45:0/2618208464 shutdown_connections 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 --2- 172.21.15.45:0/2618208464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c106f80 0x7f106c10eb30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 --2- 172.21.15.45:0/2618208464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06a410 0x7f106c06a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 --2- 172.21.15.45:0/2618208464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c06adc0 0x7f106c1068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 -- 172.21.15.45:0/2618208464 >> 172.21.15.45:0/2618208464 conn(0x7f106c076bc0 msgr2=0x7f106c076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 -- 172.21.15.45:0/2618208464 shutdown_connections 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.880+0000 7f1073422700 1 -- 172.21.15.45:0/2618208464 wait complete. 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 Processor -- start 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 -- start start 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 0x7f106c100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06adc0 0x7f106c1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c106f80 0x7f106c1016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1a6410 con 0x7f106c06a410 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1a6580 con 0x7f106c06adc0 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f1073422700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1a6740 con 0x7f106c106f80 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f10711be700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 0x7f106c100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f10711be700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 0x7f106c100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48362/0 (socket says 172.21.15.45:48362) 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f10711be700 1 -- 172.21.15.45:0/984470582 learned_addr learned my addr 172.21.15.45:0/984470582 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:20.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.881+0000 7f10709bd700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06adc0 0x7f106c1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 -- 172.21.15.45:0/984470582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c106f80 msgr2=0x7f106c1016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10719bf700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c106f80 0x7f106c1016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c106f80 0x7f106c1016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 -- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06adc0 msgr2=0x7f106c1011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06adc0 0x7f106c1011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10640095f0 con 0x7f106c06a410 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10709bd700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f106c06adc0 0x7f106c1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10719bf700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f106c106f80 0x7f106c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f10711be700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 0x7f106c100c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ada0 tx=0x7f105c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00f070 con 0x7f106c06a410 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c1a6a20 con 0x7f106c06a410 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c1a6f70 con 0x7f106c06a410 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f105c00ae90 con 0x7f106c06a410 2024-03-12T05:09:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.882+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c005320 con 0x7f106c06a410 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.883+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f106c111ea0 con 0x7f106c06a410 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f105c00c980 con 0x7f106c06a410 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f10540a1bd0 0x7f10540a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f10709bd700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f10540a1bd0 0x7f10540a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f105c0cd7c0 con 0x7f106c06a410 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] conn(0x7f10540ab3b0 0x7f10540ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10540ade80 con 0x7f10540ab3b0 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.885+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f105c0cdc60 con 0x7f106c06a410 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.886+0000 7f10709bd700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f10540a1bd0 0x7f10540a4090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f106c1084b0 tx=0x7f106000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.886+0000 7f10719bf700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] conn(0x7f10540ab3b0 0x7f10540ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:20.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.886+0000 7f10719bf700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] conn(0x7f10540ab3b0 0x7f10540ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:20.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.887+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== osd.4 v2:172.21.15.116:6808/3946228497 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10540ade80 con 0x7f10540ab3b0 2024-03-12T05:09:20.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.904+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 --> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f106c062300 con 0x7f10540ab3b0 2024-03-12T05:09:20.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f106a7fc700 1 -- 172.21.15.45:0/984470582 <== osd.4 v2:172.21.15.116:6808/3946228497 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f106c062300 con 0x7f10540ab3b0 2024-03-12T05:09:20.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] conn(0x7f10540ab3b0 msgr2=0x7f10540ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] conn(0x7f10540ab3b0 0x7f10540ad7d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f10540a1bd0 msgr2=0x7f10540a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f10540a1bd0 0x7f10540a4090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f106c1084b0 tx=0x7f106000b410 comp rx=0 tx=0).stop 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 msgr2=0x7f106c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f1073422700 1 --2- 172.21.15.45:0/984470582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f106c06a410 0x7f106c100c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f105c00ada0 tx=0x7f105c00c330 comp rx=0 tx=0).stop 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.905+0000 7f10719bf700 1 -- 172.21.15.45:0/984470582 reap_dead start 2024-03-12T05:09:20.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.906+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 shutdown_connections 2024-03-12T05:09:20.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.906+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 >> 172.21.15.45:0/984470582 conn(0x7f106c076bc0 msgr2=0x7f106c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:20.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.906+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 shutdown_connections 2024-03-12T05:09:20.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:20.906+0000 7f1073422700 1 -- 172.21.15.45:0/984470582 wait complete. 2024-03-12T05:09:21.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:21 smithi130 ceph-mon[35398]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:21.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:21 smithi116 ceph-mon[35486]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:21.611 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:21 smithi045 ceph-mon[32174]: pgmap v304: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:21.645 INFO:teuthology.orchestra.run.smithi045.stdout:163208757299 2024-03-12T05:09:21.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T05:09:22.057 INFO:teuthology.orchestra.run.smithi045.stdout:253403070500 2024-03-12T05:09:22.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-12T05:09:22.738 INFO:teuthology.orchestra.run.smithi045.stdout:141733920825 2024-03-12T05:09:22.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 -- 172.21.15.45:0/3954793089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c106ff0 msgr2=0x7fe56c10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 --2- 172.21.15.45:0/3954793089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c106ff0 0x7fe56c10eb70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe56400b0a0 tx=0x7fe56400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 -- 172.21.15.45:0/3954793089 shutdown_connections 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 --2- 172.21.15.45:0/3954793089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c106ff0 0x7fe56c10eb70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 --2- 172.21.15.45:0/3954793089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 0x7fe56c06a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 --2- 172.21.15.45:0/3954793089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c06adc0 0x7fe56c1068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 -- 172.21.15.45:0/3954793089 >> 172.21.15.45:0/3954793089 conn(0x7fe56c076bc0 msgr2=0x7fe56c076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 -- 172.21.15.45:0/3954793089 shutdown_connections 2024-03-12T05:09:22.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.979+0000 7fe570f73700 1 -- 172.21.15.45:0/3954793089 wait complete. 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 Processor -- start 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 -- start start 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 0x7fe56c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 0x7fe56c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c106ff0 0x7fe56c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19d110 con 0x7fe56c06adc0 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19d280 con 0x7fe56c106ff0 2024-03-12T05:09:22.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe570f73700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19d440 con 0x7fe56c06a410 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe569d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 0x7fe56c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe569d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 0x7fe56c198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48380/0 (socket says 172.21.15.45:48380) 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe569d9b700 1 -- 172.21.15.45:0/4116462390 learned_addr learned my addr 172.21.15.45:0/4116462390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe56ad9d700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c106ff0 0x7fe56c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.980+0000 7fe56a59c700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 0x7fe56c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 -- 172.21.15.45:0/4116462390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 msgr2=0x7fe56c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 0x7fe56c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 -- 172.21.15.45:0/4116462390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c106ff0 msgr2=0x7fe56c19cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe56c106ff0 0x7fe56c19cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009d00 con 0x7fe56c06adc0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe56a59c700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe56c06a410 0x7fe56c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe569d9b700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 0x7fe56c198770 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c8c0 tx=0x7fe56000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560004220 con 0x7fe56c06adc0 2024-03-12T05:09:22.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c19d720 con 0x7fe56c06adc0 2024-03-12T05:09:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe560004ce0 con 0x7fe56c06adc0 2024-03-12T05:09:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600056e0 con 0x7fe56c06adc0 2024-03-12T05:09:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.981+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c1a4f80 con 0x7fe56c06adc0 2024-03-12T05:09:22.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.982+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe56c111f30 con 0x7fe56c06adc0 2024-03-12T05:09:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.983+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe560004750 con 0x7fe56c06adc0 2024-03-12T05:09:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe55b7fe700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5540a1b50 0x7fe5540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe56a59c700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5540a1b50 0x7fe5540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe5600cd2e0 con 0x7fe56c06adc0 2024-03-12T05:09:22.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe55b7fe700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] conn(0x7fe5540ab330 0x7fe5540ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5540ade00 con 0x7fe5540ab330 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5600cd780 con 0x7fe56c06adc0 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe56ad9d700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] conn(0x7fe5540ab330 0x7fe5540ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe56a59c700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5540a1b50 0x7fe5540a4010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0099d0 tx=0x7fe55c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.984+0000 7fe56ad9d700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] conn(0x7fe5540ab330 0x7fe5540ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:22.985+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== osd.3 v2:172.21.15.45:6810/1948936218 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe5540ade00 con 0x7fe5540ab330 2024-03-12T05:09:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.003+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 --> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe56c062300 con 0x7fe5540ab330 2024-03-12T05:09:23.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe55b7fe700 1 -- 172.21.15.45:0/4116462390 <== osd.3 v2:172.21.15.45:6810/1948936218 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe56c062300 con 0x7fe5540ab330 2024-03-12T05:09:23.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] conn(0x7fe5540ab330 msgr2=0x7fe5540ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] conn(0x7fe5540ab330 0x7fe5540ad750 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5540a1b50 msgr2=0x7fe5540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe5540a1b50 0x7fe5540a4010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0099d0 tx=0x7fe55c008040 comp rx=0 tx=0).stop 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 msgr2=0x7fe56c198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 --2- 172.21.15.45:0/4116462390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe56c06adc0 0x7fe56c198770 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe56000c8c0 tx=0x7fe56000cbd0 comp rx=0 tx=0).stop 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe56ad9d700 1 -- 172.21.15.45:0/4116462390 reap_dead start 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 shutdown_connections 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 >> 172.21.15.45:0/4116462390 conn(0x7fe56c076bc0 msgr2=0x7fe56c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 shutdown_connections 2024-03-12T05:09:23.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:23.004+0000 7fe570f73700 1 -- 172.21.15.45:0/4116462390 wait complete. 2024-03-12T05:09:23.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:23 smithi045 ceph-mon[32174]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:23.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:23 smithi130 ceph-mon[35398]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:23.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:23 smithi116 ceph-mon[35486]: pgmap v305: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:23.967 INFO:teuthology.orchestra.run.smithi045.stdout:107374182462 2024-03-12T05:09:23.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T05:09:24.215 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:24.215 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:24.630 INFO:teuthology.orchestra.run.smithi045.stdout:219043332138 2024-03-12T05:09:24.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T05:09:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 -- 172.21.15.45:0/2280036866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40100e50 msgr2=0x7f3f401012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2280036866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40100e50 0x7f3f401012d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009990 tx=0x7f3f34009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 -- 172.21.15.45:0/2280036866 shutdown_connections 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2280036866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f40101810 0x7f3f4010da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2280036866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f40106ca0 0x7f3f40107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2280036866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40100e50 0x7f3f401012d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 -- 172.21.15.45:0/2280036866 >> 172.21.15.45:0/2280036866 conn(0x7f3f40077a30 msgr2=0x7f3f40077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 -- 172.21.15.45:0/2280036866 shutdown_connections 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.986+0000 7f3f481cc700 1 -- 172.21.15.45:0/2280036866 wait complete. 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 Processor -- start 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 -- start start 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f40100e50 0x7f3f40198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:25.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 0x7f3f401985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f40106ca0 0x7f3f4019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4019cf60 con 0x7f3f40101810 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4019d0d0 con 0x7f3f40106ca0 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f481cc700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4019d290 con 0x7f3f40100e50 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f45767700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 0x7f3f401985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f45767700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 0x7f3f401985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48410/0 (socket says 172.21.15.45:48410) 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f45767700 1 -- 172.21.15.45:0/2613169492 learned_addr learned my addr 172.21.15.45:0/2613169492 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f45f68700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f40100e50 0x7f3f40198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.987+0000 7f3f46769700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f40106ca0 0x7f3f4019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 -- 172.21.15.45:0/2613169492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f40100e50 msgr2=0x7f3f40198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f40100e50 0x7f3f40198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 -- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f40106ca0 msgr2=0x7f3f4019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f40106ca0 0x7f3f4019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f340095f0 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f45767700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 0x7f3f401985c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f34000c00 tx=0x7f3f3400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3401d070 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4019d510 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3f3400bdc0 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34022d60 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.988+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f401a4dd0 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.989+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f401022a0 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.990+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3f34022500 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.990+0000 7f3f3effd700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3f280a1b50 0x7f3f280a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f45f68700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3f280a1b50 0x7f3f280a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3f340cea90 con 0x7f3f40101810 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f3effd700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] conn(0x7f3f280ab330 0x7f3f280ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f280ade00 con 0x7f3f280ab330 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f340cee00 con 0x7f3f40101810 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f45f68700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3f280a1b50 0x7f3f280a4010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009550 tx=0x7f3f380091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.991+0000 7f3f46769700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] conn(0x7f3f280ab330 0x7f3f280ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.992+0000 7f3f46769700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] conn(0x7f3f280ab330 0x7f3f280ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:24.992+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== osd.1 v2:172.21.15.116:6800/1071649441 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3f280ade00 con 0x7f3f280ab330 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.009+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 --> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3f40111f40 con 0x7f3f280ab330 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.009+0000 7f3f3effd700 1 -- 172.21.15.45:0/2613169492 <== osd.1 v2:172.21.15.116:6800/1071649441 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3f40111f40 con 0x7f3f280ab330 2024-03-12T05:09:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] conn(0x7f3f280ab330 msgr2=0x7f3f280ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] conn(0x7f3f280ab330 0x7f3f280ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3f280a1b50 msgr2=0x7f3f280a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3f280a1b50 0x7f3f280a4010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009550 tx=0x7f3f380091b0 comp rx=0 tx=0).stop 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 msgr2=0x7f3f401985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 --2- 172.21.15.45:0/2613169492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f40101810 0x7f3f401985c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3f34000c00 tx=0x7f3f3400f690 comp rx=0 tx=0).stop 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f46769700 1 -- 172.21.15.45:0/2613169492 reap_dead start 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 shutdown_connections 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 >> 172.21.15.45:0/2613169492 conn(0x7f3f40077a30 msgr2=0x7f3f400ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 shutdown_connections 2024-03-12T05:09:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:25.010+0000 7f3f481cc700 1 -- 172.21.15.45:0/2613169492 wait complete. 2024-03-12T05:09:25.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:25 smithi130 ceph-mon[35398]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-12T05:09:25.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:25 smithi116 ceph-mon[35486]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-12T05:09:25.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:25 smithi045 ceph-mon[32174]: pgmap v306: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-12T05:09:25.818 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:25.819 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:25.845 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:25.845 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:25.848 INFO:teuthology.orchestra.run.smithi045.stdout:55834574920 2024-03-12T05:09:25.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T05:09:27.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:27 smithi130 ceph-mon[35398]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:27.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:27 smithi116 ceph-mon[35486]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:27.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:27 smithi045 ceph-mon[32174]: pgmap v307: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:29.193 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:29 smithi045 ceph-mon[32174]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-12T05:09:29.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:29 smithi130 ceph-mon[35398]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-12T05:09:29.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:29 smithi116 ceph-mon[35486]: pgmap v308: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 -- 172.21.15.45:0/1678539727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64101eb0 msgr2=0x7f3a64102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/1678539727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64101eb0 0x7f3a64102290 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009990 tx=0x7f3a60009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 -- 172.21.15.45:0/1678539727 shutdown_connections 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/1678539727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64106f90 0x7f3a6410eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/1678539727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64101eb0 0x7f3a64102290 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/1678539727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64102860 0x7f3a641068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 -- 172.21.15.45:0/1678539727 >> 172.21.15.45:0/1678539727 conn(0x7f3a640fd720 msgr2=0x7f3a640ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 -- 172.21.15.45:0/1678539727 shutdown_connections 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.672+0000 7f3a6c02a700 1 -- 172.21.15.45:0/1678539727 wait complete. 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 Processor -- start 2024-03-12T05:09:29.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 -- start start 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 0x7f3a64198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 0x7f3a641986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6419d040 con 0x7f3a64106f90 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6419d1b0 con 0x7f3a64102860 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6c02a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6419d370 con 0x7f3a64101eb0 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6a5c7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6a5c7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45386/0 (socket says 172.21.15.45:45386) 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.673+0000 7f3a6a5c7700 1 -- 172.21.15.45:0/3706100385 learned_addr learned my addr 172.21.15.45:0/3706100385 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a695c5700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 0x7f3a641986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a69dc6700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 0x7f3a64198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 -- 172.21.15.45:0/3706100385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 msgr2=0x7f3a64198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 0x7f3a64198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 -- 172.21.15.45:0/3706100385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 msgr2=0x7f3a641986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 0x7f3a641986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 -- 172.21.15.45:0/3706100385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600095f0 con 0x7f3a64106f90 2024-03-12T05:09:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a695c5700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 0x7f3a641986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a69dc6700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 0x7f3a64198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6a5c7700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00ada0 tx=0x7f3a5c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c00f070 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6419d650 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3a5c007b30 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c005420 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.674+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a641a4eb0 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.675+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6410a1c0 con 0x7f3a64106f90 2024-03-12T05:09:29.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.677+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3a5c00c980 con 0x7f3a64106f90 2024-03-12T05:09:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.678+0000 7f3a5affd700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 0x7f3a500a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.678+0000 7f3a69dc6700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 0x7f3a500a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.678+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3a5c0ccad0 con 0x7f3a64106f90 2024-03-12T05:09:29.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.678+0000 7f3a69dc6700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 0x7f3a500a4010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a6000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:29.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.686+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3a5c095dc0 con 0x7f3a64106f90 2024-03-12T05:09:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.891+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f3a641994c0 con 0x7f3a64106f90 2024-03-12T05:09:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.891+0000 7f3a5affd700 1 -- 172.21.15.45:0/3706100385 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f3a5c099480 con 0x7f3a64106f90 2024-03-12T05:09:29.890 INFO:teuthology.orchestra.run.smithi045.stdout:343597383702 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 msgr2=0x7f3a500a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 0x7f3a500a4010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3a60006010 tx=0x7f3a6000b540 comp rx=0 tx=0).stop 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 msgr2=0x7f3a6419ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00ada0 tx=0x7f3a5c00c330 comp rx=0 tx=0).stop 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 shutdown_connections 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f3a500a1b50 0x7f3a500a4010 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a64106f90 0x7f3a6419ca30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a64101eb0 0x7f3a64198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 --2- 172.21.15.45:0/3706100385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a64102860 0x7f3a641986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.894+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 >> 172.21.15.45:0/3706100385 conn(0x7f3a640fd720 msgr2=0x7f3a64106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.895+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 shutdown_connections 2024-03-12T05:09:29.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.895+0000 7f3a6c02a700 1 -- 172.21.15.45:0/3706100385 wait complete. 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 -- 172.21.15.45:0/3754069106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66806a410 msgr2=0x7fe66806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/3754069106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66806a410 0x7fe66806a7f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe650009990 tx=0x7fe650009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 -- 172.21.15.45:0/3754069106 shutdown_connections 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/3754069106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe668106f80 0x7fe66810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/3754069106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe66806a410 0x7fe66806a7f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/3754069106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806adc0 0x7fe6681068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 -- 172.21.15.45:0/3754069106 >> 172.21.15.45:0/3754069106 conn(0x7fe668076bc0 msgr2=0x7fe668076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 -- 172.21.15.45:0/3754069106 shutdown_connections 2024-03-12T05:09:29.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.911+0000 7fe66c8e5700 1 -- 172.21.15.45:0/3754069106 wait complete. 2024-03-12T05:09:29.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 Processor -- start 2024-03-12T05:09:29.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 -- start start 2024-03-12T05:09:29.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 0x7fe6681985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe668106f80 0x7fe66819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66819cf60 con 0x7fe668106f80 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66819d0d0 con 0x7fe66806adc0 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.912+0000 7fe66c8e5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66819d290 con 0x7fe66806a410 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50822/0 (socket says 172.21.15.45:50822) 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 -- 172.21.15.45:0/2345724508 learned_addr learned my addr 172.21.15.45:0/2345724508 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe665d9b700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 0x7fe6681985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe666d9d700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe668106f80 0x7fe66819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 -- 172.21.15.45:0/2345724508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 msgr2=0x7fe6681985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 0x7fe6681985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 -- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe668106f80 msgr2=0x7fe66819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe668106f80 0x7fe66819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 -- 172.21.15.45:0/2345724508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500095f0 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe665d9b700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 0x7fe6681985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe66659c700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe650000c00 tx=0x7fe65000f510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.913+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65001d070 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.914+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe650022470 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.914+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe650003a20 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.914+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66819d510 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.914+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe668100ba0 con 0x7fe66806a410 2024-03-12T05:09:29.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.915+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66810a1b0 con 0x7fe66806a410 2024-03-12T05:09:29.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.917+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe6500225e0 con 0x7fe66806a410 2024-03-12T05:09:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.917+0000 7fe65f7fe700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 0x7fe6540a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.917+0000 7fe665d9b700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 0x7fe6540a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.917+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe6500ce760 con 0x7fe66806a410 2024-03-12T05:09:29.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.918+0000 7fe665d9b700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 0x7fe6540a4090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe668108490 tx=0x7fe658005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:29.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:29.922+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe650097a50 con 0x7fe66806a410 2024-03-12T05:09:30.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.130+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe6681993e0 con 0x7fe66806a410 2024-03-12T05:09:30.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.131+0000 7fe65f7fe700 1 -- 172.21.15.45:0/2345724508 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe65009b110 con 0x7fe66806a410 2024-03-12T05:09:30.129 INFO:teuthology.orchestra.run.smithi045.stdout:197568495664 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 msgr2=0x7fe6540a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 0x7fe6540a4090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe668108490 tx=0x7fe658005f50 comp rx=0 tx=0).stop 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 msgr2=0x7fe668198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe650000c00 tx=0x7fe65000f510 comp rx=0 tx=0).stop 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 shutdown_connections 2024-03-12T05:09:30.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe6540a1bd0 0x7fe6540a4090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe668106f80 0x7fe66819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe66806a410 0x7fe668198080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 --2- 172.21.15.45:0/2345724508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe66806adc0 0x7fe6681985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 >> 172.21.15.45:0/2345724508 conn(0x7fe668076bc0 msgr2=0x7fe668100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 shutdown_connections 2024-03-12T05:09:30.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:30.133+0000 7fe66c8e5700 1 -- 172.21.15.45:0/2345724508 wait complete. 2024-03-12T05:09:30.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3706100385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:09:30.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2345724508' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:09:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3706100385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:09:30.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2345724508' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:09:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3706100385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:09:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2345724508' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:09:31.209 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383699 got 343597383702 for osd.11 2024-03-12T05:09:31.209 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:31.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:31 smithi130 ceph-mon[35398]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-12T05:09:31.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:31 smithi116 ceph-mon[35486]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-12T05:09:31.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:31 smithi045 ceph-mon[32174]: pgmap v309: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-12T05:09:33.134 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495662 got 197568495664 for osd.6 2024-03-12T05:09:33.134 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:33.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:33 smithi045 ceph-mon[32174]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:33.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:33 smithi130 ceph-mon[35398]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:33.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:33 smithi116 ceph-mon[35486]: pgmap v310: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:34.394 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:34.395 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:34.395 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:34.407 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:35.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:35 smithi045 ceph-mon[32174]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-12T05:09:35.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:35 smithi130 ceph-mon[35398]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-12T05:09:35.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:35 smithi116 ceph-mon[35486]: pgmap v311: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-12T05:09:36.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:36 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:09:36.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:36 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:09:36.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:36 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:09:37.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- 172.21.15.45:0/3457095759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 msgr2=0x7f085406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:37.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3457095759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085406a840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f08440099c0 tx=0x7f0844009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- 172.21.15.45:0/3457095759 shutdown_connections 2024-03-12T05:09:37.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3457095759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085406ad80 0x7f085410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3457095759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854107ca0 0x7f0854108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3457095759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085406a840 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- 172.21.15.45:0/3457095759 >> 172.21.15.45:0/3457095759 conn(0x7f0854077b80 msgr2=0x7f0854077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- 172.21.15.45:0/3457095759 shutdown_connections 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- 172.21.15.45:0/3457095759 wait complete. 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 Processor -- start 2024-03-12T05:09:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.534+0000 7f085b2c6700 1 -- start start 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085406ad80 0x7f085419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854107ca0 0x7f085419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419f160 con 0x7f085406a3c0 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419f2d0 con 0x7f085406ad80 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085419f490 con 0x7f0854107ca0 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46642/0 (socket says 172.21.15.45:46642) 2024-03-12T05:09:37.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 -- 172.21.15.45:0/3899522480 learned_addr learned my addr 172.21.15.45:0/3899522480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0858861700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085406ad80 0x7f085419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859863700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854107ca0 0x7f085419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 -- 172.21.15.45:0/3899522480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854107ca0 msgr2=0x7f085419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854107ca0 0x7f085419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 -- 172.21.15.45:0/3899522480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085406ad80 msgr2=0x7f085419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085406ad80 0x7f085419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 -- 172.21.15.45:0/3899522480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440095f0 con 0x7f085406a3c0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f0859062700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f085000e9a0 tx=0x7f085000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850009790 con 0x7f085406a3c0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0850010430 con 0x7f085406a3c0 2024-03-12T05:09:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085419f770 con 0x7f085406a3c0 2024-03-12T05:09:37.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.535+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850018680 con 0x7f085406a3c0 2024-03-12T05:09:37.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.536+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541a6fd0 con 0x7f085406a3c0 2024-03-12T05:09:37.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.537+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0854068930 con 0x7f085406a3c0 2024-03-12T05:09:37.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.538+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f08500105a0 con 0x7f085406a3c0 2024-03-12T05:09:37.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.538+0000 7f084a7fc700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 0x7f08400a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:37.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.538+0000 7f0858861700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 0x7f08400a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:37.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.538+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0850014070 con 0x7f085406a3c0 2024-03-12T05:09:37.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.539+0000 7f0858861700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 0x7f08400a4090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:37.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.543+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0850052d20 con 0x7f085406a3c0 2024-03-12T05:09:37.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:37 smithi130 ceph-mon[35398]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:37.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.747+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f0854111b60 con 0x7f085406a3c0 2024-03-12T05:09:37.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.748+0000 7f084a7fc700 1 -- 172.21.15.45:0/3899522480 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f0850096730 con 0x7f085406a3c0 2024-03-12T05:09:37.746 INFO:teuthology.orchestra.run.smithi045.stdout:309237645341 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 msgr2=0x7f08400a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 0x7f08400a4090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844005fb0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 msgr2=0x7f085419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f085000e9a0 tx=0x7f085000ecb0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 shutdown_connections 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08400a1bd0 0x7f08400a4090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0854107ca0 0x7f085419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f085406a3c0 0x7f085419a280 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 --2- 172.21.15.45:0/3899522480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085406ad80 0x7f085419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:37.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 >> 172.21.15.45:0/3899522480 conn(0x7f0854077b80 msgr2=0x7f085410cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:37.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 shutdown_connections 2024-03-12T05:09:37.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:37.750+0000 7f085b2c6700 1 -- 172.21.15.45:0/3899522480 wait complete. 2024-03-12T05:09:37.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:37 smithi045 ceph-mon[32174]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:37.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:37 smithi116 ceph-mon[35486]: pgmap v312: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:38.683 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3899522480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4269599344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:09:38.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3899522480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:09:38.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4269599344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:09:38.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3899522480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:09:38.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:38.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4269599344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- 172.21.15.45:0/2269345862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 msgr2=0x7f0afc106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 --2- 172.21.15.45:0/2269345862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc106f90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009990 tx=0x7f0aec009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- 172.21.15.45:0/2269345862 shutdown_connections 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 --2- 172.21.15.45:0/2269345862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc101560 0x7f0afc10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 --2- 172.21.15.45:0/2269345862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc106f90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 --2- 172.21.15.45:0/2269345862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc100ba0 0x7f0afc101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- 172.21.15.45:0/2269345862 >> 172.21.15.45:0/2269345862 conn(0x7f0afc076b80 msgr2=0x7f0afc076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- 172.21.15.45:0/2269345862 shutdown_connections 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- 172.21.15.45:0/2269345862 wait complete. 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 Processor -- start 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.387+0000 7f0b0289e700 1 -- start start 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc100ba0 0x7f0afc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc101560 0x7f0afc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc19cf30 con 0x7f0afc106bb0 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc19d0a0 con 0x7f0afc100ba0 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc19d260 con 0x7f0afc101560 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:39.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46672/0 (socket says 172.21.15.45:46672) 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 -- 172.21.15.45:0/1492165008 learned_addr learned my addr 172.21.15.45:0/1492165008 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0afbfff700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc100ba0 0x7f0afc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0afb7fe700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc101560 0x7f0afc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 -- 172.21.15.45:0/1492165008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc101560 msgr2=0x7f0afc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc101560 0x7f0afc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 -- 172.21.15.45:0/1492165008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc100ba0 msgr2=0x7f0afc198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc100ba0 0x7f0afc198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 -- 172.21.15.45:0/1492165008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec0095f0 con 0x7f0afc106bb0 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b00e3b700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0af4007ee0 tx=0x7f0af400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4017070 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0afc19d540 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.388+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0af400f040 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.389+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af40055e0 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.389+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc1a4da0 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.390+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc1090b0 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.391+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0af400db20 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.391+0000 7f0af97fa700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 0x7f0ae40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.391+0000 7f0afbfff700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 0x7f0ae40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.391+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0af40ccb20 con 0x7f0afc106bb0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.392+0000 7f0afbfff700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 0x7f0ae40a4010 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00b5c0 tx=0x7f0aec005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:39.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.396+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0af4095d80 con 0x7f0afc106bb0 2024-03-12T05:09:39.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.604+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0afc1993b0 con 0x7f0afc106bb0 2024-03-12T05:09:39.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.604+0000 7f0af97fa700 1 -- 172.21.15.45:0/1492165008 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0af4099440 con 0x7f0afc106bb0 2024-03-12T05:09:39.602 INFO:teuthology.orchestra.run.smithi045.stdout:34359738447 2024-03-12T05:09:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 msgr2=0x7f0ae40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 0x7f0ae40a4010 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00b5c0 tx=0x7f0aec005960 comp rx=0 tx=0).stop 2024-03-12T05:09:39.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 msgr2=0x7f0afc19c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0af4007ee0 tx=0x7f0af400d3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 shutdown_connections 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ae40a1b50 0x7f0ae40a4010 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0afc106bb0 0x7f0afc19c920 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0afc100ba0 0x7f0afc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 --2- 172.21.15.45:0/1492165008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0afc101560 0x7f0afc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.606+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 >> 172.21.15.45:0/1492165008 conn(0x7f0afc076b80 msgr2=0x7f0afc0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.607+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 shutdown_connections 2024-03-12T05:09:39.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:39.607+0000 7f0b0289e700 1 -- 172.21.15.45:0/1492165008 wait complete. 2024-03-12T05:09:39.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:39 smithi045 ceph-mon[32174]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:09:39.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1492165008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:09:39.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:39 smithi130 ceph-mon[35398]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:09:39.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1492165008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:09:39.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:39 smithi116 ceph-mon[35486]: pgmap v313: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:09:39.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1492165008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:09:40.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 -- 172.21.15.45:0/514730083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 msgr2=0x7f65881068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 --2- 172.21.15.45:0/514730083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f65881068d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6570009990 tx=0x7f6570009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 -- 172.21.15.45:0/514730083 shutdown_connections 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 --2- 172.21.15.45:0/514730083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 0x7f658810eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 --2- 172.21.15.45:0/514730083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 0x7f658806a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 --2- 172.21.15.45:0/514730083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f65881068d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 -- 172.21.15.45:0/514730083 >> 172.21.15.45:0/514730083 conn(0x7f6588077a30 msgr2=0x7f6588077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 -- 172.21.15.45:0/514730083 shutdown_connections 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.133+0000 7f658c9c4700 1 -- 172.21.15.45:0/514730083 wait complete. 2024-03-12T05:09:40.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 Processor -- start 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 -- start start 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 0x7f6588100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 0x7f65881019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65881a6490 con 0x7f658806af20 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65881a6600 con 0x7f658806a570 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658c9c4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65881a67c0 con 0x7f6588106fb0 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46690/0 (socket says 172.21.15.45:46690) 2024-03-12T05:09:40.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 -- 172.21.15.45:0/145281017 learned_addr learned my addr 172.21.15.45:0/145281017 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f658659c700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 0x7f6588100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6586d9d700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 0x7f65881019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:40.561 INFO:teuthology.orchestra.run.smithi045.stdout:253403070504 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 -- 172.21.15.45:0/145281017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 msgr2=0x7f65881019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 0x7f65881019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 -- 172.21.15.45:0/145281017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 msgr2=0x7f6588100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 0x7f6588100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 -- 172.21.15.45:0/145281017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700095f0 con 0x7f658806af20 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.134+0000 7f6585d9b700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6570000c00 tx=0x7f65700057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.135+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657001d070 con 0x7f658806af20 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.135+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f657000fb20 con 0x7f658806af20 2024-03-12T05:09:40.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.135+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65881a6a40 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.135+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570022b40 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.135+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65881a6ee0 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.136+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658810a1b0 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.137+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f657000fc90 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.137+0000 7f657f7fe700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 0x7f65740a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.137+0000 7f658659c700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 0x7f65740a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.137+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f65700cee90 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.138+0000 7f658659c700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 0x7f65740a4010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6580009550 tx=0x7f65800091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.141+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6570098180 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.348+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f6588068930 con 0x7f658806af20 2024-03-12T05:09:40.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.348+0000 7f657f7fe700 1 -- 172.21.15.45:0/145281017 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6570027020 con 0x7f658806af20 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 msgr2=0x7f65740a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 0x7f65740a4010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6580009550 tx=0x7f65800091b0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 msgr2=0x7f6588101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6570000c00 tx=0x7f65700057c0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 shutdown_connections 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f65740a1b50 0x7f65740a4010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6588106fb0 0x7f65881019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f658806a570 0x7f6588100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 --2- 172.21.15.45:0/145281017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f658806af20 0x7f6588101460 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 >> 172.21.15.45:0/145281017 conn(0x7f6588077a30 msgr2=0x7f65880ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 shutdown_connections 2024-03-12T05:09:40.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:40.350+0000 7f658c9c4700 1 -- 172.21.15.45:0/145281017 wait complete. 2024-03-12T05:09:40.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/145281017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:09:40.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/145281017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:09:41.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/145281017' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:09:41.342 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070500 got 253403070504 for osd.8 2024-03-12T05:09:41.343 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:41.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:41 smithi045 ceph-mon[32174]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:41.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:41 smithi130 ceph-mon[35398]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:41.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:41 smithi116 ceph-mon[35486]: pgmap v314: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:09:42.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738447 for osd.0 2024-03-12T05:09:42.151 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-12T05:09:43.292 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:43 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-12T05:09:43.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:43 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-12T05:09:43.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:09:43.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-12T05:09:43.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-12T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-12T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]: dispatch 2024-03-12T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-12T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-12T05:09:43.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:43 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 -- 172.21.15.45:0/4198717590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea241026b0 msgr2=0x7fea2410eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 --2- 172.21.15.45:0/4198717590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea241026b0 0x7fea2410eb10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00b0a0 tx=0x7fea1c00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 -- 172.21.15.45:0/4198717590 shutdown_connections 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 --2- 172.21.15.45:0/4198717590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea241026b0 0x7fea2410eb10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 --2- 172.21.15.45:0/4198717590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24107dd0 0x7fea241081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 --2- 172.21.15.45:0/4198717590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea24101cf0 0x7fea24102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 -- 172.21.15.45:0/4198717590 >> 172.21.15.45:0/4198717590 conn(0x7fea24078d30 msgr2=0x7fea24079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:43.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 -- 172.21.15.45:0/4198717590 shutdown_connections 2024-03-12T05:09:43.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.299+0000 7fea28eb8700 1 -- 172.21.15.45:0/4198717590 wait complete. 2024-03-12T05:09:43.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 Processor -- start 2024-03-12T05:09:43.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 -- start start 2024-03-12T05:09:43.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24101cf0 0x7fea2419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:43.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea241026b0 0x7fea2419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:43.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:43.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2419f1d0 con 0x7fea24107dd0 2024-03-12T05:09:43.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2419f340 con 0x7fea24101cf0 2024-03-12T05:09:43.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea28eb8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2419f500 con 0x7fea241026b0 2024-03-12T05:09:43.983 INFO:teuthology.orchestra.run.smithi045.stdout:55834574924 2024-03-12T05:09:43.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea22d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:43.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea22d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46706/0 (socket says 172.21.15.45:46706) 2024-03-12T05:09:43.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea22d9d700 1 -- 172.21.15.45:0/2380842472 learned_addr learned my addr 172.21.15.45:0/2380842472 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:43.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea2259c700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24101cf0 0x7fea2419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.300+0000 7fea21d9b700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea241026b0 0x7fea2419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 -- 172.21.15.45:0/2380842472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea241026b0 msgr2=0x7fea2419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea241026b0 0x7fea2419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 -- 172.21.15.45:0/2380842472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24101cf0 msgr2=0x7fea2419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24101cf0 0x7fea2419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 -- 172.21.15.45:0/2380842472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c009d00 con 0x7fea24107dd0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea22d9d700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fea1c006130 tx=0x7fea1c009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00e040 con 0x7fea24107dd0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2419f780 con 0x7fea24107dd0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea1c004c60 con 0x7fea24107dd0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0052a0 con 0x7fea24107dd0 2024-03-12T05:09:43.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.301+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea241a7040 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.302+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea24068930 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.304+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fea1c01b990 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.304+0000 7fea137fe700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 0x7fea0c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.305+0000 7fea2259c700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 0x7fea0c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.305+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea1c0cf480 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.305+0000 7fea2259c700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 0x7fea0c0a4010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea14005950 tx=0x7fea1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.308+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea1c098570 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.513+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fea24111bd0 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.513+0000 7fea137fe700 1 -- 172.21.15.45:0/2380842472 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea1c09bc30 con 0x7fea24107dd0 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 msgr2=0x7fea0c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 0x7fea0c0a4010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea14005950 tx=0x7fea1400b410 comp rx=0 tx=0).stop 2024-03-12T05:09:43.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 msgr2=0x7fea2419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fea1c006130 tx=0x7fea1c009380 comp rx=0 tx=0).stop 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 shutdown_connections 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea0c0a1b50 0x7fea0c0a4010 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea24107dd0 0x7fea2419ebc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea24101cf0 0x7fea2419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 --2- 172.21.15.45:0/2380842472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea241026b0 0x7fea2419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 >> 172.21.15.45:0/2380842472 conn(0x7fea24078d30 msgr2=0x7fea2410d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.515+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 shutdown_connections 2024-03-12T05:09:43.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:43.516+0000 7fea28eb8700 1 -- 172.21.15.45:0/2380842472 wait complete. 2024-03-12T05:09:44.357 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2380842472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:09:44.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:44 smithi045 ceph-mon[32174]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:09:44.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2380842472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:09:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:44 smithi130 ceph-mon[35398]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:09:44.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: pgmap v315: 97 pgs: 97 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:09:44.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:09:44.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-12T05:09:44.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-12T05:09:44.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9, 4, 6]}]': finished 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2380842472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:09:44.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:44 smithi116 ceph-mon[35486]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:09:44.522 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645334 got 309237645341 for osd.10 2024-03-12T05:09:44.522 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 -- 172.21.15.45:0/2942871253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580100ba0 msgr2=0x7fc580101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:44.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 --2- 172.21.15.45:0/2942871253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580100ba0 0x7fc580101020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc5740099c0 tx=0x7fc574009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 -- 172.21.15.45:0/2942871253 shutdown_connections 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 --2- 172.21.15.45:0/2942871253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 0x7fc58010daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 --2- 172.21.15.45:0/2942871253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580102b00 0x7fc580102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 --2- 172.21.15.45:0/2942871253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580100ba0 0x7fc580101020 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 -- 172.21.15.45:0/2942871253 >> 172.21.15.45:0/2942871253 conn(0x7fc580076b90 msgr2=0x7fc580076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 -- 172.21.15.45:0/2942871253 shutdown_connections 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 -- 172.21.15.45:0/2942871253 wait complete. 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.918+0000 7fc587ed3700 1 Processor -- start 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 -- start start 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580100ba0 0x7fc580196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:44.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 0x7fc5801972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801a6490 con 0x7fc580102b00 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801a65d0 con 0x7fc580101560 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc587ed3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801a6790 con 0x7fc580100ba0 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46728/0 (socket says 172.21.15.45:46728) 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 -- 172.21.15.45:0/1552827468 learned_addr learned my addr 172.21.15.45:0/1552827468 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc585c6f700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580100ba0 0x7fc580196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc58546e700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 0x7fc5801972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:44.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 -- 172.21.15.45:0/1552827468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580100ba0 msgr2=0x7fc580196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580100ba0 0x7fc580196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 -- 172.21.15.45:0/1552827468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 msgr2=0x7fc5801972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 0x7fc5801972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 -- 172.21.15.45:0/1552827468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740095f0 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc586470700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc57800b550 tx=0x7fc57800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.919+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578011830 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.920+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc578011e40 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.920+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578010de0 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.920+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5801a6a70 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.920+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5801a6ef0 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.921+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580109240 con 0x7fc580102b00 2024-03-12T05:09:44.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.922+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc578011990 con 0x7fc580102b00 2024-03-12T05:09:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.922+0000 7fc57effd700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 0x7fc5680a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.922+0000 7fc585c6f700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 0x7fc5680a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.922+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fc5780cd1b0 con 0x7fc580102b00 2024-03-12T05:09:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.923+0000 7fc585c6f700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 0x7fc5680a4010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f440 tx=0x7fc5700079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:44.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:44.926+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc578096380 con 0x7fc580102b00 2024-03-12T05:09:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.133+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc580068930 con 0x7fc580102b00 2024-03-12T05:09:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.134+0000 7fc57effd700 1 -- 172.21.15.45:0/1552827468 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc578099a40 con 0x7fc580102b00 2024-03-12T05:09:45.132 INFO:teuthology.orchestra.run.smithi045.stdout:163208757305 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 msgr2=0x7fc5680a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 0x7fc5680a4010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f440 tx=0x7fc5700079e0 comp rx=0 tx=0).stop 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 msgr2=0x7fc580197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc57800b550 tx=0x7fc57800b860 comp rx=0 tx=0).stop 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 shutdown_connections 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fc5680a1b50 0x7fc5680a4010 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc580102b00 0x7fc580197800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc580100ba0 0x7fc580196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 --2- 172.21.15.45:0/1552827468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc580101560 0x7fc5801972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 >> 172.21.15.45:0/1552827468 conn(0x7fc580076b90 msgr2=0x7fc5800ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 shutdown_connections 2024-03-12T05:09:45.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:45.136+0000 7fc587ed3700 1 -- 172.21.15.45:0/1552827468 wait complete. 2024-03-12T05:09:45.353 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:45 smithi045 ceph-mon[32174]: pgmap v318: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:45.376 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574924 for osd.1 2024-03-12T05:09:45.376 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:45.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:45 smithi130 ceph-mon[35398]: pgmap v318: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:45.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:45 smithi116 ceph-mon[35486]: pgmap v318: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 -- 172.21.15.45:0/16777633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 msgr2=0x7f367810eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 --2- 172.21.15.45:0/16777633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f367810eb70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3670009990 tx=0x7f3670009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 -- 172.21.15.45:0/16777633 shutdown_connections 2024-03-12T05:09:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 --2- 172.21.15.45:0/16777633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f367810eb70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 --2- 172.21.15.45:0/16777633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806a410 0x7f367806a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 --2- 172.21.15.45:0/16777633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806adc0 0x7f36781068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 -- 172.21.15.45:0/16777633 >> 172.21.15.45:0/16777633 conn(0x7f3678076bc0 msgr2=0x7f3678076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 -- 172.21.15.45:0/16777633 shutdown_connections 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 -- 172.21.15.45:0/16777633 wait complete. 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.335+0000 7f367f784700 1 Processor -- start 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 -- start start 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806a410 0x7f3678100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806adc0 0x7f36781011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36781a6470 con 0x7f3678106ff0 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36781a65e0 con 0x7f367806adc0 2024-03-12T05:09:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36781a67a0 con 0x7f367806a410 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46652/0 (socket says 172.21.15.45:46652) 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 -- 172.21.15.45:0/4269599344 learned_addr learned my addr 172.21.15.45:0/4269599344 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367d520700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806a410 0x7f3678100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367cd1f700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806adc0 0x7f36781011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 -- 172.21.15.45:0/4269599344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806a410 msgr2=0x7f3678100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806a410 0x7f3678100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 -- 172.21.15.45:0/4269599344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806adc0 msgr2=0x7f36781011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806adc0 0x7f36781011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 -- 172.21.15.45:0/4269599344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36700095f0 con 0x7f3678106ff0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367dd21700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3670004730 tx=0x7f3670004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367001d070 con 0x7f3678106ff0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36781a6a20 con 0x7f3678106ff0 2024-03-12T05:09:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3670022470 con 0x7f3678106ff0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.336+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670017650 con 0x7f3678106ff0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.337+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36781a6e90 con 0x7f3678106ff0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.337+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367810a1e0 con 0x7f3678106ff0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.339+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f36700225e0 con 0x7f3678106ff0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.339+0000 7f36767fc700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 0x7f36600a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stdout:287762808867 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.339+0000 7f367d520700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 0x7f36600a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:46.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.339+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f36700ce670 con 0x7f3678106ff0 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.340+0000 7f367d520700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 0x7f36600a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f366800f540 tx=0x7f366800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.344+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3670097960 con 0x7f3678106ff0 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.563+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f3678068930 con 0x7f3678106ff0 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.564+0000 7f36767fc700 1 -- 172.21.15.45:0/4269599344 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f367009b020 con 0x7f3678106ff0 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 msgr2=0x7f36600a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 0x7f36600a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f366800f540 tx=0x7f366800b040 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 msgr2=0x7f36781016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3670004730 tx=0x7f3670004760 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 shutdown_connections 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36600a1b50 0x7f36600a4010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3678106ff0 0x7f36781016f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f367806a410 0x7f3678100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 --2- 172.21.15.45:0/4269599344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367806adc0 0x7f36781011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 >> 172.21.15.45:0/4269599344 conn(0x7f3678076bc0 msgr2=0x7f36780fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:46.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 shutdown_connections 2024-03-12T05:09:46.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:38.566+0000 7f367f784700 1 -- 172.21.15.45:0/4269599344 wait complete. 2024-03-12T05:09:46.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:46 smithi130 ceph-mon[35398]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T05:09:46.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:46 smithi130 ceph-mon[35398]: Health check failed: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-03-12T05:09:46.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1552827468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:09:46.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:46 smithi116 ceph-mon[35486]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T05:09:46.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:46 smithi116 ceph-mon[35486]: Health check failed: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-03-12T05:09:46.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1552827468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:09:46.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:46 smithi045 ceph-mon[32174]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-12T05:09:46.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:46 smithi045 ceph-mon[32174]: Health check failed: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-03-12T05:09:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1552827468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:09:46.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808862 got 287762808867 for osd.9 2024-03-12T05:09:46.836 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:47.442 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:47 smithi045 ceph-mon[32174]: pgmap v319: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 -- 172.21.15.45:0/328814299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04106f90 msgr2=0x7fae0410eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 --2- 172.21.15.45:0/328814299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04106f90 0x7fae0410eb60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009990 tx=0x7fadfc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 -- 172.21.15.45:0/328814299 shutdown_connections 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 --2- 172.21.15.45:0/328814299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04106f90 0x7fae0410eb60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 --2- 172.21.15.45:0/328814299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04101eb0 0x7fae04102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 --2- 172.21.15.45:0/328814299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 0x7fae041068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 -- 172.21.15.45:0/328814299 >> 172.21.15.45:0/328814299 conn(0x7fae040fd720 msgr2=0x7fae040ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 -- 172.21.15.45:0/328814299 shutdown_connections 2024-03-12T05:09:47.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.454+0000 7fae0be3f700 1 -- 172.21.15.45:0/328814299 wait complete. 2024-03-12T05:09:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 Processor -- start 2024-03-12T05:09:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 -- start start 2024-03-12T05:09:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:47.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 0x7fae04198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 0x7fae0419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0419cf20 con 0x7fae04101eb0 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0419d090 con 0x7fae04106f90 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae0be3f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0419d250 con 0x7fae04102860 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae09bdb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae09bdb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42966/0 (socket says 172.21.15.45:42966) 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.455+0000 7fae09bdb700 1 -- 172.21.15.45:0/1776008382 learned_addr learned my addr 172.21.15.45:0/1776008382 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae093da700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 0x7fae04198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae0a3dc700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 0x7fae0419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 -- 172.21.15.45:0/1776008382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 msgr2=0x7fae04198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 0x7fae04198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 -- 172.21.15.45:0/1776008382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 msgr2=0x7fae0419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:47.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 0x7fae0419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 -- 172.21.15.45:0/1776008382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc0095f0 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae0a3dc700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 0x7fae0419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae09bdb700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadf40060b0 tx=0x7fadf400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400ddd0 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0419d530 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fadf400f040 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4014670 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.456+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae041a4d90 con 0x7fae04101eb0 2024-03-12T05:09:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.458+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0410a2b0 con 0x7fae04101eb0 2024-03-12T05:09:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.459+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fadf4015400 con 0x7fae04101eb0 2024-03-12T05:09:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.460+0000 7fae02ffd700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 0x7fadec0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.460+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fadf40cd420 con 0x7fae04101eb0 2024-03-12T05:09:47.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.460+0000 7fae093da700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 0x7fadec0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:47.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.461+0000 7fae093da700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 0x7fadec0a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae04100f10 tx=0x7fadf800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.464+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadf40964e0 con 0x7fae04101eb0 2024-03-12T05:09:47.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:47 smithi130 ceph-mon[35398]: pgmap v319: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:47.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:47 smithi116 ceph-mon[35486]: pgmap v319: 97 pgs: 1 activating+degraded, 1 peering, 95 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 0 objects/s recovering 2024-03-12T05:09:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.675+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fae04068930 con 0x7fae04101eb0 2024-03-12T05:09:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.676+0000 7fae02ffd700 1 -- 172.21.15.45:0/1776008382 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fadf4099ba0 con 0x7fae04101eb0 2024-03-12T05:09:47.674 INFO:teuthology.orchestra.run.smithi045.stdout:141733920830 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 msgr2=0x7fadec0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 0x7fadec0a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae04100f10 tx=0x7fadf800b410 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 msgr2=0x7fae04198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadf40060b0 tx=0x7fadf400d3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 shutdown_connections 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadec0a1b50 0x7fadec0a4010 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04106f90 0x7fae0419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae04101eb0 0x7fae04198040 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 --2- 172.21.15.45:0/1776008382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae04102860 0x7fae04198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 >> 172.21.15.45:0/1776008382 conn(0x7fae040fd720 msgr2=0x7fae040ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 shutdown_connections 2024-03-12T05:09:47.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.678+0000 7fae0be3f700 1 -- 172.21.15.45:0/1776008382 wait complete. 2024-03-12T05:09:48.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 -- 172.21.15.45:0/3237572745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640101cc0 msgr2=0x7f5640102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:48.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 --2- 172.21.15.45:0/3237572745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640101cc0 0x7f5640102140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f56300099c0 tx=0x7f5630009cd0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 -- 172.21.15.45:0/3237572745 shutdown_connections 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 --2- 172.21.15.45:0/3237572745 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f564010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 --2- 172.21.15.45:0/3237572745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640107ca0 0x7f5640108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 --2- 172.21.15.45:0/3237572745 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640101cc0 0x7f5640102140 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 -- 172.21.15.45:0/3237572745 >> 172.21.15.45:0/3237572745 conn(0x7f5640078da0 msgr2=0x7f56400791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 -- 172.21.15.45:0/3237572745 shutdown_connections 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.011+0000 7f5645f7b700 1 -- 172.21.15.45:0/3237572745 wait complete. 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 Processor -- start 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 -- start start 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640101cc0 0x7f5640197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f5640198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564019ceb0 con 0x7f5640107ca0 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564019d020 con 0x7f5640102680 2024-03-12T05:09:48.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f5645f7b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564019d1e0 con 0x7f5640101cc0 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f563ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f563ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43000/0 (socket says 172.21.15.45:43000) 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f563ffff700 1 -- 172.21.15.45:0/151914230 learned_addr learned my addr 172.21.15.45:0/151914230 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f563f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640101cc0 0x7f5640197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.012+0000 7f563effd700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f5640198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 -- 172.21.15.45:0/151914230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640101cc0 msgr2=0x7f5640197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640101cc0 0x7f5640197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 -- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 msgr2=0x7f564019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 -- 172.21.15.45:0/151914230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56300095f0 con 0x7f5640102680 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563ffff700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563effd700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f5640198510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5630009f50 tx=0x7f56300057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563001d070 con 0x7f5640102680 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564019d460 con 0x7f5640102680 2024-03-12T05:09:48.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f563000fc20 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630022d40 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.013+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56401a4d20 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.014+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564010a180 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.015+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5630022470 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.016+0000 7f563cff9700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 0x7f56280a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.016+0000 7f563f7fe700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 0x7f56280a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.016+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f56300ceff0 con 0x7f5640102680 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.017+0000 7f563f7fe700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 0x7f56280a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f563400ac50 tx=0x7f563400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:48.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.020+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5630098140 con 0x7f5640102680 2024-03-12T05:09:48.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.230+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5640199330 con 0x7f5640102680 2024-03-12T05:09:48.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.231+0000 7f563cff9700 1 -- 172.21.15.45:0/151914230 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f563009b800 con 0x7f5640102680 2024-03-12T05:09:48.228 INFO:teuthology.orchestra.run.smithi045.stdout:107374182467 2024-03-12T05:09:48.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 msgr2=0x7f56280a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 0x7f56280a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f563400ac50 tx=0x7f563400a380 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 msgr2=0x7f5640198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f5640198510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5630009f50 tx=0x7f56300057b0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 shutdown_connections 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56280a1b50 0x7f56280a4010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5640107ca0 0x7f564019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5640101cc0 0x7f5640197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 --2- 172.21.15.45:0/151914230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5640102680 0x7f5640198510 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 >> 172.21.15.45:0/151914230 conn(0x7f5640078da0 msgr2=0x7f56401008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 shutdown_connections 2024-03-12T05:09:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.233+0000 7f5645f7b700 1 -- 172.21.15.45:0/151914230 wait complete. 2024-03-12T05:09:48.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:48 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1776008382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:09:48.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:48 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2791139026' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:09:48.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:48 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1776008382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:09:48.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:48 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2791139026' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:09:48.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:48 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1776008382' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:09:48.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:48 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2791139026' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:09:48.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182462 got 107374182467 for osd.3 2024-03-12T05:09:48.880 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 -- 172.21.15.45:0/3417028664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 msgr2=0x7f60cc10eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 --2- 172.21.15.45:0/3417028664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc10eac0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f60c400b0d0 tx=0x7f60c400b3e0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 -- 172.21.15.45:0/3417028664 shutdown_connections 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 --2- 172.21.15.45:0/3417028664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc10eac0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 --2- 172.21.15.45:0/3417028664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc107ca0 0x7f60cc108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 --2- 172.21.15.45:0/3417028664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc101cc0 0x7f60cc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.892+0000 7f60d4629700 1 -- 172.21.15.45:0/3417028664 >> 172.21.15.45:0/3417028664 conn(0x7f60cc078da0 msgr2=0x7f60cc0791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- 172.21.15.45:0/3417028664 shutdown_connections 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- 172.21.15.45:0/3417028664 wait complete. 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 Processor -- start 2024-03-12T05:09:49.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- start start 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc101cc0 0x7f60cc19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 0x7f60cc19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc19f2b0 con 0x7f60cc102680 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc19f420 con 0x7f60cc101cc0 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d4629700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc19f5e0 con 0x7f60cc107ca0 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d1bc4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d1bc4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42980/0 (socket says 172.21.15.45:42980) 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.893+0000 7f60d1bc4700 1 -- 172.21.15.45:0/2791139026 learned_addr learned my addr 172.21.15.45:0/2791139026 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:49.233 INFO:teuthology.orchestra.run.smithi045.stdout:219043332143 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d23c5700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc101cc0 0x7f60cc19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d2bc6700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 0x7f60cc19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 -- 172.21.15.45:0/2791139026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 msgr2=0x7f60cc19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 0x7f60cc19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 -- 172.21.15.45:0/2791139026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc101cc0 msgr2=0x7f60cc19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc101cc0 0x7f60cc19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 -- 172.21.15.45:0/2791139026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c4009d00 con 0x7f60cc102680 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d2bc6700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 0x7f60cc19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d1bc4700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00d950 tx=0x7f60bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc009940 con 0x7f60cc102680 2024-03-12T05:09:49.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f60bc010460 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc19f8c0 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00f620 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.894+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc1a7120 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.895+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc068930 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.896+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f60bc0105d0 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.897+0000 7f60c37fe700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 0x7f60b80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.897+0000 7f60d23c5700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 0x7f60b80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.897+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f60bc0cd320 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.897+0000 7f60d23c5700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 0x7f60b80a4090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f60c8005950 tx=0x7f60c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:47.900+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f60bc0963e0 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.110+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f60cc111bc0 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.111+0000 7f60c37fe700 1 -- 172.21.15.45:0/2791139026 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f60bc020090 con 0x7f60cc102680 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 msgr2=0x7f60b80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 0x7f60b80a4090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f60c8005950 tx=0x7f60c800b410 comp rx=0 tx=0).stop 2024-03-12T05:09:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 msgr2=0x7f60cc19a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00d950 tx=0x7f60bc00dc60 comp rx=0 tx=0).stop 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 shutdown_connections 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f60b80a1bd0 0x7f60b80a4090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60cc107ca0 0x7f60cc19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60cc101cc0 0x7f60cc19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 --2- 172.21.15.45:0/2791139026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60cc102680 0x7f60cc19a910 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 >> 172.21.15.45:0/2791139026 conn(0x7f60cc078da0 msgr2=0x7f60cc074130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 shutdown_connections 2024-03-12T05:09:49.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:48.113+0000 7f60d4629700 1 -- 172.21.15.45:0/2791139026 wait complete. 2024-03-12T05:09:49.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:49 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/151914230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:09:49.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:49 smithi130 ceph-mon[35398]: pgmap v320: 97 pgs: 1 activating+degraded, 96 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 1 objects/s recovering 2024-03-12T05:09:49.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:49 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/151914230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:09:49.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:49 smithi116 ceph-mon[35486]: pgmap v320: 97 pgs: 1 activating+degraded, 96 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 1 objects/s recovering 2024-03-12T05:09:49.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:49 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/151914230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:09:49.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:49 smithi045 ceph-mon[32174]: pgmap v320: 97 pgs: 1 activating+degraded, 96 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 2/306 objects degraded (0.654%); 83 B/s, 1 objects/s recovering 2024-03-12T05:09:49.848 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332143 for osd.7 2024-03-12T05:09:49.848 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:50.427 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:50 smithi045 ceph-mon[32174]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T05:09:50.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:50 smithi130 ceph-mon[35398]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T05:09:50.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:50 smithi116 ceph-mon[35486]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-12T05:09:50.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757299 got 163208757305 for osd.5 2024-03-12T05:09:50.685 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:51.358 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:51 smithi045 ceph-mon[32174]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 83 B/s, 1 objects/s recovering 2024-03-12T05:09:51.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920830 for osd.4 2024-03-12T05:09:51.388 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:51.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:51 smithi130 ceph-mon[35398]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 83 B/s, 1 objects/s recovering 2024-03-12T05:09:51.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:51 smithi116 ceph-mon[35486]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 83 B/s, 1 objects/s recovering 2024-03-12T05:09:51.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 -- 172.21.15.45:0/703314841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 msgr2=0x7f5b40106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:51.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 --2- 172.21.15.45:0/703314841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b40106f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000b0a0 tx=0x7f5b3000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 -- 172.21.15.45:0/703314841 shutdown_connections 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 --2- 172.21.15.45:0/703314841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 0x7f5b4010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 --2- 172.21.15.45:0/703314841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b40106f60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 --2- 172.21.15.45:0/703314841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 0x7f5b40100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 -- 172.21.15.45:0/703314841 >> 172.21.15.45:0/703314841 conn(0x7f5b40077aa0 msgr2=0x7f5b40077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 -- 172.21.15.45:0/703314841 shutdown_connections 2024-03-12T05:09:51.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.575+0000 7f5b450f8700 1 -- 172.21.15.45:0/703314841 wait complete. 2024-03-12T05:09:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.576+0000 7f5b450f8700 1 Processor -- start 2024-03-12T05:09:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.576+0000 7f5b450f8700 1 -- start start 2024-03-12T05:09:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 0x7f5b40198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 0x7f5b40198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4019d000 con 0x7f5b40106b80 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4019d170 con 0x7f5b40100b70 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b450f8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4019d330 con 0x7f5b40101530 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43014/0 (socket says 172.21.15.45:43014) 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 -- 172.21.15.45:0/3279514452 learned_addr learned my addr 172.21.15.45:0/3279514452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3ed9d700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 0x7f5b40198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3e59c700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 0x7f5b40198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 -- 172.21.15.45:0/3279514452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 msgr2=0x7f5b40198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 0x7f5b40198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 -- 172.21.15.45:0/3279514452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 msgr2=0x7f5b40198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 0x7f5b40198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 -- 172.21.15.45:0/3279514452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30009d00 con 0x7f5b40106b80 2024-03-12T05:09:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b3f59e700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800e940 tx=0x7f5b3800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b380097b0 con 0x7f5b40106b80 2024-03-12T05:09:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.577+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5b38005020 con 0x7f5b40106b80 2024-03-12T05:09:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.578+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800cc80 con 0x7f5b40106b80 2024-03-12T05:09:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.578+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4019d610 con 0x7f5b40106b80 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.578+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b401a4e70 con 0x7f5b40106b80 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.579+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b40068930 con 0x7f5b40106b80 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.580+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5b38005190 con 0x7f5b40106b80 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.580+0000 7f5b37fff700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 0x7f5b2c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.581+0000 7f5b3ed9d700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 0x7f5b2c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.581+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5b38014070 con 0x7f5b40106b80 2024-03-12T05:09:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.581+0000 7f5b3ed9d700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 0x7f5b2c0a4010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009fd0 tx=0x7f5b3000b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.585+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b38096800 con 0x7f5b40106b80 2024-03-12T05:09:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.795+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5b40199480 con 0x7f5b40106b80 2024-03-12T05:09:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.796+0000 7f5b37fff700 1 -- 172.21.15.45:0/3279514452 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5b38099ec0 con 0x7f5b40106b80 2024-03-12T05:09:51.793 INFO:teuthology.orchestra.run.smithi045.stdout:77309411401 2024-03-12T05:09:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 msgr2=0x7f5b2c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 0x7f5b2c0a4010 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009fd0 tx=0x7f5b3000b720 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 msgr2=0x7f5b4019c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800e940 tx=0x7f5b3800ec50 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 shutdown_connections 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5b2c0a1b50 0x7f5b2c0a4010 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b40106b80 0x7f5b4019c9f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b40100b70 0x7f5b40198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 --2- 172.21.15.45:0/3279514452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b40101530 0x7f5b40198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 >> 172.21.15.45:0/3279514452 conn(0x7f5b40077aa0 msgr2=0x7f5b400ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 shutdown_connections 2024-03-12T05:09:51.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:51.798+0000 7f5b450f8700 1 -- 172.21.15.45:0/3279514452 wait complete. 2024-03-12T05:09:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:52 smithi130 ceph-mon[35398]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded) 2024-03-12T05:09:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:52 smithi130 ceph-mon[35398]: Cluster is now healthy 2024-03-12T05:09:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:52 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3279514452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:09:52.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:52 smithi045 ceph-mon[32174]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded) 2024-03-12T05:09:52.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:52 smithi045 ceph-mon[32174]: Cluster is now healthy 2024-03-12T05:09:52.470 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:52 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3279514452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:09:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:52 smithi116 ceph-mon[35486]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/306 objects degraded (0.654%), 1 pg degraded) 2024-03-12T05:09:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:52 smithi116 ceph-mon[35486]: Cluster is now healthy 2024-03-12T05:09:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:52 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3279514452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:09:52.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411401 for osd.2 2024-03-12T05:09:52.604 DEBUG:teuthology.parallel:result is None 2024-03-12T05:09:52.604 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T05:09:52.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T05:09:52.998 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:53.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:53 smithi045 ceph-mon[32174]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 1 objects/s recovering 2024-03-12T05:09:53.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:53 smithi130 ceph-mon[35398]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 1 objects/s recovering 2024-03-12T05:09:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:53 smithi116 ceph-mon[35486]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 1 objects/s recovering 2024-03-12T05:09:54.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 -- 172.21.15.45:0/1860744230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29401016b0 msgr2=0x7f2940101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:54.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 --2- 172.21.15.45:0/1860744230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29401016b0 0x7f2940101b30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2930009990 tx=0x7f2930009ca0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 -- 172.21.15.45:0/1860744230 shutdown_connections 2024-03-12T05:09:54.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 --2- 172.21.15.45:0/1860744230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2940102070 0x7f294010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 --2- 172.21.15.45:0/1860744230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 0x7f2940105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 --2- 172.21.15.45:0/1860744230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29401016b0 0x7f2940101b30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 -- 172.21.15.45:0/1860744230 >> 172.21.15.45:0/1860744230 conn(0x7f2940075ea0 msgr2=0x7f29400762b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 -- 172.21.15.45:0/1860744230 shutdown_connections 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.754+0000 7f29484de700 1 -- 172.21.15.45:0/1860744230 wait complete. 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 Processor -- start 2024-03-12T05:09:54.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 -- start start 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 0x7f29401980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 0x7f294019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294019cfc0 con 0x7f2940102070 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294019d130 con 0x7f2940105ba0 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f29484de700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294019d2f0 con 0x7f29401016b0 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f2945a79700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f2945a79700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43036/0 (socket says 172.21.15.45:43036) 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f2945a79700 1 -- 172.21.15.45:0/209635592 learned_addr learned my addr 172.21.15.45:0/209635592 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f2946a7b700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 0x7f294019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.755+0000 7f294627a700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 0x7f29401980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:54.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 -- 172.21.15.45:0/209635592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 msgr2=0x7f29401980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 0x7f29401980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 -- 172.21.15.45:0/209635592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 msgr2=0x7f294019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 0x7f294019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 -- 172.21.15.45:0/209635592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300095f0 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f294627a700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 0x7f29401980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f2945a79700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f293000f6c0 tx=0x7f293000f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293001d070 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294019d570 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f293000bdc0 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930022cd0 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.756+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29401a4e30 con 0x7f2940102070 2024-03-12T05:09:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.757+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2940068930 con 0x7f2940102070 2024-03-12T05:09:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.758+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2930022470 con 0x7f2940102070 2024-03-12T05:09:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.759+0000 7f29377fe700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 0x7f292c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.759+0000 7f294627a700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 0x7f292c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.759+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f29300cf150 con 0x7f2940102070 2024-03-12T05:09:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.759+0000 7f294627a700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 0x7f292c0a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f293c007900 tx=0x7f293c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.762+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2930098240 con 0x7f2940102070 2024-03-12T05:09:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.978+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2940111c50 con 0x7f292c0a1b50 2024-03-12T05:09:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.986+0000 7f29377fe700 1 -- 172.21.15.45:0/209635592 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258989 (secure 0 0 0) 0x7f2940111c50 con 0x7f292c0a1b50 2024-03-12T05:09:54.987 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 msgr2=0x7f292c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 0x7f292c0a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f293c007900 tx=0x7f293c008040 comp rx=0 tx=0).stop 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 msgr2=0x7f2940198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f293000f6c0 tx=0x7f293000f6f0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 shutdown_connections 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f292c0a1b50 0x7f292c0a4010 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2940105ba0 0x7f294019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29401016b0 0x7f29401980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 --2- 172.21.15.45:0/209635592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2940102070 0x7f2940198620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 >> 172.21.15.45:0/209635592 conn(0x7f2940075ea0 msgr2=0x7f29400fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 shutdown_connections 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:54.990+0000 7f29484de700 1 -- 172.21.15.45:0/209635592 wait complete. 2024-03-12T05:09:54.990 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:09:55.656 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-12T05:09:53.117547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657109,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":831,"num_read_kb":536,"num_write":419,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181408,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369260,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572564,"statfs":{"total":1151940427776,"available":1151562305536,"internally_reserved":0,"allocated":11980800,"data_stored":7434459,"data_compressed":113928,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":95,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.995474"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128692+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:09:44.128692+0000","last_peered":"2024-03-12T05:09:44.128692+0000","last_clean":"2024-03-12T05:09:44.128692+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:09:44.128692+0000","last_undegraded":"2024-03-12T05:09:44.128692+0000","last_fullsized":"2024-03-12T05:09:44.128692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124710+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:09:44.124710+0000","last_peered":"2024-03-12T05:09:44.124710+0000","last_clean":"2024-03-12T05:09:44.124710+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:09:44.124710+0000","last_undegraded":"2024-03-12T05:09:44.124710+0000","last_fullsized":"2024-03-12T05:09:44.124710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124617+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:09:44.124617+0000","last_peered":"2024-03-12T05:09:44.124617+0000","last_clean":"2024-03-12T05:09:44.124617+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:09:44.124617+0000","last_undegraded":"2024-03-12T05:09:44.124617+0000","last_fullsized":"2024-03-12T05:09:44.124617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310504+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:09:44.310504+0000","last_peered":"2024-03-12T05:09:44.310504+0000","last_clean":"2024-03-12T05:09:44.310504+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:09:44.310504+0000","last_undegraded":"2024-03-12T05:09:44.310504+0000","last_fullsized":"2024-03-12T05:09:44.310504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129408+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:09:44.129408+0000","last_peered":"2024-03-12T05:09:44.129408+0000","last_clean":"2024-03-12T05:09:44.129408+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:09:44.129408+0000","last_undegraded":"2024-03-12T05:09:44.129408+0000","last_fullsized":"2024-03-12T05:09:44.129408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381264+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:09:44.381264+0000","last_peered":"2024-03-12T05:09:44.381264+0000","last_clean":"2024-03-12T05:09:44.381264+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:09:44.381264+0000","last_undegraded":"2024-03-12T05:09:44.381264+0000","last_fullsized":"2024-03-12T05:09:44.381264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125325+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:09:44.125325+0000","last_peered":"2024-03-12T05:09:44.125325+0000","last_clean":"2024-03-12T05:09:44.125325+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:09:44.125325+0000","last_undegraded":"2024-03-12T05:09:44.125325+0000","last_fullsized":"2024-03-12T05:09:44.125325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381417+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:09:44.381417+0000","last_peered":"2024-03-12T05:09:44.381417+0000","last_clean":"2024-03-12T05:09:44.381417+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:09:44.381417+0000","last_undegraded":"2024-03-12T05:09:44.381417+0000","last_fullsized":"2024-03-12T05:09:44.381417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127078+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:09:44.127078+0000","last_peered":"2024-03-12T05:09:44.127078+0000","last_clean":"2024-03-12T05:09:44.127078+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:09:44.127078+0000","last_undegraded":"2024-03-12T05:09:44.127078+0000","last_fullsized":"2024-03-12T05:09:44.127078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381586+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:09:44.381586+0000","last_peered":"2024-03-12T05:09:44.381586+0000","last_clean":"2024-03-12T05:09:44.381586+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:09:44.381586+0000","last_undegraded":"2024-03-12T05:09:44.381586+0000","last_fullsized":"2024-03-12T05:09:44.381586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125698+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:09:44.125698+0000","last_peered":"2024-03-12T05:09:44.125698+0000","last_clean":"2024-03-12T05:09:44.125698+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:09:44.125698+0000","last_undegraded":"2024-03-12T05:09:44.125698+0000","last_fullsized":"2024-03-12T05:09:44.125698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310513+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:09:44.310513+0000","last_peered":"2024-03-12T05:09:44.310513+0000","last_clean":"2024-03-12T05:09:44.310513+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:09:44.310513+0000","last_undegraded":"2024-03-12T05:09:44.310513+0000","last_fullsized":"2024-03-12T05:09:44.310513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126454+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:09:44.126454+0000","last_peered":"2024-03-12T05:09:44.126454+0000","last_clean":"2024-03-12T05:09:44.126454+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:09:44.126454+0000","last_undegraded":"2024-03-12T05:09:44.126454+0000","last_fullsized":"2024-03-12T05:09:44.126454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124416+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:09:44.124416+0000","last_peered":"2024-03-12T05:09:44.124416+0000","last_clean":"2024-03-12T05:09:44.124416+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:09:44.124416+0000","last_undegraded":"2024-03-12T05:09:44.124416+0000","last_fullsized":"2024-03-12T05:09:44.124416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124334+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:09:44.124334+0000","last_peered":"2024-03-12T05:09:44.124334+0000","last_clean":"2024-03-12T05:09:44.124334+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:09:44.124334+0000","last_undegraded":"2024-03-12T05:09:44.124334+0000","last_fullsized":"2024-03-12T05:09:44.124334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126896+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:09:44.126896+0000","last_peered":"2024-03-12T05:09:44.126896+0000","last_clean":"2024-03-12T05:09:44.126896+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:09:44.126896+0000","last_undegraded":"2024-03-12T05:09:44.126896+0000","last_fullsized":"2024-03-12T05:09:44.126896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128810+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:09:44.128810+0000","last_peered":"2024-03-12T05:09:44.128810+0000","last_clean":"2024-03-12T05:09:44.128810+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:09:44.128810+0000","last_undegraded":"2024-03-12T05:09:44.128810+0000","last_fullsized":"2024-03-12T05:09:44.128810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128345+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:09:44.128345+0000","last_peered":"2024-03-12T05:09:44.128345+0000","last_clean":"2024-03-12T05:09:44.128345+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:09:44.128345+0000","last_undegraded":"2024-03-12T05:09:44.128345+0000","last_fullsized":"2024-03-12T05:09:44.128345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131740+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:09:44.131740+0000","last_peered":"2024-03-12T05:09:44.131740+0000","last_clean":"2024-03-12T05:09:44.131740+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:09:44.131740+0000","last_undegraded":"2024-03-12T05:09:44.131740+0000","last_fullsized":"2024-03-12T05:09:44.131740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":119,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:48.135729+0000","last_change":"2024-03-12T05:09:44.142398+0000","last_active":"2024-03-12T05:09:48.135729+0000","last_peered":"2024-03-12T05:09:48.135729+0000","last_clean":"2024-03-12T05:09:48.135729+0000","last_became_active":"2024-03-12T05:09:44.129576+0000","last_became_peered":"2024-03-12T05:09:44.129576+0000","last_unstale":"2024-03-12T05:09:48.135729+0000","last_undegraded":"2024-03-12T05:09:48.135729+0000","last_fullsized":"2024-03-12T05:09:48.135729+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:32.791210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":65,"num_read_kb":65,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.135418+0000","last_change":"2024-03-12T05:09:44.135049+0000","last_active":"2024-03-12T05:09:44.135418+0000","last_peered":"2024-03-12T05:09:44.135418+0000","last_clean":"2024-03-12T05:09:44.135418+0000","last_became_active":"2024-03-12T05:09:44.134638+0000","last_became_peered":"2024-03-12T05:09:44.134638+0000","last_unstale":"2024-03-12T05:09:44.135418+0000","last_undegraded":"2024-03-12T05:09:44.135418+0000","last_fullsized":"2024-03-12T05:09:44.135418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:28:59.785062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381917+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:09:44.381917+0000","last_peered":"2024-03-12T05:09:44.381917+0000","last_clean":"2024-03-12T05:09:44.381917+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:09:44.381917+0000","last_undegraded":"2024-03-12T05:09:44.381917+0000","last_fullsized":"2024-03-12T05:09:44.381917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124824+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:09:44.124824+0000","last_peered":"2024-03-12T05:09:44.124824+0000","last_clean":"2024-03-12T05:09:44.124824+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:09:44.124824+0000","last_undegraded":"2024-03-12T05:09:44.124824+0000","last_fullsized":"2024-03-12T05:09:44.124824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128279+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:09:44.128279+0000","last_peered":"2024-03-12T05:09:44.128279+0000","last_clean":"2024-03-12T05:09:44.128279+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:09:44.128279+0000","last_undegraded":"2024-03-12T05:09:44.128279+0000","last_fullsized":"2024-03-12T05:09:44.128279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126669+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:09:44.126669+0000","last_peered":"2024-03-12T05:09:44.126669+0000","last_clean":"2024-03-12T05:09:44.126669+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:09:44.126669+0000","last_undegraded":"2024-03-12T05:09:44.126669+0000","last_fullsized":"2024-03-12T05:09:44.126669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128527+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:09:44.128527+0000","last_peered":"2024-03-12T05:09:44.128527+0000","last_clean":"2024-03-12T05:09:44.128527+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:09:44.128527+0000","last_undegraded":"2024-03-12T05:09:44.128527+0000","last_fullsized":"2024-03-12T05:09:44.128527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.133756+0000","last_change":"2024-03-12T05:09:44.133756+0000","last_active":"2024-03-12T05:09:44.133756+0000","last_peered":"2024-03-12T05:09:44.133756+0000","last_clean":"2024-03-12T05:09:44.133756+0000","last_became_active":"2024-03-12T05:09:44.133312+0000","last_became_peered":"2024-03-12T05:09:44.133312+0000","last_unstale":"2024-03-12T05:09:44.133756+0000","last_undegraded":"2024-03-12T05:09:44.133756+0000","last_fullsized":"2024-03-12T05:09:44.133756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:54:24.882257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:47.446032+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:09:47.446032+0000","last_peered":"2024-03-12T05:09:47.446032+0000","last_clean":"2024-03-12T05:09:47.446032+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:09:47.446032+0000","last_undegraded":"2024-03-12T05:09:47.446032+0000","last_fullsized":"2024-03-12T05:09:47.446032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128072+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:09:44.128072+0000","last_peered":"2024-03-12T05:09:44.128072+0000","last_clean":"2024-03-12T05:09:44.128072+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:09:44.128072+0000","last_undegraded":"2024-03-12T05:09:44.128072+0000","last_fullsized":"2024-03-12T05:09:44.128072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381204+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:09:44.381204+0000","last_peered":"2024-03-12T05:09:44.381204+0000","last_clean":"2024-03-12T05:09:44.381204+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:09:44.381204+0000","last_undegraded":"2024-03-12T05:09:44.381204+0000","last_fullsized":"2024-03-12T05:09:44.381204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129783+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:09:44.129783+0000","last_peered":"2024-03-12T05:09:44.129783+0000","last_clean":"2024-03-12T05:09:44.129783+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:09:44.129783+0000","last_undegraded":"2024-03-12T05:09:44.129783+0000","last_fullsized":"2024-03-12T05:09:44.129783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126104+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:09:44.126104+0000","last_peered":"2024-03-12T05:09:44.126104+0000","last_clean":"2024-03-12T05:09:44.126104+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:09:44.126104+0000","last_undegraded":"2024-03-12T05:09:44.126104+0000","last_fullsized":"2024-03-12T05:09:44.126104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381349+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:09:44.381349+0000","last_peered":"2024-03-12T05:09:44.381349+0000","last_clean":"2024-03-12T05:09:44.381349+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:09:44.381349+0000","last_undegraded":"2024-03-12T05:09:44.381349+0000","last_fullsized":"2024-03-12T05:09:44.381349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128856+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:09:44.128856+0000","last_peered":"2024-03-12T05:09:44.128856+0000","last_clean":"2024-03-12T05:09:44.128856+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:09:44.128856+0000","last_undegraded":"2024-03-12T05:09:44.128856+0000","last_fullsized":"2024-03-12T05:09:44.128856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127902+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:09:44.127902+0000","last_peered":"2024-03-12T05:09:44.127902+0000","last_clean":"2024-03-12T05:09:44.127902+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:09:44.127902+0000","last_undegraded":"2024-03-12T05:09:44.127902+0000","last_fullsized":"2024-03-12T05:09:44.127902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":155,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:48.446669+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:09:48.446669+0000","last_peered":"2024-03-12T05:09:48.446669+0000","last_clean":"2024-03-12T05:09:48.446669+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:09:48.446669+0000","last_undegraded":"2024-03-12T05:09:48.446669+0000","last_fullsized":"2024-03-12T05:09:48.446669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127231+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:09:44.127231+0000","last_peered":"2024-03-12T05:09:44.127231+0000","last_clean":"2024-03-12T05:09:44.127231+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:09:44.127231+0000","last_undegraded":"2024-03-12T05:09:44.127231+0000","last_fullsized":"2024-03-12T05:09:44.127231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.315313+0000","last_change":"2024-03-12T05:09:44.315313+0000","last_active":"2024-03-12T05:09:44.315313+0000","last_peered":"2024-03-12T05:09:44.315313+0000","last_clean":"2024-03-12T05:09:44.315313+0000","last_became_active":"2024-03-12T05:09:44.311453+0000","last_became_peered":"2024-03-12T05:09:44.311453+0000","last_unstale":"2024-03-12T05:09:44.315313+0000","last_undegraded":"2024-03-12T05:09:44.315313+0000","last_fullsized":"2024-03-12T05:09:44.315313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:21.022485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124802+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:09:44.124802+0000","last_peered":"2024-03-12T05:09:44.124802+0000","last_clean":"2024-03-12T05:09:44.124802+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:09:44.124802+0000","last_undegraded":"2024-03-12T05:09:44.124802+0000","last_fullsized":"2024-03-12T05:09:44.124802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132030+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:09:44.132030+0000","last_peered":"2024-03-12T05:09:44.132030+0000","last_clean":"2024-03-12T05:09:44.132030+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:09:44.132030+0000","last_undegraded":"2024-03-12T05:09:44.132030+0000","last_fullsized":"2024-03-12T05:09:44.132030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126851+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:09:44.126851+0000","last_peered":"2024-03-12T05:09:44.126851+0000","last_clean":"2024-03-12T05:09:44.126851+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:09:44.126851+0000","last_undegraded":"2024-03-12T05:09:44.126851+0000","last_fullsized":"2024-03-12T05:09:44.126851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'28","reported_seq":185,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:48.446426+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:09:48.446426+0000","last_peered":"2024-03-12T05:09:48.446426+0000","last_clean":"2024-03-12T05:09:48.446426+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:09:48.446426+0000","last_undegraded":"2024-03-12T05:09:48.446426+0000","last_fullsized":"2024-03-12T05:09:48.446426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":126,"num_write":30,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127947+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:09:44.127947+0000","last_peered":"2024-03-12T05:09:44.127947+0000","last_clean":"2024-03-12T05:09:44.127947+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:09:44.127947+0000","last_undegraded":"2024-03-12T05:09:44.127947+0000","last_fullsized":"2024-03-12T05:09:44.127947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131709+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:09:44.131709+0000","last_peered":"2024-03-12T05:09:44.131709+0000","last_clean":"2024-03-12T05:09:44.131709+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:09:44.131709+0000","last_undegraded":"2024-03-12T05:09:44.131709+0000","last_fullsized":"2024-03-12T05:09:44.131709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126153+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:09:44.126153+0000","last_peered":"2024-03-12T05:09:44.126153+0000","last_clean":"2024-03-12T05:09:44.126153+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:09:44.126153+0000","last_undegraded":"2024-03-12T05:09:44.126153+0000","last_fullsized":"2024-03-12T05:09:44.126153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128625+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:09:44.128625+0000","last_peered":"2024-03-12T05:09:44.128625+0000","last_clean":"2024-03-12T05:09:44.128625+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:09:44.128625+0000","last_undegraded":"2024-03-12T05:09:44.128625+0000","last_fullsized":"2024-03-12T05:09:44.128625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125826+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:09:44.125826+0000","last_peered":"2024-03-12T05:09:44.125826+0000","last_clean":"2024-03-12T05:09:44.125826+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:09:44.125826+0000","last_undegraded":"2024-03-12T05:09:44.125826+0000","last_fullsized":"2024-03-12T05:09:44.125826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:47.445624+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:09:47.445624+0000","last_peered":"2024-03-12T05:09:47.445624+0000","last_clean":"2024-03-12T05:09:47.445624+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:09:47.445624+0000","last_undegraded":"2024-03-12T05:09:47.445624+0000","last_fullsized":"2024-03-12T05:09:47.445624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125250+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:09:44.125250+0000","last_peered":"2024-03-12T05:09:44.125250+0000","last_clean":"2024-03-12T05:09:44.125250+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:09:44.125250+0000","last_undegraded":"2024-03-12T05:09:44.125250+0000","last_fullsized":"2024-03-12T05:09:44.125250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127773+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:09:44.127773+0000","last_peered":"2024-03-12T05:09:44.127773+0000","last_clean":"2024-03-12T05:09:44.127773+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:09:44.127773+0000","last_undegraded":"2024-03-12T05:09:44.127773+0000","last_fullsized":"2024-03-12T05:09:44.127773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129690+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:09:44.129690+0000","last_peered":"2024-03-12T05:09:44.129690+0000","last_clean":"2024-03-12T05:09:44.129690+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:09:44.129690+0000","last_undegraded":"2024-03-12T05:09:44.129690+0000","last_fullsized":"2024-03-12T05:09:44.129690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128902+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:09:44.128902+0000","last_peered":"2024-03-12T05:09:44.128902+0000","last_clean":"2024-03-12T05:09:44.128902+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:09:44.128902+0000","last_undegraded":"2024-03-12T05:09:44.128902+0000","last_fullsized":"2024-03-12T05:09:44.128902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128731+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:09:44.128731+0000","last_peered":"2024-03-12T05:09:44.128731+0000","last_clean":"2024-03-12T05:09:44.128731+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:09:44.128731+0000","last_undegraded":"2024-03-12T05:09:44.128731+0000","last_fullsized":"2024-03-12T05:09:44.128731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128017+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:09:44.128017+0000","last_peered":"2024-03-12T05:09:44.128017+0000","last_clean":"2024-03-12T05:09:44.128017+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:09:44.128017+0000","last_undegraded":"2024-03-12T05:09:44.128017+0000","last_fullsized":"2024-03-12T05:09:44.128017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381570+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:09:44.381570+0000","last_peered":"2024-03-12T05:09:44.381570+0000","last_clean":"2024-03-12T05:09:44.381570+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:09:44.381570+0000","last_undegraded":"2024-03-12T05:09:44.381570+0000","last_fullsized":"2024-03-12T05:09:44.381570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:48.136408+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:09:48.136408+0000","last_peered":"2024-03-12T05:09:48.136408+0000","last_clean":"2024-03-12T05:09:48.136408+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:09:48.136408+0000","last_undegraded":"2024-03-12T05:09:48.136408+0000","last_fullsized":"2024-03-12T05:09:48.136408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381394+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:09:44.381394+0000","last_peered":"2024-03-12T05:09:44.381394+0000","last_clean":"2024-03-12T05:09:44.381394+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:09:44.381394+0000","last_undegraded":"2024-03-12T05:09:44.381394+0000","last_fullsized":"2024-03-12T05:09:44.381394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.134305+0000","last_change":"2024-03-12T05:09:44.133911+0000","last_active":"2024-03-12T05:09:44.134305+0000","last_peered":"2024-03-12T05:09:44.134305+0000","last_clean":"2024-03-12T05:09:44.134305+0000","last_became_active":"2024-03-12T05:09:44.133488+0000","last_became_peered":"2024-03-12T05:09:44.133488+0000","last_unstale":"2024-03-12T05:09:44.134305+0000","last_undegraded":"2024-03-12T05:09:44.134305+0000","last_fullsized":"2024-03-12T05:09:44.134305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:44.969275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:48.446767+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:09:48.446767+0000","last_peered":"2024-03-12T05:09:48.446767+0000","last_clean":"2024-03-12T05:09:48.446767+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:09:48.446767+0000","last_undegraded":"2024-03-12T05:09:48.446767+0000","last_fullsized":"2024-03-12T05:09:48.446767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381913+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:09:44.381913+0000","last_peered":"2024-03-12T05:09:44.381913+0000","last_clean":"2024-03-12T05:09:44.381913+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:09:44.381913+0000","last_undegraded":"2024-03-12T05:09:44.381913+0000","last_fullsized":"2024-03-12T05:09:44.381913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310523+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:09:44.310523+0000","last_peered":"2024-03-12T05:09:44.310523+0000","last_clean":"2024-03-12T05:09:44.310523+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:09:44.310523+0000","last_undegraded":"2024-03-12T05:09:44.310523+0000","last_fullsized":"2024-03-12T05:09:44.310523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125386+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:09:44.125386+0000","last_peered":"2024-03-12T05:09:44.125386+0000","last_clean":"2024-03-12T05:09:44.125386+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:09:44.125386+0000","last_undegraded":"2024-03-12T05:09:44.125386+0000","last_fullsized":"2024-03-12T05:09:44.125386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125969+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:09:44.125969+0000","last_peered":"2024-03-12T05:09:44.125969+0000","last_clean":"2024-03-12T05:09:44.125969+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:09:44.125969+0000","last_undegraded":"2024-03-12T05:09:44.125969+0000","last_fullsized":"2024-03-12T05:09:44.125969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125565+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:09:44.125565+0000","last_peered":"2024-03-12T05:09:44.125565+0000","last_clean":"2024-03-12T05:09:44.125565+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:09:44.125565+0000","last_undegraded":"2024-03-12T05:09:44.125565+0000","last_fullsized":"2024-03-12T05:09:44.125565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128082+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:09:44.128082+0000","last_peered":"2024-03-12T05:09:44.128082+0000","last_clean":"2024-03-12T05:09:44.128082+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:09:44.128082+0000","last_undegraded":"2024-03-12T05:09:44.128082+0000","last_fullsized":"2024-03-12T05:09:44.128082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126476+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:09:44.126476+0000","last_peered":"2024-03-12T05:09:44.126476+0000","last_clean":"2024-03-12T05:09:44.126476+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:09:44.126476+0000","last_undegraded":"2024-03-12T05:09:44.126476+0000","last_fullsized":"2024-03-12T05:09:44.126476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'12","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128192+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:09:44.128192+0000","last_peered":"2024-03-12T05:09:44.128192+0000","last_clean":"2024-03-12T05:09:44.128192+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:09:44.128192+0000","last_undegraded":"2024-03-12T05:09:44.128192+0000","last_fullsized":"2024-03-12T05:09:44.128192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381816+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:09:44.381816+0000","last_peered":"2024-03-12T05:09:44.381816+0000","last_clean":"2024-03-12T05:09:44.381816+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:09:44.381816+0000","last_undegraded":"2024-03-12T05:09:44.381816+0000","last_fullsized":"2024-03-12T05:09:44.381816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310664+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:09:44.310664+0000","last_peered":"2024-03-12T05:09:44.310664+0000","last_clean":"2024-03-12T05:09:44.310664+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:09:44.310664+0000","last_undegraded":"2024-03-12T05:09:44.310664+0000","last_fullsized":"2024-03-12T05:09:44.310664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124630+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:09:44.124630+0000","last_peered":"2024-03-12T05:09:44.124630+0000","last_clean":"2024-03-12T05:09:44.124630+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:09:44.124630+0000","last_undegraded":"2024-03-12T05:09:44.124630+0000","last_fullsized":"2024-03-12T05:09:44.124630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.136091+0000","last_change":"2024-03-12T05:09:44.136091+0000","last_active":"2024-03-12T05:09:44.136091+0000","last_peered":"2024-03-12T05:09:44.136091+0000","last_clean":"2024-03-12T05:09:44.136091+0000","last_became_active":"2024-03-12T05:09:44.130865+0000","last_became_peered":"2024-03-12T05:09:44.130865+0000","last_unstale":"2024-03-12T05:09:44.136091+0000","last_undegraded":"2024-03-12T05:09:44.136091+0000","last_fullsized":"2024-03-12T05:09:44.136091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:13.588722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127748+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:09:44.127748+0000","last_peered":"2024-03-12T05:09:44.127748+0000","last_clean":"2024-03-12T05:09:44.127748+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:09:44.127748+0000","last_undegraded":"2024-03-12T05:09:44.127748+0000","last_fullsized":"2024-03-12T05:09:44.127748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381493+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:09:44.381493+0000","last_peered":"2024-03-12T05:09:44.381493+0000","last_clean":"2024-03-12T05:09:44.381493+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:09:44.381493+0000","last_undegraded":"2024-03-12T05:09:44.381493+0000","last_fullsized":"2024-03-12T05:09:44.381493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129221+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:09:44.129221+0000","last_peered":"2024-03-12T05:09:44.129221+0000","last_clean":"2024-03-12T05:09:44.129221+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:09:44.129221+0000","last_undegraded":"2024-03-12T05:09:44.129221+0000","last_fullsized":"2024-03-12T05:09:44.129221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131374+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:09:44.131374+0000","last_peered":"2024-03-12T05:09:44.131374+0000","last_clean":"2024-03-12T05:09:44.131374+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:09:44.131374+0000","last_undegraded":"2024-03-12T05:09:44.131374+0000","last_fullsized":"2024-03-12T05:09:44.131374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381765+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:09:44.381765+0000","last_peered":"2024-03-12T05:09:44.381765+0000","last_clean":"2024-03-12T05:09:44.381765+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:09:44.381765+0000","last_undegraded":"2024-03-12T05:09:44.381765+0000","last_fullsized":"2024-03-12T05:09:44.381765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.145784+0000","last_change":"2024-03-12T05:09:44.145284+0000","last_active":"2024-03-12T05:09:44.145784+0000","last_peered":"2024-03-12T05:09:44.145784+0000","last_clean":"2024-03-12T05:09:44.145784+0000","last_became_active":"2024-03-12T05:09:44.134640+0000","last_became_peered":"2024-03-12T05:09:44.134640+0000","last_unstale":"2024-03-12T05:09:44.145784+0000","last_undegraded":"2024-03-12T05:09:44.145784+0000","last_fullsized":"2024-03-12T05:09:44.145784+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:17.719372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125538+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:09:44.125538+0000","last_peered":"2024-03-12T05:09:44.125538+0000","last_clean":"2024-03-12T05:09:44.125538+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:09:44.125538+0000","last_undegraded":"2024-03-12T05:09:44.125538+0000","last_fullsized":"2024-03-12T05:09:44.125538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:49.448671+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:09:49.448671+0000","last_peered":"2024-03-12T05:09:49.448671+0000","last_clean":"2024-03-12T05:09:49.448671+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:09:49.448671+0000","last_undegraded":"2024-03-12T05:09:49.448671+0000","last_fullsized":"2024-03-12T05:09:49.448671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125831+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:09:44.125831+0000","last_peered":"2024-03-12T05:09:44.125831+0000","last_clean":"2024-03-12T05:09:44.125831+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:09:44.125831+0000","last_undegraded":"2024-03-12T05:09:44.125831+0000","last_fullsized":"2024-03-12T05:09:44.125831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131890+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:09:44.131890+0000","last_peered":"2024-03-12T05:09:44.131890+0000","last_clean":"2024-03-12T05:09:44.131890+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:09:44.131890+0000","last_undegraded":"2024-03-12T05:09:44.131890+0000","last_fullsized":"2024-03-12T05:09:44.131890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125642+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:09:44.125642+0000","last_peered":"2024-03-12T05:09:44.125642+0000","last_clean":"2024-03-12T05:09:44.125642+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:09:44.125642+0000","last_undegraded":"2024-03-12T05:09:44.125642+0000","last_fullsized":"2024-03-12T05:09:44.125642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126596+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:09:44.126596+0000","last_peered":"2024-03-12T05:09:44.126596+0000","last_clean":"2024-03-12T05:09:44.126596+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:09:44.126596+0000","last_undegraded":"2024-03-12T05:09:44.126596+0000","last_fullsized":"2024-03-12T05:09:44.126596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125880+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:09:44.125880+0000","last_peered":"2024-03-12T05:09:44.125880+0000","last_clean":"2024-03-12T05:09:44.125880+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:09:44.125880+0000","last_undegraded":"2024-03-12T05:09:44.125880+0000","last_fullsized":"2024-03-12T05:09:44.125880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129251+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:09:44.129251+0000","last_peered":"2024-03-12T05:09:44.129251+0000","last_clean":"2024-03-12T05:09:44.129251+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:09:44.129251+0000","last_undegraded":"2024-03-12T05:09:44.129251+0000","last_fullsized":"2024-03-12T05:09:44.129251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131447+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:09:44.131447+0000","last_peered":"2024-03-12T05:09:44.131447+0000","last_clean":"2024-03-12T05:09:44.131447+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:09:44.131447+0000","last_undegraded":"2024-03-12T05:09:44.131447+0000","last_fullsized":"2024-03-12T05:09:44.131447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131203+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:09:44.131203+0000","last_peered":"2024-03-12T05:09:44.131203+0000","last_clean":"2024-03-12T05:09:44.131203+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:09:44.131203+0000","last_undegraded":"2024-03-12T05:09:44.131203+0000","last_fullsized":"2024-03-12T05:09:44.131203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124551+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:09:44.124551+0000","last_peered":"2024-03-12T05:09:44.124551+0000","last_clean":"2024-03-12T05:09:44.124551+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:09:44.124551+0000","last_undegraded":"2024-03-12T05:09:44.124551+0000","last_fullsized":"2024-03-12T05:09:44.124551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129436+0000","last_change":"2024-03-12T05:09:44.129436+0000","last_active":"2024-03-12T05:09:44.129436+0000","last_peered":"2024-03-12T05:09:44.129436+0000","last_clean":"2024-03-12T05:09:44.129436+0000","last_became_active":"2024-03-12T05:09:44.129063+0000","last_became_peered":"2024-03-12T05:09:44.129063+0000","last_unstale":"2024-03-12T05:09:44.129436+0000","last_undegraded":"2024-03-12T05:09:44.129436+0000","last_fullsized":"2024-03-12T05:09:44.129436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:06.724357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129493+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:09:44.129493+0000","last_peered":"2024-03-12T05:09:44.129493+0000","last_clean":"2024-03-12T05:09:44.129493+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:09:44.129493+0000","last_undegraded":"2024-03-12T05:09:44.129493+0000","last_fullsized":"2024-03-12T05:09:44.129493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132630+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:09:44.132630+0000","last_peered":"2024-03-12T05:09:44.132630+0000","last_clean":"2024-03-12T05:09:44.132630+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:09:44.132630+0000","last_undegraded":"2024-03-12T05:09:44.132630+0000","last_fullsized":"2024-03-12T05:09:44.132630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381766+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:09:44.381766+0000","last_peered":"2024-03-12T05:09:44.381766+0000","last_clean":"2024-03-12T05:09:44.381766+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:09:44.381766+0000","last_undegraded":"2024-03-12T05:09:44.381766+0000","last_fullsized":"2024-03-12T05:09:44.381766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128940+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:09:44.128940+0000","last_peered":"2024-03-12T05:09:44.128940+0000","last_clean":"2024-03-12T05:09:44.128940+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:09:44.128940+0000","last_undegraded":"2024-03-12T05:09:44.128940+0000","last_fullsized":"2024-03-12T05:09:44.128940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381482+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:09:44.381482+0000","last_peered":"2024-03-12T05:09:44.381482+0000","last_clean":"2024-03-12T05:09:44.381482+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:09:44.381482+0000","last_undegraded":"2024-03-12T05:09:44.381482+0000","last_fullsized":"2024-03-12T05:09:44.381482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126791+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:09:44.126791+0000","last_peered":"2024-03-12T05:09:44.126791+0000","last_clean":"2024-03-12T05:09:44.126791+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:09:44.126791+0000","last_undegraded":"2024-03-12T05:09:44.126791+0000","last_fullsized":"2024-03-12T05:09:44.126791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126227+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:09:44.126227+0000","last_peered":"2024-03-12T05:09:44.126227+0000","last_clean":"2024-03-12T05:09:44.126227+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:09:44.126227+0000","last_undegraded":"2024-03-12T05:09:44.126227+0000","last_fullsized":"2024-03-12T05:09:44.126227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128442+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:09:44.128442+0000","last_peered":"2024-03-12T05:09:44.128442+0000","last_clean":"2024-03-12T05:09:44.128442+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:09:44.128442+0000","last_undegraded":"2024-03-12T05:09:44.128442+0000","last_fullsized":"2024-03-12T05:09:44.128442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66507,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":695,"num_read_kb":417,"num_write":236,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":199521,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383707,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":456844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645344,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1106785,"data_compressed":37976,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":456383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070506,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":456089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332144,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1049670,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":456100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":516597,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":457830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":516927,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":455925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1048957,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61336,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61666,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":61039,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:09:55.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T05:09:55.909 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:56.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:56.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 61 B/s, 1 objects/s recovering 2024-03-12T05:09:56.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:09:56.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:09:56.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:56 smithi045 ceph-mon[32174]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 61 B/s, 1 objects/s recovering 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:09:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:56 smithi130 ceph-mon[35398]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 61 B/s, 1 objects/s recovering 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:09:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:56 smithi116 ceph-mon[35486]: from='mgr.14152 172.21.15.45:0/277169288' entity='mgr.x' 2024-03-12T05:09:57.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:57 smithi045 ceph-mon[32174]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:09:57.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:57 smithi130 ceph-mon[35398]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:09:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:57 smithi116 ceph-mon[35486]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 -- 172.21.15.45:0/168371147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760101530 msgr2=0x7fa76010d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 --2- 172.21.15.45:0/168371147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760101530 0x7fa76010d9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b0a0 tx=0x7fa75800b3b0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 -- 172.21.15.45:0/168371147 shutdown_connections 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 --2- 172.21.15.45:0/168371147 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760101530 0x7fa76010d9c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 --2- 172.21.15.45:0/168371147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760106b80 0x7fa760106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 --2- 172.21.15.45:0/168371147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 0x7fa760100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 -- 172.21.15.45:0/168371147 >> 172.21.15.45:0/168371147 conn(0x7fa760077aa0 msgr2=0x7fa760077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 -- 172.21.15.45:0/168371147 shutdown_connections 2024-03-12T05:09:57.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.606+0000 7fa7679a9700 1 -- 172.21.15.45:0/168371147 wait complete. 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 Processor -- start 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 -- start start 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 0x7fa760198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760101530 0x7fa760198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76019d1f0 con 0x7fa760106b80 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76019d360 con 0x7fa760100b70 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa7679a9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76019d520 con 0x7fa760101530 2024-03-12T05:09:57.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45444/0 (socket says 172.21.15.45:45444) 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 -- 172.21.15.45:0/523235185 learned_addr learned my addr 172.21.15.45:0/523235185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765745700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 0x7fa760198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa764f44700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760101530 0x7fa760198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 -- 172.21.15.45:0/523235185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760101530 msgr2=0x7fa760198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760101530 0x7fa760198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 -- 172.21.15.45:0/523235185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 msgr2=0x7fa760198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.607+0000 7fa765f46700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 0x7fa760198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa765f46700 1 -- 172.21.15.45:0/523235185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa758009d00 con 0x7fa760106b80 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa765f46700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b830 tx=0x7fa758009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800e040 con 0x7fa760106b80 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76019d7a0 con 0x7fa760106b80 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa758004e60 con 0x7fa760106b80 2024-03-12T05:09:57.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7580053c0 con 0x7fa760106b80 2024-03-12T05:09:57.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.608+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7601a5060 con 0x7fa760106b80 2024-03-12T05:09:57.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.609+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7601021f0 con 0x7fa760106b80 2024-03-12T05:09:57.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.610+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa75801bb20 con 0x7fa760106b80 2024-03-12T05:09:57.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.611+0000 7fa7567fc700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 0x7fa74c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:09:57.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.611+0000 7fa765745700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 0x7fa74c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:09:57.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.611+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa7580cedb0 con 0x7fa760106b80 2024-03-12T05:09:57.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.611+0000 7fa765745700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 0x7fa74c0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009a10 tx=0x7fa75c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:09:57.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.614+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa758097f00 con 0x7fa760106b80 2024-03-12T05:09:57.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.816+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa760110b20 con 0x7fa74c0a1b50 2024-03-12T05:09:57.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.824+0000 7fa7567fc700 1 -- 172.21.15.45:0/523235185 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258966 (secure 0 0 0) 0x7fa760110b20 con 0x7fa74c0a1b50 2024-03-12T05:09:57.825 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:09:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.826+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 msgr2=0x7fa74c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:57.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.826+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 0x7fa74c0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009a10 tx=0x7fa75c009190 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.826+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 msgr2=0x7fa76019cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.826+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b830 tx=0x7fa758009380 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 shutdown_connections 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa74c0a1b50 0x7fa74c0a4010 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa760106b80 0x7fa76019cbe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa760100b70 0x7fa760198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 --2- 172.21.15.45:0/523235185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa760101530 0x7fa760198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 >> 172.21.15.45:0/523235185 conn(0x7fa760077aa0 msgr2=0x7fa760069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 shutdown_connections 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:09:57.827+0000 7fa7679a9700 1 -- 172.21.15.45:0/523235185 wait complete. 2024-03-12T05:09:57.827 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:09:58.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:58 smithi130 ceph-mon[35398]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:58 smithi116 ceph-mon[35486]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:58.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:58 smithi045 ceph-mon[32174]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:09:58.536 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-12T05:09:57.118460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657109,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":881,"num_read_kb":561,"num_write":419,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181408,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369260,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572564,"statfs":{"total":1151940427776,"available":1151562305536,"internally_reserved":0,"allocated":11980800,"data_stored":7434459,"data_compressed":113928,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003167"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128692+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:09:44.128692+0000","last_peered":"2024-03-12T05:09:44.128692+0000","last_clean":"2024-03-12T05:09:44.128692+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:09:44.128692+0000","last_undegraded":"2024-03-12T05:09:44.128692+0000","last_fullsized":"2024-03-12T05:09:44.128692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124710+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:09:44.124710+0000","last_peered":"2024-03-12T05:09:44.124710+0000","last_clean":"2024-03-12T05:09:44.124710+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:09:44.124710+0000","last_undegraded":"2024-03-12T05:09:44.124710+0000","last_fullsized":"2024-03-12T05:09:44.124710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124617+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:09:44.124617+0000","last_peered":"2024-03-12T05:09:44.124617+0000","last_clean":"2024-03-12T05:09:44.124617+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:09:44.124617+0000","last_undegraded":"2024-03-12T05:09:44.124617+0000","last_fullsized":"2024-03-12T05:09:44.124617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310504+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:09:44.310504+0000","last_peered":"2024-03-12T05:09:44.310504+0000","last_clean":"2024-03-12T05:09:44.310504+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:09:44.310504+0000","last_undegraded":"2024-03-12T05:09:44.310504+0000","last_fullsized":"2024-03-12T05:09:44.310504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129408+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:09:44.129408+0000","last_peered":"2024-03-12T05:09:44.129408+0000","last_clean":"2024-03-12T05:09:44.129408+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:09:44.129408+0000","last_undegraded":"2024-03-12T05:09:44.129408+0000","last_fullsized":"2024-03-12T05:09:44.129408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381264+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:09:44.381264+0000","last_peered":"2024-03-12T05:09:44.381264+0000","last_clean":"2024-03-12T05:09:44.381264+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:09:44.381264+0000","last_undegraded":"2024-03-12T05:09:44.381264+0000","last_fullsized":"2024-03-12T05:09:44.381264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125325+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:09:44.125325+0000","last_peered":"2024-03-12T05:09:44.125325+0000","last_clean":"2024-03-12T05:09:44.125325+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:09:44.125325+0000","last_undegraded":"2024-03-12T05:09:44.125325+0000","last_fullsized":"2024-03-12T05:09:44.125325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381417+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:09:44.381417+0000","last_peered":"2024-03-12T05:09:44.381417+0000","last_clean":"2024-03-12T05:09:44.381417+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:09:44.381417+0000","last_undegraded":"2024-03-12T05:09:44.381417+0000","last_fullsized":"2024-03-12T05:09:44.381417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127078+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:09:44.127078+0000","last_peered":"2024-03-12T05:09:44.127078+0000","last_clean":"2024-03-12T05:09:44.127078+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:09:44.127078+0000","last_undegraded":"2024-03-12T05:09:44.127078+0000","last_fullsized":"2024-03-12T05:09:44.127078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381586+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:09:44.381586+0000","last_peered":"2024-03-12T05:09:44.381586+0000","last_clean":"2024-03-12T05:09:44.381586+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:09:44.381586+0000","last_undegraded":"2024-03-12T05:09:44.381586+0000","last_fullsized":"2024-03-12T05:09:44.381586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125698+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:09:44.125698+0000","last_peered":"2024-03-12T05:09:44.125698+0000","last_clean":"2024-03-12T05:09:44.125698+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:09:44.125698+0000","last_undegraded":"2024-03-12T05:09:44.125698+0000","last_fullsized":"2024-03-12T05:09:44.125698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310513+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:09:44.310513+0000","last_peered":"2024-03-12T05:09:44.310513+0000","last_clean":"2024-03-12T05:09:44.310513+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:09:44.310513+0000","last_undegraded":"2024-03-12T05:09:44.310513+0000","last_fullsized":"2024-03-12T05:09:44.310513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126454+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:09:44.126454+0000","last_peered":"2024-03-12T05:09:44.126454+0000","last_clean":"2024-03-12T05:09:44.126454+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:09:44.126454+0000","last_undegraded":"2024-03-12T05:09:44.126454+0000","last_fullsized":"2024-03-12T05:09:44.126454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124416+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:09:44.124416+0000","last_peered":"2024-03-12T05:09:44.124416+0000","last_clean":"2024-03-12T05:09:44.124416+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:09:44.124416+0000","last_undegraded":"2024-03-12T05:09:44.124416+0000","last_fullsized":"2024-03-12T05:09:44.124416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124334+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:09:44.124334+0000","last_peered":"2024-03-12T05:09:44.124334+0000","last_clean":"2024-03-12T05:09:44.124334+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:09:44.124334+0000","last_undegraded":"2024-03-12T05:09:44.124334+0000","last_fullsized":"2024-03-12T05:09:44.124334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126896+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:09:44.126896+0000","last_peered":"2024-03-12T05:09:44.126896+0000","last_clean":"2024-03-12T05:09:44.126896+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:09:44.126896+0000","last_undegraded":"2024-03-12T05:09:44.126896+0000","last_fullsized":"2024-03-12T05:09:44.126896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128810+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:09:44.128810+0000","last_peered":"2024-03-12T05:09:44.128810+0000","last_clean":"2024-03-12T05:09:44.128810+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:09:44.128810+0000","last_undegraded":"2024-03-12T05:09:44.128810+0000","last_fullsized":"2024-03-12T05:09:44.128810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128345+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:09:44.128345+0000","last_peered":"2024-03-12T05:09:44.128345+0000","last_clean":"2024-03-12T05:09:44.128345+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:09:44.128345+0000","last_undegraded":"2024-03-12T05:09:44.128345+0000","last_fullsized":"2024-03-12T05:09:44.128345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131740+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:09:44.131740+0000","last_peered":"2024-03-12T05:09:44.131740+0000","last_clean":"2024-03-12T05:09:44.131740+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:09:44.131740+0000","last_undegraded":"2024-03-12T05:09:44.131740+0000","last_fullsized":"2024-03-12T05:09:44.131740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":124,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:53.142401+0000","last_change":"2024-03-12T05:09:44.142398+0000","last_active":"2024-03-12T05:09:53.142401+0000","last_peered":"2024-03-12T05:09:53.142401+0000","last_clean":"2024-03-12T05:09:53.142401+0000","last_became_active":"2024-03-12T05:09:44.129576+0000","last_became_peered":"2024-03-12T05:09:44.129576+0000","last_unstale":"2024-03-12T05:09:53.142401+0000","last_undegraded":"2024-03-12T05:09:53.142401+0000","last_fullsized":"2024-03-12T05:09:53.142401+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:32.791210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.135418+0000","last_change":"2024-03-12T05:09:44.135049+0000","last_active":"2024-03-12T05:09:44.135418+0000","last_peered":"2024-03-12T05:09:44.135418+0000","last_clean":"2024-03-12T05:09:44.135418+0000","last_became_active":"2024-03-12T05:09:44.134638+0000","last_became_peered":"2024-03-12T05:09:44.134638+0000","last_unstale":"2024-03-12T05:09:44.135418+0000","last_undegraded":"2024-03-12T05:09:44.135418+0000","last_fullsized":"2024-03-12T05:09:44.135418+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:28:59.785062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381917+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:09:44.381917+0000","last_peered":"2024-03-12T05:09:44.381917+0000","last_clean":"2024-03-12T05:09:44.381917+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:09:44.381917+0000","last_undegraded":"2024-03-12T05:09:44.381917+0000","last_fullsized":"2024-03-12T05:09:44.381917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124824+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:09:44.124824+0000","last_peered":"2024-03-12T05:09:44.124824+0000","last_clean":"2024-03-12T05:09:44.124824+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:09:44.124824+0000","last_undegraded":"2024-03-12T05:09:44.124824+0000","last_fullsized":"2024-03-12T05:09:44.124824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128279+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:09:44.128279+0000","last_peered":"2024-03-12T05:09:44.128279+0000","last_clean":"2024-03-12T05:09:44.128279+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:09:44.128279+0000","last_undegraded":"2024-03-12T05:09:44.128279+0000","last_fullsized":"2024-03-12T05:09:44.128279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126669+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:09:44.126669+0000","last_peered":"2024-03-12T05:09:44.126669+0000","last_clean":"2024-03-12T05:09:44.126669+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:09:44.126669+0000","last_undegraded":"2024-03-12T05:09:44.126669+0000","last_fullsized":"2024-03-12T05:09:44.126669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128527+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:09:44.128527+0000","last_peered":"2024-03-12T05:09:44.128527+0000","last_clean":"2024-03-12T05:09:44.128527+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:09:44.128527+0000","last_undegraded":"2024-03-12T05:09:44.128527+0000","last_fullsized":"2024-03-12T05:09:44.128527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.133756+0000","last_change":"2024-03-12T05:09:44.133756+0000","last_active":"2024-03-12T05:09:44.133756+0000","last_peered":"2024-03-12T05:09:44.133756+0000","last_clean":"2024-03-12T05:09:44.133756+0000","last_became_active":"2024-03-12T05:09:44.133312+0000","last_became_peered":"2024-03-12T05:09:44.133312+0000","last_unstale":"2024-03-12T05:09:44.133756+0000","last_undegraded":"2024-03-12T05:09:44.133756+0000","last_fullsized":"2024-03-12T05:09:44.133756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:54:24.882257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:52.452751+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:09:52.452751+0000","last_peered":"2024-03-12T05:09:52.452751+0000","last_clean":"2024-03-12T05:09:52.452751+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:09:52.452751+0000","last_undegraded":"2024-03-12T05:09:52.452751+0000","last_fullsized":"2024-03-12T05:09:52.452751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128072+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:09:44.128072+0000","last_peered":"2024-03-12T05:09:44.128072+0000","last_clean":"2024-03-12T05:09:44.128072+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:09:44.128072+0000","last_undegraded":"2024-03-12T05:09:44.128072+0000","last_fullsized":"2024-03-12T05:09:44.128072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381204+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:09:44.381204+0000","last_peered":"2024-03-12T05:09:44.381204+0000","last_clean":"2024-03-12T05:09:44.381204+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:09:44.381204+0000","last_undegraded":"2024-03-12T05:09:44.381204+0000","last_fullsized":"2024-03-12T05:09:44.381204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129783+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:09:44.129783+0000","last_peered":"2024-03-12T05:09:44.129783+0000","last_clean":"2024-03-12T05:09:44.129783+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:09:44.129783+0000","last_undegraded":"2024-03-12T05:09:44.129783+0000","last_fullsized":"2024-03-12T05:09:44.129783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126104+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:09:44.126104+0000","last_peered":"2024-03-12T05:09:44.126104+0000","last_clean":"2024-03-12T05:09:44.126104+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:09:44.126104+0000","last_undegraded":"2024-03-12T05:09:44.126104+0000","last_fullsized":"2024-03-12T05:09:44.126104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381349+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:09:44.381349+0000","last_peered":"2024-03-12T05:09:44.381349+0000","last_clean":"2024-03-12T05:09:44.381349+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:09:44.381349+0000","last_undegraded":"2024-03-12T05:09:44.381349+0000","last_fullsized":"2024-03-12T05:09:44.381349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128856+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:09:44.128856+0000","last_peered":"2024-03-12T05:09:44.128856+0000","last_clean":"2024-03-12T05:09:44.128856+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:09:44.128856+0000","last_undegraded":"2024-03-12T05:09:44.128856+0000","last_fullsized":"2024-03-12T05:09:44.128856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127902+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:09:44.127902+0000","last_peered":"2024-03-12T05:09:44.127902+0000","last_clean":"2024-03-12T05:09:44.127902+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:09:44.127902+0000","last_undegraded":"2024-03-12T05:09:44.127902+0000","last_fullsized":"2024-03-12T05:09:44.127902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":165,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:53.453457+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:09:53.453457+0000","last_peered":"2024-03-12T05:09:53.453457+0000","last_clean":"2024-03-12T05:09:53.453457+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:09:53.453457+0000","last_undegraded":"2024-03-12T05:09:53.453457+0000","last_fullsized":"2024-03-12T05:09:53.453457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127231+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:09:44.127231+0000","last_peered":"2024-03-12T05:09:44.127231+0000","last_clean":"2024-03-12T05:09:44.127231+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:09:44.127231+0000","last_undegraded":"2024-03-12T05:09:44.127231+0000","last_fullsized":"2024-03-12T05:09:44.127231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.315313+0000","last_change":"2024-03-12T05:09:44.315313+0000","last_active":"2024-03-12T05:09:44.315313+0000","last_peered":"2024-03-12T05:09:44.315313+0000","last_clean":"2024-03-12T05:09:44.315313+0000","last_became_active":"2024-03-12T05:09:44.311453+0000","last_became_peered":"2024-03-12T05:09:44.311453+0000","last_unstale":"2024-03-12T05:09:44.315313+0000","last_undegraded":"2024-03-12T05:09:44.315313+0000","last_fullsized":"2024-03-12T05:09:44.315313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:21.022485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124802+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:09:44.124802+0000","last_peered":"2024-03-12T05:09:44.124802+0000","last_clean":"2024-03-12T05:09:44.124802+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:09:44.124802+0000","last_undegraded":"2024-03-12T05:09:44.124802+0000","last_fullsized":"2024-03-12T05:09:44.124802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132030+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:09:44.132030+0000","last_peered":"2024-03-12T05:09:44.132030+0000","last_clean":"2024-03-12T05:09:44.132030+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:09:44.132030+0000","last_undegraded":"2024-03-12T05:09:44.132030+0000","last_fullsized":"2024-03-12T05:09:44.132030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126851+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:09:44.126851+0000","last_peered":"2024-03-12T05:09:44.126851+0000","last_clean":"2024-03-12T05:09:44.126851+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:09:44.126851+0000","last_undegraded":"2024-03-12T05:09:44.126851+0000","last_fullsized":"2024-03-12T05:09:44.126851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'28","reported_seq":195,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:53.453459+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:09:53.453459+0000","last_peered":"2024-03-12T05:09:53.453459+0000","last_clean":"2024-03-12T05:09:53.453459+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:09:53.453459+0000","last_undegraded":"2024-03-12T05:09:53.453459+0000","last_fullsized":"2024-03-12T05:09:53.453459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58912,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":148,"num_read_kb":131,"num_write":30,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127947+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:09:44.127947+0000","last_peered":"2024-03-12T05:09:44.127947+0000","last_clean":"2024-03-12T05:09:44.127947+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:09:44.127947+0000","last_undegraded":"2024-03-12T05:09:44.127947+0000","last_fullsized":"2024-03-12T05:09:44.127947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131709+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:09:44.131709+0000","last_peered":"2024-03-12T05:09:44.131709+0000","last_clean":"2024-03-12T05:09:44.131709+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:09:44.131709+0000","last_undegraded":"2024-03-12T05:09:44.131709+0000","last_fullsized":"2024-03-12T05:09:44.131709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126153+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:09:44.126153+0000","last_peered":"2024-03-12T05:09:44.126153+0000","last_clean":"2024-03-12T05:09:44.126153+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:09:44.126153+0000","last_undegraded":"2024-03-12T05:09:44.126153+0000","last_fullsized":"2024-03-12T05:09:44.126153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128625+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:09:44.128625+0000","last_peered":"2024-03-12T05:09:44.128625+0000","last_clean":"2024-03-12T05:09:44.128625+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:09:44.128625+0000","last_undegraded":"2024-03-12T05:09:44.128625+0000","last_fullsized":"2024-03-12T05:09:44.128625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125826+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:09:44.125826+0000","last_peered":"2024-03-12T05:09:44.125826+0000","last_clean":"2024-03-12T05:09:44.125826+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:09:44.125826+0000","last_undegraded":"2024-03-12T05:09:44.125826+0000","last_fullsized":"2024-03-12T05:09:44.125826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:52.452474+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:09:52.452474+0000","last_peered":"2024-03-12T05:09:52.452474+0000","last_clean":"2024-03-12T05:09:52.452474+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:09:52.452474+0000","last_undegraded":"2024-03-12T05:09:52.452474+0000","last_fullsized":"2024-03-12T05:09:52.452474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125250+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:09:44.125250+0000","last_peered":"2024-03-12T05:09:44.125250+0000","last_clean":"2024-03-12T05:09:44.125250+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:09:44.125250+0000","last_undegraded":"2024-03-12T05:09:44.125250+0000","last_fullsized":"2024-03-12T05:09:44.125250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127773+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:09:44.127773+0000","last_peered":"2024-03-12T05:09:44.127773+0000","last_clean":"2024-03-12T05:09:44.127773+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:09:44.127773+0000","last_undegraded":"2024-03-12T05:09:44.127773+0000","last_fullsized":"2024-03-12T05:09:44.127773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129690+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:09:44.129690+0000","last_peered":"2024-03-12T05:09:44.129690+0000","last_clean":"2024-03-12T05:09:44.129690+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:09:44.129690+0000","last_undegraded":"2024-03-12T05:09:44.129690+0000","last_fullsized":"2024-03-12T05:09:44.129690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128902+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:09:44.128902+0000","last_peered":"2024-03-12T05:09:44.128902+0000","last_clean":"2024-03-12T05:09:44.128902+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:09:44.128902+0000","last_undegraded":"2024-03-12T05:09:44.128902+0000","last_fullsized":"2024-03-12T05:09:44.128902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128731+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:09:44.128731+0000","last_peered":"2024-03-12T05:09:44.128731+0000","last_clean":"2024-03-12T05:09:44.128731+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:09:44.128731+0000","last_undegraded":"2024-03-12T05:09:44.128731+0000","last_fullsized":"2024-03-12T05:09:44.128731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128017+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:09:44.128017+0000","last_peered":"2024-03-12T05:09:44.128017+0000","last_clean":"2024-03-12T05:09:44.128017+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:09:44.128017+0000","last_undegraded":"2024-03-12T05:09:44.128017+0000","last_fullsized":"2024-03-12T05:09:44.128017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381570+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:09:44.381570+0000","last_peered":"2024-03-12T05:09:44.381570+0000","last_clean":"2024-03-12T05:09:44.381570+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:09:44.381570+0000","last_undegraded":"2024-03-12T05:09:44.381570+0000","last_fullsized":"2024-03-12T05:09:44.381570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:53.142940+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:09:53.142940+0000","last_peered":"2024-03-12T05:09:53.142940+0000","last_clean":"2024-03-12T05:09:53.142940+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:09:53.142940+0000","last_undegraded":"2024-03-12T05:09:53.142940+0000","last_fullsized":"2024-03-12T05:09:53.142940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381394+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:09:44.381394+0000","last_peered":"2024-03-12T05:09:44.381394+0000","last_clean":"2024-03-12T05:09:44.381394+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:09:44.381394+0000","last_undegraded":"2024-03-12T05:09:44.381394+0000","last_fullsized":"2024-03-12T05:09:44.381394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.134305+0000","last_change":"2024-03-12T05:09:44.133911+0000","last_active":"2024-03-12T05:09:44.134305+0000","last_peered":"2024-03-12T05:09:44.134305+0000","last_clean":"2024-03-12T05:09:44.134305+0000","last_became_active":"2024-03-12T05:09:44.133488+0000","last_became_peered":"2024-03-12T05:09:44.133488+0000","last_unstale":"2024-03-12T05:09:44.134305+0000","last_undegraded":"2024-03-12T05:09:44.134305+0000","last_fullsized":"2024-03-12T05:09:44.134305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:44.969275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:53.453475+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:09:53.453475+0000","last_peered":"2024-03-12T05:09:53.453475+0000","last_clean":"2024-03-12T05:09:53.453475+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:09:53.453475+0000","last_undegraded":"2024-03-12T05:09:53.453475+0000","last_fullsized":"2024-03-12T05:09:53.453475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381913+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:09:44.381913+0000","last_peered":"2024-03-12T05:09:44.381913+0000","last_clean":"2024-03-12T05:09:44.381913+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:09:44.381913+0000","last_undegraded":"2024-03-12T05:09:44.381913+0000","last_fullsized":"2024-03-12T05:09:44.381913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310523+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:09:44.310523+0000","last_peered":"2024-03-12T05:09:44.310523+0000","last_clean":"2024-03-12T05:09:44.310523+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:09:44.310523+0000","last_undegraded":"2024-03-12T05:09:44.310523+0000","last_fullsized":"2024-03-12T05:09:44.310523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125386+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:09:44.125386+0000","last_peered":"2024-03-12T05:09:44.125386+0000","last_clean":"2024-03-12T05:09:44.125386+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:09:44.125386+0000","last_undegraded":"2024-03-12T05:09:44.125386+0000","last_fullsized":"2024-03-12T05:09:44.125386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125969+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:09:44.125969+0000","last_peered":"2024-03-12T05:09:44.125969+0000","last_clean":"2024-03-12T05:09:44.125969+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:09:44.125969+0000","last_undegraded":"2024-03-12T05:09:44.125969+0000","last_fullsized":"2024-03-12T05:09:44.125969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125565+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:09:44.125565+0000","last_peered":"2024-03-12T05:09:44.125565+0000","last_clean":"2024-03-12T05:09:44.125565+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:09:44.125565+0000","last_undegraded":"2024-03-12T05:09:44.125565+0000","last_fullsized":"2024-03-12T05:09:44.125565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128082+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:09:44.128082+0000","last_peered":"2024-03-12T05:09:44.128082+0000","last_clean":"2024-03-12T05:09:44.128082+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:09:44.128082+0000","last_undegraded":"2024-03-12T05:09:44.128082+0000","last_fullsized":"2024-03-12T05:09:44.128082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126476+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:09:44.126476+0000","last_peered":"2024-03-12T05:09:44.126476+0000","last_clean":"2024-03-12T05:09:44.126476+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:09:44.126476+0000","last_undegraded":"2024-03-12T05:09:44.126476+0000","last_fullsized":"2024-03-12T05:09:44.126476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'12","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128192+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:09:44.128192+0000","last_peered":"2024-03-12T05:09:44.128192+0000","last_clean":"2024-03-12T05:09:44.128192+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:09:44.128192+0000","last_undegraded":"2024-03-12T05:09:44.128192+0000","last_fullsized":"2024-03-12T05:09:44.128192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381816+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:09:44.381816+0000","last_peered":"2024-03-12T05:09:44.381816+0000","last_clean":"2024-03-12T05:09:44.381816+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:09:44.381816+0000","last_undegraded":"2024-03-12T05:09:44.381816+0000","last_fullsized":"2024-03-12T05:09:44.381816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310664+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:09:44.310664+0000","last_peered":"2024-03-12T05:09:44.310664+0000","last_clean":"2024-03-12T05:09:44.310664+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:09:44.310664+0000","last_undegraded":"2024-03-12T05:09:44.310664+0000","last_fullsized":"2024-03-12T05:09:44.310664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124630+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:09:44.124630+0000","last_peered":"2024-03-12T05:09:44.124630+0000","last_clean":"2024-03-12T05:09:44.124630+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:09:44.124630+0000","last_undegraded":"2024-03-12T05:09:44.124630+0000","last_fullsized":"2024-03-12T05:09:44.124630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.136091+0000","last_change":"2024-03-12T05:09:44.136091+0000","last_active":"2024-03-12T05:09:44.136091+0000","last_peered":"2024-03-12T05:09:44.136091+0000","last_clean":"2024-03-12T05:09:44.136091+0000","last_became_active":"2024-03-12T05:09:44.130865+0000","last_became_peered":"2024-03-12T05:09:44.130865+0000","last_unstale":"2024-03-12T05:09:44.136091+0000","last_undegraded":"2024-03-12T05:09:44.136091+0000","last_fullsized":"2024-03-12T05:09:44.136091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:13.588722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127748+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:09:44.127748+0000","last_peered":"2024-03-12T05:09:44.127748+0000","last_clean":"2024-03-12T05:09:44.127748+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:09:44.127748+0000","last_undegraded":"2024-03-12T05:09:44.127748+0000","last_fullsized":"2024-03-12T05:09:44.127748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381493+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:09:44.381493+0000","last_peered":"2024-03-12T05:09:44.381493+0000","last_clean":"2024-03-12T05:09:44.381493+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:09:44.381493+0000","last_undegraded":"2024-03-12T05:09:44.381493+0000","last_fullsized":"2024-03-12T05:09:44.381493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129221+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:09:44.129221+0000","last_peered":"2024-03-12T05:09:44.129221+0000","last_clean":"2024-03-12T05:09:44.129221+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:09:44.129221+0000","last_undegraded":"2024-03-12T05:09:44.129221+0000","last_fullsized":"2024-03-12T05:09:44.129221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131374+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:09:44.131374+0000","last_peered":"2024-03-12T05:09:44.131374+0000","last_clean":"2024-03-12T05:09:44.131374+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:09:44.131374+0000","last_undegraded":"2024-03-12T05:09:44.131374+0000","last_fullsized":"2024-03-12T05:09:44.131374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381765+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:09:44.381765+0000","last_peered":"2024-03-12T05:09:44.381765+0000","last_clean":"2024-03-12T05:09:44.381765+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:09:44.381765+0000","last_undegraded":"2024-03-12T05:09:44.381765+0000","last_fullsized":"2024-03-12T05:09:44.381765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.145784+0000","last_change":"2024-03-12T05:09:44.145284+0000","last_active":"2024-03-12T05:09:44.145784+0000","last_peered":"2024-03-12T05:09:44.145784+0000","last_clean":"2024-03-12T05:09:44.145784+0000","last_became_active":"2024-03-12T05:09:44.134640+0000","last_became_peered":"2024-03-12T05:09:44.134640+0000","last_unstale":"2024-03-12T05:09:44.145784+0000","last_undegraded":"2024-03-12T05:09:44.145784+0000","last_fullsized":"2024-03-12T05:09:44.145784+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:17.719372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125538+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:09:44.125538+0000","last_peered":"2024-03-12T05:09:44.125538+0000","last_clean":"2024-03-12T05:09:44.125538+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:09:44.125538+0000","last_undegraded":"2024-03-12T05:09:44.125538+0000","last_fullsized":"2024-03-12T05:09:44.125538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:54.455345+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:09:54.455345+0000","last_peered":"2024-03-12T05:09:54.455345+0000","last_clean":"2024-03-12T05:09:54.455345+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:09:54.455345+0000","last_undegraded":"2024-03-12T05:09:54.455345+0000","last_fullsized":"2024-03-12T05:09:54.455345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125831+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:09:44.125831+0000","last_peered":"2024-03-12T05:09:44.125831+0000","last_clean":"2024-03-12T05:09:44.125831+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:09:44.125831+0000","last_undegraded":"2024-03-12T05:09:44.125831+0000","last_fullsized":"2024-03-12T05:09:44.125831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131890+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:09:44.131890+0000","last_peered":"2024-03-12T05:09:44.131890+0000","last_clean":"2024-03-12T05:09:44.131890+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:09:44.131890+0000","last_undegraded":"2024-03-12T05:09:44.131890+0000","last_fullsized":"2024-03-12T05:09:44.131890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125642+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:09:44.125642+0000","last_peered":"2024-03-12T05:09:44.125642+0000","last_clean":"2024-03-12T05:09:44.125642+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:09:44.125642+0000","last_undegraded":"2024-03-12T05:09:44.125642+0000","last_fullsized":"2024-03-12T05:09:44.125642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126596+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:09:44.126596+0000","last_peered":"2024-03-12T05:09:44.126596+0000","last_clean":"2024-03-12T05:09:44.126596+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:09:44.126596+0000","last_undegraded":"2024-03-12T05:09:44.126596+0000","last_fullsized":"2024-03-12T05:09:44.126596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125880+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:09:44.125880+0000","last_peered":"2024-03-12T05:09:44.125880+0000","last_clean":"2024-03-12T05:09:44.125880+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:09:44.125880+0000","last_undegraded":"2024-03-12T05:09:44.125880+0000","last_fullsized":"2024-03-12T05:09:44.125880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129251+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:09:44.129251+0000","last_peered":"2024-03-12T05:09:44.129251+0000","last_clean":"2024-03-12T05:09:44.129251+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:09:44.129251+0000","last_undegraded":"2024-03-12T05:09:44.129251+0000","last_fullsized":"2024-03-12T05:09:44.129251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131447+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:09:44.131447+0000","last_peered":"2024-03-12T05:09:44.131447+0000","last_clean":"2024-03-12T05:09:44.131447+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:09:44.131447+0000","last_undegraded":"2024-03-12T05:09:44.131447+0000","last_fullsized":"2024-03-12T05:09:44.131447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131203+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:09:44.131203+0000","last_peered":"2024-03-12T05:09:44.131203+0000","last_clean":"2024-03-12T05:09:44.131203+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:09:44.131203+0000","last_undegraded":"2024-03-12T05:09:44.131203+0000","last_fullsized":"2024-03-12T05:09:44.131203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.124551+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:09:44.124551+0000","last_peered":"2024-03-12T05:09:44.124551+0000","last_clean":"2024-03-12T05:09:44.124551+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:09:44.124551+0000","last_undegraded":"2024-03-12T05:09:44.124551+0000","last_fullsized":"2024-03-12T05:09:44.124551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129436+0000","last_change":"2024-03-12T05:09:44.129436+0000","last_active":"2024-03-12T05:09:44.129436+0000","last_peered":"2024-03-12T05:09:44.129436+0000","last_clean":"2024-03-12T05:09:44.129436+0000","last_became_active":"2024-03-12T05:09:44.129063+0000","last_became_peered":"2024-03-12T05:09:44.129063+0000","last_unstale":"2024-03-12T05:09:44.129436+0000","last_undegraded":"2024-03-12T05:09:44.129436+0000","last_fullsized":"2024-03-12T05:09:44.129436+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:06.724357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.129493+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:09:44.129493+0000","last_peered":"2024-03-12T05:09:44.129493+0000","last_clean":"2024-03-12T05:09:44.129493+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:09:44.129493+0000","last_undegraded":"2024-03-12T05:09:44.129493+0000","last_fullsized":"2024-03-12T05:09:44.129493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132630+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:09:44.132630+0000","last_peered":"2024-03-12T05:09:44.132630+0000","last_clean":"2024-03-12T05:09:44.132630+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:09:44.132630+0000","last_undegraded":"2024-03-12T05:09:44.132630+0000","last_fullsized":"2024-03-12T05:09:44.132630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381766+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:09:44.381766+0000","last_peered":"2024-03-12T05:09:44.381766+0000","last_clean":"2024-03-12T05:09:44.381766+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:09:44.381766+0000","last_undegraded":"2024-03-12T05:09:44.381766+0000","last_fullsized":"2024-03-12T05:09:44.381766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128940+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:09:44.128940+0000","last_peered":"2024-03-12T05:09:44.128940+0000","last_clean":"2024-03-12T05:09:44.128940+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:09:44.128940+0000","last_undegraded":"2024-03-12T05:09:44.128940+0000","last_fullsized":"2024-03-12T05:09:44.128940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381482+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:09:44.381482+0000","last_peered":"2024-03-12T05:09:44.381482+0000","last_clean":"2024-03-12T05:09:44.381482+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:09:44.381482+0000","last_undegraded":"2024-03-12T05:09:44.381482+0000","last_fullsized":"2024-03-12T05:09:44.381482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126791+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:09:44.126791+0000","last_peered":"2024-03-12T05:09:44.126791+0000","last_clean":"2024-03-12T05:09:44.126791+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:09:44.126791+0000","last_undegraded":"2024-03-12T05:09:44.126791+0000","last_fullsized":"2024-03-12T05:09:44.126791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126227+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:09:44.126227+0000","last_peered":"2024-03-12T05:09:44.126227+0000","last_clean":"2024-03-12T05:09:44.126227+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:09:44.126227+0000","last_undegraded":"2024-03-12T05:09:44.126227+0000","last_fullsized":"2024-03-12T05:09:44.126227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128442+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:09:44.128442+0000","last_peered":"2024-03-12T05:09:44.128442+0000","last_clean":"2024-03-12T05:09:44.128442+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:09:44.128442+0000","last_undegraded":"2024-03-12T05:09:44.128442+0000","last_fullsized":"2024-03-12T05:09:44.128442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66507,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":745,"num_read_kb":442,"num_write":236,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":199521,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":456844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1106785,"data_compressed":37976,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":456383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070507,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":456089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1049670,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":456100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":516597,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":978944,"data_stored":457830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":516927,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":455925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1048957,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61336,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":61666,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":61039,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:09:58.540 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T05:09:58.541 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T05:09:58.541 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T05:09:58.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T05:09:58.787 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:09:59.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:09:59 smithi130 ceph-mon[35398]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:09:59.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:09:59 smithi116 ceph-mon[35486]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:09:59.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:09:59 smithi045 ceph-mon[32174]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:10:00.258 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:00 smithi045 ceph-mon[32174]: overall HEALTH_OK 2024-03-12T05:10:00.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:00 smithi130 ceph-mon[35398]: overall HEALTH_OK 2024-03-12T05:10:00.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:00 smithi116 ceph-mon[35486]: overall HEALTH_OK 2024-03-12T05:10:00.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 -- 172.21.15.45:0/3213408154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100590 msgr2=0x7fd030100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 --2- 172.21.15.45:0/3213408154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100590 0x7fd030100970 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd020009990 tx=0x7fd020009ca0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 -- 172.21.15.45:0/3213408154 shutdown_connections 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 --2- 172.21.15.45:0/3213408154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 0x7fd03010d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 --2- 172.21.15.45:0/3213408154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100590 0x7fd030100970 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 --2- 172.21.15.45:0/3213408154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100f40 0x7fd030104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 -- 172.21.15.45:0/3213408154 >> 172.21.15.45:0/3213408154 conn(0x7fd030077170 msgr2=0x7fd0300fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 -- 172.21.15.45:0/3213408154 shutdown_connections 2024-03-12T05:10:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.517+0000 7fd0351ab700 1 -- 172.21.15.45:0/3213408154 wait complete. 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 Processor -- start 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 -- start start 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100590 0x7fd030198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 0x7fd03019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03019d030 con 0x7fd030100f40 2024-03-12T05:10:00.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03019d1a0 con 0x7fd030105670 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd0351ab700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03019d360 con 0x7fd030100590 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45456/0 (socket says 172.21.15.45:45456) 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 -- 172.21.15.45:0/2080573307 learned_addr learned my addr 172.21.15.45:0/2080573307 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02f59e700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 0x7fd03019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02ed9d700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100590 0x7fd030198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 -- 172.21.15.45:0/2080573307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100590 msgr2=0x7fd030198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100590 0x7fd030198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 -- 172.21.15.45:0/2080573307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 msgr2=0x7fd03019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 0x7fd03019ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.518+0000 7fd02e59c700 1 -- 172.21.15.45:0/2080573307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200095f0 con 0x7fd030100f40 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd02e59c700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c9e0 tx=0x7fd02400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:00.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024007aa0 con 0x7fd030100f40 2024-03-12T05:10:00.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03019d640 con 0x7fd030100f40 2024-03-12T05:10:00.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd02400ce80 con 0x7fd030100f40 2024-03-12T05:10:00.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024018420 con 0x7fd030100f40 2024-03-12T05:10:00.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.519+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0301a4ea0 con 0x7fd030100f40 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.520+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0301088a0 con 0x7fd030100f40 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.521+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd02400f450 con 0x7fd030100f40 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.522+0000 7fd017fff700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 0x7fd0180a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.522+0000 7fd02ed9d700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 0x7fd0180a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.522+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd0240cd4b0 con 0x7fd030100f40 2024-03-12T05:10:00.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.522+0000 7fd02ed9d700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 0x7fd0180a3ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098e0 tx=0x7fd020005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.526+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd024096570 con 0x7fd030100f40 2024-03-12T05:10:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.773+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd030199510 con 0x7fd030100f40 2024-03-12T05:10:00.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.774+0000 7fd017fff700 1 -- 172.21.15.45:0/2080573307 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd024099c30 con 0x7fd030100f40 2024-03-12T05:10:00.772 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:00.772 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T05:10:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 msgr2=0x7fd0180a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 0x7fd0180a3ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098e0 tx=0x7fd020005960 comp rx=0 tx=0).stop 2024-03-12T05:10:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 msgr2=0x7fd030198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c9e0 tx=0x7fd02400ccf0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 shutdown_connections 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.776+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd0180a1b30 0x7fd0180a3ff0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd030105670 0x7fd03019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd030100590 0x7fd030198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 --2- 172.21.15.45:0/2080573307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd030100f40 0x7fd030198690 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 >> 172.21.15.45:0/2080573307 conn(0x7fd030077170 msgr2=0x7fd0300fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 shutdown_connections 2024-03-12T05:10:00.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:00.777+0000 7fd0351ab700 1 -- 172.21.15.45:0/2080573307 wait complete. 2024-03-12T05:10:01.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:01 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2080573307' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:10:01.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:01 smithi130 ceph-mon[35398]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:10:01.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:01 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2080573307' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:10:01.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:01 smithi116 ceph-mon[35486]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:10:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:01 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2080573307' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:10:01.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:01 smithi045 ceph-mon[32174]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-12T05:10:01.555 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T05:10:01.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- 172.21.15.45:0/507780590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4106e80 msgr2=0x7fd6e41092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/507780590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4106e80 0x7fd6e41092d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009ca0 tx=0x7fd6dc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- 172.21.15.45:0/507780590 shutdown_connections 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/507780590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4106e80 0x7fd6e41092d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/507780590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 0x7fd6e4106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/507780590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6e4101c20 0x7fd6e4104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- 172.21.15.45:0/507780590 >> 172.21.15.45:0/507780590 conn(0x7fd6e40fb610 msgr2=0x7fd6e40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- 172.21.15.45:0/507780590 shutdown_connections 2024-03-12T05:10:01.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- 172.21.15.45:0/507780590 wait complete. 2024-03-12T05:10:01.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 Processor -- start 2024-03-12T05:10:01.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.669+0000 7fd6ea40f640 1 -- start start 2024-03-12T05:10:01.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 0x7fd6e40ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6e4106e80 0x7fd6e4100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4100870 con 0x7fd6e4101c20 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41009e0 con 0x7fd6e4104550 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6ea40f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41a11c0 con 0x7fd6e4106e80 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45466/0 (socket says 172.21.15.45:45466) 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 -- 172.21.15.45:0/1097119700 learned_addr learned my addr 172.21.15.45:0/1097119700 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e37fe640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 0x7fd6e40ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 -- 172.21.15.45:0/1097119700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6e4106e80 msgr2=0x7fd6e4100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6e4106e80 0x7fd6e4100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 -- 172.21.15.45:0/1097119700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 msgr2=0x7fd6e40ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 0x7fd6e40ffbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 -- 172.21.15.45:0/1097119700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009900 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e3fff640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c9a0 tx=0x7fd6d400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.670+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4007bb0 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.671+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e41a1470 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.671+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd6d4007d10 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.671+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4004e80 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.671+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e41a1980 con 0x7fd6e4101c20 2024-03-12T05:10:01.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.672+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e40fd100 con 0x7fd6e4101c20 2024-03-12T05:10:01.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.673+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd6d40046c0 con 0x7fd6e4101c20 2024-03-12T05:10:01.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.674+0000 7fd6e17fa640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 0x7fd6d00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.674+0000 7fd6e37fe640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 0x7fd6d00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.674+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd6d40cd390 con 0x7fd6e4101c20 2024-03-12T05:10:01.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.674+0000 7fd6e37fe640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 0x7fd6d00a3f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005fd0 tx=0x7fd6cc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:01.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.678+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd6d4096450 con 0x7fd6e4101c20 2024-03-12T05:10:01.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.861+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6e402ce70 con 0x7fd6e4101c20 2024-03-12T05:10:01.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.862+0000 7fd6e17fa640 1 -- 172.21.15.45:0/1097119700 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7fd6d4099b10 con 0x7fd6e4101c20 2024-03-12T05:10:01.860 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:01.860 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 msgr2=0x7fd6d00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 0x7fd6d00a3f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc005fd0 tx=0x7fd6cc005850 comp rx=0 tx=0).stop 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 msgr2=0x7fd6e4101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c9a0 tx=0x7fd6d400ce60 comp rx=0 tx=0).stop 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 shutdown_connections 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd6d00a1ac0 0x7fd6d00a3f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6e4106e80 0x7fd6e4100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6e4104550 0x7fd6e40ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 --2- 172.21.15.45:0/1097119700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6e4101c20 0x7fd6e4101590 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 >> 172.21.15.45:0/1097119700 conn(0x7fd6e40fb610 msgr2=0x7fd6e4102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 shutdown_connections 2024-03-12T05:10:01.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.864+0000 7fd6ea40f640 1 -- 172.21.15.45:0/1097119700 wait complete. 2024-03-12T05:10:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:01.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 -- 172.21.15.45:0/1982978597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102380 msgr2=0x7f8308102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 --2- 172.21.15.45:0/1982978597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102380 0x7f8308102800 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009900 tx=0x7f82f002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 -- 172.21.15.45:0/1982978597 shutdown_connections 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 --2- 172.21.15.45:0/1982978597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102380 0x7f8308102800 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 --2- 172.21.15.45:0/1982978597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102d40 0x7f83081095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 --2- 172.21.15.45:0/1982978597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 0x7f83081015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 -- 172.21.15.45:0/1982978597 >> 172.21.15.45:0/1982978597 conn(0x7f83080fc700 msgr2=0x7f83080feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 -- 172.21.15.45:0/1982978597 shutdown_connections 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.984+0000 7f830db02640 1 -- 172.21.15.45:0/1982978597 wait complete. 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 Processor -- start 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 -- start start 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 0x7f8308199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102380 0x7f8308199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83081a0ea0 con 0x7f8308102d40 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83081a1010 con 0x7f8308101190 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f830db02640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83081a11d0 con 0x7f8308102380 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f8307fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f8307fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45472/0 (socket says 172.21.15.45:45472) 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.985+0000 7f8307fff640 1 -- 172.21.15.45:0/2535811625 learned_addr learned my addr 172.21.15.45:0/2535811625 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:01.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8306ffd640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102380 0x7f8308199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f83077fe640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 0x7f8308199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 -- 172.21.15.45:0/2535811625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102380 msgr2=0x7f8308199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102380 0x7f8308199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 -- 172.21.15.45:0/2535811625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 msgr2=0x7f8308199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 0x7f8308199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 -- 172.21.15.45:0/2535811625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0009560 con 0x7f8308102d40 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8307fff640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f830000c970 tx=0x7f830000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300007bb0 con 0x7f8308102d40 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8300007d10 con 0x7f8308102d40 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83081a14b0 con 0x7f8308102d40 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300004e40 con 0x7f8308102d40 2024-03-12T05:10:01.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.986+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83081a19c0 con 0x7f8308102d40 2024-03-12T05:10:01.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.987+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830810caa0 con 0x7f8308102d40 2024-03-12T05:10:01.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.988+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f8300005020 con 0x7f8308102d40 2024-03-12T05:10:01.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.989+0000 7f8304ff9640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 0x7f82f40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:01.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.989+0000 7f83077fe640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 0x7f82f40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:01.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.989+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f83000cd210 con 0x7f8308102d40 2024-03-12T05:10:01.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.990+0000 7f83077fe640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 0x7f82f40a4000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f83081021e0 tx=0x7f82f8009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:01.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:01.994+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83000962d0 con 0x7f8308102d40 2024-03-12T05:10:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.176+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83081a1c40 con 0x7f8308102d40 2024-03-12T05:10:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.177+0000 7f8304ff9640 1 -- 172.21.15.45:0/2535811625 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f830001d020 con 0x7f8308102d40 2024-03-12T05:10:02.175 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 msgr2=0x7f82f40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 0x7f82f40a4000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f83081021e0 tx=0x7f82f8009460 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 msgr2=0x7f83081a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f830000c970 tx=0x7f830000ce30 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 shutdown_connections 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f82f40a1b40 0x7f82f40a4000 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8308102d40 0x7f83081a0960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8308102380 0x7f8308199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 --2- 172.21.15.45:0/2535811625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8308101190 0x7f8308199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 >> 172.21.15.45:0/2535811625 conn(0x7f83080fc700 msgr2=0x7f83080733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 shutdown_connections 2024-03-12T05:10:02.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.179+0000 7f830db02640 1 -- 172.21.15.45:0/2535811625 wait complete. 2024-03-12T05:10:02.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 -- 172.21.15.45:0/1931167622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101c30 msgr2=0x7fdac01084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 --2- 172.21.15.45:0/1931167622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101c30 0x7fdac01084c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009ca0 tx=0x7fdab002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 -- 172.21.15.45:0/1931167622 shutdown_connections 2024-03-12T05:10:02.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 --2- 172.21.15.45:0/1931167622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101c30 0x7fdac01084c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 --2- 172.21.15.45:0/1931167622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101270 0x7fdac01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 --2- 172.21.15.45:0/1931167622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 0x7fdac01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 -- 172.21.15.45:0/1931167622 >> 172.21.15.45:0/1931167622 conn(0x7fdac00fb610 msgr2=0x7fdac00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 -- 172.21.15.45:0/1931167622 shutdown_connections 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.297+0000 7fdac8365640 1 -- 172.21.15.45:0/1931167622 wait complete. 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 Processor -- start 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 -- start start 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 0x7fdac01994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 0x7fdac01a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a0fe0 con 0x7fdac0101270 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a1150 con 0x7fdac0101c30 2024-03-12T05:10:02.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac8365640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a1310 con 0x7fdac0100080 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45480/0 (socket says 172.21.15.45:45480) 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 -- 172.21.15.45:0/4176187585 learned_addr learned my addr 172.21.15.45:0/4176187585 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac6101640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 0x7fdac01994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac6902640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 0x7fdac01a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 -- 172.21.15.45:0/4176187585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 msgr2=0x7fdac01994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 0x7fdac01994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 -- 172.21.15.45:0/4176187585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 msgr2=0x7fdac01a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 0x7fdac01a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.298+0000 7fdac5900640 1 -- 172.21.15.45:0/4176187585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009900 con 0x7fdac0101270 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdac6902640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 0x7fdac01a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdac5900640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdab400b4a0 tx=0x7fdab400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4010610 con 0x7fdac0101270 2024-03-12T05:10:02.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdab4010c20 con 0x7fdac0101270 2024-03-12T05:10:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac01a15f0 con 0x7fdac0101270 2024-03-12T05:10:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4017d00 con 0x7fdac0101270 2024-03-12T05:10:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.299+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac01a1b80 con 0x7fdac0101270 2024-03-12T05:10:02.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.300+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac010cb20 con 0x7fdac0101270 2024-03-12T05:10:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.301+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdab4010770 con 0x7fdac0101270 2024-03-12T05:10:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.302+0000 7fdabf7fe640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 0x7fdaa80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.302+0000 7fdac6101640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 0x7fdaa80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.302+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdab40cd1f0 con 0x7fdac0101270 2024-03-12T05:10:02.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.302+0000 7fdac6101640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 0x7fdaa80a3f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdac01010d0 tx=0x7fdab8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.305+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdab40963c0 con 0x7fdac0101270 2024-03-12T05:10:02.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:02 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1097119700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:02.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:02 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2535811625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:02.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.487+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdac01a2010 con 0x7fdac0101270 2024-03-12T05:10:02.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.487+0000 7fdabf7fe640 1 -- 172.21.15.45:0/4176187585 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdab4099a80 con 0x7fdac0101270 2024-03-12T05:10:02.485 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 msgr2=0x7fdaa80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 0x7fdaa80a3f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdac01010d0 tx=0x7fdab8008040 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 msgr2=0x7fdac0199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdab400b4a0 tx=0x7fdab400b960 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 shutdown_connections 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1ac0 0x7fdaa80a3f80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac0101270 0x7fdac0199a30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0101c30 0x7fdac01a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 --2- 172.21.15.45:0/4176187585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0100080 0x7fdac01994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 >> 172.21.15.45:0/4176187585 conn(0x7fdac00fb610 msgr2=0x7fdac0106a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 shutdown_connections 2024-03-12T05:10:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.489+0000 7fdac8365640 1 -- 172.21.15.45:0/4176187585 wait complete. 2024-03-12T05:10:02.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:02 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1097119700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:02.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:02 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2535811625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:02.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:02 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1097119700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:02.496 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:02 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2535811625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:02.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 -- 172.21.15.45:0/946969797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01011b0 msgr2=0x7fbbc01015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/946969797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01011b0 0x7fbbc01015d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009900 tx=0x7fbbb002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 -- 172.21.15.45:0/946969797 shutdown_connections 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/946969797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 0x7fbbc01095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/946969797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01023a0 0x7fbbc0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/946969797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01011b0 0x7fbbc01015d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 -- 172.21.15.45:0/946969797 >> 172.21.15.45:0/946969797 conn(0x7fbbc00fc720 msgr2=0x7fbbc00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 -- 172.21.15.45:0/946969797 shutdown_connections 2024-03-12T05:10:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.606+0000 7fbbc86dd640 1 -- 172.21.15.45:0/946969797 wait complete. 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 Processor -- start 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 -- start start 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01011b0 0x7fbbc01971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 0x7fbbc019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc019eca0 con 0x7fbbc01023a0 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc019ee10 con 0x7fbbc0102d60 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc86dd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc019efd0 con 0x7fbbc01011b0 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45504/0 (socket says 172.21.15.45:45504) 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 -- 172.21.15.45:0/3802554260 learned_addr learned my addr 172.21.15.45:0/3802554260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc6479640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01011b0 0x7fbbc01971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc6c7a640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 0x7fbbc019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 -- 172.21.15.45:0/3802554260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01011b0 msgr2=0x7fbbc01971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01011b0 0x7fbbc01971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 -- 172.21.15.45:0/3802554260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 msgr2=0x7fbbc019e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 0x7fbbc019e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 -- 172.21.15.45:0/3802554260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0009560 con 0x7fbbc01023a0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.607+0000 7fbbc5c78640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00d770 tx=0x7fbbbc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.608+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc009c30 con 0x7fbbc01023a0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.608+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc019f2b0 con 0x7fbbc01023a0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.608+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbbbc00de80 con 0x7fbbc01023a0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.608+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc004e00 con 0x7fbbc01023a0 2024-03-12T05:10:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.608+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc019f7c0 con 0x7fbbc01023a0 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.609+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc0067f70 con 0x7fbbc01023a0 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.610+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbbbc004f60 con 0x7fbbc01023a0 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.611+0000 7fbbb77fe640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 0x7fbbac0a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.611+0000 7fbbc6479640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 0x7fbbac0a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.611+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbbbc0ccc30 con 0x7fbbc01023a0 2024-03-12T05:10:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.611+0000 7fbbc6479640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 0x7fbbac0a3e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002c7d0 tx=0x7fbbb0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.615+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbbbc095d40 con 0x7fbbc01023a0 2024-03-12T05:10:02.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.798+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbbc010ca70 con 0x7fbbc01023a0 2024-03-12T05:10:02.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.798+0000 7fbbb77fe640 1 -- 172.21.15.45:0/3802554260 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbbbc0200a0 con 0x7fbbc01023a0 2024-03-12T05:10:02.799 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:02.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 msgr2=0x7fbbac0a3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 0x7fbbac0a3e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002c7d0 tx=0x7fbbb0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 msgr2=0x7fbbc01976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00d770 tx=0x7fbbbc00dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 shutdown_connections 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbbac0a19a0 0x7fbbac0a3e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbc01023a0 0x7fbbc01976f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbc0102d60 0x7fbbc019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 --2- 172.21.15.45:0/3802554260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbc01011b0 0x7fbbc01971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 >> 172.21.15.45:0/3802554260 conn(0x7fbbc00fc720 msgr2=0x7fbbc0105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 shutdown_connections 2024-03-12T05:10:02.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.801+0000 7fbbc86dd640 1 -- 172.21.15.45:0/3802554260 wait complete. 2024-03-12T05:10:02.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 -- 172.21.15.45:0/1594674314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 msgr2=0x7fe298102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 --2- 172.21.15.45:0/1594674314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe298102810 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009900 tx=0x7fe28c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 -- 172.21.15.45:0/1594674314 shutdown_connections 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 --2- 172.21.15.45:0/1594674314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe298102810 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 --2- 172.21.15.45:0/1594674314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe298102d50 0x7fe2981095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 --2- 172.21.15.45:0/1594674314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 0x7fe2981015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 -- 172.21.15.45:0/1594674314 >> 172.21.15.45:0/1594674314 conn(0x7fe2980fc730 msgr2=0x7fe2980feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 -- 172.21.15.45:0/1594674314 shutdown_connections 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.917+0000 7fe29e165640 1 -- 172.21.15.45:0/1594674314 wait complete. 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 Processor -- start 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 -- start start 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 0x7fe298197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe298102d50 0x7fe29819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29819ec60 con 0x7fe298102390 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29819edd0 con 0x7fe2981011a0 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe29e165640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29819ef90 con 0x7fe298102d50 2024-03-12T05:10:02.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45526/0 (socket says 172.21.15.45:45526) 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 -- 172.21.15.45:0/2727433873 learned_addr learned my addr 172.21.15.45:0/2727433873 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe2977fe640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 0x7fe298197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 -- 172.21.15.45:0/2727433873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe298102d50 msgr2=0x7fe29819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe298102d50 0x7fe29819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 -- 172.21.15.45:0/2727433873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 msgr2=0x7fe298197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 0x7fe298197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.918+0000 7fe296ffd640 1 -- 172.21.15.45:0/2727433873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c009560 con 0x7fe298102390 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe296ffd640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe28c036040 tx=0x7fe28c004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003800 con 0x7fe298102390 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe28c02de90 con 0x7fe298102390 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c005600 con 0x7fe298102390 2024-03-12T05:10:02.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29819f210 con 0x7fe298102390 2024-03-12T05:10:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.919+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29819f7a0 con 0x7fe298102390 2024-03-12T05:10:02.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.920+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2981084c0 con 0x7fe298102390 2024-03-12T05:10:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.921+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe28c004dc0 con 0x7fe298102390 2024-03-12T05:10:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.922+0000 7fe294ff9640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 0x7fe2800a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:02.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.922+0000 7fe2977fe640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 0x7fe2800a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:02.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.922+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe28c0efff0 con 0x7fe298102390 2024-03-12T05:10:02.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.922+0000 7fe2977fe640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 0x7fe2800a4000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2981021f0 tx=0x7fe288005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:02.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:02.925+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe28c0b92a0 con 0x7fe298102390 2024-03-12T05:10:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.144+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe29810caa0 con 0x7fe298102390 2024-03-12T05:10:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.145+0000 7fe294ff9640 1 -- 172.21.15.45:0/2727433873 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fe28c043020 con 0x7fe298102390 2024-03-12T05:10:03.143 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:03.143 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 msgr2=0x7fe2800a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 0x7fe2800a4000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2981021f0 tx=0x7fe288005f90 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 msgr2=0x7fe2981976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe28c036040 tx=0x7fe28c004370 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 shutdown_connections 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe2800a1b40 0x7fe2800a4000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298102390 0x7fe2981976b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe298102d50 0x7fe29819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 --2- 172.21.15.45:0/2727433873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2981011a0 0x7fe298197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 >> 172.21.15.45:0/2727433873 conn(0x7fe2980fc730 msgr2=0x7fe298105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 shutdown_connections 2024-03-12T05:10:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.147+0000 7fe29e165640 1 -- 172.21.15.45:0/2727433873 wait complete. 2024-03-12T05:10:03.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:03.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.268+0000 7fd2663d7640 1 -- 172.21.15.45:0/1882439152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 msgr2=0x7fd260101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.268+0000 7fd2663d7640 1 --2- 172.21.15.45:0/1882439152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd260101730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd250009900 tx=0x7fd25002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- 172.21.15.45:0/1882439152 shutdown_connections 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- 172.21.15.45:0/1882439152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd260101730 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- 172.21.15.45:0/1882439152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd260101c70 0x7fd260108500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- 172.21.15.45:0/1882439152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2601000c0 0x7fd2601004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- 172.21.15.45:0/1882439152 >> 172.21.15.45:0/1882439152 conn(0x7fd2600fb630 msgr2=0x7fd2600fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- 172.21.15.45:0/1882439152 shutdown_connections 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- 172.21.15.45:0/1882439152 wait complete. 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 Processor -- start 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- start start 2024-03-12T05:10:03.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 0x7fd26006a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd260101c70 0x7fd26019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26019ecd0 con 0x7fd2601012b0 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26019ee40 con 0x7fd2601000c0 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.269+0000 7fd2663d7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26019f000 con 0x7fd260101c70 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45536/0 (socket says 172.21.15.45:45536) 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 -- 172.21.15.45:0/3490574254 learned_addr learned my addr 172.21.15.45:0/3490574254 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25ffff640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 0x7fd26006a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 -- 172.21.15.45:0/3490574254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd260101c70 msgr2=0x7fd26019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd264974640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd260101c70 0x7fd26019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd260101c70 0x7fd26019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 -- 172.21.15.45:0/3490574254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 msgr2=0x7fd26006a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 0x7fd26006a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 -- 172.21.15.45:0/3490574254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250009560 con 0x7fd2601012b0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25ffff640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 0x7fd26006a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25f7fe640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd250036040 tx=0x7fd250004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250003800 con 0x7fd2601012b0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd25002de90 con 0x7fd2601012b0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26019f280 con 0x7fd2601012b0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250005600 con 0x7fd2601012b0 2024-03-12T05:10:03.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.270+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26019f810 con 0x7fd2601012b0 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.271+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2600683c0 con 0x7fd2601012b0 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.272+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd250004dc0 con 0x7fd2601012b0 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.273+0000 7fd25d7fa640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 0x7fd24c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.273+0000 7fd25ffff640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 0x7fd24c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.273+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd2500ef890 con 0x7fd2601012b0 2024-03-12T05:10:03.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.274+0000 7fd25ffff640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 0x7fd24c0a3f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd260101110 tx=0x7fd248005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.278+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd2500b8a60 con 0x7fd2601012b0 2024-03-12T05:10:03.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.460+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd260107430 con 0x7fd2601012b0 2024-03-12T05:10:03.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.462+0000 7fd25d7fa640 1 -- 172.21.15.45:0/3490574254 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7fd2500bc120 con 0x7fd2601012b0 2024-03-12T05:10:03.459 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:03.460 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 msgr2=0x7fd24c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 0x7fd24c0a3f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd260101110 tx=0x7fd248005d30 comp rx=0 tx=0).stop 2024-03-12T05:10:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 msgr2=0x7fd26019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd250036040 tx=0x7fd250004370 comp rx=0 tx=0).stop 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 shutdown_connections 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd24c0a1ac0 0x7fd24c0a3f80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2601012b0 0x7fd26019bf00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd260101c70 0x7fd26019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 --2- 172.21.15.45:0/3490574254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2601000c0 0x7fd26006a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 >> 172.21.15.45:0/3490574254 conn(0x7fd2600fb630 msgr2=0x7fd260104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 shutdown_connections 2024-03-12T05:10:03.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.464+0000 7fd2663d7640 1 -- 172.21.15.45:0/3490574254 wait complete. 2024-03-12T05:10:03.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:03 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4176187585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:03 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3802554260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:03 smithi130 ceph-mon[35398]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:03.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:03 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2727433873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:03.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:03 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4176187585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:03 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3802554260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:03 smithi045 ceph-mon[32174]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:03.473 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:03 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2727433873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:03.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:03.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:03.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:03 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4176187585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:03 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3802554260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:03.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:03 smithi116 ceph-mon[35486]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:03.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:03 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2727433873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:03.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 -- 172.21.15.45:0/732404098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 msgr2=0x7fdec41084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 --2- 172.21.15.45:0/732404098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec41084c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009ca0 tx=0x7fdeb402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 -- 172.21.15.45:0/732404098 shutdown_connections 2024-03-12T05:10:03.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 --2- 172.21.15.45:0/732404098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec41084c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 --2- 172.21.15.45:0/732404098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4101270 0x7fdec41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 --2- 172.21.15.45:0/732404098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4100080 0x7fdec41004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 -- 172.21.15.45:0/732404098 >> 172.21.15.45:0/732404098 conn(0x7fdec40fb610 msgr2=0x7fdec40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 -- 172.21.15.45:0/732404098 shutdown_connections 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.584+0000 7fdec94c3640 1 -- 172.21.15.45:0/732404098 wait complete. 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 Processor -- start 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 -- start start 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4100080 0x7fdec4104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 0x7fdec4102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec41032f0 con 0x7fdec4101c30 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4103460 con 0x7fdec4101270 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec94c3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec41a11b0 con 0x7fdec4100080 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45550/0 (socket says 172.21.15.45:45550) 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec37fe640 1 -- 172.21.15.45:0/2007987515 learned_addr learned my addr 172.21.15.45:0/2007987515 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.585+0000 7fdec2ffd640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4100080 0x7fdec4104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec27fc640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 0x7fdec4102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 -- 172.21.15.45:0/2007987515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4100080 msgr2=0x7fdec4104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4100080 0x7fdec4104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 -- 172.21.15.45:0/2007987515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 msgr2=0x7fdec4102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 0x7fdec4102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 -- 172.21.15.45:0/2007987515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4009900 con 0x7fdec4101c30 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec27fc640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 0x7fdec4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec37fe640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4033000 tx=0x7fdeb4032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4034e70 con 0x7fdec4101c30 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdeb4003e20 con 0x7fdec4101c30 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec41a1400 con 0x7fdec4101c30 2024-03-12T05:10:03.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4004880 con 0x7fdec4101c30 2024-03-12T05:10:03.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.586+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec41a18b0 con 0x7fdec4101c30 2024-03-12T05:10:03.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.587+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec410b940 con 0x7fdec4101c30 2024-03-12T05:10:03.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.588+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdeb403b040 con 0x7fdec4101c30 2024-03-12T05:10:03.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.589+0000 7fdebbfff640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 0x7fdeb00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.589+0000 7fdec2ffd640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 0x7fdeb00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.589+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdeb40efb80 con 0x7fdec4101c30 2024-03-12T05:10:03.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.589+0000 7fdec2ffd640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 0x7fdeb00a3f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdebc005fd0 tx=0x7fdebc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.593+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdeb40b8cf0 con 0x7fdec4101c30 2024-03-12T05:10:03.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.775+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdec41a1d70 con 0x7fdec4101c30 2024-03-12T05:10:03.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.776+0000 7fdebbfff640 1 -- 172.21.15.45:0/2007987515 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdeb40bc3b0 con 0x7fdec4101c30 2024-03-12T05:10:03.773 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:03.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 msgr2=0x7fdeb00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 0x7fdeb00a3f80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdebc005fd0 tx=0x7fdebc00b410 comp rx=0 tx=0).stop 2024-03-12T05:10:03.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 msgr2=0x7fdec4102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4033000 tx=0x7fdeb4032980 comp rx=0 tx=0).stop 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 shutdown_connections 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdeb00a1ac0 0x7fdeb00a3f80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdec4101c30 0x7fdec4102b90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdec4101270 0x7fdec4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.777+0000 7fdec94c3640 1 --2- 172.21.15.45:0/2007987515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4100080 0x7fdec4104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.778+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 >> 172.21.15.45:0/2007987515 conn(0x7fdec40fb610 msgr2=0x7fdec40fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.778+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 shutdown_connections 2024-03-12T05:10:03.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.778+0000 7fdec94c3640 1 -- 172.21.15.45:0/2007987515 wait complete. 2024-03-12T05:10:03.786 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 -- 172.21.15.45:0/124794198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 msgr2=0x7efdc8107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 --2- 172.21.15.45:0/124794198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc8107c60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009ca0 tx=0x7efdc002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 -- 172.21.15.45:0/124794198 shutdown_connections 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 --2- 172.21.15.45:0/124794198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc8107c60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 --2- 172.21.15.45:0/124794198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8100a10 0x7efdc8100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 --2- 172.21.15.45:0/124794198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdc80ff820 0x7efdc80ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 -- 172.21.15.45:0/124794198 >> 172.21.15.45:0/124794198 conn(0x7efdc80fad70 msgr2=0x7efdc80fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 -- 172.21.15.45:0/124794198 shutdown_connections 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.894+0000 7efdcc840640 1 -- 172.21.15.45:0/124794198 wait complete. 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 Processor -- start 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 -- start start 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 0x7efdc8197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdc8100a10 0x7efdc8197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc819ebf0 con 0x7efdc81013d0 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc819ed60 con 0x7efdc80ff820 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdcc840640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc819ef20 con 0x7efdc8100a10 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45576/0 (socket says 172.21.15.45:45576) 2024-03-12T05:10:03.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 -- 172.21.15.45:0/1308647043 learned_addr learned my addr 172.21.15.45:0/1308647043 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc659c640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 0x7efdc8197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 -- 172.21.15.45:0/1308647043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdc8100a10 msgr2=0x7efdc8197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdc8100a10 0x7efdc8197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 -- 172.21.15.45:0/1308647043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 msgr2=0x7efdc8197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 0x7efdc8197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc6d9d640 1 -- 172.21.15.45:0/1308647043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0009900 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.895+0000 7efdc659c640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 0x7efdc8197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdc6d9d640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efdc0037040 tx=0x7efdc0032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0034e70 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc819f1a0 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdc0003e20 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc003d3f0 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.896+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc819f6a0 con 0x7efdc81013d0 2024-03-12T05:10:03.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.897+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc8106b40 con 0x7efdc81013d0 2024-03-12T05:10:03.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.898+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7efdc003b040 con 0x7efdc81013d0 2024-03-12T05:10:03.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.898+0000 7efdb77fe640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 0x7efdb00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:03.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.898+0000 7efdc659c640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 0x7efdb00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:03.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.899+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7efdc00f00b0 con 0x7efdc81013d0 2024-03-12T05:10:03.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.899+0000 7efdc659c640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 0x7efdb00a3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdb80058c0 tx=0x7efdb8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:03.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:03.902+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efdc00b9280 con 0x7efdc81013d0 2024-03-12T05:10:04.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.082+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efdc810aa20 con 0x7efdc81013d0 2024-03-12T05:10:04.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.082+0000 7efdb77fe640 1 -- 172.21.15.45:0/1308647043 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7efdc00bc940 con 0x7efdc81013d0 2024-03-12T05:10:04.080 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 msgr2=0x7efdb00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 0x7efdb00a3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdb80058c0 tx=0x7efdb8005850 comp rx=0 tx=0).stop 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 msgr2=0x7efdc819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efdc0037040 tx=0x7efdc0032980 comp rx=0 tx=0).stop 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 shutdown_connections 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7efdb00a1ac0 0x7efdb00a3f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efdc81013d0 0x7efdc819e6b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efdc8100a10 0x7efdc8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 --2- 172.21.15.45:0/1308647043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc80ff820 0x7efdc8197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 >> 172.21.15.45:0/1308647043 conn(0x7efdc80fad70 msgr2=0x7efdc8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 shutdown_connections 2024-03-12T05:10:04.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.084+0000 7efdcc840640 1 -- 172.21.15.45:0/1308647043 wait complete. 2024-03-12T05:10:04.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:04.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 -- 172.21.15.45:0/3264198472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1388106ee0 msgr2=0x7f13881092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 --2- 172.21.15.45:0/3264198472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1388106ee0 0x7f13881092d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1380009ca0 tx=0x7f138002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 -- 172.21.15.45:0/3264198472 shutdown_connections 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 --2- 172.21.15.45:0/3264198472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1388106ee0 0x7f13881092d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 --2- 172.21.15.45:0/3264198472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13881045b0 0x7f13881069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 --2- 172.21.15.45:0/3264198472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 0x7f1388104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 -- 172.21.15.45:0/3264198472 >> 172.21.15.45:0/3264198472 conn(0x7f13880fb630 msgr2=0x7f13880fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 -- 172.21.15.45:0/3264198472 shutdown_connections 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.200+0000 7f138de1c640 1 -- 172.21.15.45:0/3264198472 wait complete. 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 Processor -- start 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 -- start start 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 0x7f1388197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388106ee0 0x7f138819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138819ec10 con 0x7f13881045b0 2024-03-12T05:10:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138819ed80 con 0x7f1388106ee0 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f138de1c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138819ef40 con 0x7f1388101c80 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45586/0 (socket says 172.21.15.45:45586) 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f13877fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 0x7f1388197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f1386ffd640 1 -- 172.21.15.45:0/1262810426 learned_addr learned my addr 172.21.15.45:0/1262810426 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.201+0000 7f1387fff640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388106ee0 0x7f138819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 -- 172.21.15.45:0/1262810426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 msgr2=0x7f1388197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 0x7f1388197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 -- 172.21.15.45:0/1262810426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388106ee0 msgr2=0x7f138819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388106ee0 0x7f138819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 -- 172.21.15.45:0/1262810426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1380009900 con 0x7f13881045b0 2024-03-12T05:10:04.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.202+0000 7f1386ffd640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f137c00d770 tx=0x7f137c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.204+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c009c30 con 0x7f13881045b0 2024-03-12T05:10:04.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.204+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138819f220 con 0x7f13881045b0 2024-03-12T05:10:04.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.204+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f137c00de80 con 0x7f13881045b0 2024-03-12T05:10:04.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.204+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c00f600 con 0x7f13881045b0 2024-03-12T05:10:04.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.204+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138819f730 con 0x7f13881045b0 2024-03-12T05:10:04.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.205+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13880fd120 con 0x7f13881045b0 2024-03-12T05:10:04.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.206+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f137c010460 con 0x7f13881045b0 2024-03-12T05:10:04.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.206+0000 7f1384ff9640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 0x7f13700a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.206+0000 7f13877fe640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 0x7f13700a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.206+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f137c0cd220 con 0x7f13881045b0 2024-03-12T05:10:04.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.207+0000 7f13877fe640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 0x7f13700a3f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1378005d00 tx=0x7f137800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.211+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f137c096390 con 0x7f13881045b0 2024-03-12T05:10:04.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:04 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3490574254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:04.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:04 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2007987515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:04 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1308647043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.391+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f138802cda0 con 0x7f13881045b0 2024-03-12T05:10:04.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.392+0000 7f1384ff9640 1 -- 172.21.15.45:0/1262810426 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f137c099a50 con 0x7f13881045b0 2024-03-12T05:10:04.389 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:04.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 msgr2=0x7f13700a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 0x7f13700a3f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1378005d00 tx=0x7f137800b410 comp rx=0 tx=0).stop 2024-03-12T05:10:04.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 msgr2=0x7f1388197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f137c00d770 tx=0x7f137c00dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 shutdown_connections 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f13700a1ac0 0x7f13700a3f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13881045b0 0x7f1388197660 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1388106ee0 0x7f138819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 --2- 172.21.15.45:0/1262810426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1388101c80 0x7f1388197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.393+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 >> 172.21.15.45:0/1262810426 conn(0x7f13880fb630 msgr2=0x7f13881025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.394+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 shutdown_connections 2024-03-12T05:10:04.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.394+0000 7f138de1c640 1 -- 172.21.15.45:0/1262810426 wait complete. 2024-03-12T05:10:04.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:04.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:04 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3490574254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:04.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:04 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2007987515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:04 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1308647043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:04 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3490574254' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:04.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:04 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2007987515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:04 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1308647043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 -- 172.21.15.45:0/2658146125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea801000a0 msgr2=0x7fea801004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 --2- 172.21.15.45:0/2658146125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea801000a0 0x7fea801004c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fea78009900 tx=0x7fea7802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 -- 172.21.15.45:0/2658146125 shutdown_connections 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 --2- 172.21.15.45:0/2658146125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 --2- 172.21.15.45:0/2658146125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea80101290 0x7fea80101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 --2- 172.21.15.45:0/2658146125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea801000a0 0x7fea801004c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 -- 172.21.15.45:0/2658146125 >> 172.21.15.45:0/2658146125 conn(0x7fea800fb610 msgr2=0x7fea800fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 -- 172.21.15.45:0/2658146125 shutdown_connections 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.507+0000 7fea84f8a640 1 -- 172.21.15.45:0/2658146125 wait complete. 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 Processor -- start 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 -- start start 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea801000a0 0x7fea8010a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801093a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80109b00 con 0x7fea80101290 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80109c70 con 0x7fea80101c50 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea84f8a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea801a1380 con 0x7fea801000a0 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea7dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea7dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45598/0 (socket says 172.21.15.45:45598) 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea7dd9b640 1 -- 172.21.15.45:0/1430333308 learned_addr learned my addr 172.21.15.45:0/1430333308 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.508+0000 7fea7ed9d640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7e59c640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea801000a0 0x7fea8010a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 -- 172.21.15.45:0/1430333308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea801000a0 msgr2=0x7fea8010a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea801000a0 0x7fea8010a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 -- 172.21.15.45:0/1430333308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 msgr2=0x7fea801093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801093a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 -- 172.21.15.45:0/1430333308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78009560 con 0x7fea80101290 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7ed9d640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea7dd9b640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b4d0 tx=0x7fea6800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68010870 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fea68010e80 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68017360 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea801a1630 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.509+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea801a1b70 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.510+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea80107500 con 0x7fea80101290 2024-03-12T05:10:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.511+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fea680109d0 con 0x7fea80101290 2024-03-12T05:10:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.511+0000 7fea777fe640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 0x7fea6c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.512+0000 7fea7e59c640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 0x7fea6c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.512+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fea680ccf60 con 0x7fea80101290 2024-03-12T05:10:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.512+0000 7fea7e59c640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 0x7fea6c0a3f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea78036040 tx=0x7fea78005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.516+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea68096130 con 0x7fea80101290 2024-03-12T05:10:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.741+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea8010cb20 con 0x7fea80101290 2024-03-12T05:10:04.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.741+0000 7fea777fe640 1 -- 172.21.15.45:0/1430333308 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fea680997f0 con 0x7fea80101290 2024-03-12T05:10:04.739 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:04.740 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:04.740 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.743+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 msgr2=0x7fea6c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.743+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 0x7fea6c0a3f80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea78036040 tx=0x7fea78005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.743+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 msgr2=0x7fea80108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.743+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b4d0 tx=0x7fea6800b990 comp rx=0 tx=0).stop 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 shutdown_connections 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fea6c0a1ac0 0x7fea6c0a3f80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea80101290 0x7fea80108e60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea80101c50 0x7fea801093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 --2- 172.21.15.45:0/1430333308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea801000a0 0x7fea8010a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 >> 172.21.15.45:0/1430333308 conn(0x7fea800fb610 msgr2=0x7fea801046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 shutdown_connections 2024-03-12T05:10:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.744+0000 7fea84f8a640 1 -- 172.21.15.45:0/1430333308 wait complete. 2024-03-12T05:10:04.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:04.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 -- 172.21.15.45:0/1382285865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 msgr2=0x7f42281084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 --2- 172.21.15.45:0/1382285865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f42281084e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4220009ca0 tx=0x7f422002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 -- 172.21.15.45:0/1382285865 shutdown_connections 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 --2- 172.21.15.45:0/1382285865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f42281084e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 --2- 172.21.15.45:0/1382285865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4228101290 0x7f4228101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 --2- 172.21.15.45:0/1382285865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42281000a0 0x7f42281004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 -- 172.21.15.45:0/1382285865 >> 172.21.15.45:0/1382285865 conn(0x7f42280fb610 msgr2=0x7f42280fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 -- 172.21.15.45:0/1382285865 shutdown_connections 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.858+0000 7f422cb2c640 1 -- 172.21.15.45:0/1382285865 wait complete. 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 Processor -- start 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 -- start start 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42281000a0 0x7f42281970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4228101290 0x7f4228197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819eb80 con 0x7f4228101c50 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819ecf0 con 0x7f42281000a0 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422cb2c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819eeb0 con 0x7f4228101290 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45610/0 (socket says 172.21.15.45:45610) 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 -- 172.21.15.45:0/2265349089 learned_addr learned my addr 172.21.15.45:0/2265349089 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f422659c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42281000a0 0x7f42281970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f421ffff640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4228101290 0x7f4228197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 -- 172.21.15.45:0/2265349089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4228101290 msgr2=0x7f4228197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4228101290 0x7f4228197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 -- 172.21.15.45:0/2265349089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42281000a0 msgr2=0x7f42281970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:04.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42281000a0 0x7f42281970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.859+0000 7f4226d9d640 1 -- 172.21.15.45:0/2265349089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4220009900 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f4226d9d640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4220037040 tx=0x7f4220032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220034e70 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4220003e20 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422819f130 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422003d3f0 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.860+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422819f5e0 con 0x7f4228101c50 2024-03-12T05:10:04.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.861+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422810b920 con 0x7f4228101c50 2024-03-12T05:10:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.863+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f422003b040 con 0x7f4228101c50 2024-03-12T05:10:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.863+0000 7f421f7fe640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 0x7f42140a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.863+0000 7f422659c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 0x7f42140a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.863+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f42200f00b0 con 0x7f4228101c50 2024-03-12T05:10:04.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.864+0000 7f422659c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 0x7f42140a3f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42180058c0 tx=0x7f4218005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:04.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:04.867+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f42200b9280 con 0x7f4228101c50 2024-03-12T05:10:05.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.050+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f422819fac0 con 0x7f4228101c50 2024-03-12T05:10:05.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.051+0000 7f421f7fe640 1 -- 172.21.15.45:0/2265349089 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f42200bc940 con 0x7f4228101c50 2024-03-12T05:10:05.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:05.049 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 msgr2=0x7f42140a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 0x7f42140a3f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f42180058c0 tx=0x7f4218005850 comp rx=0 tx=0).stop 2024-03-12T05:10:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 msgr2=0x7f422819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4220037040 tx=0x7f4220032980 comp rx=0 tx=0).stop 2024-03-12T05:10:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 shutdown_connections 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42140a1ac0 0x7f42140a3f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4228101c50 0x7f422819e640 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4228101290 0x7f4228197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 --2- 172.21.15.45:0/2265349089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42281000a0 0x7f42281970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.053+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 >> 172.21.15.45:0/2265349089 conn(0x7f42280fb610 msgr2=0x7f42280fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.054+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 shutdown_connections 2024-03-12T05:10:05.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.054+0000 7f422cb2c640 1 -- 172.21.15.45:0/2265349089 wait complete. 2024-03-12T05:10:05.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 -- 172.21.15.45:0/1851658784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 msgr2=0x7fbc64104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 --2- 172.21.15.45:0/1851658784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc64104010 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009900 tx=0x7fbc6002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 -- 172.21.15.45:0/1851658784 shutdown_connections 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 --2- 172.21.15.45:0/1851658784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64106e80 0x7fbc641092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 --2- 172.21.15.45:0/1851658784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64104550 0x7fbc64106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 --2- 172.21.15.45:0/1851658784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc64104010 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 -- 172.21.15.45:0/1851658784 >> 172.21.15.45:0/1851658784 conn(0x7fbc640fb610 msgr2=0x7fbc640fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 -- 172.21.15.45:0/1851658784 shutdown_connections 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.167+0000 7fbc6b981640 1 -- 172.21.15.45:0/1851658784 wait complete. 2024-03-12T05:10:05.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 Processor -- start 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 -- start start 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64104550 0x7fbc64197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64106e80 0x7fbc6419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419ecb0 con 0x7fbc64101c20 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419ee20 con 0x7fbc64104550 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6b981640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419efe0 con 0x7fbc64106e80 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6971d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6971d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45622/0 (socket says 172.21.15.45:45622) 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc6971d640 1 -- 172.21.15.45:0/3271756237 learned_addr learned my addr 172.21.15.45:0/3271756237 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc69f1e640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64106e80 0x7fbc6419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.168+0000 7fbc68f1c640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64104550 0x7fbc64197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 -- 172.21.15.45:0/3271756237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64106e80 msgr2=0x7fbc6419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64106e80 0x7fbc6419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 -- 172.21.15.45:0/3271756237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64104550 msgr2=0x7fbc64197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64104550 0x7fbc64197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 -- 172.21.15.45:0/3271756237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60009560 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6971d640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbc60036040 tx=0x7fbc600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600044e0 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc6002de90 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6419f260 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60005470 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.169+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6419f7c0 con 0x7fbc64101c20 2024-03-12T05:10:05.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.170+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc640fd100 con 0x7fbc64101c20 2024-03-12T05:10:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.171+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbc60003800 con 0x7fbc64101c20 2024-03-12T05:10:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.172+0000 7fbc5a7fc640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 0x7fbc500a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.172+0000 7fbc68f1c640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 0x7fbc500a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.172+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbc600efce0 con 0x7fbc64101c20 2024-03-12T05:10:05.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.172+0000 7fbc68f1c640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 0x7fbc500a4000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc641986d0 tx=0x7fbc5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.176+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc600b8eb0 con 0x7fbc64101c20 2024-03-12T05:10:05.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:05 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1262810426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:05.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:05 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1430333308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:05 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2265349089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:05 smithi045 ceph-mon[32174]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:05.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.397+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6410ca60 con 0x7fbc64101c20 2024-03-12T05:10:05.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.398+0000 7fbc5a7fc640 1 -- 172.21.15.45:0/3271756237 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fbc600bc570 con 0x7fbc64101c20 2024-03-12T05:10:05.395 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:05.396 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:05.396 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 msgr2=0x7fbc500a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 0x7fbc500a4000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc641986d0 tx=0x7fbc5400b390 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 msgr2=0x7fbc641971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbc60036040 tx=0x7fbc600042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 shutdown_connections 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbc500a1b40 0x7fbc500a4000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc64106e80 0x7fbc6419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc64104550 0x7fbc64197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 --2- 172.21.15.45:0/3271756237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc64101c20 0x7fbc641971c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 >> 172.21.15.45:0/3271756237 conn(0x7fbc640fb610 msgr2=0x7fbc641026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 shutdown_connections 2024-03-12T05:10:05.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.400+0000 7fbc6b981640 1 -- 172.21.15.45:0/3271756237 wait complete. 2024-03-12T05:10:05.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-12T05:08:37.112985+0000', 'modified': '2024-03-12T05:08:43.349662+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24472, 'mds_3': 24467, 'mds_4': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.45:6839/1651909238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6838', 'nonce': 1651909238}, {'type': 'v1', 'addr': '172.21.15.45:6839', 'nonce': 1651909238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.45:6841/3025141330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6840', 'nonce': 3025141330}, {'type': 'v1', 'addr': '172.21.15.45:6841', 'nonce': 3025141330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/2049389812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 2049389812}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 2049389812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24421': {'gid': 24421, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/1602847227', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 1602847227}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 1602847227}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/4238162743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 4238162743}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 4238162743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.130:6835/1592150426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 1592150426}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 1592150426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6837/2581569461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2581569461}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2581569461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.130:6837/2852741522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2852741522}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2852741522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.130:6839/2939102833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2939102833}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2939102833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6839/397791649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 397791649}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 397791649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-12T05:10:05.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-12T05:10:05.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-12T05:10:05.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:05 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1262810426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:05.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:05 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1430333308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:05 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2265349089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:05 smithi130 ceph-mon[35398]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:05.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:05 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1262810426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:05.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:05 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1430333308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:05 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2265349089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:05.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:05 smithi116 ceph-mon[35486]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 -- 172.21.15.45:0/2460757037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 msgr2=0x7f6f6c075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 --2- 172.21.15.45:0/2460757037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c075f60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009900 tx=0x7f6f6002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 -- 172.21.15.45:0/2460757037 shutdown_connections 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 --2- 172.21.15.45:0/2460757037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c077720 0x7f6f6c077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 --2- 172.21.15.45:0/2460757037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c076d80 0x7f6f6c0771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 --2- 172.21.15.45:0/2460757037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c075f60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 -- 172.21.15.45:0/2460757037 >> 172.21.15.45:0/2460757037 conn(0x7f6f6c0fe930 msgr2=0x7f6f6c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 -- 172.21.15.45:0/2460757037 shutdown_connections 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.515+0000 7f6f72840640 1 -- 172.21.15.45:0/2460757037 wait complete. 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 Processor -- start 2024-03-12T05:10:05.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 -- start start 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c076d80 0x7f6f6c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 0x7f6f6c1a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1a5270 con 0x7f6f6c075b40 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1a53e0 con 0x7f6f6c077720 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f72840640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1a55a0 con 0x7f6f6c076d80 2024-03-12T05:10:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45640/0 (socket says 172.21.15.45:45640) 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.516+0000 7f6f6bfff640 1 -- 172.21.15.45:0/1115111679 learned_addr learned my addr 172.21.15.45:0/1115111679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c076d80 0x7f6f6c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 0x7f6f6c1a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 -- 172.21.15.45:0/1115111679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c076d80 msgr2=0x7f6f6c19dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c076d80 0x7f6f6c19dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 -- 172.21.15.45:0/1115111679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 msgr2=0x7f6f6c1a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 0x7f6f6c1a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 -- 172.21.15.45:0/1115111679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009560 con 0x7f6f6c075b40 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 0x7f6f6c1a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6f60036040 tx=0x7f6f600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f697fa640 1 -- 172.21.15.45:0/1115111679 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f600044e0 con 0x7f6f6c075b40 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f697fa640 1 -- 172.21.15.45:0/1115111679 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f6002de90 con 0x7f6f6c075b40 2024-03-12T05:10:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c1a5820 con 0x7f6f6c075b40 2024-03-12T05:10:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f697fa640 1 -- 172.21.15.45:0/1115111679 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60005470 con 0x7f6f6c075b40 2024-03-12T05:10:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.517+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c1a5d80 con 0x7f6f6c075b40 2024-03-12T05:10:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.519+0000 7f6f697fa640 1 -- 172.21.15.45:0/1115111679 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6f60003800 con 0x7f6f6c075b40 2024-03-12T05:10:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 Processor -- start 2024-03-12T05:10:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 -- start start 2024-03-12T05:10:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f697fa640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 0x7f6f540a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a013a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 0x556652a06470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566529bccd0 con 0x556652a06ae0 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665296ff20 con 0x556652a01970 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f75413740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665296d450 con 0x556652a00f80 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f697fa640 1 -- 172.21.15.45:0/1115111679 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6f600ef650 con 0x7f6f6c075b40 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 0x7f6f540a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45656/0 (socket says 172.21.15.45:45656) 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 -- 172.21.15.45:0/977987239 learned_addr learned my addr 172.21.15.45:0/977987239 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 0x556652a06470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 -- 172.21.15.45:0/977987239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 msgr2=0x556652a013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 -- 172.21.15.45:0/977987239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 msgr2=0x556652a06470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 0x556652a06470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f70ddd640 1 -- 172.21.15.45:0/977987239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55665291df20 con 0x556652a06ae0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.520+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 0x556652a06470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 0x7f6f540a4000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f64005c40 tx=0x7f6f64009e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f64010bd0 tx=0x7f6f6401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db74fe1e4e02aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f527fc640 1 -- 172.21.15.45:0/977987239 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6400cb40 con 0x556652a06ae0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f527fc640 1 -- 172.21.15.45:0/977987239 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f6400cca0 con 0x556652a06ae0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f527fc640 1 -- 172.21.15.45:0/977987239 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64005790 con 0x556652a06ae0 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 -- 172.21.15.45:0/977987239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 msgr2=0x556652a08f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f64010bd0 tx=0x7f6f6401c000 comp rx=0 tx=0).stop 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 -- 172.21.15.45:0/977987239 shutdown_connections 2024-03-12T05:10:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a08f70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a01970 0x556652a06470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 --2- 172.21.15.45:0/977987239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 -- 172.21.15.45:0/977987239 >> 172.21.15.45:0/977987239 conn(0x5566529fc4f0 msgr2=0x5566529fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 -- 172.21.15.45:0/977987239 shutdown_connections 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.521+0000 7f6f75413740 1 -- 172.21.15.45:0/977987239 wait complete. 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 -- start start 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 Processor -- start 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a0b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a12710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556652968c30 con 0x556652a01970 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665290dd30 con 0x556652a06ae0 2024-03-12T05:10:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665263c310 con 0x556652a00f80 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f75413740 10 client.0 _ll_register_callbacks cb 0x556652915d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45670/0 (socket says 172.21.15.45:45670) 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f6b7fe640 1 -- 172.21.15.45:0/2729822681 learned_addr learned my addr 172.21.15.45:0/2729822681 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a12710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.522+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a0b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 -- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 msgr2=0x556652a0b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a0b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 -- 172.21.15.45:0/2729822681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 msgr2=0x556652a12710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a12710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 -- 172.21.15.45:0/2729822681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6401a040 con 0x556652a01970 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a0b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00bad0 tx=0x7f6f5c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c5c4bf12dc0a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c016070 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6f5c003dc0 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566526686c0 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f5c021030 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 10 client.15048 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5566529672b0 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5566526686c0 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f5c011640 con 0x556652a01970 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 10 client.15048 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 10 client.15048 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 20 client.15048 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a175c0 0x556652a16e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 4 client.15048 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.523+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566526b9270 con 0x556652a175c0 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.524+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a175c0 0x556652a16e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.524+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a175c0 0x556652a16e90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=867f0166de7030a1 server_cookie=3f63c0d65e8445d3 in_seq=0 out_seq=0 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.524+0000 7f6f4b7fe640 10 client.15048 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/2729822681 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566526b9270 con 0x556652a175c0 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f4b7fe640 10 client.15048 handle_command_reply: tid=0 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 client.15048 shutdown 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 msgr2=0x556652a0b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00bad0 tx=0x7f6f5c00bf90 comp rx=0 tx=0).stop 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 shutdown_connections 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a175c0 0x556652a16e90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a01970 0x556652a0b6d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a12710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f4b7fe640 0 client.15048 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 --2- 172.21.15.45:0/2729822681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a00f80 0x556652a0b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.525+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 >> 172.21.15.45:0/2729822681 conn(0x5566529fc4f0 msgr2=0x5566529fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.526+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 shutdown_connections 2024-03-12T05:10:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.526+0000 7f6f75413740 1 -- 172.21.15.45:0/2729822681 wait complete. 2024-03-12T05:10:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.526+0000 7f6f75413740 20 client.15048 trim_cache size 0 max 16384 2024-03-12T05:10:05.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 Processor -- start 2024-03-12T05:10:05.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 -- start start 2024-03-12T05:10:05.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a00f80 0x5566529fc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566528ee490 con 0x556652a06ae0 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566526686c0 con 0x556652a00f80 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f75413740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566529d28c0 con 0x556652a01970 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f70ddd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f70ddd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45678/0 (socket says 172.21.15.45:45678) 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.541+0000 7f6f70ddd640 1 -- 172.21.15.45:0/4082589254 learned_addr learned my addr 172.21.15.45:0/4082589254 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 -- 172.21.15.45:0/4082589254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 msgr2=0x556652a126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 -- 172.21.15.45:0/4082589254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a00f80 msgr2=0x5566529fc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a00f80 0x5566529fc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a00f80 0x5566529fc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 -- 172.21.15.45:0/4082589254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556652903a20 con 0x556652a06ae0 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f64024000 tx=0x7f6f640129f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344bd0b5e9b82f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f51ffb640 1 -- 172.21.15.45:0/4082589254 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6400cb40 con 0x556652a06ae0 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f51ffb640 1 -- 172.21.15.45:0/4082589254 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f6400ce10 con 0x556652a06ae0 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f51ffb640 1 -- 172.21.15.45:0/4082589254 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640058a0 con 0x556652a06ae0 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 -- 172.21.15.45:0/4082589254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 msgr2=0x556652a0ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f64024000 tx=0x7f6f640129f0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 -- 172.21.15.45:0/4082589254 shutdown_connections 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a06ae0 0x556652a0ada0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.542+0000 7f6f75413740 1 --2- 172.21.15.45:0/4082589254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a00f80 0x5566529fc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- 172.21.15.45:0/4082589254 >> 172.21.15.45:0/4082589254 conn(0x5566529fc4f0 msgr2=0x556652a023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- 172.21.15.45:0/4082589254 shutdown_connections 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- 172.21.15.45:0/4082589254 wait complete. 2024-03-12T05:10:05.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- start start 2024-03-12T05:10:05.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 Processor -- start 2024-03-12T05:10:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a10240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556652a11720 con 0x556652a00f80 2024-03-12T05:10:05.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556652a11890 con 0x556652a06ae0 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556652a11a50 con 0x556652a01970 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f75413740 10 client.0 _ll_register_callbacks cb 0x7f6f640127b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45680/0 (socket says 172.21.15.45:45680) 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f6bfff640 1 -- 172.21.15.45:0/3465625797 learned_addr learned my addr 172.21.15.45:0/3465625797 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.543+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a10240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 -- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 msgr2=0x556652a10240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a10240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 -- 172.21.15.45:0/3465625797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 msgr2=0x556652a111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 -- 172.21.15.45:0/3465625797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55665291df20 con 0x556652a00f80 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f70ddd640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f6bfff640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f600bc510 tx=0x7f6f60004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385c5af48d3e2549 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f600bc580 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6f600c06d0 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556652a17f70 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f600c4d20 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 10 client.15054 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556652929780 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556652a17f70 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f600d6080 con 0x556652a00f80 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 10 client.15054 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 10 client.15054 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 20 client.15054 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a73640 0x556652a6faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 4 client.15054 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.544+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6f6400cca0 con 0x556652a73640 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a73640 0x556652a6faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f4b7fe640 10 client.15054 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f6b7fe640 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a73640 0x556652a6faf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11fd6dd66276d79c server_cookie=8887176e180c6762 in_seq=0 out_seq=0 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f4b7fe640 1 -- 172.21.15.45:0/3465625797 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f6f6400cca0 con 0x556652a73640 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f4b7fe640 10 client.15054 handle_command_reply: tid=0 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f75413740 1 client.15054 shutdown 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 msgr2=0x556652a0fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f600bc510 tx=0x7f6f60004e80 comp rx=0 tx=0).stop 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.545+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 shutdown_connections 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556652a73640 0x556652a6faf0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556652a06ae0 0x556652a111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652a01970 0x556652a10240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 --2- 172.21.15.45:0/3465625797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556652a00f80 0x556652a0fd00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f4b7fe640 0 client.15054 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 >> 172.21.15.45:0/3465625797 conn(0x5566529fc4f0 msgr2=0x5566529fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 shutdown_connections 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 1 -- 172.21.15.45:0/3465625797 wait complete. 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f75413740 20 client.15054 trim_cache size 0 max 16384 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 4, 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24460, 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:05.550 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 83.200010085000002, 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 109.900253397 2024-03-12T05:10:05.551 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 msgr2=0x7f6f540a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 0x7f6f540a4000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f64005c40 tx=0x7f6f64009e40 comp rx=0 tx=0).stop 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 msgr2=0x7f6f6c19d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6f60036040 tx=0x7f6f600042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 shutdown_connections 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6f540a1b40 0x7f6f540a4000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f6c077720 0x7f6f6c1a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6c076d80 0x7f6f6c19dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 --2- 172.21.15.45:0/1115111679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6c075b40 0x7f6f6c19d780 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 >> 172.21.15.45:0/1115111679 conn(0x7f6f6c0fe930 msgr2=0x7f6f6c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 shutdown_connections 2024-03-12T05:10:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.546+0000 7f6f72840640 1 -- 172.21.15.45:0/1115111679 wait complete. 2024-03-12T05:10:05.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.660+0000 7f41f3e9d640 1 -- 172.21.15.45:0/2749497392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101c30 msgr2=0x7f41ec1084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.660+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/2749497392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101c30 0x7f41ec1084c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009ca0 tx=0x7f41e802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- 172.21.15.45:0/2749497392 shutdown_connections 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/2749497392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101c30 0x7f41ec1084c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/2749497392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ec101270 0x7f41ec1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/2749497392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- 172.21.15.45:0/2749497392 >> 172.21.15.45:0/2749497392 conn(0x7f41ec0fb610 msgr2=0x7f41ec0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- 172.21.15.45:0/2749497392 shutdown_connections 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- 172.21.15.45:0/2749497392 wait complete. 2024-03-12T05:10:05.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 Processor -- start 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- start start 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ec101c30 0x7f41ec19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19eb90 con 0x7f41ec101270 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19ed00 con 0x7f41ec100080 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f3e9d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec19eec0 con 0x7f41ec101c30 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45708/0 (socket says 172.21.15.45:45708) 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.661+0000 7f41f1438640 1 -- 172.21.15.45:0/955426282 learned_addr learned my addr 172.21.15.45:0/955426282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1c39640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 -- 172.21.15.45:0/955426282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ec101c30 msgr2=0x7f41ec19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ec101c30 0x7f41ec19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 -- 172.21.15.45:0/955426282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 msgr2=0x7f41ec068eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec068eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 -- 172.21.15.45:0/955426282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8009900 con 0x7f41ec101270 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1c39640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec068eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f1438640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00d770 tx=0x7f41dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41e2ffd640 1 -- 172.21.15.45:0/955426282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc009c30 con 0x7f41ec101270 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41e2ffd640 1 -- 172.21.15.45:0/955426282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41dc00de80 con 0x7f41ec101270 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41e2ffd640 1 -- 172.21.15.45:0/955426282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004e00 con 0x7f41ec101270 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec19f1a0 con 0x7f41ec101270 2024-03-12T05:10:05.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.662+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec19f660 con 0x7f41ec101270 2024-03-12T05:10:05.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.664+0000 7f41e2ffd640 1 -- 172.21.15.45:0/955426282 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f41dc004f60 con 0x7f41ec101270 2024-03-12T05:10:05.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.664+0000 7f41f6a7b740 1 Processor -- start 2024-03-12T05:10:05.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 -- start start 2024-03-12T05:10:05.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e4b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41e2ffd640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 0x7f41d80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e50880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e02cd0 con 0x55ee74e50ef0 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74db5f20 con 0x55ee74e4bd80 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74db3450 con 0x55ee74e4b390 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f1c39640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 0x7f41d80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45724/0 (socket says 172.21.15.45:45724) 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 -- 172.21.15.45:0/3107751736 learned_addr learned my addr 172.21.15.45:0/3107751736 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e4b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41e2ffd640 1 -- 172.21.15.45:0/955426282 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41dc0cd010 con 0x7f41ec101270 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f1438640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e50880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 -- 172.21.15.45:0/3107751736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 msgr2=0x55ee74e4b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e4b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 -- 172.21.15.45:0/3107751736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 msgr2=0x55ee74e50880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e50880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f243a640 1 -- 172.21.15.45:0/3107751736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee74d63f20 con 0x55ee74e50ef0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f1438640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e50880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.665+0000 7f41f1c39640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 0x7f41d80a4000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1010d0 tx=0x7f41e4005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f243a640 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55ee74e4c5e0 tx=0x7f41e8005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bd7d891803b873 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41d7fff640 1 -- 172.21.15.45:0/3107751736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8046030 con 0x55ee74e50ef0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41d7fff640 1 -- 172.21.15.45:0/3107751736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41e8032e90 con 0x55ee74e50ef0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41d7fff640 1 -- 172.21.15.45:0/3107751736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8003970 con 0x55ee74e50ef0 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3107751736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 msgr2=0x55ee74e53380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55ee74e4c5e0 tx=0x7f41e8005da0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3107751736 shutdown_connections 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e53380 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e50880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3107751736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e4b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.666+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3107751736 >> 172.21.15.45:0/3107751736 conn(0x55ee74e424f0 msgr2=0x55ee74e428f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3107751736 shutdown_connections 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3107751736 wait complete. 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- start start 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 Processor -- start 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e55580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74e5cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74daec30 con 0x55ee74e4bd80 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74d53d30 con 0x55ee74e4b390 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74a82310 con 0x55ee74e50ef0 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f6a7b740 10 client.0 _ll_register_callbacks cb 0x55ee74d5bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45738/0 (socket says 172.21.15.45:45738) 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.667+0000 7f41f1438640 1 -- 172.21.15.45:0/3909545721 learned_addr learned my addr 172.21.15.45:0/3909545721 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e55580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 -- 172.21.15.45:0/3909545721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 msgr2=0x55ee74e5cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f243a640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74e5cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74e5cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 -- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 msgr2=0x55ee74e55580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e55580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 -- 172.21.15.45:0/3909545721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec108dc0 con 0x55ee74e4bd80 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e55580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f243a640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74e5cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f1438640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f41dc096800 tx=0x7f41dc099e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20584625856a1bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc09f220 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41dc09f680 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee74aae6c0 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41dc0a25d0 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f6a7b740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ee74dad2b0 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41dc0b6020 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f41dc0a2140 con 0x55ee74e4bd80 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.668+0000 7f41f6a7b740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f6a7b740 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f6a7b740 20 client.15066 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74e61f70 0x55ee74e61230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f6a7b740 4 client.15066 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee74aff270 con 0x55ee74e61f70 2024-03-12T05:10:05.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74e61f70 0x55ee74e61230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74e61f70 0x55ee74e61230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c59a0a4df1d03fba server_cookie=771ea0361ad7b40a in_seq=0 out_seq=0 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.669+0000 7f41d57fa640 10 client.15066 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41d57fa640 1 -- 172.21.15.45:0/3909545721 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ee74aff270 con 0x55ee74e61f70 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41d57fa640 10 client.15066 handle_command_reply: tid=0 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 client.15066 shutdown 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 msgr2=0x55ee74e55ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f41dc096800 tx=0x7f41dc099e30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 shutdown_connections 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74e61f70 0x55ee74e61230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e55ac0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74e5cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3909545721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e55580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41d57fa640 0 client.15066 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 >> 172.21.15.45:0/3909545721 conn(0x55ee74e424f0 msgr2=0x55ee74e417d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.670+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 shutdown_connections 2024-03-12T05:10:05.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.671+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3909545721 wait complete. 2024-03-12T05:10:05.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.671+0000 7f41f6a7b740 20 client.15066 trim_cache size 0 max 16384 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 Processor -- start 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 -- start start 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e3d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e5cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74d34490 con 0x55ee74e50ef0 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e43490 con 0x55ee74e4bd80 2024-03-12T05:10:05.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e5d780 con 0x55ee74e4b390 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f243a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f243a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45748/0 (socket says 172.21.15.45:45748) 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.686+0000 7f41f243a640 1 -- 172.21.15.45:0/1376983353 learned_addr learned my addr 172.21.15.45:0/1376983353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f1438640 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e5cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 -- 172.21.15.45:0/1376983353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 msgr2=0x55ee74e3d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e3d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 -- 172.21.15.45:0/1376983353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 msgr2=0x55ee74e5cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e5cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 -- 172.21.15.45:0/1376983353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee74d49a20 con 0x55ee74e50ef0 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f1438640 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e5cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f243a640 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f41e8002230 tx=0x7f41e8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780c9112a1e06be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41d77fe640 1 -- 172.21.15.45:0/1376983353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8046040 con 0x55ee74e50ef0 2024-03-12T05:10:05.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41d77fe640 1 -- 172.21.15.45:0/1376983353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41e8034640 con 0x55ee74e50ef0 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41d77fe640 1 -- 172.21.15.45:0/1376983353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e804b690 con 0x55ee74e50ef0 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 -- 172.21.15.45:0/1376983353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 msgr2=0x55ee74e428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f41e8002230 tx=0x7f41e8009220 comp rx=0 tx=0).stop 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 -- 172.21.15.45:0/1376983353 shutdown_connections 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e50ef0 0x55ee74e428d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4bd80 0x55ee74e5cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.687+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/1376983353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e4b390 0x55ee74e3d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- 172.21.15.45:0/1376983353 >> 172.21.15.45:0/1376983353 conn(0x55ee74e424f0 msgr2=0x55ee74e4c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- 172.21.15.45:0/1376983353 shutdown_connections 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- 172.21.15.45:0/1376983353 wait complete. 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- start start 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 Processor -- start 2024-03-12T05:10:05.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74eb7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e61b40 con 0x55ee74e4bd80 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e5cc20 con 0x55ee74e4b390 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee74e5cde0 con 0x55ee74e50ef0 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f6a7b740 10 client.0 _ll_register_callbacks cb 0x7f41e8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f1438640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45758/0 (socket says 172.21.15.45:45758) 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f1438640 1 -- 172.21.15.45:0/3317237223 learned_addr learned my addr 172.21.15.45:0/3317237223 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.688+0000 7f41f243a640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74eb7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 -- 172.21.15.45:0/3317237223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 msgr2=0x55ee74eb7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74eb7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 -- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 msgr2=0x55ee74e596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 -- 172.21.15.45:0/3317237223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee74d63f20 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f243a640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74eb7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f1438640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009610 tx=0x7f41dc09ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3772a4473134cf58 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0ade80 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f41dc0b7080 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee74e61c80 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41dc083420 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 10 client.15072 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ee74d6f780 con 0x55ee74e4bd80 2024-03-12T05:10:05.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41dc0a6080 con 0x55ee74e4bd80 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f41dc0b2b60 con 0x55ee74e4bd80 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 10 client.15072 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 10 client.15072 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 20 client.15072 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74ebcc40 0x55ee74eb9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 4 client.15072 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.689+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f41e8032e90 con 0x55ee74ebcc40 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74ebcc40 0x55ee74eb9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f1c39640 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74ebcc40 0x55ee74eb9120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=552699285d895d26 server_cookie=9f012cae9036e63b in_seq=0 out_seq=0 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41d57fa640 10 client.15072 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41d57fa640 1 -- 172.21.15.45:0/3317237223 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f41e8032e90 con 0x55ee74ebcc40 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41d57fa640 10 client.15072 handle_command_reply: tid=0 2024-03-12T05:10:05.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 client.15072 shutdown 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 msgr2=0x55ee74e59c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009610 tx=0x7f41dc09ed70 comp rx=0 tx=0).stop 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 shutdown_connections 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55ee74ebcc40 0x55ee74eb9120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee74e4bd80 0x55ee74e59c00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee74e50ef0 0x55ee74eb7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41d57fa640 0 client.15072 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 --2- 172.21.15.45:0/3317237223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee74e4b390 0x55ee74e596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.690+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 >> 172.21.15.45:0/3317237223 conn(0x55ee74e424f0 msgr2=0x55ee74e4c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 shutdown_connections 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f6a7b740 1 -- 172.21.15.45:0/3317237223 wait complete. 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f6a7b740 20 client.15072 trim_cache size 0 max 16384 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:05.696 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 3, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24467, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 84.351284911999997, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 104.433433593 2024-03-12T05:10:05.697 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 msgr2=0x7f41d80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 0x7f41d80a4000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1010d0 tx=0x7f41e4005db0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 msgr2=0x7f41ec19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00d770 tx=0x7f41dc00dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 shutdown_connections 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f41d80a1b40 0x7f41d80a4000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41ec101270 0x7f41ec19bd90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ec101c30 0x7f41ec19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 --2- 172.21.15.45:0/955426282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41ec100080 0x7f41ec068eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 >> 172.21.15.45:0/955426282 conn(0x7f41ec0fb610 msgr2=0x7f41ec104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 shutdown_connections 2024-03-12T05:10:05.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.691+0000 7f41f3e9d640 1 -- 172.21.15.45:0/955426282 wait complete. 2024-03-12T05:10:05.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 -- 172.21.15.45:0/3309922604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ff1f0 msgr2=0x7f31340ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 --2- 172.21.15.45:0/3309922604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ff1f0 0x7f31340ff670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3128009900 tx=0x7f312802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 -- 172.21.15.45:0/3309922604 shutdown_connections 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 --2- 172.21.15.45:0/3309922604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ff1f0 0x7f31340ff670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 --2- 172.21.15.45:0/3309922604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f31340ffbb0 0x7f3134109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 --2- 172.21.15.45:0/3309922604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3134102780 0x7f3134102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 -- 172.21.15.45:0/3309922604 >> 172.21.15.45:0/3309922604 conn(0x7f31340fadb0 msgr2=0x7f31340fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 -- 172.21.15.45:0/3309922604 shutdown_connections 2024-03-12T05:10:05.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.808+0000 7f313bc88640 1 -- 172.21.15.45:0/3309922604 wait complete. 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 Processor -- start 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 -- start start 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31340ff1f0 0x7f3134197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3134102780 0x7f313419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313419ec80 con 0x7f31340ffbb0 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313419edf0 con 0x7f31340ff1f0 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313bc88640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313419efb0 con 0x7f3134102780 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45782/0 (socket says 172.21.15.45:45782) 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 -- 172.21.15.45:0/1221444705 learned_addr learned my addr 172.21.15.45:0/1221444705 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139a24640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31340ff1f0 0x7f3134197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f313a225640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3134102780 0x7f313419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 -- 172.21.15.45:0/1221444705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3134102780 msgr2=0x7f313419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3134102780 0x7f313419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 -- 172.21.15.45:0/1221444705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31340ff1f0 msgr2=0x7f3134197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31340ff1f0 0x7f3134197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.809+0000 7f3139223640 1 -- 172.21.15.45:0/1221444705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3128009560 con 0x7f31340ffbb0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f3139223640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3128036040 tx=0x7f31280043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f3132ffd640 1 -- 172.21.15.45:0/1221444705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128003800 con 0x7f31340ffbb0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313419f230 con 0x7f31340ffbb0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f3132ffd640 1 -- 172.21.15.45:0/1221444705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3128003e10 con 0x7f31340ffbb0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f3132ffd640 1 -- 172.21.15.45:0/1221444705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312803e450 con 0x7f31340ffbb0 2024-03-12T05:10:05.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.810+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313419f810 con 0x7f31340ffbb0 2024-03-12T05:10:05.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.812+0000 7f3132ffd640 1 -- 172.21.15.45:0/1221444705 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3128004fe0 con 0x7f31340ffbb0 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.812+0000 7f313e867740 1 Processor -- start 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 -- start start 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3132ffd640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 0x7f311c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a486503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a48657f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a4860bcd0 con 0x561a48650970 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a485bef20 con 0x561a4864ff80 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313e867740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a485bc450 con 0x561a48655ae0 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313a225640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 0x7f311c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45784/0 (socket says 172.21.15.45:45784) 2024-03-12T05:10:05.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 -- 172.21.15.45:0/2320662599 learned_addr learned my addr 172.21.15.45:0/2320662599 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3132ffd640 1 -- 172.21.15.45:0/1221444705 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f31280ef960 con 0x7f31340ffbb0 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313a225640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a48657f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139a24640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a486503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 -- 172.21.15.45:0/2320662599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 msgr2=0x561a48657f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a48657f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 -- 172.21.15.45:0/2320662599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 msgr2=0x561a486503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a486503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 -- 172.21.15.45:0/2320662599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a4856cf20 con 0x561a48650970 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313a225640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a48657f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f3139223640 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3128000c00 tx=0x7f31280b1fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3a855cd12a880f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.813+0000 7f313a225640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 0x7f311c0a3f80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f312400afb0 tx=0x7f3124006f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f311bfff640 1 -- 172.21.15.45:0/2320662599 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280bc3d0 con 0x561a48650970 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f311bfff640 1 -- 172.21.15.45:0/2320662599 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31280bc1f0 con 0x561a48650970 2024-03-12T05:10:05.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f311bfff640 1 -- 172.21.15.45:0/2320662599 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280ed0b0 con 0x561a48650970 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 -- 172.21.15.45:0/2320662599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 msgr2=0x561a48655470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3128000c00 tx=0x7f31280b1fb0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 -- 172.21.15.45:0/2320662599 shutdown_connections 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a48655470 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a48657f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 --2- 172.21.15.45:0/2320662599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a486503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 -- 172.21.15.45:0/2320662599 >> 172.21.15.45:0/2320662599 conn(0x561a4864b4f0 msgr2=0x561a4864b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 -- 172.21.15.45:0/2320662599 shutdown_connections 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.814+0000 7f313e867740 1 -- 172.21.15.45:0/2320662599 wait complete. 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 -- start start 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 Processor -- start 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48659e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a4865a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a485b7c30 con 0x561a48655ae0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a4855cd30 con 0x561a4864ff80 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a4828b310 con 0x561a48650970 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313e867740 10 client.0 _ll_register_callbacks cb 0x561a48564d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313a225640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313a225640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45786/0 (socket says 172.21.15.45:45786) 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f313a225640 1 -- 172.21.15.45:0/3294499639 learned_addr learned my addr 172.21.15.45:0/3294499639 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.815+0000 7f3139a24640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48659e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 -- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 msgr2=0x561a4865a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a4865a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 -- 172.21.15.45:0/3294499639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 msgr2=0x561a48659e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48659e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 -- 172.21.15.45:0/3294499639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313419f810 con 0x561a48655ae0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313a225640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3124016b40 tx=0x7f31240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82767b81d775494c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240128f0 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f3124012a50 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a482b76c0 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3124012c10 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561a485b62b0 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561a482b76c0 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3124006480 con 0x561a48655ae0 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 10 client.15084 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 20 client.15084 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486668f0 0x561a48665b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 4 client.15084 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:05.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.816+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a48308270 con 0x561a486668f0 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.817+0000 7f3139a24640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486668f0 0x561a48665b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.817+0000 7f3139a24640 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486668f0 0x561a48665b40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99a96fe91f413f29 server_cookie=63a503173e49581a in_seq=0 out_seq=0 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.817+0000 7f31197fa640 10 client.15084 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f31197fa640 1 -- 172.21.15.45:0/3294499639 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561a48308270 con 0x561a486668f0 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f31197fa640 10 client.15084 handle_command_reply: tid=0 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 client.15084 shutdown 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 msgr2=0x561a48661410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3124016b40 tx=0x7f31240046c0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 shutdown_connections 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486668f0 0x561a48665b40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48655ae0 0x561a48661410 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a4865a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 --2- 172.21.15.45:0/3294499639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48659e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f31197fa640 0 client.15084 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 >> 172.21.15.45:0/3294499639 conn(0x561a4864b4f0 msgr2=0x561a4864a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 shutdown_connections 2024-03-12T05:10:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 1 -- 172.21.15.45:0/3294499639 wait complete. 2024-03-12T05:10:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.818+0000 7f313e867740 20 client.15084 trim_cache size 0 max 16384 2024-03-12T05:10:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 Processor -- start 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 -- start start 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48665b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a4866ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a4853d490 con 0x561a48650970 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a48659cd0 con 0x561a4864ff80 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f313e867740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a48308270 con 0x561a48655ae0 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f3139223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45800/0 (socket says 172.21.15.45:45800) 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.834+0000 7f3139223640 1 -- 172.21.15.45:0/4050921062 learned_addr learned my addr 172.21.15.45:0/4050921062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 -- 172.21.15.45:0/4050921062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 msgr2=0x561a4866ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a4866ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 -- 172.21.15.45:0/4050921062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 msgr2=0x561a48665b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48665b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 -- 172.21.15.45:0/4050921062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a48552a20 con 0x561a48650970 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f3139223640 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31280bc820 tx=0x7f3128047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37ec80bc306bcc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f311b7fe640 1 -- 172.21.15.45:0/4050921062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280bc890 con 0x561a48650970 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f311b7fe640 1 -- 172.21.15.45:0/4050921062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31280c09e0 con 0x561a48650970 2024-03-12T05:10:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f311b7fe640 1 -- 172.21.15.45:0/4050921062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280eb0d0 con 0x561a48650970 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f313e867740 1 -- 172.21.15.45:0/4050921062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 msgr2=0x561a486613d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.835+0000 7f313e867740 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31280bc820 tx=0x7f3128047000 comp rx=0 tx=0).stop 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- 172.21.15.45:0/4050921062 shutdown_connections 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a48650970 0x561a486613d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48655ae0 0x561a4866ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- 172.21.15.45:0/4050921062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a4864ff80 0x561a48665b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- 172.21.15.45:0/4050921062 >> 172.21.15.45:0/4050921062 conn(0x561a4864b4f0 msgr2=0x561a48651200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- 172.21.15.45:0/4050921062 shutdown_connections 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- 172.21.15.45:0/4050921062 wait complete. 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- start start 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 Processor -- start 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a48660ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 0x561a48661120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a48661820 con 0x561a4864ff80 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a486bc520 con 0x561a48655ae0 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a486bc6e0 con 0x561a48650970 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f313e867740 10 client.0 _ll_register_callbacks cb 0x7f3128005290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f3139a24640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f3139a24640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45802/0 (socket says 172.21.15.45:45802) 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.836+0000 7f3139a24640 1 -- 172.21.15.45:0/1943636643 learned_addr learned my addr 172.21.15.45:0/1943636643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139223640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a48660ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313a225640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 0x561a48661120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 -- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 msgr2=0x561a48660ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a48660ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 -- 172.21.15.45:0/1943636643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 msgr2=0x561a48661120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 0x561a48661120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 -- 172.21.15.45:0/1943636643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a4856cf20 con 0x561a4864ff80 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139223640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a48660ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313a225640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 0x561a48661120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f3139a24640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f312c000f80 tx=0x7f312c012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e28e37c444cdbf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c016040 con 0x561a4864ff80 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f312c01d7d0 con 0x561a4864ff80 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a48666600 con 0x561a4864ff80 2024-03-12T05:10:05.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312c004b20 con 0x561a4864ff80 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 10 client.15090 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561a48578780 con 0x561a4864ff80 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561a48666600 con 0x561a4864ff80 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f312c02f070 con 0x561a4864ff80 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 10 client.15090 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 10 client.15090 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 20 client.15090 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486c1af0 0x561a486bdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 4 client.15090 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.837+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f31280bc1f0 con 0x561a486c1af0 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f3139223640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486c1af0 0x561a486bdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f31197fa640 10 client.15090 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f3139223640 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486c1af0 0x561a486bdf10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ff2fd4093d5da37 server_cookie=e77a016357fdcf30 in_seq=0 out_seq=0 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f31197fa640 1 -- 172.21.15.45:0/1943636643 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f31280bc1f0 con 0x561a486c1af0 2024-03-12T05:10:05.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f31197fa640 10 client.15090 handle_command_reply: tid=0 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 client.15090 shutdown 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 msgr2=0x561a48660660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f312c000f80 tx=0x7f312c012bf0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 shutdown_connections 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561a486c1af0 0x561a486bdf10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a48655ae0 0x561a48661120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a48650970 0x561a48660ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 --2- 172.21.15.45:0/1943636643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561a4864ff80 0x561a48660660 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f31197fa640 0 client.15090 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 >> 172.21.15.45:0/1943636643 conn(0x561a4864b4f0 msgr2=0x561a4866ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.838+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 shutdown_connections 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313e867740 1 -- 172.21.15.45:0/1943636643 wait complete. 2024-03-12T05:10:05.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313e867740 20 client.15090 trim_cache size 0 max 16384 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 2, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24472, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 85.506126968999993, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:05.843 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 107.436915055 2024-03-12T05:10:05.844 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 msgr2=0x7f311c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 0x7f311c0a3f80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f312400afb0 tx=0x7f3124006f10 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 msgr2=0x7f31341976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3128036040 tx=0x7f31280043e0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 shutdown_connections 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f311c0a1ac0 0x7f311c0a3f80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31340ffbb0 0x7f31341976d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3134102780 0x7f313419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 --2- 172.21.15.45:0/1221444705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31340ff1f0 0x7f3134197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 >> 172.21.15.45:0/1221444705 conn(0x7f31340fadb0 msgr2=0x7f3134105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 shutdown_connections 2024-03-12T05:10:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.839+0000 7f313bc88640 1 -- 172.21.15.45:0/1221444705 wait complete. 2024-03-12T05:10:05.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-12T05:10:05.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.952+0000 7fb7cf69d640 1 -- 172.21.15.45:0/2624765369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 msgr2=0x7fb7c81016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.952+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/2624765369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c81016f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009900 tx=0x7fb7b802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.952+0000 7fb7cf69d640 1 -- 172.21.15.45:0/2624765369 shutdown_connections 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/2624765369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c81016f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/2624765369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8101c30 0x7fb7c81084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/2624765369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8100080 0x7fb7c81004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- 172.21.15.45:0/2624765369 >> 172.21.15.45:0/2624765369 conn(0x7fb7c80fb610 msgr2=0x7fb7c80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- 172.21.15.45:0/2624765369 shutdown_connections 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- 172.21.15.45:0/2624765369 wait complete. 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 Processor -- start 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- start start 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 0x7fb7c81971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8101c30 0x7fb7c819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819ecd0 con 0x7fb7c8101270 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819ee40 con 0x7fb7c8101c30 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7cf69d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c819f000 con 0x7fb7c8100080 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7ccc38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7ccc38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45812/0 (socket says 172.21.15.45:45812) 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.953+0000 7fb7ccc38640 1 -- 172.21.15.45:0/86700416 learned_addr learned my addr 172.21.15.45:0/86700416 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7cd439640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 0x7fb7c81971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8101c30 0x7fb7c819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 -- 172.21.15.45:0/86700416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 msgr2=0x7fb7c81971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 0x7fb7c81971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 -- 172.21.15.45:0/86700416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8101c30 msgr2=0x7fb7c819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8101c30 0x7fb7c819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 -- 172.21.15.45:0/86700416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8009560 con 0x7fb7c8101270 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7cd439640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 0x7fb7c81971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:05.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8036040 tx=0x7fb7b8004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7be7fc640 1 -- 172.21.15.45:0/86700416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8003800 con 0x7fb7c8101270 2024-03-12T05:10:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7be7fc640 1 -- 172.21.15.45:0/86700416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7b802de90 con 0x7fb7c8101270 2024-03-12T05:10:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c819f280 con 0x7fb7c8101270 2024-03-12T05:10:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7be7fc640 1 -- 172.21.15.45:0/86700416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8005600 con 0x7fb7c8101270 2024-03-12T05:10:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.954+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c819f860 con 0x7fb7c8101270 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.956+0000 7fb7be7fc640 1 -- 172.21.15.45:0/86700416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb7b8004dc0 con 0x7fb7c8101270 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 Processor -- start 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 -- start start 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7be7fc640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 0x7fb7b40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2db83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dbd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2d73cd0 con 0x5604c2dbdae0 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2d26f20 con 0x5604c2db8970 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7d227b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2d24450 con 0x5604c2db7f80 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 0x7fb7b40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7be7fc640 1 -- 172.21.15.45:0/86700416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7b80ef840 con 0x7fb7c8101270 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45828/0 (socket says 172.21.15.45:45828) 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/747049609 learned_addr learned my addr 172.21.15.45:0/747049609 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dbd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cd439640 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2db83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/747049609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 msgr2=0x5604c2db83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2db83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/747049609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 msgr2=0x5604c2dbd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dbd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.957+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/747049609 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604c2cd4f20 con 0x5604c2dbdae0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000af30 tx=0x7fb7c0016700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f22e5495b6b4556 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 0x7fb7b40a4000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0016c00 tx=0x7fb7c000c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7b37fe640 1 -- 172.21.15.45:0/747049609 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000f070 con 0x5604c2dbdae0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7b37fe640 1 -- 172.21.15.45:0/747049609 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7c0016ed0 con 0x5604c2dbdae0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7b37fe640 1 -- 172.21.15.45:0/747049609 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0019bf0 con 0x5604c2dbdae0 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 -- 172.21.15.45:0/747049609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 msgr2=0x5604c2dbff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000af30 tx=0x7fb7c0016700 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 -- 172.21.15.45:0/747049609 shutdown_connections 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dbff70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dbd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 --2- 172.21.15.45:0/747049609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2db83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 -- 172.21.15.45:0/747049609 >> 172.21.15.45:0/747049609 conn(0x5604c2db34f0 msgr2=0x5604c2db38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 -- 172.21.15.45:0/747049609 shutdown_connections 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.958+0000 7fb7d227b740 1 -- 172.21.15.45:0/747049609 wait complete. 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 -- start start 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 Processor -- start 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db7f80 0x5604c2dc21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2dbdae0 0x5604c2dc9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2d1fc30 con 0x5604c2db8970 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2cc4d30 con 0x5604c2db7f80 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c29f3310 con 0x5604c2dbdae0 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7d227b740 10 client.0 _ll_register_callbacks cb 0x5604c2cccd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7ccc38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7ccc38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45844/0 (socket says 172.21.15.45:45844) 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7ccc38640 1 -- 172.21.15.45:0/148388574 learned_addr learned my addr 172.21.15.45:0/148388574 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.959+0000 7fb7cd439640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db7f80 0x5604c2dc21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2dbdae0 0x5604c2dc9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 -- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2dbdae0 msgr2=0x5604c2dc9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2dbdae0 0x5604c2dc9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 -- 172.21.15.45:0/148388574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db7f80 msgr2=0x5604c2dc21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db7f80 0x5604c2dc21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 -- 172.21.15.45:0/148388574 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c000aae0 con 0x5604c2db8970 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80bc6a0 tx=0x7fb7b80bc310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fc9c52670415b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80bc710 con 0x5604c2db8970 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7b80c0860 con 0x5604c2db8970 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604c2a1f6c0 con 0x5604c2db8970 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7b80c4eb0 con 0x5604c2db8970 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 10 client.15102 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5604c2d1e2b0 con 0x5604c2db8970 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5604c2a1f6c0 con 0x5604c2db8970 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb7b803c070 con 0x5604c2db8970 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 10 client.15102 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 20 client.15102 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2dcdbd0 0x5604c2dd2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 4 client.15102 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.960+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604c2a70270 con 0x5604c2dcdbd0 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.961+0000 7fb7cd439640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2dcdbd0 0x5604c2dd2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.961+0000 7fb7b0ff9640 10 client.15102 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.961+0000 7fb7cd439640 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2dcdbd0 0x5604c2dd2cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aad53fe8015aed7c server_cookie=ae33b70a51a0086a in_seq=0 out_seq=0 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/148388574 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5604c2a70270 con 0x5604c2dcdbd0 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7b0ff9640 10 client.15102 handle_command_reply: tid=0 2024-03-12T05:10:05.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 client.15102 shutdown 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 msgr2=0x5604c2dc2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80bc6a0 tx=0x7fb7b80bc310 comp rx=0 tx=0).stop 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 shutdown_connections 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2db8970 0x5604c2dc2730 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2dcdbd0 0x5604c2dd2cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2dbdae0 0x5604c2dc9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 --2- 172.21.15.45:0/148388574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db7f80 0x5604c2dc21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7b0ff9640 0 client.15102 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 >> 172.21.15.45:0/148388574 conn(0x5604c2db34f0 msgr2=0x5604c2db27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 shutdown_connections 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 1 -- 172.21.15.45:0/148388574 wait complete. 2024-03-12T05:10:05.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.962+0000 7fb7d227b740 20 client.15102 trim_cache size 0 max 16384 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 Processor -- start 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 -- start start 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2ca5490 con 0x5604c2dbdae0 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2dd0ad0 con 0x5604c2db8970 2024-03-12T05:10:05.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7d227b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2a1f6c0 con 0x5604c2db7f80 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45852/0 (socket says 172.21.15.45:45852) 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/3246407327 learned_addr learned my addr 172.21.15.45:0/3246407327 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/3246407327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 msgr2=0x5604c2dca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/3246407327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 msgr2=0x5604c2dc9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/3246407327 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604c2cbaa20 con 0x5604c2dbdae0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.978+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000bfd0 tx=0x7fb7c00093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891172ce8ccd3355 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7b2ffd640 1 -- 172.21.15.45:0/3246407327 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c002f070 con 0x5604c2dbdae0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7b2ffd640 1 -- 172.21.15.45:0/3246407327 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7c0009c90 con 0x5604c2dbdae0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7b2ffd640 1 -- 172.21.15.45:0/3246407327 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000f040 con 0x5604c2dbdae0 2024-03-12T05:10:05.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- 172.21.15.45:0/3246407327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 msgr2=0x5604c2db38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000bfd0 tx=0x7fb7c00093d0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- 172.21.15.45:0/3246407327 shutdown_connections 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2db38d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 --2- 172.21.15.45:0/3246407327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- 172.21.15.45:0/3246407327 >> 172.21.15.45:0/3246407327 conn(0x5604c2db34f0 msgr2=0x5604c2db93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- 172.21.15.45:0/3246407327 shutdown_connections 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- 172.21.15.45:0/3246407327 wait complete. 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.979+0000 7fb7d227b740 1 -- start start 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 Processor -- start 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dc93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2dc6910 con 0x5604c2dbdae0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2dc6a80 con 0x5604c2db8970 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604c2e24980 con 0x5604c2db7f80 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 10 client.0 _ll_register_callbacks cb 0x7fb7c0009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45860/0 (socket says 172.21.15.45:45860) 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/439088335 learned_addr learned my addr 172.21.15.45:0/439088335 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7ccc38640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cd439640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dc93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 msgr2=0x5604c2dc93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dc93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/439088335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 msgr2=0x5604c2dc5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 -- 172.21.15.45:0/439088335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604c2cd4f20 con 0x5604c2dbdae0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7cdc3a640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000a790 tx=0x7fb7c0019be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b857b041a88d835c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:05.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0037070 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7c0030070 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.980+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604c2dce550 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c002f030 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 10 client.15108 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5604c2ce0780 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb7c0042020 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb7c003a020 con 0x5604c2dbdae0 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 10 client.15108 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 20 client.15108 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2e29770 0x5604c2e25c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 4 client.15108 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:05.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5604c2dcebb0 con 0x5604c2e29770 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.981+0000 7fb7cd439640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2e29770 0x5604c2e25c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7cd439640 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2e29770 0x5604c2e25c30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c87ed2895a6854bf server_cookie=2eb3f5a716ba2a7a in_seq=0 out_seq=0 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7b0ff9640 10 client.15108 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7b0ff9640 1 -- 172.21.15.45:0/439088335 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5604c2dcebb0 con 0x5604c2e29770 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7b0ff9640 10 client.15108 handle_command_reply: tid=0 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 client.15108 shutdown 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 msgr2=0x5604c2dc63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000a790 tx=0x7fb7c0019be0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 shutdown_connections 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604c2dbdae0 0x5604c2dc63d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5604c2e29770 0x5604c2e25c30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604c2db8970 0x5604c2dc5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 --2- 172.21.15.45:0/439088335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604c2db7f80 0x5604c2dc93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7b0ff9640 0 client.15108 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 >> 172.21.15.45:0/439088335 conn(0x5604c2db34f0 msgr2=0x5604c2db9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 shutdown_connections 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 1 -- 172.21.15.45:0/439088335 wait complete. 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.982+0000 7fb7d227b740 20 client.15108 trim_cache size 0 max 16384 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24487, 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:05.986 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 86.660757111999999, 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 101.966246774 2024-03-12T05:10:05.987 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 msgr2=0x7fb7b40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 0x7fb7b40a4000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0016c00 tx=0x7fb7c000c000 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 msgr2=0x7fb7c8197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8036040 tx=0x7fb7b8004370 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 shutdown_connections 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7b40a1b40 0x7fb7b40a4000 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7c8101270 0x7fb7c8197720 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c8101c30 0x7fb7c819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 --2- 172.21.15.45:0/86700416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8100080 0x7fb7c81971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 >> 172.21.15.45:0/86700416 conn(0x7fb7c80fb610 msgr2=0x7fb7c8104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 shutdown_connections 2024-03-12T05:10:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:05.983+0000 7fb7cf69d640 1 -- 172.21.15.45:0/86700416 wait complete. 2024-03-12T05:10:05.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-12T05:10:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- 172.21.15.45:0/2435692484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101c50 msgr2=0x7f91181084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- 172.21.15.45:0/2435692484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101c50 0x7f91181084e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9110009ca0 tx=0x7f911002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- 172.21.15.45:0/2435692484 shutdown_connections 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- 172.21.15.45:0/2435692484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101c50 0x7f91181084e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- 172.21.15.45:0/2435692484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118101290 0x7f9118101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- 172.21.15.45:0/2435692484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91181000a0 0x7f91181004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- 172.21.15.45:0/2435692484 >> 172.21.15.45:0/2435692484 conn(0x7f91180fb630 msgr2=0x7f91180fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- 172.21.15.45:0/2435692484 shutdown_connections 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- 172.21.15.45:0/2435692484 wait complete. 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 Processor -- start 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- start start 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91181000a0 0x7f911810a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101c50 0x7f91181094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118109c40 con 0x7f9118101290 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118109db0 con 0x7f91181000a0 2024-03-12T05:10:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f9120481640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181a1150 con 0x7f9118101c50 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.097+0000 7f911da1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45882/0 (socket says 172.21.15.45:45882) 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 -- 172.21.15.45:0/3451454795 learned_addr learned my addr 172.21.15.45:0/3451454795 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101c50 0x7f91181094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911e21d640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91181000a0 0x7f911810a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 -- 172.21.15.45:0/3451454795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101c50 msgr2=0x7f91181094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101c50 0x7f91181094e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 -- 172.21.15.45:0/3451454795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91181000a0 msgr2=0x7f911810a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91181000a0 0x7f911810a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 -- 172.21.15.45:0/3451454795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110009900 con 0x7f9118101290 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f911da1c640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f910800d980 tx=0x7f910800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f910f7fe640 1 -- 172.21.15.45:0/3451454795 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108009a30 con 0x7f9118101290 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f910f7fe640 1 -- 172.21.15.45:0/3451454795 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f91080044d0 con 0x7f9118101290 2024-03-12T05:10:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f910f7fe640 1 -- 172.21.15.45:0/3451454795 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91080047a0 con 0x7f9118101290 2024-03-12T05:10:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91181a1400 con 0x7f9118101290 2024-03-12T05:10:06.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.098+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91181a1910 con 0x7f9118101290 2024-03-12T05:10:06.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.100+0000 7f910f7fe640 1 -- 172.21.15.45:0/3451454795 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9108004020 con 0x7f9118101290 2024-03-12T05:10:06.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 Processor -- start 2024-03-12T05:10:06.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 -- start start 2024-03-12T05:10:06.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 0x5600c11fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 0x5600c11fff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c11b3cd0 con 0x5600c11f7f80 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c1166f20 con 0x5600c11f8970 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f912304e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c1164450 con 0x5600c11fdae0 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f911ea1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f910f7fe640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 0x7f91040a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f911ea1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45888/0 (socket says 172.21.15.45:45888) 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f911ea1e640 1 -- 172.21.15.45:0/3250785137 learned_addr learned my addr 172.21.15.45:0/3250785137 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.101+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 0x7f91040a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f910f7fe640 1 -- 172.21.15.45:0/3451454795 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f91080cd0c0 con 0x7f9118101290 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911da1c640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 0x5600c11fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911e21d640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 0x5600c11fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 -- 172.21.15.45:0/3250785137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 msgr2=0x5600c11fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 0x5600c11fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 -- 172.21.15.45:0/3250785137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 msgr2=0x5600c11fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 0x5600c11fd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 -- 172.21.15.45:0/3250785137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600c1114f20 con 0x5600c11f7f80 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911da1c640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 0x5600c11fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911e21d640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 0x5600c11fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f91100326d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de34a04a293fe974 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 0x7f91040a3f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9110032460 tx=0x7f9110048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f910cbfe640 1 -- 172.21.15.45:0/3250785137 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110004d40 con 0x5600c11f7f80 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f910cbfe640 1 -- 172.21.15.45:0/3250785137 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9110032ea0 con 0x5600c11f7f80 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.102+0000 7f910cbfe640 1 -- 172.21.15.45:0/3250785137 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91100042a0 con 0x5600c11f7f80 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- 172.21.15.45:0/3250785137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 msgr2=0x5600c11f83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f91100098d0 tx=0x7f91100326d0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- 172.21.15.45:0/3250785137 shutdown_connections 2024-03-12T05:10:06.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11fdae0 0x5600c11fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f8970 0x5600c11fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 --2- 172.21.15.45:0/3250785137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c11f83a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- 172.21.15.45:0/3250785137 >> 172.21.15.45:0/3250785137 conn(0x5600c11f34f0 msgr2=0x5600c11f38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- 172.21.15.45:0/3250785137 shutdown_connections 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- 172.21.15.45:0/3250785137 wait complete. 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 -- start start 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.103+0000 7f912304e740 1 Processor -- start 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c12026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1209720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c115fc30 con 0x5600c11f7f80 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c1104d30 con 0x5600c11fdae0 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c0e33310 con 0x5600c11f8970 2024-03-12T05:10:06.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 10 client.0 _ll_register_callbacks cb 0x5600c110cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45902/0 (socket says 172.21.15.45:45902) 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 -- 172.21.15.45:0/3211841824 learned_addr learned my addr 172.21.15.45:0/3211841824 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911da1c640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c12026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 -- 172.21.15.45:0/3211841824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 msgr2=0x5600c12026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c12026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 -- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 msgr2=0x5600c1209720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1209720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 -- 172.21.15.45:0/3211841824 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9118069790 con 0x5600c11f7f80 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911da1c640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c12026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f911e21d640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f10 tx=0x7f91140127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8903758b6028b4e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f9101ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114010810 con 0x5600c11f7f80 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600c0e5f6c0 con 0x5600c11f7f80 2024-03-12T05:10:06.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9114010970 con 0x5600c11f7f80 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9114010b30 con 0x5600c11f7f80 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 10 client.15120 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.104+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5600c115e2b0 con 0x5600c11f7f80 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5600c0e5f6c0 con 0x5600c11f7f80 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f91140253f0 con 0x5600c11f7f80 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 10 client.15120 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 20 client.15120 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c120ec00 0x5600c120de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 4 client.15120 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600c0eb0270 con 0x5600c120ec00 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f911da1c640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c120ec00 0x5600c120de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f9101ffb640 10 client.15120 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:06.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.105+0000 7f911da1c640 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c120ec00 0x5600c120de50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e0c9b86233b28db server_cookie=84f61436db7c0f2e in_seq=0 out_seq=0 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f9101ffb640 1 -- 172.21.15.45:0/3211841824 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5600c0eb0270 con 0x5600c120ec00 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f9101ffb640 10 client.15120 handle_command_reply: tid=0 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 client.15120 shutdown 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 msgr2=0x5600c12021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f10 tx=0x7f91140127c0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 shutdown_connections 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c120ec00 0x5600c120de50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1209720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f9101ffb640 0 client.15120 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c12026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 --2- 172.21.15.45:0/3211841824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f7f80 0x5600c12021a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 >> 172.21.15.45:0/3211841824 conn(0x5600c11f34f0 msgr2=0x5600c11f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 shutdown_connections 2024-03-12T05:10:06.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.106+0000 7f912304e740 1 -- 172.21.15.45:0/3211841824 wait complete. 2024-03-12T05:10:06.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.107+0000 7f912304e740 20 client.15120 trim_cache size 0 max 16384 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 Processor -- start 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 -- start start 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f7f80 0x5600c12170a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c1209640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c10e5490 con 0x5600c11fdae0 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c11f7b40 con 0x5600c11f7f80 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f912304e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c11f3fe0 con 0x5600c11f8970 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f911ea1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f911ea1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45908/0 (socket says 172.21.15.45:45908) 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.122+0000 7f911ea1e640 1 -- 172.21.15.45:0/2664129390 learned_addr learned my addr 172.21.15.45:0/2664129390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911e21d640 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f7f80 0x5600c12170a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 -- 172.21.15.45:0/2664129390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 msgr2=0x5600c1209640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c1209640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 -- 172.21.15.45:0/2664129390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f7f80 msgr2=0x5600c12170a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f7f80 0x5600c12170a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 -- 172.21.15.45:0/2664129390 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600c11fd870 con 0x5600c11fdae0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f911ea1e640 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f911002c630 tx=0x7f91100095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f751dd886210da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f9103fff640 1 -- 172.21.15.45:0/2664129390 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110032a40 con 0x5600c11fdae0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f9103fff640 1 -- 172.21.15.45:0/2664129390 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9110034560 con 0x5600c11fdae0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f9103fff640 1 -- 172.21.15.45:0/2664129390 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110003bd0 con 0x5600c11fdae0 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f912304e740 1 -- 172.21.15.45:0/2664129390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 msgr2=0x5600c11f9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.123+0000 7f912304e740 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f911002c630 tx=0x7f91100095c0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- 172.21.15.45:0/2664129390 shutdown_connections 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11fdae0 0x5600c11f9ed0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f8970 0x5600c1209640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- 172.21.15.45:0/2664129390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11f7f80 0x5600c12170a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- 172.21.15.45:0/2664129390 >> 172.21.15.45:0/2664129390 conn(0x5600c11f34f0 msgr2=0x5600c11f38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- 172.21.15.45:0/2664129390 shutdown_connections 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- 172.21.15.45:0/2664129390 wait complete. 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- start start 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 Processor -- start 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f7f80 0x5600c12062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1264800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c1209830 con 0x5600c11f8970 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c12099a0 con 0x5600c11fdae0 2024-03-12T05:10:06.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c1209b60 con 0x5600c11f7f80 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f912304e740 10 client.0 _ll_register_callbacks cb 0x7f9110037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45920/0 (socket says 172.21.15.45:45920) 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 -- 172.21.15.45:0/3267896356 learned_addr learned my addr 172.21.15.45:0/3267896356 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911ea1e640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1264800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 -- 172.21.15.45:0/3267896356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f7f80 msgr2=0x5600c12062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f7f80 0x5600c12062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 -- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 msgr2=0x5600c1264800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1264800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.124+0000 7f911da1c640 1 -- 172.21.15.45:0/3267896356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600c1114f20 con 0x5600c11f8970 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f911da1c640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9108005820 tx=0x7f910809dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50234c553a44e85e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108099e80 con 0x5600c11f8970 2024-03-12T05:10:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f9108098180 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600c120f5d0 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91080964f0 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 10 client.15126 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5600c10faa20 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5600c120f5d0 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f91080a2190 con 0x5600c11f8970 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 10 client.15126 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 10 client.15126 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 20 client.15126 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c1269ea0 0x5600c1266300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 4 client.15126 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.125+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f9110032ea0 con 0x5600c1269ea0 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f911e21d640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c1269ea0 0x5600c1266300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f911e21d640 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c1269ea0 0x5600c1266300 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aaee59af60fbe3 server_cookie=241b77b1fe3157ca in_seq=0 out_seq=0 2024-03-12T05:10:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f9101ffb640 10 client.15126 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f9101ffb640 1 -- 172.21.15.45:0/3267896356 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f9110032ea0 con 0x5600c1269ea0 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f9101ffb640 10 client.15126 handle_command_reply: tid=0 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 client.15126 shutdown 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 msgr2=0x5600c1206810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9108005820 tx=0x7f910809dfd0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 shutdown_connections 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5600c1269ea0 0x5600c1266300 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5600c11f8970 0x5600c1206810 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600c11fdae0 0x5600c1264800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f9101ffb640 0 client.15126 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:06.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 --2- 172.21.15.45:0/3267896356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600c11f7f80 0x5600c12062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 >> 172.21.15.45:0/3267896356 conn(0x5600c11f34f0 msgr2=0x5600c12170a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 shutdown_connections 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.126+0000 7f912304e740 1 -- 172.21.15.45:0/3267896356 wait complete. 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f912304e740 20 client.15126 trim_cache size 0 max 16384 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24503, 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:06.131 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 88.358495508999994, 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 93.556648886000005 2024-03-12T05:10:06.132 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 msgr2=0x7f91040a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 0x7f91040a3f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9110032460 tx=0x7f9110048000 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 msgr2=0x7f9118108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f910800d980 tx=0x7f910800de40 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 shutdown_connections 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f91040a1ac0 0x7f91040a3f80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9118101290 0x7f9118108fa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101c50 0x7f91181094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 --2- 172.21.15.45:0/3451454795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91181000a0 0x7f911810a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 >> 172.21.15.45:0/3451454795 conn(0x7f91180fb630 msgr2=0x7f9118104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 shutdown_connections 2024-03-12T05:10:06.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:06.127+0000 7f9120481640 1 -- 172.21.15.45:0/3451454795 wait complete. 2024-03-12T05:10:06.135 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T05:10:06.135 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T05:10:06.145 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:10:06.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-12T05:10:06.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:06 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3271756237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:06.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:06 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3271756237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:06.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:06 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3271756237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:07.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:07 smithi130 ceph-mon[35398]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:07.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:07 smithi116 ceph-mon[35486]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:07.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:07 smithi045 ceph-mon[32174]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 -- 172.21.15.45:0/2011035981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 msgr2=0x7f1f68109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/2011035981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f68109e00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000b0a0 tx=0x7f1f6000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 -- 172.21.15.45:0/2011035981 shutdown_connections 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/2011035981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f68109e00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/2011035981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68102730 0x7f1f68104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/2011035981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68105060 0x7f1f68107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 -- 172.21.15.45:0/2011035981 >> 172.21.15.45:0/2011035981 conn(0x7f1f680fc160 msgr2=0x7f1f680fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 -- 172.21.15.45:0/2011035981 shutdown_connections 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 -- 172.21.15.45:0/2011035981 wait complete. 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.073+0000 7f1f6e08d700 1 Processor -- start 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 -- start start 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68102730 0x7f1f68197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 0x7f1f681981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6819f800 con 0x7f1f68107990 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6819f940 con 0x7f1f68105060 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f6e08d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6819fb00 con 0x7f1f68102730 2024-03-12T05:10:08.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58454/0 (socket says 172.21.15.45:58454) 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 -- 172.21.15.45:0/3942088030 learned_addr learned my addr 172.21.15.45:0/3942088030 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f677fe700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68102730 0x7f1f68197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f66ffd700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 0x7f1f681981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 -- 172.21.15.45:0/3942088030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68102730 msgr2=0x7f1f68197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68102730 0x7f1f68197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 -- 172.21.15.45:0/3942088030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 msgr2=0x7f1f681981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 0x7f1f681981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 -- 172.21.15.45:0/3942088030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60009d00 con 0x7f1f68107990 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f66ffd700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 0x7f1f681981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f67fff700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1f600094c0 tx=0x7f1f600094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.074+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6000e040 con 0x7f1f68107990 2024-03-12T05:10:08.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.075+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f6001d930 con 0x7f1f68107990 2024-03-12T05:10:08.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.075+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6819fd80 con 0x7f1f68107990 2024-03-12T05:10:08.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.075+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6001b9e0 con 0x7f1f68107990 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.075+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f681a01a0 con 0x7f1f68107990 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.076+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f680fdce0 con 0x7f1f68107990 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.077+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1f6001daa0 con 0x7f1f68107990 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.077+0000 7f1f64ff9700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 0x7f1f500a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.077+0000 7f1f677fe700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 0x7f1f500a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.077+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1f600cf3c0 con 0x7f1f68107990 2024-03-12T05:10:08.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.078+0000 7f1f677fe700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 0x7f1f500a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f58005950 tx=0x7f1f5800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:08.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.082+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1f60098510 con 0x7f1f68107990 2024-03-12T05:10:08.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.303+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f6802d870 con 0x7f1f500a1b50 2024-03-12T05:10:08.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.304+0000 7f1f64ff9700 1 -- 172.21.15.45:0/3942088030 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1f6802d870 con 0x7f1f500a1b50 2024-03-12T05:10:08.301 INFO:teuthology.orchestra.run.smithi045.stdout:Backend: cephadm 2024-03-12T05:10:08.301 INFO:teuthology.orchestra.run.smithi045.stdout:Available: Yes 2024-03-12T05:10:08.301 INFO:teuthology.orchestra.run.smithi045.stdout:Paused: No 2024-03-12T05:10:08.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 msgr2=0x7f1f500a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 0x7f1f500a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f58005950 tx=0x7f1f5800a300 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 msgr2=0x7f1f6819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1f600094c0 tx=0x7f1f600094f0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 shutdown_connections 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1f500a1b50 0x7f1f500a4010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f68107990 0x7f1f6819f230 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68102730 0x7f1f68197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 --2- 172.21.15.45:0/3942088030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f68105060 0x7f1f681981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 >> 172.21.15.45:0/3942088030 conn(0x7f1f680fc160 msgr2=0x7f1f68103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 shutdown_connections 2024-03-12T05:10:08.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:08.306+0000 7f1f6e08d700 1 -- 172.21.15.45:0/3942088030 wait complete. 2024-03-12T05:10:08.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-12T05:10:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:09.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:09 smithi045 ceph-mon[32174]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:09.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:09 smithi045 ceph-mon[32174]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:09.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:09 smithi130 ceph-mon[35398]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:09.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:09 smithi130 ceph-mon[35398]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:09.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:09 smithi116 ceph-mon[35486]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:09.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:09 smithi116 ceph-mon[35486]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:10.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 -- 172.21.15.45:0/2445731039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0850075ca0 msgr2=0x7f0850074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:10.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 --2- 172.21.15.45:0/2445731039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0850075ca0 0x7f0850074100 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0840009990 tx=0x7f0840009ca0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 -- 172.21.15.45:0/2445731039 shutdown_connections 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 --2- 172.21.15.45:0/2445731039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 0x7f0850110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 --2- 172.21.15.45:0/2445731039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0850075ca0 0x7f0850074100 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 --2- 172.21.15.45:0/2445731039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08500746d0 0x7f0850074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 -- 172.21.15.45:0/2445731039 >> 172.21.15.45:0/2445731039 conn(0x7f08500fd250 msgr2=0x7f08500ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 -- 172.21.15.45:0/2445731039 shutdown_connections 2024-03-12T05:10:10.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.902+0000 7f08551d6700 1 -- 172.21.15.45:0/2445731039 wait complete. 2024-03-12T05:10:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.903+0000 7f08551d6700 1 Processor -- start 2024-03-12T05:10:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.903+0000 7f08551d6700 1 -- start start 2024-03-12T05:10:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 0x7f085019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:10.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 0x7f08501a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08501a5d30 con 0x7f08500746d0 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08501a5e70 con 0x7f085010e270 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08501a6000 con 0x7f0850075ca0 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58472/0 (socket says 172.21.15.45:58472) 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 -- 172.21.15.45:0/413082328 learned_addr learned my addr 172.21.15.45:0/413082328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084f59e700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 0x7f08501a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084e59c700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 0x7f085019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 -- 172.21.15.45:0/413082328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 msgr2=0x7f085019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 0x7f085019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 -- 172.21.15.45:0/413082328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 msgr2=0x7f08501a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 0x7f08501a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 -- 172.21.15.45:0/413082328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400095f0 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084e59c700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 0x7f085019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f084ed9d700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0840005f50 tx=0x7f08400038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084001d070 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08501a6280 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0840022470 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084000f6c0 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.904+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08501a66f0 con 0x7f08500746d0 2024-03-12T05:10:10.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.905+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08501044d0 con 0x7f08500746d0 2024-03-12T05:10:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.906+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f08400225e0 con 0x7f08500746d0 2024-03-12T05:10:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.907+0000 7f0837fff700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 0x7f08380a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.907+0000 7f084e59c700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 0x7f08380a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.907+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f08400cf1d0 con 0x7f08500746d0 2024-03-12T05:10:10.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.908+0000 7f084e59c700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 0x7f08380a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08500759d0 tx=0x7f0844008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:10.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:10.911+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0840098320 con 0x7f08500746d0 2024-03-12T05:10:11.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.123+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f085002d7d0 con 0x7f08380a1b50 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.137+0000 7f0837fff700 1 -- 172.21.15.45:0/413082328 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f085002d7d0 con 0x7f08380a1b50 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:mds.a smithi045 running (101s) 16s ago 101s 17.3M - 19.0.0-1607-g94a53970 a6477a474d71 2fc266cb6911 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:mds.b smithi116 running (98s) 32s ago 98s 25.9M - 19.0.0-1607-g94a53970 a6477a474d71 1eb159864d3b 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:mds.c smithi130 running (113s) 32s ago 112s 19.7M - 19.0.0-1607-g94a53970 a6477a474d71 0f87dde9cd09 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:mds.d smithi045 running (2m) 16s ago 2m 15.3M - 19.0.0-1607-g94a53970 a6477a474d71 7f8c61f1d379 2024-03-12T05:10:11.135 INFO:teuthology.orchestra.run.smithi045.stdout:mds.e smithi116 running (118s) 32s ago 118s 20.0M - 19.0.0-1607-g94a53970 a6477a474d71 b6df71652c2a 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.f smithi130 running (115s) 32s ago 115s 17.5M - 19.0.0-1607-g94a53970 a6477a474d71 7e342701e597 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.g smithi045 running (104s) 16s ago 104s 17.0M - 19.0.0-1607-g94a53970 a6477a474d71 798b2fd4c2d8 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.h smithi116 running (110s) 32s ago 109s 19.6M - 19.0.0-1607-g94a53970 a6477a474d71 8f372197e694 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.i smithi130 running (107s) 32s ago 107s 20.1M - 19.0.0-1607-g94a53970 a6477a474d71 651a89d1fe4a 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.j smithi045 running (2m) 16s ago 2m 13.0M - 19.0.0-1607-g94a53970 a6477a474d71 19c6da4d4b8d 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.k smithi116 running (2m) 32s ago 2m 16.9M - 19.0.0-1607-g94a53970 a6477a474d71 4681eced2e4f 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mds.l smithi130 running (2m) 32s ago 2m 16.6M - 19.0.0-1607-g94a53970 a6477a474d71 e87bf456e869 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.x smithi045 *:9283,8765 running (9m) 16s ago 9m 513M - 19.0.0-1607-g94a53970 a6477a474d71 fd1d6e2148cb 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.y smithi116 *:8443,8765 running (7m) 32s ago 7m 453M - 19.0.0-1607-g94a53970 a6477a474d71 f93e594effb6 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.z smithi130 *:8443,8765 running (7m) 32s ago 7m 450M - 19.0.0-1607-g94a53970 a6477a474d71 15fd0feb25c7 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a smithi045 running (9m) 16s ago 9m 48.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 20415fd31915 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b smithi116 running (7m) 32s ago 7m 39.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 3136956a4350 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c smithi130 running (7m) 32s ago 7m 38.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 2543392497e6 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (6m) 16s ago 6m 56.6M 4273M 19.0.0-1607-g94a53970 a6477a474d71 cb87d553c9b8 2024-03-12T05:10:11.136 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi116 running (6m) 32s ago 6m 45.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 1797fa2b141d 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi130 running (6m) 32s ago 6m 57.2M 4272M 19.0.0-1607-g94a53970 a6477a474d71 bf548f5da58c 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (5m) 16s ago 5m 54.0M 4273M 19.0.0-1607-g94a53970 a6477a474d71 b35959c6f82a 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi116 running (5m) 32s ago 5m 42.3M 4272M 19.0.0-1607-g94a53970 a6477a474d71 3222ef06758f 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi130 running (5m) 32s ago 4m 43.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 c45dbddf9925 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi045 running (4m) 16s ago 4m 53.9M 4273M 19.0.0-1607-g94a53970 a6477a474d71 d2d7925406d8 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi116 running (4m) 32s ago 4m 44.0M 4272M 19.0.0-1607-g94a53970 a6477a474d71 5e6258d98f32 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.8 smithi130 running (3m) 32s ago 3m 45.0M 4272M 19.0.0-1607-g94a53970 a6477a474d71 016d10eea39a 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.9 smithi045 running (3m) 16s ago 3m 53.0M 4273M 19.0.0-1607-g94a53970 a6477a474d71 0b17c685f6c4 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.10 smithi116 running (2m) 32s ago 2m 44.3M 4272M 19.0.0-1607-g94a53970 a6477a474d71 71a911bf747b 2024-03-12T05:10:11.137 INFO:teuthology.orchestra.run.smithi045.stdout:osd.11 smithi130 running (2m) 32s ago 2m 40.1M 4272M 19.0.0-1607-g94a53970 a6477a474d71 933ff8d8c446 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 msgr2=0x7f08380a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 0x7f08380a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08500759d0 tx=0x7f0844008040 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 msgr2=0x7f085019e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0840005f50 tx=0x7f08400038c0 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 shutdown_connections 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f08380a1b50 0x7f08380a4010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f085010e270 0x7f08501a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08500746d0 0x7f085019e1a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 --2- 172.21.15.45:0/413082328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0850075ca0 0x7f085019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:11.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 >> 172.21.15.45:0/413082328 conn(0x7f08500fd250 msgr2=0x7f0850101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 shutdown_connections 2024-03-12T05:10:11.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:11.140+0000 7f08551d6700 1 -- 172.21.15.45:0/413082328 wait complete. 2024-03-12T05:10:11.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:11 smithi130 ceph-mon[35398]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:11.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:11 smithi130 ceph-mon[35398]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:11.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:11 smithi116 ceph-mon[35486]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:11.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:11 smithi116 ceph-mon[35486]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:11.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:11 smithi045 ceph-mon[32174]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:11.557 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:11 smithi045 ceph-mon[32174]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:11.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-12T05:10:12.108 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:13.446 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:13 smithi045 ceph-mon[32174]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:13.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:13 smithi116 ceph-mon[35486]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:13.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:13 smithi130 ceph-mon[35398]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:13.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 -- 172.21.15.45:0/1003122348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 msgr2=0x7fadc41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 --2- 172.21.15.45:0/1003122348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc41020c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fadb40099c0 tx=0x7fadb4009cd0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 -- 172.21.15.45:0/1003122348 shutdown_connections 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 --2- 172.21.15.45:0/1003122348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4103840 0x7fadc410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 --2- 172.21.15.45:0/1003122348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc41020c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 --2- 172.21.15.45:0/1003122348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4102ea0 0x7fadc4103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 -- 172.21.15.45:0/1003122348 >> 172.21.15.45:0/1003122348 conn(0x7fadc40fd220 msgr2=0x7fadc40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 -- 172.21.15.45:0/1003122348 shutdown_connections 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.798+0000 7fadcaa22700 1 -- 172.21.15.45:0/1003122348 wait complete. 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 Processor -- start 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 -- start start 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:13.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4102ea0 0x7fadc4074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 0x7fadc40745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc4074d20 con 0x7fadc4101ca0 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc4074e90 con 0x7fadc4102ea0 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadcaa22700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc41a1ba0 con 0x7fadc4103840 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58492/0 (socket says 172.21.15.45:58492) 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 -- 172.21.15.45:0/3114640718 learned_addr learned my addr 172.21.15.45:0/3114640718 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:13.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc37fe700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4102ea0 0x7fadc4074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc8fbf700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 0x7fadc40745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 -- 172.21.15.45:0/3114640718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 msgr2=0x7fadc40745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 0x7fadc40745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 -- 172.21.15.45:0/3114640718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4102ea0 msgr2=0x7fadc4074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4102ea0 0x7fadc4074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 -- 172.21.15.45:0/3114640718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb40095f0 con 0x7fadc4101ca0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc8fbf700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 0x7fadc40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.799+0000 7fadc3fff700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009f50 tx=0x7fadb4005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.800+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb401d070 con 0x7fadc4101ca0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.800+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fadb40041c0 con 0x7fadc4101ca0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.800+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc41a1e20 con 0x7fadc4101ca0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.800+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb400f460 con 0x7fadc4101ca0 2024-03-12T05:10:13.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.800+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc41a2200 con 0x7fadc4101ca0 2024-03-12T05:10:13.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.801+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc4068930 con 0x7fadc4101ca0 2024-03-12T05:10:13.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.802+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fadb4022700 con 0x7fadc4101ca0 2024-03-12T05:10:13.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.802+0000 7fadc17fa700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 0x7fadac0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:13.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.802+0000 7fadc37fe700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 0x7fadac0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:13.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.802+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fadb40ce950 con 0x7fadc4101ca0 2024-03-12T05:10:13.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.803+0000 7fadc37fe700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 0x7fadac0a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fadb8005950 tx=0x7fadb800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:13.807+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadb4097aa0 con 0x7fadc4101ca0 2024-03-12T05:10:14.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.017+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fadc402d950 con 0x7fadac0a1b50 2024-03-12T05:10:14.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.023+0000 7fadc17fa700 1 -- 172.21.15.45:0/3114640718 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fadc402d950 con 0x7fadac0a1b50 2024-03-12T05:10:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T05:10:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:mds.all 12/12 35s ago 2m smithi045=a;smithi045=d;smithi045=g;smithi045=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-03-12T05:10:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:mgr 3/3 35s ago 7m smithi045=x;smithi116=y;smithi130=z;count:3 2024-03-12T05:10:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:mon 3/3 35s ago 7m smithi045:172.21.15.45=a;smithi116:172.21.15.116=b;smithi130:172.21.15.130=c;count:3 2024-03-12T05:10:14.021 INFO:teuthology.orchestra.run.smithi045.stdout:osd 12 35s ago - 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.025+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 msgr2=0x7fadac0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.025+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 0x7fadac0a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fadb8005950 tx=0x7fadb800a300 comp rx=0 tx=0).stop 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.025+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 msgr2=0x7fadc4075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.025+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009f50 tx=0x7fadb4005ee0 comp rx=0 tx=0).stop 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 shutdown_connections 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fadac0a1b50 0x7fadac0a4010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:14.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fadc4103840 0x7fadc40745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadc4101ca0 0x7fadc4075a10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 --2- 172.21.15.45:0/3114640718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fadc4102ea0 0x7fadc4074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 >> 172.21.15.45:0/3114640718 conn(0x7fadc40fd220 msgr2=0x7fadc41062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 shutdown_connections 2024-03-12T05:10:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:14.026+0000 7fadcaa22700 1 -- 172.21.15.45:0/3114640718 wait complete. 2024-03-12T05:10:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:14 smithi130 ceph-mon[35398]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:14.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:14 smithi116 ceph-mon[35486]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:14.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:14 smithi045 ceph-mon[32174]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:14.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-12T05:10:15.047 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:15.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:15 smithi045 ceph-mon[32174]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:10:15.422 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:15 smithi045 ceph-mon[32174]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-12T05:10:15.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:15 smithi130 ceph-mon[35398]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:10:15.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:15 smithi130 ceph-mon[35398]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-12T05:10:15.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:15 smithi116 ceph-mon[35486]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:10:15.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:15 smithi116 ceph-mon[35486]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-12T05:10:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 -- 172.21.15.45:0/724414038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08101c90 msgr2=0x7f0e081020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 --2- 172.21.15.45:0/724414038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08101c90 0x7f0e081020b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0df800b0d0 tx=0x7f0df800b3e0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 -- 172.21.15.45:0/724414038 shutdown_connections 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 --2- 172.21.15.45:0/724414038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08103830 0x7f0e0810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 --2- 172.21.15.45:0/724414038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08101c90 0x7f0e081020b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 --2- 172.21.15.45:0/724414038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08102e90 0x7f0e081032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 -- 172.21.15.45:0/724414038 >> 172.21.15.45:0/724414038 conn(0x7f0e080fd250 msgr2=0x7f0e080ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 -- 172.21.15.45:0/724414038 shutdown_connections 2024-03-12T05:10:16.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.743+0000 7f0e0c911700 1 -- 172.21.15.45:0/724414038 wait complete. 2024-03-12T05:10:16.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 Processor -- start 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 -- start start 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08101c90 0x7f0e08197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 0x7f0e0819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0819f950 con 0x7f0e08102e90 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0819fa90 con 0x7f0e08103830 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e0c911700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0819fc20 con 0x7f0e08101c90 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e05d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.744+0000 7f0e05d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58508/0 (socket says 172.21.15.45:58508) 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 -- 172.21.15.45:0/3655703735 learned_addr learned my addr 172.21.15.45:0/3655703735 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e0659c700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08101c90 0x7f0e08197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e06d9d700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 0x7f0e0819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:16.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 -- 172.21.15.45:0/3655703735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08101c90 msgr2=0x7f0e08197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08101c90 0x7f0e08197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 -- 172.21.15.45:0/3655703735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 msgr2=0x7f0e0819f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 0x7f0e0819f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 -- 172.21.15.45:0/3655703735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8009d00 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e06d9d700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 0x7f0e0819f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e05d9b700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c950 tx=0x7f0df000cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0007ac0 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0819ff00 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0df0004ce0 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0005570 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.745+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e081a03f0 con 0x7f0e08102e90 2024-03-12T05:10:16.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.746+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08068930 con 0x7f0e08102e90 2024-03-12T05:10:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.747+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0df0004750 con 0x7f0e08102e90 2024-03-12T05:10:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.748+0000 7f0dff7fe700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 0x7f0df40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.748+0000 7f0e0659c700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 0x7f0df40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.748+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0df00cd450 con 0x7f0e08102e90 2024-03-12T05:10:16.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.749+0000 7f0e0659c700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 0x7f0df40a3f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009fd0 tx=0x7f0df800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:16.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.752+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0df0096670 con 0x7f0e08102e90 2024-03-12T05:10:16.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.959+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e0802d950 con 0x7f0df40a1ac0 2024-03-12T05:10:16.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.961+0000 7f0dff7fe700 1 -- 172.21.15.45:0/3655703735 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f0e0802d950 con 0x7f0df40a1ac0 2024-03-12T05:10:16.958 INFO:teuthology.orchestra.run.smithi045.stdout:HOST ADDR LABELS STATUS 2024-03-12T05:10:16.959 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 172.21.15.45 2024-03-12T05:10:16.959 INFO:teuthology.orchestra.run.smithi045.stdout:smithi116 172.21.15.116 2024-03-12T05:10:16.959 INFO:teuthology.orchestra.run.smithi045.stdout:smithi130 172.21.15.130 2024-03-12T05:10:16.959 INFO:teuthology.orchestra.run.smithi045.stdout:3 hosts in cluster 2024-03-12T05:10:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 msgr2=0x7f0df40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:16.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 0x7f0df40a3f80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009fd0 tx=0x7f0df800b040 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 msgr2=0x7f0e08198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c950 tx=0x7f0df000cd10 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 shutdown_connections 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0df40a1ac0 0x7f0df40a3f80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0e08103830 0x7f0e0819f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e08101c90 0x7f0e08197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 --2- 172.21.15.45:0/3655703735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e08102e90 0x7f0e08198300 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 >> 172.21.15.45:0/3655703735 conn(0x7f0e080fd250 msgr2=0x7f0e08106260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 shutdown_connections 2024-03-12T05:10:16.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:16.963+0000 7f0e0c911700 1 -- 172.21.15.45:0/3655703735 wait complete. 2024-03-12T05:10:17.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:17 smithi130 ceph-mon[35398]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:17.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:17 smithi130 ceph-mon[35398]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:17.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:17 smithi116 ceph-mon[35486]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:17.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:17 smithi116 ceph-mon[35486]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:17.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:17 smithi045 ceph-mon[32174]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:17.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:17 smithi045 ceph-mon[32174]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:17.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-12T05:10:17.908 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:19.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:19 smithi130 ceph-mon[35398]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:19.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:19 smithi116 ceph-mon[35486]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:19.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:19 smithi045 ceph-mon[32174]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 -- 172.21.15.45:0/2629245704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c101da0 msgr2=0x7fe94c102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 --2- 172.21.15.45:0/2629245704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c101da0 0x7fe94c102200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe9400099c0 tx=0x7fe940009cd0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 -- 172.21.15.45:0/2629245704 shutdown_connections 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 --2- 172.21.15.45:0/2629245704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c102740 0x7fe94c108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 --2- 172.21.15.45:0/2629245704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c100ba0 0x7fe94c100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 --2- 172.21.15.45:0/2629245704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c101da0 0x7fe94c102200 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 -- 172.21.15.45:0/2629245704 >> 172.21.15.45:0/2629245704 conn(0x7fe94c0fc140 msgr2=0x7fe94c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:19.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 -- 172.21.15.45:0/2629245704 shutdown_connections 2024-03-12T05:10:19.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.754+0000 7fe952470700 1 -- 172.21.15.45:0/2629245704 wait complete. 2024-03-12T05:10:19.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 Processor -- start 2024-03-12T05:10:19.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 -- start start 2024-03-12T05:10:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 0x7fe94c103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:19.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 0x7fe94c1036c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c103e40 con 0x7fe94c100ba0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c103fb0 con 0x7fe94c102740 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe952470700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c1a1bf0 con 0x7fe94c101da0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48278/0 (socket says 172.21.15.45:48278) 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 -- 172.21.15.45:0/1446584079 learned_addr learned my addr 172.21.15.45:0/1446584079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94b7fe700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 0x7fe94c103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe950a0d700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 0x7fe94c1036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 -- 172.21.15.45:0/1446584079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 msgr2=0x7fe94c103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 0x7fe94c103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 -- 172.21.15.45:0/1446584079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 msgr2=0x7fe94c1036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 0x7fe94c1036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 -- 172.21.15.45:0/1446584079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400095f0 con 0x7fe94c100ba0 2024-03-12T05:10:19.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe950a0d700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 0x7fe94c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94b7fe700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 0x7fe94c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.755+0000 7fe94bfff700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00d980 tx=0x7fe93c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.756+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00bb60 con 0x7fe94c100ba0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.756+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c1a1ed0 con 0x7fe94c100ba0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.756+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe93c007620 con 0x7fe94c100ba0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.756+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00f620 con 0x7fe94c100ba0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.756+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c1a22f0 con 0x7fe94c100ba0 2024-03-12T05:10:19.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.757+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c068930 con 0x7fe94c100ba0 2024-03-12T05:10:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.758+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe93c010460 con 0x7fe94c100ba0 2024-03-12T05:10:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.758+0000 7fe9497fa700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 0x7fe9340a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.758+0000 7fe94b7fe700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 0x7fe9340a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.758+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe93c0cdc20 con 0x7fe94c100ba0 2024-03-12T05:10:19.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.759+0000 7fe94b7fe700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 0x7fe9340a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9400037f0 tx=0x7fe940005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:19.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.762+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe93c096e40 con 0x7fe94c100ba0 2024-03-12T05:10:19.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.975+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe94c0fe360 con 0x7fe9340a1ac0 2024-03-12T05:10:19.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.977+0000 7fe9497fa700 1 -- 172.21.15.45:0/1446584079 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fe94c0fe360 con 0x7fe9340a1ac0 2024-03-12T05:10:19.975 INFO:teuthology.orchestra.run.smithi045.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-12T05:10:19.975 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-12T05:10:19.975 INFO:teuthology.orchestra.run.smithi045.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-12T05:10:19.975 INFO:teuthology.orchestra.run.smithi045.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-12T05:10:19.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 msgr2=0x7fe9340a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:19.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 0x7fe9340a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9400037f0 tx=0x7fe940005fb0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 msgr2=0x7fe94c104b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:19.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00d980 tx=0x7fe93c00dd40 comp rx=0 tx=0).stop 2024-03-12T05:10:19.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 shutdown_connections 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe9340a1ac0 0x7fe9340a3f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe94c102740 0x7fe94c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe94c100ba0 0x7fe94c104b30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 --2- 172.21.15.45:0/1446584079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe94c101da0 0x7fe94c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 >> 172.21.15.45:0/1446584079 conn(0x7fe94c0fc140 msgr2=0x7fe94c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 shutdown_connections 2024-03-12T05:10:19.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:19.979+0000 7fe952470700 1 -- 172.21.15.45:0/1446584079 wait complete. 2024-03-12T05:10:20.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:20 smithi045 ceph-mon[32174]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:20.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:20 smithi130 ceph-mon[35398]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:20.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:20 smithi116 ceph-mon[35486]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:20.775 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T05:10:20.784 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:10:20.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-12T05:10:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:21.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:21 smithi130 ceph-mon[35398]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:21.493 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:21 smithi045 ceph-mon[32174]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:21.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:21 smithi116 ceph-mon[35486]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-12T05:10:22.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/3046740349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4100c10 msgr2=0x7fd9c4101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/3046740349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4100c10 0x7fd9c4101030 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40099c0 tx=0x7fd9b4009cd0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/3046740349 shutdown_connections 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/3046740349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 0x7fd9c4109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/3046740349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4100c10 0x7fd9c4101030 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/3046740349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4101e10 0x7fd9c4102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/3046740349 >> 172.21.15.45:0/3046740349 conn(0x7fd9c40fc1d0 msgr2=0x7fd9c40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/3046740349 shutdown_connections 2024-03-12T05:10:22.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/3046740349 wait complete. 2024-03-12T05:10:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 Processor -- start 2024-03-12T05:10:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.800+0000 7fd9c9bbe700 1 -- start start 2024-03-12T05:10:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 0x7fd9c4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 0x7fd9c419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c419f740 con 0x7fd9c4101e10 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c419f880 con 0x7fd9c4100c10 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c9bbe700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c419fa10 con 0x7fd9c41027b0 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48298/0 (socket says 172.21.15.45:48298) 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 -- 172.21.15.45:0/1635342570 learned_addr learned my addr 172.21.15.45:0/1635342570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c3fff700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 0x7fd9c419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c37fe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 0x7fd9c4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 -- 172.21.15.45:0/1635342570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 msgr2=0x7fd9c419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 0x7fd9c419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 -- 172.21.15.45:0/1635342570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 msgr2=0x7fd9c4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 0x7fd9c4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 -- 172.21.15.45:0/1635342570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b40095f0 con 0x7fd9c4101e10 2024-03-12T05:10:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c37fe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 0x7fd9c4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.801+0000 7fd9c2ffd700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c9e0 tx=0x7fd9b800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.802+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8007ac0 con 0x7fd9c4101e10 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.802+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd9b8007c20 con 0x7fd9c4101e10 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.802+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c419fcf0 con 0x7fd9c4101e10 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.802+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8018660 con 0x7fd9c4101e10 2024-03-12T05:10:22.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.802+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c41a01a0 con 0x7fd9c4101e10 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.803+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c4068930 con 0x7fd9c4101e10 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.805+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd9b800f450 con 0x7fd9c4101e10 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.805+0000 7fd9c0ff9700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 0x7fd9ac0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.805+0000 7fd9c37fe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 0x7fd9ac0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.805+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd9b80cd660 con 0x7fd9c4101e10 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.806+0000 7fd9c37fe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 0x7fd9ac0a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4000c00 tx=0x7fd9b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:22.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:22.808+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd9b8096910 con 0x7fd9c4101e10 2024-03-12T05:10:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.055+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd9c410c3e0 con 0x7fd9c4101e10 2024-03-12T05:10:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.055+0000 7fd9c0ff9700 1 -- 172.21.15.45:0/1635342570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2973 (secure 0 0 0) 0x7fd9b800f700 con 0x7fd9c4101e10 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:e24 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:legacy client fscid: 1 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:Filesystem 'cephfs' (1) 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:fs_name cephfs 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 24 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-12T05:10:23.054 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-12T05:08:37.112985+0000 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2024-03-12T05:08:43.349662+0000 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:tableserver 0 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:root 0 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:session_timeout 300 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:session_autoclose 300 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:max_file_size 1099511627776 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:max_xattr_size 65536 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:required_client_features {} 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure 0 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure_osd_epoch 0 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:max_mds 5 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:in 0,1,2,3,4 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:up {0=24503,1=24487,2=24472,3=24467,4=24460} 2024-03-12T05:10:23.055 INFO:teuthology.orchestra.run.smithi045.stdout:failed 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:damaged 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:stopped 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:data_pools [3] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:metadata_pool 2 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:inline_data disabled 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:balancer 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:bal_rank_mask -1 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:standby_count_wanted 0 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.e{0:24443} state up:standby-replay seq 1 addr [v2:172.21.15.116:6834/4238162743,v1:172.21.15.116:6835/4238162743] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.i{1:24487} state up:active seq 5 addr [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.l{1:24421} state up:standby-replay seq 1 addr [v2:172.21.15.130:6832/1602847227,v1:172.21.15.130:6833/1602847227] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.c{2:24472} state up:active seq 7 addr [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.k{2:24419} state up:standby-replay seq 1 addr [v2:172.21.15.116:6832/2049389812,v1:172.21.15.116:6833/2049389812] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.056 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.h{3:24467} state up:active seq 7 addr [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.a{3:14631} state up:standby-replay seq 1 addr [v2:172.21.15.45:6840/3025141330,v1:172.21.15.45:6841/3025141330] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.f{4:24460} state up:active seq 8 addr [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.g{4:14619} state up:standby-replay seq 1 addr [v2:172.21.15.45:6838/1651909238,v1:172.21.15.45:6839/1651909238] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:Standby daemons: 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.45:6834/1645552275,v1:172.21.15.45:6835/1645552275] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.45:6836/378571831,v1:172.21.15.45:6837/378571831] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 msgr2=0x7fd9ac0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 0x7fd9ac0a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4000c00 tx=0x7fd9b4005fb0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 msgr2=0x7fd9c4198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c9e0 tx=0x7fd9b800ccf0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 shutdown_connections 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd9ac0a1ac0 0x7fd9ac0a3f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9c41027b0 0x7fd9c419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9c4100c10 0x7fd9c4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 --2- 172.21.15.45:0/1635342570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9c4101e10 0x7fd9c4198160 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:23.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.057+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 >> 172.21.15.45:0/1635342570 conn(0x7fd9c40fc1d0 msgr2=0x7fd9c40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:23.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.058+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 shutdown_connections 2024-03-12T05:10:23.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:23.058+0000 7fd9c9bbe700 1 -- 172.21.15.45:0/1635342570 wait complete. 2024-03-12T05:10:23.059 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:23.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:23 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1635342570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:10:23.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:23 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1635342570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:10:23.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:23 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1635342570' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:10:23.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-12T05:10:23.970 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:24.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:24 smithi045 ceph-mon[32174]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:24.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:24 smithi130 ceph-mon[35398]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:24.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:24 smithi116 ceph-mon[35486]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-12T05:10:25.390 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:25 smithi045 ceph-mon[32174]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-12T05:10:25.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:25 smithi130 ceph-mon[35398]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-12T05:10:25.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:25 smithi116 ceph-mon[35486]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-12T05:10:25.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 -- 172.21.15.45:0/3295439296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648101cd0 msgr2=0x7f56481020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:25.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 --2- 172.21.15.45:0/3295439296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648101cd0 0x7f56481020f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f56380099c0 tx=0x7f5638009cd0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 -- 172.21.15.45:0/3295439296 shutdown_connections 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 --2- 172.21.15.45:0/3295439296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648103870 0x7f564810a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 --2- 172.21.15.45:0/3295439296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648101cd0 0x7f56481020f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 --2- 172.21.15.45:0/3295439296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 0x7f5648103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 -- 172.21.15.45:0/3295439296 >> 172.21.15.45:0/3295439296 conn(0x7f56480fd270 msgr2=0x7f56480ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 -- 172.21.15.45:0/3295439296 shutdown_connections 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.745+0000 7f564d6a9700 1 -- 172.21.15.45:0/3295439296 wait complete. 2024-03-12T05:10:25.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.746+0000 7f564d6a9700 1 Processor -- start 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.746+0000 7f564d6a9700 1 -- start start 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 0x7f5648197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 0x7f5648198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564819f690 con 0x7f5648103870 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564819f7d0 con 0x7f5648101cd0 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564819f960 con 0x7f5648102ed0 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48318/0 (socket says 172.21.15.45:48318) 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 -- 172.21.15.45:0/2111858830 learned_addr learned my addr 172.21.15.45:0/2111858830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56467fc700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 0x7f5648198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f5646ffd700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 0x7f5648197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 -- 172.21.15.45:0/2111858830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 msgr2=0x7f5648198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:25.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 0x7f5648198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 -- 172.21.15.45:0/2111858830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 msgr2=0x7f5648197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 0x7f5648197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 -- 172.21.15.45:0/2111858830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380095f0 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f5646ffd700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 0x7f5648197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f56477fe700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5640007da0 tx=0x7f564000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000ad00 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f564000ae60 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564819fc40 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640011640 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.747+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481a00f0 con 0x7f5648103870 2024-03-12T05:10:25.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.748+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5648109090 con 0x7f5648103870 2024-03-12T05:10:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.749+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5640004ad0 con 0x7f5648103870 2024-03-12T05:10:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.750+0000 7f562ffff700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.750+0000 7f5646ffd700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.750+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f56400cda40 con 0x7f5648103870 2024-03-12T05:10:25.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.750+0000 7f5646ffd700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 0x7f56300a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:25.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.754+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5640096c60 con 0x7f5648103870 2024-03-12T05:10:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.960+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f5648068980 con 0x7f5648103870 2024-03-12T05:10:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.961+0000 7f562ffff700 1 -- 172.21.15.45:0/2111858830 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5276 (secure 0 0 0) 0x7f5640016070 con 0x7f5648103870 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 87 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:fsid 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-12T05:00:48.629903+0000 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2024-03-12T05:09:44.121222+0000 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:crush_version 26 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:full_ratio 0.95 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:backfillfull_ratio 0.9 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:nearfull_ratio 0.85 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:require_min_compat_client luminous 2024-03-12T05:10:25.960 INFO:teuthology.orchestra.run.smithi045.stdout:min_compat_client luminous 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:require_osd_release squid 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:stretch_mode_enabled false 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:max_osd 12 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6802/55762477,v1:172.21.15.45:6803/55762477] [v2:172.21.15.45:6804/55762477,v1:172.21.15.45:6805/55762477] exists,up 4e690af1-6483-47d3-8fcb-240b9f1bec6b 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/1071649441,v1:172.21.15.116:6801/1071649441] [v2:172.21.15.116:6802/1071649441,v1:172.21.15.116:6803/1071649441] exists,up 50765f29-2dd5-499e-aeb8-529a3344ab49 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/845680146,v1:172.21.15.130:6801/845680146] [v2:172.21.15.130:6802/845680146,v1:172.21.15.130:6803/845680146] exists,up 8d3b4de1-376d-4209-bef9-a48e51f77a93 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6810/1948936218,v1:172.21.15.45:6811/1948936218] [v2:172.21.15.45:6812/1948936218,v1:172.21.15.45:6813/1948936218] exists,up 522e40da-a98f-4f92-bc68-60f665e2f962 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6808/3946228497,v1:172.21.15.116:6809/3946228497] [v2:172.21.15.116:6810/3946228497,v1:172.21.15.116:6811/3946228497] exists,up c172804f-1d0d-4ab9-9dc9-a67b20f80b52 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/2846635408,v1:172.21.15.130:6809/2846635408] [v2:172.21.15.130:6810/2846635408,v1:172.21.15.130:6811/2846635408] exists,up 5bf734df-d459-46bb-a3a2-432bdcf349fb 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6818/3221804389,v1:172.21.15.45:6819/3221804389] [v2:172.21.15.45:6820/3221804389,v1:172.21.15.45:6821/3221804389] exists,up 84d0438a-d41f-43a0-b705-636605037f1c 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/1067697558,v1:172.21.15.116:6817/1067697558] [v2:172.21.15.116:6818/1067697558,v1:172.21.15.116:6819/1067697558] exists,up da01102a-465e-47b8-bc0a-8d4119260bac 2024-03-12T05:10:25.961 INFO:teuthology.orchestra.run.smithi045.stdout:osd.8 up in weight 1 up_from 59 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/1919686935,v1:172.21.15.130:6817/1919686935] [v2:172.21.15.130:6818/1919686935,v1:172.21.15.130:6819/1919686935] exists,up ae6cce03-2fb7-4fe5-a943-33e47113ef19 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6826/15901149,v1:172.21.15.45:6827/15901149] [v2:172.21.15.45:6828/15901149,v1:172.21.15.45:6829/15901149] exists,up 8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6824/2890731133,v1:172.21.15.116:6825/2890731133] [v2:172.21.15.116:6826/2890731133,v1:172.21.15.116:6827/2890731133] exists,up 00932bcc-b7f2-4a9c-9421-ce264276091b 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/627874902,v1:172.21.15.130:6825/627874902] [v2:172.21.15.130:6826/627874902,v1:172.21.15.130:6827/627874902] exists,up 57499d1f-cdfc-473e-b09c-9c906168826d 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.8 [2,11] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.b [8,11] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.16 [2,4] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 2.1f [8,1] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.0 [5,9,4,6] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.1d [4,8] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.1e [3,9] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 3.24 [4,9] 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/2524215406 expires 2024-03-13T05:01:43.021024+0000 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/2524215406 expires 2024-03-13T05:01:43.021024+0000 2024-03-12T05:10:25.962 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2481316973 expires 2024-03-13T05:01:43.021024+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2630663966 expires 2024-03-13T05:01:43.021024+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/1305486259 expires 2024-03-13T05:01:16.183699+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2281722728 expires 2024-03-13T05:01:16.183699+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/4138375301 expires 2024-03-13T05:01:16.183699+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/4294356868 expires 2024-03-13T05:01:16.183699+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/1305486259 expires 2024-03-13T05:01:16.183699+0000 2024-03-12T05:10:25.963 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/662273801 expires 2024-03-13T05:01:43.021024+0000 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 msgr2=0x7f56300a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 0x7f56300a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5638000c00 tx=0x7f5638005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 msgr2=0x7f564819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5640007da0 tx=0x7f564000a340 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 shutdown_connections 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5648103870 0x7f564819f0c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5648101cd0 0x7f5648197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 --2- 172.21.15.45:0/2111858830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5648102ed0 0x7f5648198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 >> 172.21.15.45:0/2111858830 conn(0x7f56480fd270 msgr2=0x7f56481061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 shutdown_connections 2024-03-12T05:10:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:25.963+0000 7f564d6a9700 1 -- 172.21.15.45:0/2111858830 wait complete. 2024-03-12T05:10:26.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:26 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2111858830' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:10:26.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:26 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2111858830' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:10:26.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:26 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2111858830' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:10:26.684 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-12T05:10:26.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 -- 172.21.15.45:0/1317620958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518106f40 msgr2=0x7f3518109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 --2- 172.21.15.45:0/1317620958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518106f40 0x7f3518109330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3510009ca0 tx=0x7f351002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 -- 172.21.15.45:0/1317620958 shutdown_connections 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 --2- 172.21.15.45:0/1317620958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518106f40 0x7f3518109330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 --2- 172.21.15.45:0/1317620958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518104610 0x7f3518106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 --2- 172.21.15.45:0/1317620958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518101ce0 0x7f35181040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 -- 172.21.15.45:0/1317620958 >> 172.21.15.45:0/1317620958 conn(0x7f35180fb630 msgr2=0x7f35180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 -- 172.21.15.45:0/1317620958 shutdown_connections 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.804+0000 7f351e84e640 1 -- 172.21.15.45:0/1317620958 wait complete. 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 Processor -- start 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 -- start start 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518104610 0x7f35180ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518106f40 0x7f35181001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518100930 con 0x7f3518101ce0 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518100aa0 con 0x7f3518104610 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351e84e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35181a11d0 con 0x7f3518106f40 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:26.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48330/0 (socket says 172.21.15.45:48330) 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 -- 172.21.15.45:0/1469955515 learned_addr learned my addr 172.21.15.45:0/1469955515 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f35177fe640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518104610 0x7f35180ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f351cdeb640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518106f40 0x7f35181001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 -- 172.21.15.45:0/1469955515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518106f40 msgr2=0x7f35181001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518106f40 0x7f35181001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 -- 172.21.15.45:0/1469955515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518104610 msgr2=0x7f35180ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518104610 0x7f35180ffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.805+0000 7f3517fff640 1 -- 172.21.15.45:0/1469955515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510009900 con 0x7f3518101ce0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f3517fff640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f350800d980 tx=0x7f350800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508009c30 con 0x7f3518101ce0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35181a1480 con 0x7f3518101ce0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f350800b660 con 0x7f3518101ce0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800f620 con 0x7f3518101ce0 2024-03-12T05:10:26.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.806+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35181a1990 con 0x7f3518101ce0 2024-03-12T05:10:26.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.807+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180fd120 con 0x7f3518101ce0 2024-03-12T05:10:26.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.808+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3508010460 con 0x7f3518101ce0 2024-03-12T05:10:26.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.808+0000 7f35157fa640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 0x7f35000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.808+0000 7f35177fe640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 0x7f35000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.808+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f35080ccfd0 con 0x7f3518101ce0 2024-03-12T05:10:26.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.809+0000 7f35177fe640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 0x7f35000a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f35181013c0 tx=0x7f350c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:26.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.812+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f35080961f0 con 0x7f3518101ce0 2024-03-12T05:10:26.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.995+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f351810cc20 con 0x7f3518101ce0 2024-03-12T05:10:26.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.996+0000 7f35157fa640 1 -- 172.21.15.45:0/1469955515 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f351810cc20 con 0x7f3518101ce0 2024-03-12T05:10:26.995 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:26.995 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 msgr2=0x7f35000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 0x7f35000a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f35181013c0 tx=0x7f350c00b390 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 msgr2=0x7f3518101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f350800d980 tx=0x7f350800de40 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 shutdown_connections 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f35000a1a30 0x7f35000a3ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3518106f40 0x7f35181001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3518104610 0x7f35180ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 --2- 172.21.15.45:0/1469955515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3518101ce0 0x7f3518101650 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 >> 172.21.15.45:0/1469955515 conn(0x7f35180fb630 msgr2=0x7f3518102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:26.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 shutdown_connections 2024-03-12T05:10:26.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:26.998+0000 7f351e84e640 1 -- 172.21.15.45:0/1469955515 wait complete. 2024-03-12T05:10:27.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:27.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 -- 172.21.15.45:0/3263588251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 msgr2=0x7fcce81004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 --2- 172.21.15.45:0/3263588251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce81004a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009900 tx=0x7fcce002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 -- 172.21.15.45:0/3263588251 shutdown_connections 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 --2- 172.21.15.45:0/3263588251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce81084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 --2- 172.21.15.45:0/3263588251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce81016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 --2- 172.21.15.45:0/3263588251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce81004a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 -- 172.21.15.45:0/3263588251 >> 172.21.15.45:0/3263588251 conn(0x7fcce80fb610 msgr2=0x7fcce80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 -- 172.21.15.45:0/3263588251 shutdown_connections 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.116+0000 7fccef50c640 1 -- 172.21.15.45:0/3263588251 wait complete. 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 Processor -- start 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 -- start start 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce819ed70 con 0x7fcce8100080 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce819eee0 con 0x7fcce8101270 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccef50c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce819f0a0 con 0x7fcce8101c30 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fcced2a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fcced2a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48354/0 (socket says 172.21.15.45:48354) 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fcced2a8640 1 -- 172.21.15.45:0/3381642807 learned_addr learned my addr 172.21.15.45:0/3381642807 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccecaa7640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fcced2a8640 1 -- 172.21.15.45:0/3381642807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 msgr2=0x7fcce819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.117+0000 7fccedaa9640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fcced2a8640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fcced2a8640 1 -- 172.21.15.45:0/3381642807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 msgr2=0x7fcce819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fcced2a8640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce819bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fcced2a8640 1 -- 172.21.15.45:0/3381642807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0009560 con 0x7fcce8100080 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccecaa7640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccedaa9640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fcced2a8640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcce002c7d0 tx=0x7fcce0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0003800 con 0x7fcce8100080 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce819f2c0 con 0x7fcce8100080 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcce002de90 con 0x7fcce8100080 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00055f0 con 0x7fcce8100080 2024-03-12T05:10:27.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.118+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce819f8a0 con 0x7fcce8100080 2024-03-12T05:10:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.119+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce81074c0 con 0x7fcce8100080 2024-03-12T05:10:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.120+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcce0004dc0 con 0x7fcce8100080 2024-03-12T05:10:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.121+0000 7fccde7fc640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 0x7fccd40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.121+0000 7fccecaa7640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 0x7fccd40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.121+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fcce00efdb0 con 0x7fcce8100080 2024-03-12T05:10:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.121+0000 7fccecaa7640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 0x7fccd40a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcce8068bc0 tx=0x7fccd8005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.124+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcce00b8fd0 con 0x7fcce8100080 2024-03-12T05:10:27.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.308+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcce810bb10 con 0x7fcce8100080 2024-03-12T05:10:27.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.309+0000 7fccde7fc640 1 -- 172.21.15.45:0/3381642807 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcce810bb10 con 0x7fcce8100080 2024-03-12T05:10:27.308 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 msgr2=0x7fccd40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 0x7fccd40a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcce8068bc0 tx=0x7fccd8005ec0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 msgr2=0x7fcce8068e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcce002c7d0 tx=0x7fcce0004360 comp rx=0 tx=0).stop 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 shutdown_connections 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fccd40a1a30 0x7fccd40a3ef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcce8101c30 0x7fcce819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcce8101270 0x7fcce819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 --2- 172.21.15.45:0/3381642807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcce8100080 0x7fcce8068e50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 >> 172.21.15.45:0/3381642807 conn(0x7fcce80fb610 msgr2=0x7fcce8104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 shutdown_connections 2024-03-12T05:10:27.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.311+0000 7fccef50c640 1 -- 172.21.15.45:0/3381642807 wait complete. 2024-03-12T05:10:27.319 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:27 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1469955515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:27.320 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:27 smithi045 ceph-mon[32174]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:27.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 -- 172.21.15.45:0/2600085236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 msgr2=0x7fbb981084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/2600085236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb981084c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009ca0 tx=0x7fbb9002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 -- 172.21.15.45:0/2600085236 shutdown_connections 2024-03-12T05:10:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/2600085236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb981084c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/2600085236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98101270 0x7fbb981016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/2600085236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98100080 0x7fbb981004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 -- 172.21.15.45:0/2600085236 >> 172.21.15.45:0/2600085236 conn(0x7fbb980fb630 msgr2=0x7fbb980fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 -- 172.21.15.45:0/2600085236 shutdown_connections 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 -- 172.21.15.45:0/2600085236 wait complete. 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.428+0000 7fbb9e19a640 1 Processor -- start 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 -- start start 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98100080 0x7fbb98197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98101270 0x7fbb981977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9819ed60 con 0x7fbb98101c30 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9819eed0 con 0x7fbb98101270 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9819f090 con 0x7fbb98100080 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60962/0 (socket says 172.21.15.45:60962) 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 -- 172.21.15.45:0/4232944453 learned_addr learned my addr 172.21.15.45:0/4232944453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb977fe640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98100080 0x7fbb98197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb96ffd640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98101270 0x7fbb981977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 -- 172.21.15.45:0/4232944453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98100080 msgr2=0x7fbb98197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98100080 0x7fbb98197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 -- 172.21.15.45:0/4232944453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98101270 msgr2=0x7fbb981977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98101270 0x7fbb981977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 -- 172.21.15.45:0/4232944453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb90009900 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb97fff640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb90037040 tx=0x7fbb90032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90034e70 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb90003e20 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9003d3f0 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.429+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9819f310 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.430+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9819f840 con 0x7fbb98101c30 2024-03-12T05:10:27.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.431+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb981074c0 con 0x7fbb98101c30 2024-03-12T05:10:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.432+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbb9003b040 con 0x7fbb98101c30 2024-03-12T05:10:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.432+0000 7fbb94ff9640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 0x7fbb800a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.432+0000 7fbb977fe640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 0x7fbb800a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.432+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fbb900efd70 con 0x7fbb98101c30 2024-03-12T05:10:27.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.433+0000 7fbb977fe640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 0x7fbb800a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbb981010d0 tx=0x7fbb88009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.437+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb900b8f90 con 0x7fbb98101c30 2024-03-12T05:10:27.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:27 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1469955515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:27.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:27 smithi130 ceph-mon[35398]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:27.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:27 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1469955515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:27.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:27 smithi116 ceph-mon[35486]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:27.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.617+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb9802ce40 con 0x7fbb98101c30 2024-03-12T05:10:27.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.617+0000 7fbb94ff9640 1 -- 172.21.15.45:0/4232944453 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb900bc650 con 0x7fbb98101c30 2024-03-12T05:10:27.616 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 msgr2=0x7fbb800a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 0x7fbb800a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbb981010d0 tx=0x7fbb88009110 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 msgr2=0x7fbb9819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb90037040 tx=0x7fbb90032980 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 shutdown_connections 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fbb800a1a30 0x7fbb800a3ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb98101c30 0x7fbb9819e820 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb98101270 0x7fbb981977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 --2- 172.21.15.45:0/4232944453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb98100080 0x7fbb98197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 >> 172.21.15.45:0/4232944453 conn(0x7fbb980fb630 msgr2=0x7fbb98104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 shutdown_connections 2024-03-12T05:10:27.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.619+0000 7fbb9e19a640 1 -- 172.21.15.45:0/4232944453 wait complete. 2024-03-12T05:10:27.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 -- 172.21.15.45:0/2119367133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 msgr2=0x7fdac010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 --2- 172.21.15.45:0/2119367133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac010fb50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009cd0 tx=0x7fdab002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 -- 172.21.15.45:0/2119367133 shutdown_connections 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 --2- 172.21.15.45:0/2119367133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac010fb50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 --2- 172.21.15.45:0/2119367133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 0x7fdac0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 --2- 172.21.15.45:0/2119367133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 0x7fdac0073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 -- 172.21.15.45:0/2119367133 >> 172.21.15.45:0/2119367133 conn(0x7fdac00fc720 msgr2=0x7fdac00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 -- 172.21.15.45:0/2119367133 shutdown_connections 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.736+0000 7fdac7beb640 1 -- 172.21.15.45:0/2119367133 wait complete. 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 Processor -- start 2024-03-12T05:10:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 -- start start 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 0x7fdac019d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 0x7fdac019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a52b0 con 0x7fdac010d760 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a5420 con 0x7fdac0073be0 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac7beb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac01a55e0 con 0x7fdac00751c0 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60982/0 (socket says 172.21.15.45:60982) 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 -- 172.21.15.45:0/194973451 learned_addr learned my addr 172.21.15.45:0/194973451 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac5987640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 0x7fdac019d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac5186640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 0x7fdac019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 -- 172.21.15.45:0/194973451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 msgr2=0x7fdac019dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 0x7fdac019dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 -- 172.21.15.45:0/194973451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 msgr2=0x7fdac019d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 0x7fdac019d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 -- 172.21.15.45:0/194973451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009900 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdac6188640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009ca0 tx=0x7fdab0032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.737+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0034e70 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.738+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdab0003e20 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.738+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0004810 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.738+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac01a5860 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.738+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac01a5cc0 con 0x7fdac010d760 2024-03-12T05:10:27.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.738+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac0103930 con 0x7fdac010d760 2024-03-12T05:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.740+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdab003b040 con 0x7fdac010d760 2024-03-12T05:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.740+0000 7fdabeffd640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 0x7fdaa80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.740+0000 7fdac5987640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 0x7fdaa80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.740+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdab00efe50 con 0x7fdac010d760 2024-03-12T05:10:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.741+0000 7fdac5987640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 0x7fdaa80a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009990 tx=0x7fdab8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.744+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdab00b9070 con 0x7fdac010d760 2024-03-12T05:10:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.924+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdac002cda0 con 0x7fdac010d760 2024-03-12T05:10:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.925+0000 7fdabeffd640 1 -- 172.21.15.45:0/194973451 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdab00bc730 con 0x7fdac010d760 2024-03-12T05:10:27.923 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 msgr2=0x7fdaa80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 0x7fdaa80a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009990 tx=0x7fdab8009110 comp rx=0 tx=0).stop 2024-03-12T05:10:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 msgr2=0x7fdac01a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009ca0 tx=0x7fdab0032980 comp rx=0 tx=0).stop 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 shutdown_connections 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa80a1a30 0x7fdaa80a3ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.926+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdac010d760 0x7fdac01a4d70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.927+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac00751c0 0x7fdac019dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.927+0000 7fdac7beb640 1 --2- 172.21.15.45:0/194973451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdac0073be0 0x7fdac019d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.927+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 >> 172.21.15.45:0/194973451 conn(0x7fdac00fc720 msgr2=0x7fdac0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.927+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 shutdown_connections 2024-03-12T05:10:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:27.927+0000 7fdac7beb640 1 -- 172.21.15.45:0/194973451 wait complete. 2024-03-12T05:10:27.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 -- 172.21.15.45:0/2912449200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad8106e70 msgr2=0x7f0ad81092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 --2- 172.21.15.45:0/2912449200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad8106e70 0x7f0ad81092c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009ca0 tx=0x7f0ad002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 -- 172.21.15.45:0/2912449200 shutdown_connections 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 --2- 172.21.15.45:0/2912449200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad8106e70 0x7f0ad81092c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 --2- 172.21.15.45:0/2912449200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 --2- 172.21.15.45:0/2912449200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80fedf0 0x7f0ad80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 -- 172.21.15.45:0/2912449200 >> 172.21.15.45:0/2912449200 conn(0x7f0ad80fa980 msgr2=0x7f0ad80fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 -- 172.21.15.45:0/2912449200 shutdown_connections 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.040+0000 7f0adcec0640 1 -- 172.21.15.45:0/2912449200 wait complete. 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 Processor -- start 2024-03-12T05:10:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 -- start start 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad81008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8106e70 0x7f0ad8100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8101590 con 0x7f0ad80fedf0 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8101700 con 0x7f0ad8106e70 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0adcec0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81a11d0 con 0x7f0ad80ff7e0 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60996/0 (socket says 172.21.15.45:60996) 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 -- 172.21.15.45:0/1307892356 learned_addr learned my addr 172.21.15.45:0/1307892356 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad6d9d640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8106e70 0x7f0ad8100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0acffff640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad81008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 -- 172.21.15.45:0/1307892356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80ff7e0 msgr2=0x7f0ad81008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad81008c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 -- 172.21.15.45:0/1307892356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8106e70 msgr2=0x7f0ad8100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8106e70 0x7f0ad8100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 -- 172.21.15.45:0/1307892356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0009900 con 0x7f0ad80fedf0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.041+0000 7f0ad659c640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800d7a0 tx=0x7f0ac800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.042+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8009c30 con 0x7f0ad80fedf0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.042+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ac800deb0 con 0x7f0ad80fedf0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.042+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8004dd0 con 0x7f0ad80fedf0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.042+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad81a1430 con 0x7f0ad80fedf0 2024-03-12T05:10:28.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.042+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad81a1940 con 0x7f0ad80fedf0 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.043+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8105740 con 0x7f0ad80fedf0 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.044+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0ac8004f30 con 0x7f0ad80fedf0 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.044+0000 7f0acf7fe640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 0x7f0ac40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.044+0000 7f0acffff640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 0x7f0ac40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.044+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0ac80ccdd0 con 0x7f0ad80fedf0 2024-03-12T05:10:28.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.045+0000 7f0acffff640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 0x7f0ac40a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8101fa0 tx=0x7f0ac000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.049+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0ac8099b20 con 0x7f0ad80fedf0 2024-03-12T05:10:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.269+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad802ce40 con 0x7f0ad80fedf0 2024-03-12T05:10:28.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.270+0000 7f0acf7fe640 1 -- 172.21.15.45:0/1307892356 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f0ac8017090 con 0x7f0ad80fedf0 2024-03-12T05:10:28.268 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:28.268 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 msgr2=0x7f0ac40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 0x7f0ac40a3f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8101fa0 tx=0x7f0ac000a4c0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 msgr2=0x7f0ad8102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800d7a0 tx=0x7f0ac800dc60 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 shutdown_connections 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0ac40a1ab0 0x7f0ac40a3f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad8106e70 0x7f0ad8100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad81008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 --2- 172.21.15.45:0/1307892356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0ad80fedf0 0x7f0ad8102230 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 >> 172.21.15.45:0/1307892356 conn(0x7f0ad80fa980 msgr2=0x7f0ad8102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 shutdown_connections 2024-03-12T05:10:28.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.272+0000 7f0adcec0640 1 -- 172.21.15.45:0/1307892356 wait complete. 2024-03-12T05:10:28.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:28.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 -- 172.21.15.45:0/2342422363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100310 msgr2=0x7fe338100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 --2- 172.21.15.45:0/2342422363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100310 0x7fe338100790 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe32c009900 tx=0x7fe32c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 -- 172.21.15.45:0/2342422363 shutdown_connections 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 --2- 172.21.15.45:0/2342422363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100310 0x7fe338100790 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 --2- 172.21.15.45:0/2342422363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100cd0 0x7fe338108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.385+0000 7fe33d20b640 1 --2- 172.21.15.45:0/2342422363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe338101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.386+0000 7fe33d20b640 1 -- 172.21.15.45:0/2342422363 >> 172.21.15.45:0/2342422363 conn(0x7fe3380fada0 msgr2=0x7fe3380fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.386+0000 7fe33d20b640 1 -- 172.21.15.45:0/2342422363 shutdown_connections 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.386+0000 7fe33d20b640 1 -- 172.21.15.45:0/2342422363 wait complete. 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.386+0000 7fe33d20b640 1 Processor -- start 2024-03-12T05:10:28.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.386+0000 7fe33d20b640 1 -- start start 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100310 0x7fe3381971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe33819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33819ecc0 con 0x7fe338100cd0 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33819ee30 con 0x7fe338100310 2024-03-12T05:10:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33819eff0 con 0x7fe3381016c0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32786/0 (socket says 172.21.15.45:32786) 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 -- 172.21.15.45:0/1622236759 learned_addr learned my addr 172.21.15.45:0/1622236759 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 -- 172.21.15.45:0/1622236759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 msgr2=0x7fe33819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33759e640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe33819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe336d9d640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100310 0x7fe3381971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe33819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 -- 172.21.15.45:0/1622236759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100310 msgr2=0x7fe3381971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100310 0x7fe3381971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 -- 172.21.15.45:0/1622236759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c009560 con 0x7fe338100cd0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33759e640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe33819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33659c640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe32c000c00 tx=0x7fe32c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c003800 con 0x7fe338100cd0 2024-03-12T05:10:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33819f270 con 0x7fe338100cd0 2024-03-12T05:10:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe32c003e10 con 0x7fe338100cd0 2024-03-12T05:10:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c03e450 con 0x7fe338100cd0 2024-03-12T05:10:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.387+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33819f850 con 0x7fe338100cd0 2024-03-12T05:10:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.388+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe338107470 con 0x7fe338100cd0 2024-03-12T05:10:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.390+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe32c004f50 con 0x7fe338100cd0 2024-03-12T05:10:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.390+0000 7fe31ffff640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 0x7fe3200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.391+0000 7fe336d9d640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 0x7fe3200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.391+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe32c0f00c0 con 0x7fe338100cd0 2024-03-12T05:10:28.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.391+0000 7fe336d9d640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 0x7fe3200a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe3280058c0 tx=0x7fe328005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.394+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe32c0b92e0 con 0x7fe338100cd0 2024-03-12T05:10:28.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:28 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3381642807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:28 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4232944453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:28 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/194973451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:28 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3381642807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:28 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4232944453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:28 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/194973451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.576+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe33810cd10 con 0x7fe338100cd0 2024-03-12T05:10:28.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.577+0000 7fe31ffff640 1 -- 172.21.15.45:0/1622236759 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7fe32c0bc9a0 con 0x7fe338100cd0 2024-03-12T05:10:28.575 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:28.575 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 msgr2=0x7fe3200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 0x7fe3200a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe3280058c0 tx=0x7fe328005c80 comp rx=0 tx=0).stop 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 msgr2=0x7fe338197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe32c000c00 tx=0x7fe32c004360 comp rx=0 tx=0).stop 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 shutdown_connections 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe3200a1a30 0x7fe3200a3ef0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe338100cd0 0x7fe338197710 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3381016c0 0x7fe33819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 --2- 172.21.15.45:0/1622236759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe338100310 0x7fe3381971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 >> 172.21.15.45:0/1622236759 conn(0x7fe3380fada0 msgr2=0x7fe338104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 shutdown_connections 2024-03-12T05:10:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.579+0000 7fe33d20b640 1 -- 172.21.15.45:0/1622236759 wait complete. 2024-03-12T05:10:28.586 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:28 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3381642807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:28 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4232944453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.587 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:28 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/194973451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:28.588 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:28.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 -- 172.21.15.45:0/772637158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102420 msgr2=0x7f36501028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 --2- 172.21.15.45:0/772637158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102420 0x7f36501028a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3644009900 tx=0x7f364402bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 -- 172.21.15.45:0/772637158 shutdown_connections 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 --2- 172.21.15.45:0/772637158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102420 0x7f36501028a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 --2- 172.21.15.45:0/772637158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3650102de0 0x7f3650109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 --2- 172.21.15.45:0/772637158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f36500ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 -- 172.21.15.45:0/772637158 >> 172.21.15.45:0/772637158 conn(0x7f36500fada0 msgr2=0x7f36500fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 -- 172.21.15.45:0/772637158 shutdown_connections 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.696+0000 7f3655898640 1 -- 172.21.15.45:0/772637158 wait complete. 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 Processor -- start 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 -- start start 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f3650197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3650102420 0x7f36501976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365019ec70 con 0x7f3650102de0 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365019ede0 con 0x7f3650102420 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f3655898640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365019efa0 con 0x7f36500ff450 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f364f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f364f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32800/0 (socket says 172.21.15.45:32800) 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.697+0000 7f364f7fe640 1 -- 172.21.15.45:0/4197985811 learned_addr learned my addr 172.21.15.45:0/4197985811 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364effd640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f3650197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 -- 172.21.15.45:0/4197985811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 msgr2=0x7f3650197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f3650197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 -- 172.21.15.45:0/4197985811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3650102420 msgr2=0x7f36501976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3650102420 0x7f36501976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 -- 172.21.15.45:0/4197985811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3644009560 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364effd640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f3650197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f364f7fe640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f364800cce0 tx=0x7f364800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800f450 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f364800fa60 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365019f280 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648018b10 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.698+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365019f810 con 0x7f3650102de0 2024-03-12T05:10:28.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.699+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36501085d0 con 0x7f3650102de0 2024-03-12T05:10:28.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.700+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f364800fbd0 con 0x7f3650102de0 2024-03-12T05:10:28.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.701+0000 7f3654896640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 0x7f36380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:28.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.701+0000 7f364effd640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 0x7f36380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:28.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.701+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f36480cdc70 con 0x7f3650102de0 2024-03-12T05:10:28.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.701+0000 7f364effd640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 0x7f36380a3f70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3640006fd0 tx=0x7f3640008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:28.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.704+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3648096eb0 con 0x7f3650102de0 2024-03-12T05:10:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.887+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f365010cd10 con 0x7f3650102de0 2024-03-12T05:10:28.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.887+0000 7f3654896640 1 -- 172.21.15.45:0/4197985811 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f364809a570 con 0x7f3650102de0 2024-03-12T05:10:28.885 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 msgr2=0x7f36380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 0x7f36380a3f70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3640006fd0 tx=0x7f3640008040 comp rx=0 tx=0).stop 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 msgr2=0x7f365019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f364800cce0 tx=0x7f364800a620 comp rx=0 tx=0).stop 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 shutdown_connections 2024-03-12T05:10:28.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f36380a1ab0 0x7f36380a3f70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3650102de0 0x7f365019e730 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3650102420 0x7f36501976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.889+0000 7f3655898640 1 --2- 172.21.15.45:0/4197985811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36500ff450 0x7f3650197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.890+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 >> 172.21.15.45:0/4197985811 conn(0x7f36500fada0 msgr2=0x7f36501057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.890+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 shutdown_connections 2024-03-12T05:10:28.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:28.890+0000 7f3655898640 1 -- 172.21.15.45:0/4197985811 wait complete. 2024-03-12T05:10:28.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 -- 172.21.15.45:0/4031866915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c1000a0 msgr2=0x7f0c6c1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 --2- 172.21.15.45:0/4031866915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c1004c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009900 tx=0x7f0c6402bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 -- 172.21.15.45:0/4031866915 shutdown_connections 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 --2- 172.21.15.45:0/4031866915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c101c50 0x7f0c6c1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 --2- 172.21.15.45:0/4031866915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 --2- 172.21.15.45:0/4031866915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c1004c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 -- 172.21.15.45:0/4031866915 >> 172.21.15.45:0/4031866915 conn(0x7f0c6c0fb630 msgr2=0x7f0c6c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 -- 172.21.15.45:0/4031866915 shutdown_connections 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.006+0000 7f0c715c7640 1 -- 172.21.15.45:0/4031866915 wait complete. 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 Processor -- start 2024-03-12T05:10:29.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 -- start start 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c19ecf0 con 0x7f0c6c101c50 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c19ee60 con 0x7f0c6c1000a0 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c715c7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c19f020 con 0x7f0c6c101290 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32814/0 (socket says 172.21.15.45:32814) 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 -- 172.21.15.45:0/95372116 learned_addr learned my addr 172.21.15.45:0/95372116 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6affd640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6a7fc640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 -- 172.21.15.45:0/95372116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 msgr2=0x7f0c6c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 -- 172.21.15.45:0/95372116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c1000a0 msgr2=0x7f0c6c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 -- 172.21.15.45:0/95372116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64009560 con 0x7f0c6c101c50 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6a7fc640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.007+0000 7f0c6b7fe640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00b970 tx=0x7f0c5c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.008+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00cb40 con 0x7f0c6c101c50 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.008+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c19f300 con 0x7f0c6c101c50 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.008+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c5c00cca0 con 0x7f0c6c101c50 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.008+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0076e0 con 0x7f0c6c101c50 2024-03-12T05:10:29.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.008+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c19f890 con 0x7f0c6c101c50 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.009+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c107440 con 0x7f0c6c101c50 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.010+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0c5c014440 con 0x7f0c6c101c50 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.011+0000 7f0c63fff640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 0x7f0c580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.011+0000 7f0c6affd640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 0x7f0c580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.011+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0c5c0ccde0 con 0x7f0c6c101c50 2024-03-12T05:10:29.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.011+0000 7f0c6affd640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 0x7f0c580a3ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c240 tx=0x7f0c64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.015+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0c5c095fa0 con 0x7f0c6c101c50 2024-03-12T05:10:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.196+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c6c02ce40 con 0x7f0c6c101c50 2024-03-12T05:10:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.198+0000 7f0c63fff640 1 -- 172.21.15.45:0/95372116 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c5c019020 con 0x7f0c6c101c50 2024-03-12T05:10:29.196 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 msgr2=0x7f0c580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 0x7f0c580a3ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c240 tx=0x7f0c64005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 msgr2=0x7f0c6c19e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00b970 tx=0x7f0c5c00be30 comp rx=0 tx=0).stop 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 shutdown_connections 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0c580a1a30 0x7f0c580a3ef0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c6c101c50 0x7f0c6c19e7b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c6c101290 0x7f0c6c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 --2- 172.21.15.45:0/95372116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c6c1000a0 0x7f0c6c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.200+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 >> 172.21.15.45:0/95372116 conn(0x7f0c6c0fb630 msgr2=0x7f0c6c104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.201+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 shutdown_connections 2024-03-12T05:10:29.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.201+0000 7f0c715c7640 1 -- 172.21.15.45:0/95372116 wait complete. 2024-03-12T05:10:29.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 -- 172.21.15.45:0/1817739423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 msgr2=0x7fa1e01084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1817739423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e01084c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009ca0 tx=0x7fa1d802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 -- 172.21.15.45:0/1817739423 shutdown_connections 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1817739423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e01084c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1817739423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1e0101270 0x7fa1e01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1817739423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 -- 172.21.15.45:0/1817739423 >> 172.21.15.45:0/1817739423 conn(0x7fa1e00fb610 msgr2=0x7fa1e00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 -- 172.21.15.45:0/1817739423 shutdown_connections 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.317+0000 7fa1e6548640 1 -- 172.21.15.45:0/1817739423 wait complete. 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 Processor -- start 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 -- start start 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1e0101270 0x7fa1e01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e019dba0 con 0x7fa1e0101c30 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e019dd10 con 0x7fa1e0100080 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e6548640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e019ded0 con 0x7fa1e0101270 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e4ae5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e4ae5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32844/0 (socket says 172.21.15.45:32844) 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1e4ae5640 1 -- 172.21.15.45:0/1871141482 learned_addr learned my addr 172.21.15.45:0/1871141482 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.318+0000 7fa1dffff640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 -- 172.21.15.45:0/1871141482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1e0101270 msgr2=0x7fa1e01965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1e0101270 0x7fa1e01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 -- 172.21.15.45:0/1871141482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 msgr2=0x7fa1e01960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 -- 172.21.15.45:0/1871141482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8009900 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1dffff640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e4ae5640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8037040 tx=0x7fa1d8032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8034c70 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1d8003e20 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e00682e0 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8004720 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.319+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e0068810 con 0x7fa1e0101c30 2024-03-12T05:10:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.320+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e0067f70 con 0x7fa1e0101c30 2024-03-12T05:10:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.321+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa1d803b070 con 0x7fa1e0101c30 2024-03-12T05:10:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.322+0000 7fa1dd7fa640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 0x7fa1cc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.322+0000 7fa1dffff640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 0x7fa1cc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.322+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fa1d80f1ce0 con 0x7fa1e0101c30 2024-03-12T05:10:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.322+0000 7fa1dffff640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 0x7fa1cc0a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1e01010d0 tx=0x7fa1d0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.326+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1d80baea0 con 0x7fa1e0101c30 2024-03-12T05:10:29.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1307892356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1622236759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4197985811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:29.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:29 smithi130 ceph-mon[35398]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:29.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1307892356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1622236759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4197985811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:29.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:29 smithi116 ceph-mon[35486]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.507+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1e0106ed0 con 0x7fa1e0101c30 2024-03-12T05:10:29.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.508+0000 7fa1dd7fa640 1 -- 172.21.15.45:0/1871141482 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1d80390a0 con 0x7fa1e0101c30 2024-03-12T05:10:29.506 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:29.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 msgr2=0x7fa1cc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 0x7fa1cc0a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1e01010d0 tx=0x7fa1d0008040 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 msgr2=0x7fa1e019d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8037040 tx=0x7fa1d8032980 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 shutdown_connections 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fa1cc0a1a30 0x7fa1cc0a3ef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1e0101c30 0x7fa1e019d660 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1e0101270 0x7fa1e01965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 --2- 172.21.15.45:0/1871141482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1e0100080 0x7fa1e01960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 >> 172.21.15.45:0/1871141482 conn(0x7fa1e00fb610 msgr2=0x7fa1e0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 shutdown_connections 2024-03-12T05:10:29.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.510+0000 7fa1e6548640 1 -- 172.21.15.45:0/1871141482 wait complete. 2024-03-12T05:10:29.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1307892356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1622236759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:29.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4197985811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:29.517 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:29 smithi045 ceph-mon[32174]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-12T05:10:29.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:29.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 -- 172.21.15.45:0/2951782016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8101cd0 msgr2=0x7facd8108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 --2- 172.21.15.45:0/2951782016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8101cd0 0x7facd8108560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7facd0009ca0 tx=0x7facd002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 -- 172.21.15.45:0/2951782016 shutdown_connections 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 --2- 172.21.15.45:0/2951782016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8101cd0 0x7facd8108560 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 --2- 172.21.15.45:0/2951782016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd8101310 0x7facd8101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 --2- 172.21.15.45:0/2951782016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd8100120 0x7facd8100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.623+0000 7facde787640 1 -- 172.21.15.45:0/2951782016 >> 172.21.15.45:0/2951782016 conn(0x7facd80fb6b0 msgr2=0x7facd80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- 172.21.15.45:0/2951782016 shutdown_connections 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- 172.21.15.45:0/2951782016 wait complete. 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 Processor -- start 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- start start 2024-03-12T05:10:29.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd8101310 0x7facd8197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd8101cd0 0x7facd819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd819ec00 con 0x7facd8100120 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd819ed70 con 0x7facd8101cd0 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facde787640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd819ef30 con 0x7facd8101310 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32860/0 (socket says 172.21.15.45:32860) 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.624+0000 7facd7fff640 1 -- 172.21.15.45:0/1584787276 learned_addr learned my addr 172.21.15.45:0/1584787276 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 -- 172.21.15.45:0/1584787276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd8101310 msgr2=0x7facd8197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd8101310 0x7facd8197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 -- 172.21.15.45:0/1584787276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd8101cd0 msgr2=0x7facd819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd8101cd0 0x7facd819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 -- 172.21.15.45:0/1584787276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0009900 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd7fff640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7facc800c970 tx=0x7facc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8007c10 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facc8007d70 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd819f210 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc800f7d0 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.625+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd819f7f0 con 0x7facd8100120 2024-03-12T05:10:29.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.626+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd810b9b0 con 0x7facd8100120 2024-03-12T05:10:29.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.627+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7facc8018480 con 0x7facd8100120 2024-03-12T05:10:29.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.628+0000 7facd57fa640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 0x7facc40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.628+0000 7facd77fe640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 0x7facc40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.628+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7facc8013070 con 0x7facd8100120 2024-03-12T05:10:29.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.628+0000 7facd77fe640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 0x7facc40a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facd8198620 tx=0x7facc0006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.632+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facc8096a00 con 0x7facd8100120 2024-03-12T05:10:29.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.855+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd810bb60 con 0x7facd8100120 2024-03-12T05:10:29.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.855+0000 7facd57fa640 1 -- 172.21.15.45:0/1584787276 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11397 (secure 0 0 0) 0x7facc809a0c0 con 0x7facd8100120 2024-03-12T05:10:29.853 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:29.853 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:08:43.349662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:29.853 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 24 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 msgr2=0x7facc40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 0x7facc40a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facd8198620 tx=0x7facc0006c90 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 msgr2=0x7facd8197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7facc800c970 tx=0x7facc800ce30 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 shutdown_connections 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7facc40a1ab0 0x7facc40a3f70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd8101cd0 0x7facd819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd8101310 0x7facd8197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 --2- 172.21.15.45:0/1584787276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd8100120 0x7facd8197110 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 >> 172.21.15.45:0/1584787276 conn(0x7facd80fb6b0 msgr2=0x7facd80fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 shutdown_connections 2024-03-12T05:10:29.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.857+0000 7facde787640 1 -- 172.21.15.45:0/1584787276 wait complete. 2024-03-12T05:10:29.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:29.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- 172.21.15.45:0/2807834830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d81013e0 msgr2=0x7f42d8107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- 172.21.15.45:0/2807834830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d81013e0 0x7f42d8107c70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f42d000adb0 tx=0x7f42d002cff0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- 172.21.15.45:0/2807834830 shutdown_connections 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- 172.21.15.45:0/2807834830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d81013e0 0x7f42d8107c70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- 172.21.15.45:0/2807834830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d8100a20 0x7f42d8100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- 172.21.15.45:0/2807834830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d80ff830 0x7f42d80ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- 172.21.15.45:0/2807834830 >> 172.21.15.45:0/2807834830 conn(0x7f42d80fada0 msgr2=0x7f42d80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- 172.21.15.45:0/2807834830 shutdown_connections 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- 172.21.15.45:0/2807834830 wait complete. 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 Processor -- start 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 -- start start 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d8100a20 0x7f42d81976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.971+0000 7f42de80e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d81013e0 0x7f42d819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42de80e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819ec50 con 0x7f42d80ff830 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42de80e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819edc0 con 0x7f42d8100a20 2024-03-12T05:10:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42de80e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819ef80 con 0x7f42d81013e0 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32882/0 (socket says 172.21.15.45:32882) 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 -- 172.21.15.45:0/2273125434 learned_addr learned my addr 172.21.15.45:0/2273125434 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42dcdab640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d81013e0 0x7f42d819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 -- 172.21.15.45:0/2273125434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d81013e0 msgr2=0x7f42d819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d77fe640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d8100a20 0x7f42d81976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d81013e0 0x7f42d819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 -- 172.21.15.45:0/2273125434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d8100a20 msgr2=0x7f42d81976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d8100a20 0x7f42d81976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 -- 172.21.15.45:0/2273125434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d000aa10 con 0x7f42d80ff830 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d7fff640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c970 tx=0x7f42c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8004220 con 0x7f42d80ff830 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f42c8007e60 con 0x7f42d80ff830 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d819f260 con 0x7f42d80ff830 2024-03-12T05:10:29.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8003e40 con 0x7f42d80ff830 2024-03-12T05:10:29.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.972+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d819f770 con 0x7f42d80ff830 2024-03-12T05:10:29.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.973+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d810a870 con 0x7f42d80ff830 2024-03-12T05:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.974+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f42c8003610 con 0x7f42d80ff830 2024-03-12T05:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.975+0000 7f42d57fa640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 0x7f42c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.975+0000 7f42d77fe640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 0x7f42c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.975+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f42c80cd1a0 con 0x7f42d80ff830 2024-03-12T05:10:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.975+0000 7f42d77fe640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 0x7f42c40a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42d8198670 tx=0x7f42c0005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:29.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:29.979+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f42c80963c0 con 0x7f42d80ff830 2024-03-12T05:10:30.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.161+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42d819fc40 con 0x7f42d80ff830 2024-03-12T05:10:30.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.162+0000 7f42d57fa640 1 -- 172.21.15.45:0/2273125434 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f42c8099a80 con 0x7f42d80ff830 2024-03-12T05:10:30.160 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:30.160 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 msgr2=0x7f42c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 0x7f42c40a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42d8198670 tx=0x7f42c0005c60 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 msgr2=0x7f42d8197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c970 tx=0x7f42c800ce30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 shutdown_connections 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42c40a1a30 0x7f42c40a3ef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42d81013e0 0x7f42d819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42d8100a20 0x7f42d81976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 --2- 172.21.15.45:0/2273125434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42d80ff830 0x7f42d8197160 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 >> 172.21.15.45:0/2273125434 conn(0x7f42d80fada0 msgr2=0x7f42d8105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 shutdown_connections 2024-03-12T05:10:30.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.164+0000 7f42de80e640 1 -- 172.21.15.45:0/2273125434 wait complete. 2024-03-12T05:10:30.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-12T05:08:37.112985+0000', 'modified': '2024-03-12T05:08:43.349662+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24472, 'mds_3': 24467, 'mds_4': 24460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14619': {'gid': 14619, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.45:6839/1651909238', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6838', 'nonce': 1651909238}, {'type': 'v1', 'addr': '172.21.15.45:6839', 'nonce': 1651909238}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.45:6841/3025141330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6840', 'nonce': 3025141330}, {'type': 'v1', 'addr': '172.21.15.45:6841', 'nonce': 3025141330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/2049389812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 2049389812}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 2049389812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24421': {'gid': 24421, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/1602847227', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 1602847227}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 1602847227}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/4238162743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 4238162743}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 4238162743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.130:6835/1592150426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 1592150426}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 1592150426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6837/2581569461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2581569461}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2581569461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.130:6837/2852741522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2852741522}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2852741522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.130:6839/2939102833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2939102833}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2939102833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6839/397791649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 397791649}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 397791649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-12T05:10:30.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-12T05:10:30.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 -- 172.21.15.45:0/552938126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc101270 msgr2=0x7fd5dc1016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 --2- 172.21.15.45:0/552938126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc101270 0x7fd5dc1016f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009900 tx=0x7fd5d002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 -- 172.21.15.45:0/552938126 shutdown_connections 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 --2- 172.21.15.45:0/552938126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc101270 0x7fd5dc1016f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 --2- 172.21.15.45:0/552938126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5dc101c30 0x7fd5dc1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 --2- 172.21.15.45:0/552938126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc100080 0x7fd5dc1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 -- 172.21.15.45:0/552938126 >> 172.21.15.45:0/552938126 conn(0x7fd5dc0fb610 msgr2=0x7fd5dc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 -- 172.21.15.45:0/552938126 shutdown_connections 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.280+0000 7fd5e1920640 1 -- 172.21.15.45:0/552938126 wait complete. 2024-03-12T05:10:30.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 Processor -- start 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 -- start start 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc101270 0x7fd5dc197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5dc101c30 0x7fd5dc19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc19ece0 con 0x7fd5dc100080 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc19ee50 con 0x7fd5dc101270 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5e1920640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc19f010 con 0x7fd5dc101c30 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32906/0 (socket says 172.21.15.45:32906) 2024-03-12T05:10:30.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 -- 172.21.15.45:0/115914301 learned_addr learned my addr 172.21.15.45:0/115914301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc101270 0x7fd5dc197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 -- 172.21.15.45:0/115914301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5dc101c30 msgr2=0x7fd5dc19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5dc101c30 0x7fd5dc19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 -- 172.21.15.45:0/115914301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc101270 msgr2=0x7fd5dc197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc101270 0x7fd5dc197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.281+0000 7fd5daffd640 1 -- 172.21.15.45:0/115914301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0009560 con 0x7fd5dc100080 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5daffd640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400d770 tx=0x7fd5d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5e091e640 1 -- 172.21.15.45:0/115914301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4009c30 con 0x7fd5dc100080 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5e091e640 1 -- 172.21.15.45:0/115914301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5d40044d0 con 0x7fd5dc100080 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc19f2f0 con 0x7fd5dc100080 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5e091e640 1 -- 172.21.15.45:0/115914301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4003de0 con 0x7fd5dc100080 2024-03-12T05:10:30.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.282+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc19f880 con 0x7fd5dc100080 2024-03-12T05:10:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.284+0000 7fd5e091e640 1 -- 172.21.15.45:0/115914301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd5d4003f40 con 0x7fd5dc100080 2024-03-12T05:10:30.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 Processor -- start 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 -- start start 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 0x564112efa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 0x564112eff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112eb5cd0 con 0x564112effae0 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112e68f20 con 0x564112efa970 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112e66450 con 0x564112ef9f80 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32908/0 (socket says 172.21.15.45:32908) 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1802376752 learned_addr learned my addr 172.21.15.45:0/1802376752 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e091e640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 0x7fd5c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 0x564112eff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1802376752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 msgr2=0x564112efa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 0x564112efa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 0x564112efa3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1802376752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 msgr2=0x564112eff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 0x564112eff470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1802376752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564112e16f20 con 0x564112effae0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 0x564112efa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 0x564112eff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 0x7fd5c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.285+0000 7fd5e091e640 1 -- 172.21.15.45:0/115914301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5d40cce70 con 0x7fd5dc100080 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0036040 tx=0x7fd5d0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9ddc48015de93d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5c1ffb640 1 -- 172.21.15.45:0/1802376752 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00046c0 con 0x564112effae0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5c1ffb640 1 -- 172.21.15.45:0/1802376752 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5d00044e0 con 0x564112effae0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5c1ffb640 1 -- 172.21.15.45:0/1802376752 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0003ce0 con 0x564112effae0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1802376752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 msgr2=0x564112f01f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 0x7fd5c40a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x564112efb1d0 tx=0x7fd5cc004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0036040 tx=0x7fd5d0005e80 comp rx=0 tx=0).stop 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1802376752 shutdown_connections 2024-03-12T05:10:30.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f01f70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112efa970 0x564112eff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1802376752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112ef9f80 0x564112efa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1802376752 >> 172.21.15.45:0/1802376752 conn(0x564112ef54f0 msgr2=0x564112ef58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.286+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1802376752 shutdown_connections 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1802376752 wait complete. 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 -- start start 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 Processor -- start 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112effae0 0x564112f0b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112e61c30 con 0x564112ef9f80 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112e06d30 con 0x564112effae0 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112b35310 con 0x564112efa970 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5e45cc740 10 client.0 _ll_register_callbacks cb 0x564112e0ed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32910/0 (socket says 172.21.15.45:32910) 2024-03-12T05:10:30.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5daffd640 1 -- 172.21.15.45:0/1960926187 learned_addr learned my addr 172.21.15.45:0/1960926187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.287+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112effae0 0x564112f0b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 -- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 msgr2=0x564112f045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 -- 172.21.15.45:0/1960926187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112effae0 msgr2=0x564112f0b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112effae0 0x564112f0b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 -- 172.21.15.45:0/1960926187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc0682e0 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40961f0 tx=0x7fd5d40997b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d97fdff94eb3513 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4099d90 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5d409dee0 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564112b616c0 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5d40a2530 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 10 client.15246 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564112e602b0 con 0x564112ef9f80 2024-03-12T05:10:30.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x564112b616c0 con 0x564112ef9f80 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5d40a6d20 con 0x564112ef9f80 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 10 client.15246 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 10 client.15246 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 20 client.15246 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f10a70 0x564112f0fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 4 client.15246 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.288+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564112bb2270 con 0x564112f10a70 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.289+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f10a70 0x564112f0fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.289+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f10a70 0x564112f0fcb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=aafd5b2e12e7a1e1 server_cookie=b78edc462a06895b in_seq=0 out_seq=0 2024-03-12T05:10:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.289+0000 7fd5baffd640 10 client.15246 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5baffd640 1 -- 172.21.15.45:0/1960926187 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564112bb2270 con 0x564112f10a70 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5baffd640 10 client.15246 handle_command_reply: tid=0 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 client.15246 shutdown 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 msgr2=0x564112f040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd5d40961f0 tx=0x7fd5d40997b0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 shutdown_connections 2024-03-12T05:10:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f10a70 0x564112f0fcb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112effae0 0x564112f0b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1960926187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112ef9f80 0x564112f040a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5baffd640 0 client.15246 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 >> 172.21.15.45:0/1960926187 conn(0x564112ef54f0 msgr2=0x564112ef47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 shutdown_connections 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1960926187 wait complete. 2024-03-12T05:10:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.290+0000 7fd5e45cc740 20 client.15246 trim_cache size 0 max 16384 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 Processor -- start 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 -- start start 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f18f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112effae0 0x564112efbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112de7490 con 0x564112efa970 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112ecb8c0 con 0x564112ef9f80 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112e1af40 con 0x564112effae0 2024-03-12T05:10:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32916/0 (socket says 172.21.15.45:32916) 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1755787640 learned_addr learned my addr 172.21.15.45:0/1755787640 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5daffd640 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f18f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112effae0 0x564112efbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1755787640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112effae0 msgr2=0x564112efbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112effae0 0x564112efbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1755787640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 msgr2=0x564112f18f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f18f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.306+0000 7fd5da7fc640 1 -- 172.21.15.45:0/1755787640 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564112dfca20 con 0x564112efa970 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd5d002c7d0 tx=0x7fd5d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163467253bee59c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5c17fa640 1 -- 172.21.15.45:0/1755787640 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d004ede0 con 0x564112efa970 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5c17fa640 1 -- 172.21.15.45:0/1755787640 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5d00035f0 con 0x564112efa970 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5c17fa640 1 -- 172.21.15.45:0/1755787640 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0004dc0 con 0x564112efa970 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1755787640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 msgr2=0x564112f0b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd5d002c7d0 tx=0x7fd5d002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1755787640 shutdown_connections 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112efa970 0x564112f0b5e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112effae0 0x564112efbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/1755787640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f18f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1755787640 >> 172.21.15.45:0/1755787640 conn(0x564112ef54f0 msgr2=0x564112ef58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1755787640 shutdown_connections 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.307+0000 7fd5e45cc740 1 -- 172.21.15.45:0/1755787640 wait complete. 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 -- start start 2024-03-12T05:10:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 Processor -- start 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f0b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f07e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112f0b9d0 con 0x564112effae0 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112f08880 con 0x564112ef9f80 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564112f08a10 con 0x564112efa970 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5e45cc740 10 client.0 _ll_register_callbacks cb 0x7fd5d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32928/0 (socket says 172.21.15.45:32928) 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 -- 172.21.15.45:0/2962878536 learned_addr learned my addr 172.21.15.45:0/2962878536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5daffd640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f0b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f07e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 -- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 msgr2=0x564112f07e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f07e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 -- 172.21.15.45:0/2962878536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 msgr2=0x564112f0b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f0b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.308+0000 7fd5db7fe640 1 -- 172.21.15.45:0/2962878536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564112e16f20 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5da7fc640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f07e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5db7fe640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0067b0 tx=0x7fd5cc006c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814dc236aade576e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc01e410 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fd5cc015040 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5d00044e0 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5cc01e570 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 10 client.15252 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564112e22780 con 0x564112effae0 2024-03-12T05:10:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5d00044e0 con 0x564112effae0 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5cc01b040 con 0x564112effae0 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 10 client.15252 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 10 client.15252 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 20 client.15252 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f6b860 0x564112f67ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 4 client.15252 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.309+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564112b9e610 con 0x564112f6b860 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5daffd640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f6b860 0x564112f67ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5daffd640 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f6b860 0x564112f67ca0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=593b775cb9467a03 server_cookie=480ed47aeb05ad11 in_seq=0 out_seq=0 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5baffd640 10 client.15252 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5baffd640 1 -- 172.21.15.45:0/2962878536 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564112b9e610 con 0x564112f6b860 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5baffd640 10 client.15252 handle_command_reply: tid=0 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 client.15252 shutdown 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 msgr2=0x564112f08340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0067b0 tx=0x7fd5cc006c70 comp rx=0 tx=0).stop 2024-03-12T05:10:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 shutdown_connections 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x564112f6b860 0x564112f67ca0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564112effae0 0x564112f08340 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564112efa970 0x564112f07e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5baffd640 0 client.15252 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 --2- 172.21.15.45:0/2962878536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564112ef9f80 0x564112f0b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 >> 172.21.15.45:0/2962878536 conn(0x564112ef54f0 msgr2=0x564112ef58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 shutdown_connections 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.310+0000 7fd5e45cc740 1 -- 172.21.15.45:0/2962878536 wait complete. 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e45cc740 20 client.15252 trim_cache size 0 max 16384 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 4, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24460, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 107.96485196899999, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 134.66509540199999 2024-03-12T05:10:30.315 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 msgr2=0x7fd5c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 0x7fd5c40a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x564112efb1d0 tx=0x7fd5cc004b50 comp rx=0 tx=0).stop 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 msgr2=0x7fd5dc1971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400d770 tx=0x7fd5d400dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 shutdown_connections 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fd5c40a1a30 0x7fd5c40a3ef0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5dc101c30 0x7fd5dc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5dc101270 0x7fd5dc197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 --2- 172.21.15.45:0/115914301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5dc100080 0x7fd5dc1971f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 >> 172.21.15.45:0/115914301 conn(0x7fd5dc0fb610 msgr2=0x7fd5dc104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 shutdown_connections 2024-03-12T05:10:30.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.311+0000 7fd5e1920640 1 -- 172.21.15.45:0/115914301 wait complete. 2024-03-12T05:10:30.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- 172.21.15.45:0/2235085087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d81000a0 msgr2=0x7f67d81004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- 172.21.15.45:0/2235085087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d81000a0 0x7f67d81004c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009900 tx=0x7f67d402bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- 172.21.15.45:0/2235085087 shutdown_connections 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- 172.21.15.45:0/2235085087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d8101c50 0x7f67d81084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- 172.21.15.45:0/2235085087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8101290 0x7f67d8101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- 172.21.15.45:0/2235085087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d81000a0 0x7f67d81004c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- 172.21.15.45:0/2235085087 >> 172.21.15.45:0/2235085087 conn(0x7f67d80fb630 msgr2=0x7f67d80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- 172.21.15.45:0/2235085087 shutdown_connections 2024-03-12T05:10:30.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- 172.21.15.45:0/2235085087 wait complete. 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 Processor -- start 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 -- start start 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 0x7f67d8103fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.425+0000 7f67df5bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d8101290 0x7f67d8102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8103340 con 0x7f67d8101c50 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d81034b0 con 0x7f67d8101290 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d81a12b0 con 0x7f67d81000a0 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32952/0 (socket says 172.21.15.45:32952) 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 -- 172.21.15.45:0/3736397466 learned_addr learned my addr 172.21.15.45:0/3736397466 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67dd358640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 0x7f67d8103fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 -- 172.21.15.45:0/3736397466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 msgr2=0x7f67d8103fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 0x7f67d8103fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 -- 172.21.15.45:0/3736397466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d8101290 msgr2=0x7f67d8102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d8101290 0x7f67d8102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 -- 172.21.15.45:0/3736397466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4009560 con 0x7f67d8101c50 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67dd358640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 0x7f67d8103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ddb59640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f67d000cb00 tx=0x7f67d000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ce7fc640 1 -- 172.21.15.45:0/3736397466 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000ae90 con 0x7f67d8101c50 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ce7fc640 1 -- 172.21.15.45:0/3736397466 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67d0005020 con 0x7f67d8101c50 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d81a1510 con 0x7f67d8101c50 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67ce7fc640 1 -- 172.21.15.45:0/3736397466 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000f690 con 0x7f67d8101c50 2024-03-12T05:10:30.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.426+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d81a1a20 con 0x7f67d8101c50 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.428+0000 7f67ce7fc640 1 -- 172.21.15.45:0/3736397466 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f67d0005190 con 0x7f67d8101c50 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 Processor -- start 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 -- start start 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948956c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956c970 0x559489571470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559489527cd0 con 0x559489571ae0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594894daf20 con 0x55948956bf80 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67e218f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594894d8450 con 0x55948956c970 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67ce7fc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 0x7f67c40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32962/0 (socket says 172.21.15.45:32962) 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 -- 172.21.15.45:0/1548697642 learned_addr learned my addr 172.21.15.45:0/1548697642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67ddb59640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 0x7f67c40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948956c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dd358640 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956c970 0x559489571470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67ce7fc640 1 -- 172.21.15.45:0/3736397466 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67d00cd4a0 con 0x7f67d8101c50 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 -- 172.21.15.45:0/1548697642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956c970 msgr2=0x559489571470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956c970 0x559489571470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 -- 172.21.15.45:0/1548697642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 msgr2=0x55948956c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948956c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.429+0000 7f67dcb57640 1 -- 172.21.15.45:0/1548697642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559489488f20 con 0x559489571ae0 2024-03-12T05:10:30.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67dcb57640 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f67c800bfd0 tx=0x7f67c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca0d46993331771 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67ddb59640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 0x7f67c40a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67d0096880 tx=0x7f67d006d720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67c37fe640 1 -- 172.21.15.45:0/1548697642 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c80142f0 con 0x559489571ae0 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67c37fe640 1 -- 172.21.15.45:0/1548697642 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67c8014e20 con 0x559489571ae0 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67c37fe640 1 -- 172.21.15.45:0/1548697642 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c80127c0 con 0x559489571ae0 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 -- 172.21.15.45:0/1548697642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 msgr2=0x559489573f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f67c800bfd0 tx=0x7f67c800b330 comp rx=0 tx=0).stop 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 -- 172.21.15.45:0/1548697642 shutdown_connections 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489573f70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956c970 0x559489571470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 --2- 172.21.15.45:0/1548697642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948956c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 -- 172.21.15.45:0/1548697642 >> 172.21.15.45:0/1548697642 conn(0x5594895674f0 msgr2=0x5594895678f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 -- 172.21.15.45:0/1548697642 shutdown_connections 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.430+0000 7f67e218f740 1 -- 172.21.15.45:0/1548697642 wait complete. 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 -- start start 2024-03-12T05:10:30.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 Processor -- start 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x559489576610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594894d3c30 con 0x55948956bf80 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559489478d30 con 0x55948956c970 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594891a7310 con 0x559489571ae0 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67e218f740 10 client.0 _ll_register_callbacks cb 0x559489480d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32970/0 (socket says 172.21.15.45:32970) 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 -- 172.21.15.45:0/2598551862 learned_addr learned my addr 172.21.15.45:0/2598551862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dd358640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x559489576610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 -- 172.21.15.45:0/2598551862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 msgr2=0x55948957d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 -- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 msgr2=0x559489576610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.431+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x559489576610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67dcb57640 1 -- 172.21.15.45:0/2598551862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c80144f0 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67dd358640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x559489576610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f67c8014320 tx=0x7f67c8014350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1757bd04a53ecee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8011d50 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67c8000e40 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594891d36c0 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67c8012540 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 10 client.15264 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5594894d22b0 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5594891d36c0 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67c8030070 con 0x55948956bf80 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 10 client.15264 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 10 client.15264 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 20 client.15264 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559489581b90 0x559489586c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 4 client.15264 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.432+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559489224270 con 0x559489581b90 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.433+0000 7f67dd358640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559489581b90 0x559489586c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.433+0000 7f67dd358640 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559489581b90 0x559489586c70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6250d45e480b2a43 server_cookie=caf746c034b2142e in_seq=0 out_seq=0 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.433+0000 7f67c0ff9640 10 client.15264 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2598551862 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559489224270 con 0x559489581b90 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67c0ff9640 10 client.15264 handle_command_reply: tid=0 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 client.15264 shutdown 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 msgr2=0x5594895760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f67c8014320 tx=0x7f67c8014350 comp rx=0 tx=0).stop 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 shutdown_connections 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559489581b90 0x559489586c70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67c0ff9640 0 client.15264 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x559489576610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 --2- 172.21.15.45:0/2598551862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956bf80 0x5594895760d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 >> 172.21.15.45:0/2598551862 conn(0x5594895674f0 msgr2=0x5594895667d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 shutdown_connections 2024-03-12T05:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 1 -- 172.21.15.45:0/2598551862 wait complete. 2024-03-12T05:10:30.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.434+0000 7f67e218f740 20 client.15264 trim_cache size 0 max 16384 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 Processor -- start 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 -- start start 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956bf80 0x5594895678d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x55948957d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559489459490 con 0x559489571ae0 2024-03-12T05:10:30.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559489224270 con 0x55948956c970 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67e218f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594891d36c0 con 0x55948956bf80 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32980/0 (socket says 172.21.15.45:32980) 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 -- 172.21.15.45:0/796631187 learned_addr learned my addr 172.21.15.45:0/796631187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 -- 172.21.15.45:0/796631187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956bf80 msgr2=0x5594895678d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956bf80 0x5594895678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 -- 172.21.15.45:0/796631187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 msgr2=0x55948957d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x55948957d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 -- 172.21.15.45:0/796631187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55948946ea20 con 0x559489571ae0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67dcb57640 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b370 tx=0x7f67c800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2b28518e1d675e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67c2ffd640 1 -- 172.21.15.45:0/796631187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8028ea0 con 0x559489571ae0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67c2ffd640 1 -- 172.21.15.45:0/796631187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67c802d070 con 0x559489571ae0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.450+0000 7f67c2ffd640 1 -- 172.21.15.45:0/796631187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8004070 con 0x559489571ae0 2024-03-12T05:10:30.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- 172.21.15.45:0/796631187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 msgr2=0x559489575ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b370 tx=0x7f67c800b460 comp rx=0 tx=0).stop 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- 172.21.15.45:0/796631187 shutdown_connections 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559489571ae0 0x559489575ce0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956c970 0x55948957d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 --2- 172.21.15.45:0/796631187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948956bf80 0x5594895678d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- 172.21.15.45:0/796631187 >> 172.21.15.45:0/796631187 conn(0x5594895674f0 msgr2=0x559489562660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- 172.21.15.45:0/796631187 shutdown_connections 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- 172.21.15.45:0/796631187 wait complete. 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 -- start start 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.451+0000 7f67e218f740 1 Processor -- start 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948957d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948957a930 con 0x55948956c970 2024-03-12T05:10:30.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948957d880 con 0x55948956bf80 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55948957da40 con 0x559489571ae0 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 10 client.0 _ll_register_callbacks cb 0x7f67c800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32992/0 (socket says 172.21.15.45:32992) 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 -- 172.21.15.45:0/2459453441 learned_addr learned my addr 172.21.15.45:0/2459453441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948957d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 -- 172.21.15.45:0/2459453441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 msgr2=0x55948957a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 -- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 msgr2=0x55948957d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948957d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 -- 172.21.15.45:0/2459453441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594894d22b0 con 0x55948956c970 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67dd358640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000c00 tx=0x7f67d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad883540dcb5610 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4047de0 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f67d4044040 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.452+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559489582b60 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67d4004030 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 10 client.15270 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559489494780 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559489582b60 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67d404f020 con 0x55948956c970 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 10 client.15270 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 10 client.15270 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 20 client.15270 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5594895ddb10 0x5594895d9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 4 client.15270 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f67c8014e20 con 0x5594895ddb10 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.453+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5594895ddb10 0x5594895d9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67c0ff9640 10 client.15270 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67dcb57640 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5594895ddb10 0x5594895d9f90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=956adfef617eae8a server_cookie=ff570da5db29bc09 in_seq=0 out_seq=0 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67c0ff9640 1 -- 172.21.15.45:0/2459453441 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f67c8014e20 con 0x5594895ddb10 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67c0ff9640 10 client.15270 handle_command_reply: tid=0 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 client.15270 shutdown 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 msgr2=0x559489579cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000c00 tx=0x7f67d4004750 comp rx=0 tx=0).stop 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 shutdown_connections 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5594895ddb10 0x5594895d9f90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55948956c970 0x559489579cf0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67c0ff9640 0 client.15270 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:10:30.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559489571ae0 0x55948957a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 --2- 172.21.15.45:0/2459453441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948956bf80 0x55948957d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 >> 172.21.15.45:0/2459453441 conn(0x5594895674f0 msgr2=0x5594895678d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 shutdown_connections 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 1 -- 172.21.15.45:0/2459453441 wait complete. 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.454+0000 7f67e218f740 20 client.15270 trim_cache size 0 max 16384 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 3, 2024-03-12T05:10:30.460 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24467, 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 109.114831771, 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 129.19697975899999 2024-03-12T05:10:30.461 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 msgr2=0x7f67c40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 0x7f67c40a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67d0096880 tx=0x7f67d006d720 comp rx=0 tx=0).stop 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 msgr2=0x7f67d8102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f67d000cb00 tx=0x7f67d000a250 comp rx=0 tx=0).stop 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 shutdown_connections 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f67c40a1ab0 0x7f67c40a3f70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67d8101c50 0x7f67d8102bb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d8101290 0x7f67d8102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 --2- 172.21.15.45:0/3736397466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81000a0 0x7f67d8103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 >> 172.21.15.45:0/3736397466 conn(0x7f67d80fb630 msgr2=0x7f67d80697c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 shutdown_connections 2024-03-12T05:10:30.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.455+0000 7f67df5bc640 1 -- 172.21.15.45:0/3736397466 wait complete. 2024-03-12T05:10:30.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/95372116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1871141482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1584787276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:30 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2273125434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-12T05:10:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/95372116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1871141482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1584787276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:30 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2273125434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/95372116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1871141482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1584787276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:30 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2273125434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- 172.21.15.45:0/2385801396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101270 msgr2=0x7f6cac1016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2385801396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101270 0x7f6cac1016f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009900 tx=0x7f6c9c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- 172.21.15.45:0/2385801396 shutdown_connections 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2385801396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101270 0x7f6cac1016f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2385801396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac101c30 0x7f6cac1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2385801396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac100080 0x7f6cac1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- 172.21.15.45:0/2385801396 >> 172.21.15.45:0/2385801396 conn(0x7f6cac0fb610 msgr2=0x7f6cac0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- 172.21.15.45:0/2385801396 shutdown_connections 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- 172.21.15.45:0/2385801396 wait complete. 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 Processor -- start 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.571+0000 7f6cb4064640 1 -- start start 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100080 0x7f6cac197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac101270 0x7f6cac1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19eb70 con 0x7f6cac101c30 2024-03-12T05:10:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19ece0 con 0x7f6cac101270 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19eea0 con 0x7f6cac100080 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33014/0 (socket says 172.21.15.45:33014) 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 -- 172.21.15.45:0/2686768536 learned_addr learned my addr 172.21.15.45:0/2686768536 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100080 0x7f6cac197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 -- 172.21.15.45:0/2686768536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100080 msgr2=0x7f6cac197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100080 0x7f6cac197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 -- 172.21.15.45:0/2686768536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac101270 msgr2=0x7f6cac1975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac101270 0x7f6cac1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 -- 172.21.15.45:0/2686768536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009560 con 0x7f6cac101c30 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400cce0 tx=0x7f6ca400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6ca2ffd640 1 -- 172.21.15.45:0/2686768536 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca400f450 con 0x7f6cac101c30 2024-03-12T05:10:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6ca2ffd640 1 -- 172.21.15.45:0/2686768536 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ca400fa60 con 0x7f6cac101c30 2024-03-12T05:10:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac19f180 con 0x7f6cac101c30 2024-03-12T05:10:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6ca2ffd640 1 -- 172.21.15.45:0/2686768536 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4018b10 con 0x7f6cac101c30 2024-03-12T05:10:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.572+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac19f690 con 0x7f6cac101c30 2024-03-12T05:10:30.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.574+0000 7f6ca2ffd640 1 -- 172.21.15.45:0/2686768536 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6ca400fbd0 con 0x7f6cac101c30 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 Processor -- start 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 -- start start 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6ca2ffd640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 0x7f6c980a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a380cd0 con 0x55a13a3c5970 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a333f20 con 0x55a13a3c4f80 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a331450 con 0x55a13a3caae0 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 0x7f6c980a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6ca2ffd640 1 -- 172.21.15.45:0/2686768536 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6ca40cd810 con 0x7f6cac101c30 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33028/0 (socket says 172.21.15.45:33028) 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.575+0000 7f6cb15ff640 1 -- 172.21.15.45:0/201015296 learned_addr learned my addr 172.21.15.45:0/201015296 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb2601640 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 -- 172.21.15.45:0/201015296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 msgr2=0x55a13a3ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 -- 172.21.15.45:0/201015296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 msgr2=0x55a13a3c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 -- 172.21.15.45:0/201015296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a13a2e1f20 con 0x55a13a3c5970 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 0x7f6c980a3f70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005c20 tx=0x7f6c9c02c9a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005ec0 tx=0x7f6c9c045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa5ab9bfab5d2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6c97fff640 1 -- 172.21.15.45:0/201015296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0052d0 con 0x55a13a3c5970 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6c97fff640 1 -- 172.21.15.45:0/201015296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6c9c005430 con 0x55a13a3c5970 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.576+0000 7f6c97fff640 1 -- 172.21.15.45:0/201015296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c034440 con 0x55a13a3c5970 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- 172.21.15.45:0/201015296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 msgr2=0x55a13a3ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005ec0 tx=0x7f6c9c045000 comp rx=0 tx=0).stop 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- 172.21.15.45:0/201015296 shutdown_connections 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3ca470 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/201015296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- 172.21.15.45:0/201015296 >> 172.21.15.45:0/201015296 conn(0x55a13a3c04f0 msgr2=0x55a13a3c08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- 172.21.15.45:0/201015296 shutdown_connections 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- 172.21.15.45:0/201015296 wait complete. 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 -- start start 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.577+0000 7f6cb6c33740 1 Processor -- start 2024-03-12T05:10:30.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 0x55a13a3cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3caae0 0x55a13a3d66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a32cc30 con 0x55a13a3c5970 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a2d1d30 con 0x55a13a3caae0 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a000310 con 0x55a13a3c4f80 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 10 client.0 _ll_register_callbacks cb 0x55a13a2d9d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33036/0 (socket says 172.21.15.45:33036) 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 -- 172.21.15.45:0/4223194187 learned_addr learned my addr 172.21.15.45:0/4223194187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 0x55a13a3cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 -- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 msgr2=0x55a13a3cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 0x55a13a3cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 -- 172.21.15.45:0/4223194187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3caae0 msgr2=0x55a13a3d66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3caae0 0x55a13a3d66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 -- 172.21.15.45:0/4223194187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac108dc0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 0x55a13a3cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c006010 tx=0x7f6c9c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9275435e57613697 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6c957fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c043860 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6c9c049070 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a13a02c6c0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.578+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c9c0056e0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 10 client.15282 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a13a32b2b0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a13a02c6c0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6c9c0576a0 con 0x55a13a3c5970 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 10 client.15282 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 10 client.15282 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 20 client.15282 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a3dabd0 0x55a13a3dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 4 client.15282 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a13a07d270 con 0x55a13a3dabd0 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.579+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a3dabd0 0x55a13a3dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a3dabd0 0x55a13a3dfc10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c5bd81a7a2238c3d server_cookie=dac706e596132163 in_seq=0 out_seq=0 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6c957fa640 10 client.15282 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6c957fa640 1 -- 172.21.15.45:0/4223194187 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a13a07d270 con 0x55a13a3dabd0 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6c957fa640 10 client.15282 handle_command_reply: tid=0 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6cb6c33740 1 client.15282 shutdown 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 msgr2=0x55a13a3cf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.580+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c006010 tx=0x7f6c9c009830 comp rx=0 tx=0).stop 2024-03-12T05:10:30.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 shutdown_connections 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a3dabd0 0x55a13a3dfc10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3cf6a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3caae0 0x55a13a3d66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/4223194187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c4f80 0x55a13a3cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6c957fa640 0 client.15282 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 >> 172.21.15.45:0/4223194187 conn(0x55a13a3c04f0 msgr2=0x55a13a3bf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 shutdown_connections 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 1 -- 172.21.15.45:0/4223194187 wait complete. 2024-03-12T05:10:30.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.581+0000 7f6cb6c33740 20 client.15282 trim_cache size 0 max 16384 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 Processor -- start 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 -- start start 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3cced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3e4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a2b2490 con 0x55a13a3c5970 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a3d7a60 con 0x55a13a3c4f80 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.596+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a3bb9f0 con 0x55a13a3caae0 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33052/0 (socket says 172.21.15.45:33052) 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 -- 172.21.15.45:0/2873129373 learned_addr learned my addr 172.21.15.45:0/2873129373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 -- 172.21.15.45:0/2873129373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 msgr2=0x55a13a3e4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3e4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 -- 172.21.15.45:0/2873129373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 msgr2=0x55a13a3cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 -- 172.21.15.45:0/2873129373 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a13a2c7a20 con 0x55a13a3c5970 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c02c7d0 tx=0x7f6c9c02c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce27bbabecb64e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6c977fe640 1 -- 172.21.15.45:0/2873129373 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c057640 con 0x55a13a3c5970 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6c977fe640 1 -- 172.21.15.45:0/2873129373 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6c9c02ce80 con 0x55a13a3c5970 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.597+0000 7f6c977fe640 1 -- 172.21.15.45:0/2873129373 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c034c30 con 0x55a13a3c5970 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2873129373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 msgr2=0x55a13a3d66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c02c7d0 tx=0x7f6c9c02c3e0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2873129373 shutdown_connections 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3c5970 0x55a13a3d66a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3caae0 0x55a13a3e4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2873129373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2873129373 >> 172.21.15.45:0/2873129373 conn(0x55a13a3c04f0 msgr2=0x55a13a3e3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2873129373 shutdown_connections 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2873129373 wait complete. 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- start start 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 Processor -- start 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c5970 0x55a13a3d30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a3d7360 con 0x55a13a3caae0 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a3d3d20 con 0x55a13a3c4f80 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a13a431920 con 0x55a13a3c5970 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.598+0000 7f6cb6c33740 10 client.0 _ll_register_callbacks cb 0x7f6c9c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33058/0 (socket says 172.21.15.45:33058) 2024-03-12T05:10:30.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 -- 172.21.15.45:0/2180134611 learned_addr learned my addr 172.21.15.45:0/2180134611 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb15ff640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c5970 0x55a13a3d30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 -- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c5970 msgr2=0x55a13a3d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c5970 0x55a13a3d30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 -- 172.21.15.45:0/2180134611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 msgr2=0x55a13a3d65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3d65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 -- 172.21.15.45:0/2180134611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a13a32b2b0 con 0x55a13a3caae0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb2601640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6ca40cb260 tx=0x7f6ca409a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb48f505587c6792 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca409a730 con 0x55a13a3caae0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f6ca409e880 con 0x55a13a3caae0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a13a3dbb00 con 0x55a13a3caae0 2024-03-12T05:10:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ca40a2ed0 con 0x55a13a3caae0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb6c33740 10 client.15288 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a13a2ed780 con 0x55a13a3caae0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a13a3dbb00 con 0x55a13a3caae0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6ca4016020 con 0x55a13a3caae0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.599+0000 7f6cb6c33740 10 client.15288 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 10 client.15288 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 20 client.15288 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a436c90 0x55a13a433150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 4 client.15288 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a13a3ca850 con 0x55a13a436c90 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a436c90 0x55a13a433150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6c957fa640 10 client.15288 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb1e00640 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a436c90 0x55a13a433150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a87c326b41b60df1 server_cookie=a2a6f659f455b666 in_seq=0 out_seq=0 2024-03-12T05:10:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6c957fa640 1 -- 172.21.15.45:0/2180134611 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55a13a3ca850 con 0x55a13a436c90 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6c957fa640 10 client.15288 handle_command_reply: tid=0 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.600+0000 7f6cb6c33740 1 client.15288 shutdown 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 msgr2=0x55a13a3d35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6ca40cb260 tx=0x7f6ca409a620 comp rx=0 tx=0).stop 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 shutdown_connections 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a13a436c90 0x55a13a433150 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a13a3caae0 0x55a13a3d35f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a13a3c5970 0x55a13a3d30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 --2- 172.21.15.45:0/2180134611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a13a3c4f80 0x55a13a3d65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6c957fa640 0 client.15288 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 >> 172.21.15.45:0/2180134611 conn(0x55a13a3c04f0 msgr2=0x55a13a3e44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 shutdown_connections 2024-03-12T05:10:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 1 -- 172.21.15.45:0/2180134611 wait complete. 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb6c33740 20 client.15288 trim_cache size 0 max 16384 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 2, 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24472, 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 110.268374134, 2024-03-12T05:10:30.606 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:30.607 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:30.607 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:30.607 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 132.199161645 2024-03-12T05:10:30.607 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 msgr2=0x7f6c980a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 0x7f6c980a3f70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005c20 tx=0x7f6c9c02c9a0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 msgr2=0x7f6cac19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400cce0 tx=0x7f6ca400a620 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 shutdown_connections 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f6c980a1ab0 0x7f6c980a3f70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac101c30 0x7f6cac19e630 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac101270 0x7f6cac1975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 --2- 172.21.15.45:0/2686768536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100080 0x7f6cac197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 >> 172.21.15.45:0/2686768536 conn(0x7f6cac0fb610 msgr2=0x7f6cac0682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 shutdown_connections 2024-03-12T05:10:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.601+0000 7f6cb4064640 1 -- 172.21.15.45:0/2686768536 wait complete. 2024-03-12T05:10:30.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-12T05:10:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2446430917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101c30 msgr2=0x7ff0b01084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2446430917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101c30 0x7ff0b01084c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009ca0 tx=0x7ff0a002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2446430917 shutdown_connections 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2446430917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101c30 0x7ff0b01084c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2446430917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101270 0x7ff0b01016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2446430917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 0x7ff0b01004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2446430917 >> 172.21.15.45:0/2446430917 conn(0x7ff0b00fb610 msgr2=0x7ff0b00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2446430917 shutdown_connections 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.715+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2446430917 wait complete. 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 Processor -- start 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 -- start start 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 0x7ff0b0196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 0x7ff0b019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b019db20 con 0x7ff0b0101270 2024-03-12T05:10:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b019dc90 con 0x7ff0b0101c30 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0b49bc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b019de50 con 0x7ff0b0100080 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0add9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0add9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33070/0 (socket says 172.21.15.45:33070) 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.716+0000 7ff0add9b640 1 -- 172.21.15.45:0/2004881629 learned_addr learned my addr 172.21.15.45:0/2004881629 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 0x7ff0b0196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0aed9d640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 0x7ff0b019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 -- 172.21.15.45:0/2004881629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 msgr2=0x7ff0b0196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 0x7ff0b0196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 -- 172.21.15.45:0/2004881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 msgr2=0x7ff0b019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 0x7ff0b019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 -- 172.21.15.45:0/2004881629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a0009900 con 0x7ff0b0101270 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0aed9d640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 0x7ff0b019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0add9b640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff098007910 tx=0x7ff098007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0a77fe640 1 -- 172.21.15.45:0/2004881629 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098011070 con 0x7ff0b0101270 2024-03-12T05:10:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0a77fe640 1 -- 172.21.15.45:0/2004881629 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0980044d0 con 0x7ff0b0101270 2024-03-12T05:10:30.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0068340 con 0x7ff0b0101270 2024-03-12T05:10:30.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0a77fe640 1 -- 172.21.15.45:0/2004881629 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098004a00 con 0x7ff0b0101270 2024-03-12T05:10:30.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.717+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b0068850 con 0x7ff0b0101270 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.719+0000 7ff0a77fe640 1 -- 172.21.15.45:0/2004881629 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff098004020 con 0x7ff0b0101270 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.719+0000 7ff0b7591740 1 Processor -- start 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 -- start start 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141f7f80 0x55de141f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 0x55de141fff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de141b3cd0 con 0x55de141f8970 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de14166f20 con 0x55de141fdae0 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0b7591740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de14164450 con 0x55de141f7f80 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0a77fe640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 0x7ff09c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33074/0 (socket says 172.21.15.45:33074) 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 -- 172.21.15.45:0/547977131 learned_addr learned my addr 172.21.15.45:0/547977131 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0aed9d640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 0x7ff09c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0aed9d640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141f7f80 0x55de141f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0a77fe640 1 -- 172.21.15.45:0/2004881629 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0980ce4f0 con 0x7ff0b0101270 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0add9b640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 0x55de141fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 -- 172.21.15.45:0/547977131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141f7f80 msgr2=0x55de141f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141f7f80 0x55de141f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 -- 172.21.15.45:0/547977131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 msgr2=0x55de141fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 0x55de141fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.720+0000 7ff0ae59c640 1 -- 172.21.15.45:0/547977131 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de14114f20 con 0x55de141f8970 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0add9b640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 0x55de141fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800bd10 tx=0x7ff0a800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1123c167404e31a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0aed9d640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 0x7ff09c0a3ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009bf0 tx=0x7ff0a00344e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0a4ff9640 1 -- 172.21.15.45:0/547977131 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a801a460 con 0x55de141f8970 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0a4ff9640 1 -- 172.21.15.45:0/547977131 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0a801aa70 con 0x55de141f8970 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.721+0000 7ff0a4ff9640 1 -- 172.21.15.45:0/547977131 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a80119a0 con 0x55de141f8970 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- 172.21.15.45:0/547977131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 msgr2=0x55de141fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800bd10 tx=0x7ff0a800f730 comp rx=0 tx=0).stop 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- 172.21.15.45:0/547977131 shutdown_connections 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de141fd470 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141fdae0 0x55de141fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- 172.21.15.45:0/547977131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141f7f80 0x55de141f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- 172.21.15.45:0/547977131 >> 172.21.15.45:0/547977131 conn(0x55de141f34f0 msgr2=0x55de141f38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- 172.21.15.45:0/547977131 shutdown_connections 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- 172.21.15.45:0/547977131 wait complete. 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- start start 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 Processor -- start 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14202700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14209740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1415fc30 con 0x55de141f7f80 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de14104d30 con 0x55de141f8970 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de13e33310 con 0x55de141fdae0 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0b7591740 10 client.0 _ll_register_callbacks cb 0x55de1410cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33082/0 (socket says 172.21.15.45:33082) 2024-03-12T05:10:30.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.722+0000 7ff0ae59c640 1 -- 172.21.15.45:0/3869541654 learned_addr learned my addr 172.21.15.45:0/3869541654 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14202700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 -- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 msgr2=0x55de14209740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14209740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 -- 172.21.15.45:0/3869541654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 msgr2=0x55de14202700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14202700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 -- 172.21.15.45:0/3869541654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a800b940 con 0x55de141f7f80 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8000c00 tx=0x7ff0a801a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d26680fb48fc9e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8020070 con 0x55de141f7f80 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0a8005040 con 0x55de141f7f80 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de13e5f6c0 con 0x55de141f7f80 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0a8005200 con 0x55de141f7f80 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0b7591740 10 client.15300 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55de1415e2b0 con 0x55de141f7f80 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.723+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55de13e5f6c0 con 0x55de141f7f80 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0a8004680 con 0x55de141f7f80 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 10 client.15300 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 10 client.15300 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 20 client.15300 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1420ea20 0x55de14212ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 4 client.15300 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de13eb0270 con 0x55de1420ea20 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1420ea20 0x55de14212ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1420ea20 0x55de14212ea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfa88ef2c7e310bd server_cookie=d694baa9a2861e17 in_seq=0 out_seq=0 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.724+0000 7ff095ffb640 10 client.15300 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff095ffb640 1 -- 172.21.15.45:0/3869541654 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55de13eb0270 con 0x55de1420ea20 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff095ffb640 10 client.15300 handle_command_reply: tid=0 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 client.15300 shutdown 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 msgr2=0x55de142021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8000c00 tx=0x7ff0a801a690 comp rx=0 tx=0).stop 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 shutdown_connections 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1420ea20 0x55de14212ea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14209740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14202700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff095ffb640 0 client.15300 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3869541654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de142021c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 >> 172.21.15.45:0/3869541654 conn(0x55de141f34f0 msgr2=0x55de141f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 shutdown_connections 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 1 -- 172.21.15.45:0/3869541654 wait complete. 2024-03-12T05:10:30.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.725+0000 7ff0b7591740 20 client.15300 trim_cache size 0 max 16384 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 Processor -- start 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 -- start start 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14209700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14217400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de140e5490 con 0x55de141f7f80 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1420c9c0 con 0x55de141f8970 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0b7591740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1420f2b0 con 0x55de141fdae0 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33094/0 (socket says 172.21.15.45:33094) 2024-03-12T05:10:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 -- 172.21.15.45:0/1897714025 learned_addr learned my addr 172.21.15.45:0/1897714025 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 -- 172.21.15.45:0/1897714025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 msgr2=0x55de14217400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14217400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 -- 172.21.15.45:0/1897714025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 msgr2=0x55de14209700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14209700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 -- 172.21.15.45:0/1897714025 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de140faa20 con 0x55de141f7f80 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8004fd0 tx=0x7ff0a800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307c6776ace840c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff097fff640 1 -- 172.21.15.45:0/1897714025 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8020070 con 0x55de141f7f80 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff097fff640 1 -- 172.21.15.45:0/1897714025 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0a800b580 con 0x55de141f7f80 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.748+0000 7ff097fff640 1 -- 172.21.15.45:0/1897714025 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8025470 con 0x55de141f7f80 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- 172.21.15.45:0/1897714025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 msgr2=0x55de14216ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8004fd0 tx=0x7ff0a800fc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- 172.21.15.45:0/1897714025 shutdown_connections 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14217400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f8970 0x55de14209700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 --2- 172.21.15.45:0/1897714025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f7f80 0x55de14216ec0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- 172.21.15.45:0/1897714025 >> 172.21.15.45:0/1897714025 conn(0x55de141f34f0 msgr2=0x55de141f38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- 172.21.15.45:0/1897714025 shutdown_connections 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- 172.21.15.45:0/1897714025 wait complete. 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 -- start start 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.749+0000 7ff0b7591740 1 Processor -- start 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f7f80 0x55de14206600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14264b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de14200100 con 0x55de141f8970 2024-03-12T05:10:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1420a3c0 con 0x55de141f7f80 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de142650a0 con 0x55de141fdae0 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 10 client.0 _ll_register_callbacks cb 0x7ff0a800b4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33096/0 (socket says 172.21.15.45:33096) 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 -- 172.21.15.45:0/3732552335 learned_addr learned my addr 172.21.15.45:0/3732552335 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f7f80 0x55de14206600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 -- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 msgr2=0x55de14264b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14264b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 -- 172.21.15.45:0/3732552335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f7f80 msgr2=0x55de14206600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f7f80 0x55de14206600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 -- 172.21.15.45:0/3732552335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de1415e2b0 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0add9b640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff098005fd0 tx=0x7ff0980cbeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb6dfd7816ab925 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980bb070 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff09809bb50 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de1420f3f0 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff09809b450 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 10 client.15306 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.750+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55de14120780 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55de1420f3f0 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0980b9020 con 0x55de141f8970 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 10 client.15306 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 10 client.15306 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:10:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 20 client.15306 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1426a350 0x55de142667b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 4 client.15306 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7ff0a801aa70 con 0x55de1426a350 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1426a350 0x55de142667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff095ffb640 10 client.15306 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.751+0000 7ff0ae59c640 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1426a350 0x55de142667b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a075f37cbdc9122 server_cookie=166072545262a288 in_seq=0 out_seq=0 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff095ffb640 1 -- 172.21.15.45:0/3732552335 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7ff0a801aa70 con 0x55de1426a350 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff095ffb640 10 client.15306 handle_command_reply: tid=0 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 client.15306 shutdown 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 msgr2=0x55de14206b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff098005fd0 tx=0x7ff0980cbeb0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 shutdown_connections 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de141f8970 0x55de14206b40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55de1426a350 0x55de142667b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de141fdae0 0x55de14264b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff095ffb640 0 client.15306 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 --2- 172.21.15.45:0/3732552335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de141f7f80 0x55de14206600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 >> 172.21.15.45:0/3732552335 conn(0x55de141f34f0 msgr2=0x55de14216ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 shutdown_connections 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 1 -- 172.21.15.45:0/3732552335 wait complete. 2024-03-12T05:10:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b7591740 20 client.15306 trim_cache size 0 max 16384 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 1, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24487, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 111.430382158, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:30.758 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 126.735869651 2024-03-12T05:10:30.759 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 msgr2=0x7ff09c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 0x7ff09c0a3ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009bf0 tx=0x7ff0a00344e0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 msgr2=0x7ff0b0196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff098007910 tx=0x7ff098007dd0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 shutdown_connections 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff09c0a1a30 0x7ff09c0a3ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0b0101270 0x7ff0b0196570 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b0101c30 0x7ff0b019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 --2- 172.21.15.45:0/2004881629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0b0100080 0x7ff0b0196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 >> 172.21.15.45:0/2004881629 conn(0x7ff0b00fb610 msgr2=0x7ff0b0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 shutdown_connections 2024-03-12T05:10:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.752+0000 7ff0b49bc640 1 -- 172.21.15.45:0/2004881629 wait complete. 2024-03-12T05:10:30.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- 172.21.15.45:0/4158174832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc101c60 msgr2=0x7f4ffc104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- 172.21.15.45:0/4158174832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc101c60 0x7f4ffc104050 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009900 tx=0x7f4ff802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- 172.21.15.45:0/4158174832 shutdown_connections 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- 172.21.15.45:0/4158174832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- 172.21.15.45:0/4158174832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc104590 0x7f4ffc106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- 172.21.15.45:0/4158174832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc101c60 0x7f4ffc104050 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- 172.21.15.45:0/4158174832 >> 172.21.15.45:0/4158174832 conn(0x7f4ffc0fb610 msgr2=0x7f4ffc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- 172.21.15.45:0/4158174832 shutdown_connections 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- 172.21.15.45:0/4158174832 wait complete. 2024-03-12T05:10:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 Processor -- start 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- start start 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 0x7f4ffc1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ffc104590 0x7f4ffc197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19ecb0 con 0x7f4ffc106ec0 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19ee20 con 0x7f4ffc101c60 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.870+0000 7f5003a64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19efe0 con 0x7f4ffc104590 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33116/0 (socket says 172.21.15.45:33116) 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 -- 172.21.15.45:0/3456104986 learned_addr learned my addr 172.21.15.45:0/3456104986 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5001800640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 0x7f4ffc1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 -- 172.21.15.45:0/3456104986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ffc104590 msgr2=0x7f4ffc197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ffc104590 0x7f4ffc197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 -- 172.21.15.45:0/3456104986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 msgr2=0x7f4ffc1971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 0x7f4ffc1971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 -- 172.21.15.45:0/3456104986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8009560 con 0x7f4ffc106ec0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5001800640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 0x7f4ffc1971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5002001640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400cad0 tx=0x7f4ff400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f4ff27fc640 1 -- 172.21.15.45:0/3456104986 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400ae20 con 0x7f4ffc106ec0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f4ff27fc640 1 -- 172.21.15.45:0/3456104986 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4ff4005020 con 0x7f4ffc106ec0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc19f2c0 con 0x7f4ffc106ec0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f4ff27fc640 1 -- 172.21.15.45:0/3456104986 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400f690 con 0x7f4ffc106ec0 2024-03-12T05:10:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.871+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc19f7d0 con 0x7f4ffc106ec0 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.873+0000 7f4ff27fc640 1 -- 172.21.15.45:0/3456104986 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4ff4005190 con 0x7f4ffc106ec0 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 Processor -- start 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 -- start start 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39390 0x5624aac397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f4ff27fc640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 0x7f4fe80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac41380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aabf0cd0 con 0x5624aac39d80 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aaba3f20 con 0x5624aac3eef0 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f500663b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aaba1450 con 0x5624aac39390 2024-03-12T05:10:30.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33124/0 (socket says 172.21.15.45:33124) 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5001800640 1 -- 172.21.15.45:0/2045129263 learned_addr learned my addr 172.21.15.45:0/2045129263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5000fff640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 0x7f4fe80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5002001640 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac41380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f4ff27fc640 1 -- 172.21.15.45:0/3456104986 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ff40cd4d0 con 0x7f4ffc106ec0 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.874+0000 7f5000fff640 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39390 0x5624aac397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 -- 172.21.15.45:0/2045129263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39390 msgr2=0x5624aac397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39390 0x5624aac397b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 -- 172.21.15.45:0/2045129263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 msgr2=0x5624aac41380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac41380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 -- 172.21.15.45:0/2045129263 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624aab51f20 con 0x5624aac39d80 2024-03-12T05:10:30.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5000fff640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 0x7f4fe80a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4fec005d00 tx=0x7f4fec0113e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f5001800640 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009900 tx=0x7f4ff802c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201f05f12905c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f4fe77fe640 1 -- 172.21.15.45:0/2045129263 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80048c0 con 0x5624aac39d80 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f4fe77fe640 1 -- 172.21.15.45:0/2045129263 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4ff802de90 con 0x5624aac39d80 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.875+0000 7f4fe77fe640 1 -- 172.21.15.45:0/2045129263 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80039f0 con 0x5624aac39d80 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- 172.21.15.45:0/2045129263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 msgr2=0x5624aac3e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009900 tx=0x7f4ff802c970 comp rx=0 tx=0).stop 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- 172.21.15.45:0/2045129263 shutdown_connections 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac3e880 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac41380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- 172.21.15.45:0/2045129263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39390 0x5624aac397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- 172.21.15.45:0/2045129263 >> 172.21.15.45:0/2045129263 conn(0x5624aac304f0 msgr2=0x5624aac308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- 172.21.15.45:0/2045129263 shutdown_connections 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- 172.21.15.45:0/2045129263 wait complete. 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- start start 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 Processor -- start 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 0x5624aac43a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac4aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aab9cc30 con 0x5624aac39390 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aab41d30 con 0x5624aac3eef0 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.876+0000 7f500663b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aa870310 con 0x5624aac39d80 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f500663b740 10 client.0 _ll_register_callbacks cb 0x5624aab49d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33140/0 (socket says 172.21.15.45:33140) 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 -- 172.21.15.45:0/404575617 learned_addr learned my addr 172.21.15.45:0/404575617 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5000fff640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 0x5624aac43a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5002001640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac4aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 -- 172.21.15.45:0/404575617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 msgr2=0x5624aac43a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 0x5624aac43a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 -- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 msgr2=0x5624aac4aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac4aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 -- 172.21.15.45:0/404575617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc109e90 con 0x5624aac39390 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5000fff640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 0x5624aac43a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5002001640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac4aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f5001800640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8005940 tx=0x7f4ff8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e52abaf91e5ff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f4fe4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80054f0 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ff8034440 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624aa89c6c0 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ff8005650 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f500663b740 10 client.15318 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5624aab9b2b0 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.877+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5624aa89c6c0 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4ff803c040 con 0x5624aac39390 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 10 client.15318 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 10 client.15318 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:10:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 20 client.15318 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aac4fa20 0x5624aac53ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 4 client.15318 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624aa8ed270 con 0x5624aac4fa20 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f5000fff640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aac4fa20 0x5624aac53ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f5000fff640 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aac4fa20 0x5624aac53ea0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e60df81ca6040bda server_cookie=361a16a77d22fdb6 in_seq=0 out_seq=0 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.878+0000 7f4fe4ff9640 10 client.15318 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/404575617 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5624aa8ed270 con 0x5624aac4fa20 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f4fe4ff9640 10 client.15318 handle_command_reply: tid=0 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 client.15318 shutdown 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 msgr2=0x5624aac434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8005940 tx=0x7f4ff8004ff0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 shutdown_connections 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aac4fa20 0x5624aac53ea0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac3eef0 0x5624aac4aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac39d80 0x5624aac43a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 --2- 172.21.15.45:0/404575617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac434c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f4fe4ff9640 0 client.15318 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 >> 172.21.15.45:0/404575617 conn(0x5624aac304f0 msgr2=0x5624aac2f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 shutdown_connections 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 1 -- 172.21.15.45:0/404575617 wait complete. 2024-03-12T05:10:30.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.879+0000 7f500663b740 20 client.15318 trim_cache size 0 max 16384 2024-03-12T05:10:30.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 Processor -- start 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 -- start start 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 0x5624aac57ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aac58400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aab22490 con 0x5624aac39d80 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aac42290 con 0x5624aac39390 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f500663b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aac51a00 con 0x5624aac3eef0 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f5000fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f5000fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33154/0 (socket says 172.21.15.45:33154) 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f5000fff640 1 -- 172.21.15.45:0/1154089285 learned_addr learned my addr 172.21.15.45:0/1154089285 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f5002001640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aac58400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.895+0000 7f5001800640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 0x5624aac57ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 -- 172.21.15.45:0/1154089285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 msgr2=0x5624aac58400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aac58400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 -- 172.21.15.45:0/1154089285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 msgr2=0x5624aac57ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 0x5624aac57ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 -- 172.21.15.45:0/1154089285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624aab37a20 con 0x5624aac39d80 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5002001640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aac58400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5001800640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 0x5624aac57ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f5000fff640 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4fec011f40 tx=0x7f4fec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b622eb3a849005e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f4fe6ffd640 1 -- 172.21.15.45:0/1154089285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec029070 con 0x5624aac39d80 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f4fe6ffd640 1 -- 172.21.15.45:0/1154089285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4fec005a70 con 0x5624aac39d80 2024-03-12T05:10:30.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f4fe6ffd640 1 -- 172.21.15.45:0/1154089285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec024070 con 0x5624aac39d80 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 -- 172.21.15.45:0/1154089285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 msgr2=0x5624aac4a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4fec011f40 tx=0x7f4fec014000 comp rx=0 tx=0).stop 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 -- 172.21.15.45:0/1154089285 shutdown_connections 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39d80 0x5624aac4a960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aac58400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 --2- 172.21.15.45:0/1154089285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39390 0x5624aac57ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 -- 172.21.15.45:0/1154089285 >> 172.21.15.45:0/1154089285 conn(0x5624aac304f0 msgr2=0x5624aac308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.896+0000 7f500663b740 1 -- 172.21.15.45:0/1154089285 shutdown_connections 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 -- 172.21.15.45:0/1154089285 wait complete. 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 -- start start 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 Processor -- start 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39d80 0x5624aac47cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aaca5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aac4acc0 con 0x5624aac39390 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aac4ae30 con 0x5624aac39d80 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624aaca60a0 con 0x5624aac3eef0 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f500663b740 10 client.0 _ll_register_callbacks cb 0x7f4ff8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33164/0 (socket says 172.21.15.45:33164) 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 -- 172.21.15.45:0/453959453 learned_addr learned my addr 172.21.15.45:0/453959453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5000fff640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39d80 0x5624aac47cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 -- 172.21.15.45:0/453959453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 msgr2=0x5624aaca5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aaca5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 -- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39d80 msgr2=0x5624aac47cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39d80 0x5624aac47cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.897+0000 7f5001800640 1 -- 172.21.15.45:0/453959453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec024b00 con 0x5624aac39390 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f5001800640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8000c00 tx=0x7f4ff802cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b537eac1ff4de927 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff80052f0 con 0x5624aac39390 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4ff8005450 con 0x5624aac39390 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624aac503f0 con 0x5624aac39390 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ff8005610 con 0x5624aac39390 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 10 client.15324 fetch_fsmap learned FSMap version 24 2024-03-12T05:10:30.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5624aab5d780 con 0x5624aac39390 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5624aac503f0 con 0x5624aac39390 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4ff8003af0 con 0x5624aac39390 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 10 client.15324 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 10 client.15324 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 20 client.15324 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aacaafe0 0x5624aaca7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 4 client.15324 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.898+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4ff802de90 con 0x5624aacaafe0 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f5000fff640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aacaafe0 0x5624aaca7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:30.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f4fe4ff9640 10 client.15324 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f5000fff640 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aacaafe0 0x5624aaca7400 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10c579d5666dcb12 server_cookie=6eab1e2b0de87331 in_seq=0 out_seq=0 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f4fe4ff9640 1 -- 172.21.15.45:0/453959453 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7f4ff802de90 con 0x5624aacaafe0 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f4fe4ff9640 10 client.15324 handle_command_reply: tid=0 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 client.15324 shutdown 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 msgr2=0x5624aac477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8000c00 tx=0x7f4ff802cc10 comp rx=0 tx=0).stop 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 shutdown_connections 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5624aacaafe0 0x5624aaca7400 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624aac3eef0 0x5624aaca5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f4fe4ff9640 0 client.15324 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624aac39d80 0x5624aac47cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 --2- 172.21.15.45:0/453959453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5624aac39390 0x5624aac477b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 >> 172.21.15.45:0/453959453 conn(0x5624aac304f0 msgr2=0x5624aac57ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 shutdown_connections 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 1 -- 172.21.15.45:0/453959453 wait complete. 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.899+0000 7f500663b740 20 client.15324 trim_cache size 0 max 16384 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "52aed436-e02d-11ee-95c8-87774f69a715", 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-03-12T05:10:30.906 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24503, 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs", 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 113.131490333, 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 24, 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 87, 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 85, 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 118.329643613 2024-03-12T05:10:30.907 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 msgr2=0x7f4fe80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 0x7f4fe80a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4fec005d00 tx=0x7f4fec0113e0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 msgr2=0x7f4ffc19e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400cad0 tx=0x7f4ff400cf90 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 shutdown_connections 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4fe80a1a30 0x7f4fe80a3ef0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ffc106ec0 0x7f4ffc19e770 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ffc104590 0x7f4ffc197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 --2- 172.21.15.45:0/3456104986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ffc101c60 0x7f4ffc1971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 >> 172.21.15.45:0/3456104986 conn(0x7f4ffc0fb610 msgr2=0x7f4ffc102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:30.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 shutdown_connections 2024-03-12T05:10:30.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:30.900+0000 7f5003a64640 1 -- 172.21.15.45:0/3456104986 wait complete. 2024-03-12T05:10:30.910 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-12T05:10:30.937 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-12T05:10:30.937 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-12T05:10:30.937 INFO:tasks.ceph_fuse:client.0 config is {'mntopts': ['context=system_u:object_r:postgresql_db_t:s0'], 'mount_subvol_num': 0, 'mountpoint': '/tmp/cephfs'} 2024-03-12T05:10:30.937 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-12T05:10:30.938 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-03-12T05:10:30.938 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-03-12T05:10:30.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:30.938 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-03-12T05:10:30.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:30.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-03-12T05:10:31.022 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-03-12T05:10:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:10:31.024 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-12T05:10:31.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-12T05:10:31.308 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:10:31.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:31 smithi045 ceph-mon[32174]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:31.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:31 smithi130 ceph-mon[35398]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:31.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:31 smithi116 ceph-mon[35486]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 -- 172.21.15.45:0/2849765367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 msgr2=0x7f05281057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 --2- 172.21.15.45:0/2849765367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f05281057a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f051c009990 tx=0x7f051c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 -- 172.21.15.45:0/2849765367 shutdown_connections 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 --2- 172.21.15.45:0/2849765367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528105e80 0x7f052810da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 --2- 172.21.15.45:0/2849765367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528100dc0 0x7f05281011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 --2- 172.21.15.45:0/2849765367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f05281057a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 -- 172.21.15.45:0/2849765367 >> 172.21.15.45:0/2849765367 conn(0x7f0528077ae0 msgr2=0x7f0528077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:33.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 -- 172.21.15.45:0/2849765367 shutdown_connections 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.071+0000 7f052e707700 1 -- 172.21.15.45:0/2849765367 wait complete. 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 Processor -- start 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 -- start start 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528100dc0 0x7f0528102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528105e80 0x7f05281032c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05281a64b0 con 0x7f0528101770 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05281a6620 con 0x7f0528100dc0 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052e707700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05281a67e0 con 0x7f0528105e80 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33186/0 (socket says 172.21.15.45:33186) 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 -- 172.21.15.45:0/3723323475 learned_addr learned my addr 172.21.15.45:0/3723323475 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f0527fff700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528100dc0 0x7f0528102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:33.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f052cca4700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528105e80 0x7f05281032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 -- 172.21.15.45:0/3723323475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528105e80 msgr2=0x7f05281032c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528105e80 0x7f05281032c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 -- 172.21.15.45:0/3723323475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528100dc0 msgr2=0x7f0528102840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528100dc0 0x7f0528102840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.072+0000 7f05277fe700 1 -- 172.21.15.45:0/3723323475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0095f0 con 0x7f0528101770 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f05277fe700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c01d070 con 0x7f0528101770 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f051c00fd90 con 0x7f0528101770 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0037c0 con 0x7f0528101770 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05281a6a60 con 0x7f0528101770 2024-03-12T05:10:33.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.073+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05281a6e20 con 0x7f0528101770 2024-03-12T05:10:33.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.074+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052810a140 con 0x7f0528101770 2024-03-12T05:10:33.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.075+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f051c022690 con 0x7f0528101770 2024-03-12T05:10:33.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.076+0000 7f05257fa700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 0x7f05100a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:33.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.076+0000 7f0527fff700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 0x7f05100a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:33.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.076+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f051c0cef40 con 0x7f0528101770 2024-03-12T05:10:33.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.076+0000 7f0527fff700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 0x7f05100a4000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0518009a10 tx=0x7f0518009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:33.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.079+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f051c0981f0 con 0x7f0528101770 2024-03-12T05:10:33.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.329+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0528102280 con 0x7f0528101770 2024-03-12T05:10:33.327 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:33 smithi045 ceph-mon[32174]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:33.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.330+0000 7f05257fa700 1 -- 172.21.15.45:0/3723323475 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f051c09b8b0 con 0x7f0528101770 2024-03-12T05:10:33.328 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 msgr2=0x7f05100a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 0x7f05100a4000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0518009a10 tx=0x7f0518009190 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 msgr2=0x7f0528102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c00 tx=0x7f051c0047a0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 shutdown_connections 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f05100a1b40 0x7f05100a4000 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0528105e80 0x7f05281032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0528100dc0 0x7f0528102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 --2- 172.21.15.45:0/3723323475 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0528101770 0x7f0528102d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 >> 172.21.15.45:0/3723323475 conn(0x7f0528077ae0 msgr2=0x7f052806a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:33.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 shutdown_connections 2024-03-12T05:10:33.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:33.332+0000 7f052e707700 1 -- 172.21.15.45:0/3723323475 wait complete. 2024-03-12T05:10:33.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:33 smithi130 ceph-mon[35398]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:33.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:33 smithi116 ceph-mon[35486]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-12T05:10:34.242 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-03-12T05:10:34.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.243 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-03-12T05:10:34.258 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-12T05:10:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-12T05:10:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-03-12T05:10:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:10:34.258 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42050sec preferred_lft 42050sec 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:10:34.259 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:10:34.260 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-03-12T05:10:34.260 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-03-12T05:10:34.260 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-12T05:10:34.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.260 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-12T05:10:34.260 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-12T05:10:34.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-03-12T05:10:34.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-03-12T05:10:34.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-03-12T05:10:34.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-12T05:10:34.261 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-12T05:10:34.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.421 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-12T05:10:34.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3723323475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:10:34.456 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-03-12T05:10:34.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.459 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-03-12T05:10:34.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3723323475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:10:34.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3723323475' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:10:34.535 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-03-12T05:10:34.535 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-12T05:10:34.535 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-12T05:10:34.535 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-12T05:10:34.535 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-12T05:10:34.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.536 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-12T05:10:34.536 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-12T05:10:34.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-12T05:10:34.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-12T05:10:34.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-12T05:10:34.537 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-12T05:10:34.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.825 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-03-12T05:10:34.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.841 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-03-12T05:10:34.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.895 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-12T05:10:34.895 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-12T05:10:34.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--tmp-cephfs 2024-03-12T05:10:34.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-03-12T05:10:34.895 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-12T05:10:34.978 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-03-12T05:10:34.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-03-12T05:10:34.979 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-12T05:10:35.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:35.147 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2024-03-12T05:10:35.147 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-03-12T05:10:35.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-03-12T05:10:35.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-03-12T05:10:35.147 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-03-12T05:10:35.188 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mntopts': ['context=system_u:object_r:postgresql_db_t:s0'], 'mount_subvol_num': 0, 'mountpoint': '/tmp/cephfs'} 2024-03-12T05:10:35.188 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T05:10:35.188 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /tmp/cephfs 2024-03-12T05:10:35.243 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/tmp/cephfs' 2024-03-12T05:10:35.243 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T05:10:35.243 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /tmp/cephfs 2024-03-12T05:10:35.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-03-12T05:10:35.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:35 smithi045 ceph-mon[32174]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:35.364 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-03-12T05:10:35.419 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:35.420 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/3efe74d6b4801979adbbd19da3bb48413275563fda7a550bcc0072ed6d950980/merged 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d4bb01b00f29e90fc27d57e04bc007ef7d92e179c3a5c4776c5935315c5b1a16/merged 2024-03-12T05:10:35.421 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/39fb1e022294056186875e51e514896ee4067443207f64ff83657893482bcf53/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8b52f2f97f6b981a89753c0a94220b72b13b724c6f2025586987077ee1a37040/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0150e875350208b3c4090bb52a50fe7dcaaab49ec6397aad35741c61d0932362/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/70f52330b3a705c66257c1d827a01a983c8f030670e7b6b9cf567c62857fa7e7/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1567a8ea96a5841e3f58e338de8e800488e2d43ce0cb0001874f78b8c473118/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/c964ed55c5a2c29c961ee14fa6dd8dfc2ff76d3937537dce111d7f7f82ad15bc/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7aba026d324874abf32c75dea8afc6c5d55aa2428324b955edd008a23d70c901/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d9fe7e3e287b91a4f9eb509389927824e781ee83460f02dc807148b83ffb6395/merged 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:35.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:35.423 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-03-12T05:10:35.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:35 smithi130 ceph-mon[35398]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:35.474 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-12T05:10:35.475 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /tmp/cephfs --id 0 --client_mountpoint=/volumes/qa/sv_0/aa62c577-3bcc-4fae-b8a4-44471f267da8 --client_fs=cephfs -o context=system_u:object_r:postgresql_db_t:s0) 2024-03-12T05:10:35.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:35 smithi116 ceph-mon[35486]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-12T05:10:35.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-03-12T05:10:35.545 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-03-12T05:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-12T05:10:35.602 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/3efe74d6b4801979adbbd19da3bb48413275563fda7a550bcc0072ed6d950980/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d4bb01b00f29e90fc27d57e04bc007ef7d92e179c3a5c4776c5935315c5b1a16/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/39fb1e022294056186875e51e514896ee4067443207f64ff83657893482bcf53/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8b52f2f97f6b981a89753c0a94220b72b13b724c6f2025586987077ee1a37040/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0150e875350208b3c4090bb52a50fe7dcaaab49ec6397aad35741c61d0932362/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/70f52330b3a705c66257c1d827a01a983c8f030670e7b6b9cf567c62857fa7e7/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1567a8ea96a5841e3f58e338de8e800488e2d43ce0cb0001874f78b8c473118/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/c964ed55c5a2c29c961ee14fa6dd8dfc2ff76d3937537dce111d7f7f82ad15bc/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7aba026d324874abf32c75dea8afc6c5d55aa2428324b955edd008a23d70c901/merged 2024-03-12T05:10:35.603 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d9fe7e3e287b91a4f9eb509389927824e781ee83460f02dc807148b83ffb6395/merged 2024-03-12T05:10:35.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-03-12T05:10:35.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:35.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:35.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:35.604 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-03-12T05:10:35.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:2024-03-12T05:10:35.619+0000 7f107cc0c480 -1 init, newargv = 0x55f0743c7460 newargc=17 2024-03-12T05:10:35.616 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[85020]: starting ceph client 2024-03-12T05:10:35.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[85020]: starting fuse 2024-03-12T05:10:36.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-03-12T05:10:36.766 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-03-12T05:10:36.824 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-03-12T05:10:36.825 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/3efe74d6b4801979adbbd19da3bb48413275563fda7a550bcc0072ed6d950980/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d4bb01b00f29e90fc27d57e04bc007ef7d92e179c3a5c4776c5935315c5b1a16/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/39fb1e022294056186875e51e514896ee4067443207f64ff83657893482bcf53/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/8b52f2f97f6b981a89753c0a94220b72b13b724c6f2025586987077ee1a37040/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/0150e875350208b3c4090bb52a50fe7dcaaab49ec6397aad35741c61d0932362/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/70f52330b3a705c66257c1d827a01a983c8f030670e7b6b9cf567c62857fa7e7/merged 2024-03-12T05:10:36.826 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1567a8ea96a5841e3f58e338de8e800488e2d43ce0cb0001874f78b8c473118/merged 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/c964ed55c5a2c29c961ee14fa6dd8dfc2ff76d3937537dce111d7f7f82ad15bc/merged 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7aba026d324874abf32c75dea8afc6c5d55aa2428324b955edd008a23d70c901/merged 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d9fe7e3e287b91a4f9eb509389927824e781ee83460f02dc807148b83ffb6395/merged 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs 2024-03-12T05:10:36.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:36.827 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-03-12T05:10:36.881 INFO:teuthology.orchestra.run.smithi045.stdout:119 2024-03-12T05:10:36.882 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo stdin-killer -- python3 -c ' 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> import glob 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> import re 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> import os 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> import subprocess 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> 2024-03-12T05:10:36.882 DEBUG:teuthology.orchestra.run.smithi045:> def _find_admin_socket(client_name): 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> files = glob.glob(asok_path) 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> mountpoint = "/tmp/cephfs" 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> # Given a non-glob path, it better be there 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> if "*" not in asok_path: 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> assert(len(files) == 1) 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> return files[0] 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> for f in files: 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> contents = proc_f.read() 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> if mountpoint in contents: 2024-03-12T05:10:36.883 DEBUG:teuthology.orchestra.run.smithi045:> return f 2024-03-12T05:10:36.884 DEBUG:teuthology.orchestra.run.smithi045:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-12T05:10:36.884 DEBUG:teuthology.orchestra.run.smithi045:> 2024-03-12T05:10:36.884 DEBUG:teuthology.orchestra.run.smithi045:> print(_find_admin_socket("client.0")) 2024-03-12T05:10:36.884 DEBUG:teuthology.orchestra.run.smithi045:> ' 2024-03-12T05:10:37.007 INFO:teuthology.orchestra.run.smithi045.stdout:/var/run/ceph/ceph-client.0.85020.asok 2024-03-12T05:10:37.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-12T05:10:37.018 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85020.asok 2024-03-12T05:10:37.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:37.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85020.asok status 2024-03-12T05:10:37.157 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:37.157 INFO:teuthology.orchestra.run.smithi045.stdout: "metadata": { 2024-03-12T05:10:37.157 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_sha1": "94a539707733325322964c6a187c6a1b41be1daf", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "entity_id": "0", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": "smithi045", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "mount_point": "/tmp/cephfs", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pid": "85020", 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "root": "/volumes/qa/sv_0/aa62c577-3bcc-4fae-b8a4-44471f267da8" 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_count": 0, 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_pinned_count": 0, 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 15336, 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "inst": { 2024-03-12T05:10:37.158 INFO:teuthology.orchestra.run.smithi045.stdout: "name": { 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "client", 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 15336 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3445943621 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-12T05:10:37.159 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3445943621 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "inst_str": "client.15336 192.168.0.1:0/3445943621", 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "addr_str": "192.168.0.1:0/3445943621", 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "inode_count": 5, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_epoch": 24, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch": 87, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch_barrier": 0, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklisted": false, 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "cephfs" 2024-03-12T05:10:37.160 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:37.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:37.166 DEBUG:teuthology.orchestra.run.smithi045:> stat --file-system '--printf=%T 2024-03-12T05:10:37.166 DEBUG:teuthology.orchestra.run.smithi045:> ' -- /tmp/cephfs 2024-03-12T05:10:37.181 INFO:teuthology.orchestra.run.smithi045.stdout:fuseblk 2024-03-12T05:10:37.181 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/cephfs 2024-03-12T05:10:37.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:37.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /tmp/cephfs 2024-03-12T05:10:37.251 INFO:teuthology.run_tasks:Running task exec... 2024-03-12T05:10:37.263 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:10:37.263 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:10:37.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-12T05:10:37.314 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:37 smithi045 ceph-mon[32174]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- 172.21.15.45:0/2666488832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 msgr2=0x7fb8841084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 --2- 172.21.15.45:0/2666488832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb8841084f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009ca0 tx=0x7fb87c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- 172.21.15.45:0/2666488832 shutdown_connections 2024-03-12T05:10:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 --2- 172.21.15.45:0/2666488832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb8841084f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 --2- 172.21.15.45:0/2666488832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8841012a0 0x7fb884101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 --2- 172.21.15.45:0/2666488832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb8841004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- 172.21.15.45:0/2666488832 >> 172.21.15.45:0/2666488832 conn(0x7fb8840fb620 msgr2=0x7fb8840fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- 172.21.15.45:0/2666488832 shutdown_connections 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- 172.21.15.45:0/2666488832 wait complete. 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 Processor -- start 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.397+0000 7fb88b08d640 1 -- start start 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb884196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8841012a0 0x7fb884197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88419ead0 con 0x7fb884101c60 2024-03-12T05:10:37.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88419ec40 con 0x7fb8841012a0 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88419ee00 con 0x7fb8841000b0 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33210/0 (socket says 172.21.15.45:33210) 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 -- 172.21.15.45:0/1548677780 learned_addr learned my addr 172.21.15.45:0/1548677780 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb888e29640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb884196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 -- 172.21.15.45:0/1548677780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 msgr2=0x7fb884196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb884196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 -- 172.21.15.45:0/1548677780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8841012a0 msgr2=0x7fb884197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8841012a0 0x7fb884197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 -- 172.21.15.45:0/1548677780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009900 con 0x7fb884101c60 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb888e29640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb884196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88962a640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb87c037040 tx=0x7fb87c032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c003970 con 0x7fb884101c60 2024-03-12T05:10:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb87c004040 con 0x7fb884101c60 2024-03-12T05:10:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88419f080 con 0x7fb884101c60 2024-03-12T05:10:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.398+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03d630 con 0x7fb884101c60 2024-03-12T05:10:37.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.399+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88419f530 con 0x7fb884101c60 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.400+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88410b970 con 0x7fb884101c60 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.401+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb87c03b040 con 0x7fb884101c60 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.401+0000 7fb881ffb640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 0x7fb8740a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.401+0000 7fb888e29640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 0x7fb8740a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.401+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb87c0efdc0 con 0x7fb884101c60 2024-03-12T05:10:37.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.402+0000 7fb888e29640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 0x7fb8740a3f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8780058c0 tx=0x7fb878005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:37.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.406+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb87c0b8fe0 con 0x7fb884101c60 2024-03-12T05:10:37.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:37 smithi130 ceph-mon[35398]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:37.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:37 smithi116 ceph-mon[35486]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-12T05:10:37.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:37.627+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fb88419f750 con 0x7fb884101c60 2024-03-12T05:10:38.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.193+0000 7fb881ffb640 1 -- 172.21.15.45:0/1548677780 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb87c0bc6a0 con 0x7fb884101c60 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 msgr2=0x7fb8740a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 0x7fb8740a3f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8780058c0 tx=0x7fb878005ec0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 msgr2=0x7fb88419e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb87c037040 tx=0x7fb87c032980 comp rx=0 tx=0).stop 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 shutdown_connections 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb8740a1ad0 0x7fb8740a3f90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb884101c60 0x7fb88419e590 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8841012a0 0x7fb884197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 --2- 172.21.15.45:0/1548677780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8841000b0 0x7fb884196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 >> 172.21.15.45:0/1548677780 conn(0x7fb8840fb620 msgr2=0x7fb8840fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.195+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 shutdown_connections 2024-03-12T05:10:38.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.196+0000 7fb88b08d640 1 -- 172.21.15.45:0/1548677780 wait complete. 2024-03-12T05:10:38.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-12T05:10:38.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 -- 172.21.15.45:0/3013794641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 msgr2=0x7f1fb81095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3013794641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb81095f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009900 tx=0x7f1fb002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 -- 172.21.15.45:0/3013794641 shutdown_connections 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3013794641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb81095f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3013794641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 0x7f1fb8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3013794641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 0x7f1fb81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 -- 172.21.15.45:0/3013794641 >> 172.21.15.45:0/3013794641 conn(0x7f1fb80fc740 msgr2=0x7f1fb80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 -- 172.21.15.45:0/3013794641 shutdown_connections 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 -- 172.21.15.45:0/3013794641 wait complete. 2024-03-12T05:10:38.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.354+0000 7f1fbd022640 1 Processor -- start 2024-03-12T05:10:38.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 -- start start 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 0x7f1fb81970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 0x7f1fb8197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb819ebc0 con 0x7f1fb8102d60 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb819ed30 con 0x7f1fb81011b0 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fbd022640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb819eef0 con 0x7f1fb81023a0 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47394/0 (socket says 172.21.15.45:47394) 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 -- 172.21.15.45:0/3120792489 learned_addr learned my addr 172.21.15.45:0/3120792489 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb6d9d640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 0x7f1fb81970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb659c640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 0x7f1fb8197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 -- 172.21.15.45:0/3120792489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 msgr2=0x7f1fb8197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 0x7f1fb8197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 -- 172.21.15.45:0/3120792489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 msgr2=0x7f1fb81970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 0x7f1fb81970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.355+0000 7f1fb759e640 1 -- 172.21.15.45:0/3120792489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0009560 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1fb759e640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009430 tx=0x7f1fb00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00044e0 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1fb002de90 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb819f170 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0005430 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.356+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb819f680 con 0x7f1fb8102d60 2024-03-12T05:10:38.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.357+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb8108470 con 0x7f1fb8102d60 2024-03-12T05:10:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.358+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1fb0005590 con 0x7f1fb8102d60 2024-03-12T05:10:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.358+0000 7f1faffff640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 0x7f1fa40a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.359+0000 7f1fb6d9d640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 0x7f1fa40a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.359+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1fb00efb50 con 0x7f1fb8102d60 2024-03-12T05:10:38.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.359+0000 7f1fb6d9d640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 0x7f1fa40a3f90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8102200 tx=0x7f1fa800a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.362+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1fb00b8d10 con 0x7f1fb8102d60 2024-03-12T05:10:38.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:38 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-12T05:10:38.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:38 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-12T05:10:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.570+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fb802ceb0 con 0x7f1fa40a1ad0 2024-03-12T05:10:38.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-12T05:10:38.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.588+0000 7f1faffff640 1 -- 172.21.15.45:0/3120792489 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f1fb802ceb0 con 0x7f1fa40a1ad0 2024-03-12T05:10:38.586 INFO:teuthology.orchestra.run.smithi045.stdout:{} 2024-03-12T05:10:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 msgr2=0x7f1fa40a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 0x7f1fa40a3f90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8102200 tx=0x7f1fa800a590 comp rx=0 tx=0).stop 2024-03-12T05:10:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 msgr2=0x7f1fb819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0009430 tx=0x7f1fb00042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 shutdown_connections 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1fa40a1ad0 0x7f1fa40a3f90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fb8102d60 0x7f1fb819e680 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fb81023a0 0x7f1fb8197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 --2- 172.21.15.45:0/3120792489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1fb81011b0 0x7f1fb81970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.590+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 >> 172.21.15.45:0/3120792489 conn(0x7f1fb80fc740 msgr2=0x7f1fb8105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.591+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 shutdown_connections 2024-03-12T05:10:38.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.591+0000 7f1fbd022640 1 -- 172.21.15.45:0/3120792489 wait complete. 2024-03-12T05:10:38.599 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-12T05:10:38.611 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-12T05:10:38.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:38.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.731+0000 7f87dbdbc640 1 -- 172.21.15.45:0/1431672492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 msgr2=0x7f87d4101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.731+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/1431672492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4101710 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009900 tx=0x7f87c402bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:38.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- 172.21.15.45:0/1431672492 shutdown_connections 2024-03-12T05:10:38.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/1431672492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4101710 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/1431672492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4101c50 0x7f87d41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/1431672492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d41000a0 0x7f87d41004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- 172.21.15.45:0/1431672492 >> 172.21.15.45:0/1431672492 conn(0x7f87d40fb610 msgr2=0x7f87d40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- 172.21.15.45:0/1431672492 shutdown_connections 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- 172.21.15.45:0/1431672492 wait complete. 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 Processor -- start 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- start start 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d41000a0 0x7f87d4104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d4101c50 0x7f87d4102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d4103310 con 0x7f87d4101290 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d4103480 con 0x7f87d41000a0 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87dbdbc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d41a11e0 con 0x7f87d4101c50 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87d9357640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87d9357640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47436/0 (socket says 172.21.15.45:47436) 2024-03-12T05:10:38.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.732+0000 7f87d9357640 1 -- 172.21.15.45:0/3854686404 learned_addr learned my addr 172.21.15.45:0/3854686404 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9b58640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d41000a0 0x7f87d4104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87da359640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d4101c50 0x7f87d4102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 -- 172.21.15.45:0/3854686404 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d4101c50 msgr2=0x7f87d4102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d4101c50 0x7f87d4102bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 -- 172.21.15.45:0/3854686404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d41000a0 msgr2=0x7f87d4104030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d41000a0 0x7f87d4104030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 -- 172.21.15.45:0/3854686404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4009560 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87d9357640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009430 tx=0x7f87c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4004510 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f87c402de90 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d41a1430 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4003cc0 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.733+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d41a1990 con 0x7f87d4101290 2024-03-12T05:10:38.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.734+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d410b960 con 0x7f87d4101290 2024-03-12T05:10:38.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.735+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f87c4004fd0 con 0x7f87d4101290 2024-03-12T05:10:38.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.736+0000 7f87caffd640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 0x7f87c00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:38.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.736+0000 7f87d9b58640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 0x7f87c00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:38.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.736+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f87c40efd30 con 0x7f87d4101290 2024-03-12T05:10:38.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.736+0000 7f87d9b58640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 0x7f87c00a3f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87d41010f0 tx=0x7f87cc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:38.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.739+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f87c40b8ef0 con 0x7f87d4101290 2024-03-12T05:10:38.867 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:38.574+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:38.868 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:38.574+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:38.868 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:38.574+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:38.868 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:38.574+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:38.868 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:38.574+0000 7f11b8c88700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:38.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.922+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d410baf0 con 0x7f87d4101290 2024-03-12T05:10:38.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.923+0000 7f87caffd640 1 -- 172.21.15.45:0/3854686404 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f87c40bc5b0 con 0x7f87d4101290 2024-03-12T05:10:38.922 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:38.922 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:38.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 msgr2=0x7f87c00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 0x7f87c00a3f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87d41010f0 tx=0x7f87cc00a2d0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 msgr2=0x7f87d4102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009430 tx=0x7f87c40042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 shutdown_connections 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f87c00a1ab0 0x7f87c00a3f70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87d4101290 0x7f87d4102670 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87d4101c50 0x7f87d4102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 --2- 172.21.15.45:0/3854686404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d41000a0 0x7f87d4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 >> 172.21.15.45:0/3854686404 conn(0x7f87d40fb610 msgr2=0x7f87d40fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 shutdown_connections 2024-03-12T05:10:38.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:38.925+0000 7f87dbdbc640 1 -- 172.21.15.45:0/3854686404 wait complete. 2024-03-12T05:10:38.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:38.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 -- 172.21.15.45:0/1482837643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 msgr2=0x7f0a7c101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 --2- 172.21.15.45:0/1482837643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c101700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009900 tx=0x7f0a6402bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 -- 172.21.15.45:0/1482837643 shutdown_connections 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 --2- 172.21.15.45:0/1482837643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c101700 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 --2- 172.21.15.45:0/1482837643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 0x7f0a7c1084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 --2- 172.21.15.45:0/1482837643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 0x7f0a7c1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 -- 172.21.15.45:0/1482837643 >> 172.21.15.45:0/1482837643 conn(0x7f0a7c0fb620 msgr2=0x7f0a7c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 -- 172.21.15.45:0/1482837643 shutdown_connections 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 -- 172.21.15.45:0/1482837643 wait complete. 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.042+0000 7f0a8194f640 1 Processor -- start 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 -- start start 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 0x7f0a7c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 0x7f0a7c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c19ebe0 con 0x7f0a7c101280 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c19ed50 con 0x7f0a7c100090 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8194f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c19ef10 con 0x7f0a7c101c40 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47446/0 (socket says 172.21.15.45:47446) 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 -- 172.21.15.45:0/3105015333 learned_addr learned my addr 172.21.15.45:0/3105015333 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7b7fe640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 0x7f0a7c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7affd640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 0x7f0a7c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 -- 172.21.15.45:0/3105015333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 msgr2=0x7f0a7c19e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 0x7f0a7c19e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 -- 172.21.15.45:0/3105015333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 msgr2=0x7f0a7c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 0x7f0a7c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 -- 172.21.15.45:0/3105015333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64009560 con 0x7f0a7c101280 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a7a7fc640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0a64036040 tx=0x7f0a64004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.043+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640035f0 con 0x7f0a7c101280 2024-03-12T05:10:39.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.044+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a6402de90 con 0x7f0a7c101280 2024-03-12T05:10:39.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.044+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c19f190 con 0x7f0a7c101280 2024-03-12T05:10:39.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.044+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64005490 con 0x7f0a7c101280 2024-03-12T05:10:39.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.044+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c19f6a0 con 0x7f0a7c101280 2024-03-12T05:10:39.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.045+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c1073b0 con 0x7f0a7c101280 2024-03-12T05:10:39.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.046+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0a64003750 con 0x7f0a7c101280 2024-03-12T05:10:39.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.046+0000 7f0a8094d640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 0x7f0a680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.047+0000 7f0a7affd640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 0x7f0a680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.047+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f0a640ef6f0 con 0x7f0a7c101280 2024-03-12T05:10:39.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.047+0000 7f0a7affd640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 0x7f0a680a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a740058c0 tx=0x7f0a74005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.051+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a640b8910 con 0x7f0a7c101280 2024-03-12T05:10:39.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.235+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a7c02ce40 con 0x7f0a7c101280 2024-03-12T05:10:39.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.235+0000 7f0a8094d640 1 -- 172.21.15.45:0/3105015333 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a640bbfd0 con 0x7f0a7c101280 2024-03-12T05:10:39.233 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 msgr2=0x7f0a680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 0x7f0a680a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a740058c0 tx=0x7f0a74005850 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 msgr2=0x7f0a7c19be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0a64036040 tx=0x7f0a64004140 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 shutdown_connections 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f0a680a1a30 0x7f0a680a3ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a7c101280 0x7f0a7c19be10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a7c101c40 0x7f0a7c19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 --2- 172.21.15.45:0/3105015333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a7c100090 0x7f0a7c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 >> 172.21.15.45:0/3105015333 conn(0x7f0a7c0fb620 msgr2=0x7f0a7c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 shutdown_connections 2024-03-12T05:10:39.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.237+0000 7f0a8194f640 1 -- 172.21.15.45:0/3105015333 wait complete. 2024-03-12T05:10:39.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 -- 172.21.15.45:0/816039355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14100080 msgr2=0x7f8a141004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 --2- 172.21.15.45:0/816039355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14100080 0x7f8a141004a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009900 tx=0x7f8a1002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 -- 172.21.15.45:0/816039355 shutdown_connections 2024-03-12T05:10:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 --2- 172.21.15.45:0/816039355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a14101c30 0x7f8a141084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 --2- 172.21.15.45:0/816039355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14101270 0x7f8a141016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 --2- 172.21.15.45:0/816039355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14100080 0x7f8a141004a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 -- 172.21.15.45:0/816039355 >> 172.21.15.45:0/816039355 conn(0x7f8a140fb610 msgr2=0x7f8a140fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 -- 172.21.15.45:0/816039355 shutdown_connections 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 -- 172.21.15.45:0/816039355 wait complete. 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.354+0000 7f8a1b815640 1 Processor -- start 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 -- start start 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 0x7f8a141971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a14101270 0x7f8a14197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1419ecc0 con 0x7f8a14101c30 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1419ee30 con 0x7f8a14100080 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a1b815640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1419eff0 con 0x7f8a14101270 2024-03-12T05:10:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47460/0 (socket says 172.21.15.45:47460) 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 -- 172.21.15.45:0/3406671457 learned_addr learned my addr 172.21.15.45:0/3406671457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a195b1640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 0x7f8a141971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 -- 172.21.15.45:0/3406671457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a14101270 msgr2=0x7f8a14197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a14101270 0x7f8a14197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 -- 172.21.15.45:0/3406671457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 msgr2=0x7f8a141971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 0x7f8a141971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 -- 172.21.15.45:0/3406671457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a04009490 con 0x7f8a14101c30 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a195b1640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 0x7f8a141971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a19db2640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00eb10 tx=0x7f8a0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c004b20 con 0x7f8a14101c30 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.356+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a10009560 con 0x7f8a14101c30 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.355+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8a0c004c80 con 0x7f8a14101c30 2024-03-12T05:10:39.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.356+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c010690 con 0x7f8a14101c30 2024-03-12T05:10:39.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.356+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1419f650 con 0x7f8a14101c30 2024-03-12T05:10:39.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.356+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1410b940 con 0x7f8a14101c30 2024-03-12T05:10:39.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.358+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8a0c004df0 con 0x7f8a14101c30 2024-03-12T05:10:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.358+0000 7f8a0a7fc640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 0x7f8a000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.359+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f8a0c014070 con 0x7f8a14101c30 2024-03-12T05:10:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.359+0000 7f8a195b1640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 0x7f8a000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.359+0000 7f8a195b1640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 0x7f8a000a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006010 tx=0x7f8a10005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.362+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8a0c096f00 con 0x7f8a14101c30 2024-03-12T05:10:39.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-12T05:10:39.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:39 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:10:39.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:39 smithi130 ceph-mon[35398]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:39.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3854686404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:39.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:39 smithi130 ceph-mon[35398]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-12T05:10:39.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-12T05:10:39.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:39 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:10:39.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:39 smithi116 ceph-mon[35486]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:39.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3854686404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:39.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:39 smithi116 ceph-mon[35486]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-12T05:10:39.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.544+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a1419f8d0 con 0x7f8a14101c30 2024-03-12T05:10:39.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.545+0000 7f8a0a7fc640 1 -- 172.21.15.45:0/3406671457 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a0c09a5c0 con 0x7f8a14101c30 2024-03-12T05:10:39.542 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:39.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 msgr2=0x7f8a000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 0x7f8a000a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006010 tx=0x7f8a10005c00 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 msgr2=0x7f8a1419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00eb10 tx=0x7f8a0c00c2d0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 shutdown_connections 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f8a000a1a30 0x7f8a000a3ef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a14101c30 0x7f8a1419e780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a14101270 0x7f8a14197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 --2- 172.21.15.45:0/3406671457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a14100080 0x7f8a141971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 >> 172.21.15.45:0/3406671457 conn(0x7f8a140fb610 msgr2=0x7f8a140682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 shutdown_connections 2024-03-12T05:10:39.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.547+0000 7f8a1b815640 1 -- 172.21.15.45:0/3406671457 wait complete. 2024-03-12T05:10:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1548677780' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-12T05:10:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:39 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:10:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:39 smithi045 ceph-mon[32174]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3854686404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:39.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:39 smithi045 ceph-mon[32174]: pgmap v345: 97 pgs: 97 active+clean; 648 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-12T05:10:39.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 -- 172.21.15.45:0/1015598690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 msgr2=0x7f7350101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 --2- 172.21.15.45:0/1015598690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350101710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7340009900 tx=0x7f734002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 -- 172.21.15.45:0/1015598690 shutdown_connections 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 --2- 172.21.15.45:0/1015598690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350101710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 --2- 172.21.15.45:0/1015598690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7350101c50 0x7f73501084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 --2- 172.21.15.45:0/1015598690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73501000a0 0x7f73501004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 -- 172.21.15.45:0/1015598690 >> 172.21.15.45:0/1015598690 conn(0x7f73500fb630 msgr2=0x7f73500fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 -- 172.21.15.45:0/1015598690 shutdown_connections 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.664+0000 7f7358725640 1 -- 172.21.15.45:0/1015598690 wait complete. 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 Processor -- start 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 -- start start 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 0x7f7350197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7350101c50 0x7f735019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735019edf0 con 0x7f7350101290 2024-03-12T05:10:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735019ef60 con 0x7f73501000a0 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7358725640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735019f120 con 0x7f7350101c50 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47488/0 (socket says 172.21.15.45:47488) 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 -- 172.21.15.45:0/3340270124 learned_addr learned my addr 172.21.15.45:0/3340270124 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7356cc2640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7350101c50 0x7f735019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f73564c1640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 0x7f7350197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 -- 172.21.15.45:0/3340270124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7350101c50 msgr2=0x7f735019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7350101c50 0x7f735019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 -- 172.21.15.45:0/3340270124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 msgr2=0x7f7350197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 0x7f7350197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 -- 172.21.15.45:0/3340270124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7340009560 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f73564c1640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 0x7f7350197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.665+0000 7f7355cc0640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7340036040 tx=0x7f7340004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.666+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340003800 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.666+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f734002de90 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.666+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735019f3a0 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.666+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340005600 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.666+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735019f930 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.667+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735010ba40 con 0x7f7350101290 2024-03-12T05:10:39.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.668+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7340004dc0 con 0x7f7350101290 2024-03-12T05:10:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.668+0000 7f73477fe640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 0x7f733c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.668+0000 7f73564c1640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 0x7f733c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.668+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f73400effb0 con 0x7f7350101290 2024-03-12T05:10:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.669+0000 7f73564c1640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 0x7f733c0a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f73501010f0 tx=0x7f734c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.672+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f73400b92f0 con 0x7f7350101290 2024-03-12T05:10:39.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.851+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f735019fd60 con 0x7f7350101290 2024-03-12T05:10:39.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.852+0000 7f73477fe640 1 -- 172.21.15.45:0/3340270124 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f73400430a0 con 0x7f7350101290 2024-03-12T05:10:39.849 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:39.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 msgr2=0x7f733c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 0x7f733c0a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f73501010f0 tx=0x7f734c008040 comp rx=0 tx=0).stop 2024-03-12T05:10:39.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 msgr2=0x7f7350197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7340036040 tx=0x7f7340004370 comp rx=0 tx=0).stop 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 shutdown_connections 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f733c0a1a30 0x7f733c0a3ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7350101290 0x7f7350197840 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7350101c50 0x7f735019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 --2- 172.21.15.45:0/3340270124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73501000a0 0x7f7350197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 >> 172.21.15.45:0/3340270124 conn(0x7f73500fb630 msgr2=0x7f7350069800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 shutdown_connections 2024-03-12T05:10:39.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.854+0000 7f7358725640 1 -- 172.21.15.45:0/3340270124 wait complete. 2024-03-12T05:10:39.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:39.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 -- 172.21.15.45:0/333552945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ffbe0 msgr2=0x7f0984109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 --2- 172.21.15.45:0/333552945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ffbe0 0x7f0984109790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f097c009ca0 tx=0x7f097c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 -- 172.21.15.45:0/333552945 shutdown_connections 2024-03-12T05:10:39.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 --2- 172.21.15.45:0/333552945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ffbe0 0x7f0984109790 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 --2- 172.21.15.45:0/333552945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09840ff220 0x7f09840ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 --2- 172.21.15.45:0/333552945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09841027b0 0x7f0984102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 -- 172.21.15.45:0/333552945 >> 172.21.15.45:0/333552945 conn(0x7f09840fada0 msgr2=0x7f09840fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 -- 172.21.15.45:0/333552945 shutdown_connections 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.967+0000 7f098a69d640 1 -- 172.21.15.45:0/333552945 wait complete. 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 Processor -- start 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 -- start start 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09840ffbe0 0x7f0984197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09841027b0 0x7f098419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419ec10 con 0x7f09840ff220 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419ed80 con 0x7f09841027b0 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f098a69d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098419ef40 con 0x7f09840ffbe0 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f0983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f0983fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47498/0 (socket says 172.21.15.45:47498) 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.968+0000 7f0983fff640 1 -- 172.21.15.45:0/2023348936 learned_addr learned my addr 172.21.15.45:0/2023348936 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f09837fe640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09840ffbe0 0x7f0984197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0988c3a640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09841027b0 0x7f098419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 -- 172.21.15.45:0/2023348936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09840ffbe0 msgr2=0x7f0984197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09840ffbe0 0x7f0984197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 -- 172.21.15.45:0/2023348936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09841027b0 msgr2=0x7f098419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09841027b0 0x7f098419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 -- 172.21.15.45:0/2023348936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c009900 con 0x7f09840ff220 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f0983fff640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f097800b450 tx=0x7f097800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978011890 con 0x7f09840ff220 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098419f220 con 0x7f09840ff220 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09780103e0 con 0x7f09840ff220 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978011ea0 con 0x7f09840ff220 2024-03-12T05:10:39.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.969+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098419f7b0 con 0x7f09840ff220 2024-03-12T05:10:39.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.970+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0984108610 con 0x7f09840ff220 2024-03-12T05:10:39.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.971+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0978010550 con 0x7f09840ff220 2024-03-12T05:10:39.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.972+0000 7f09817fa640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 0x7f096c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:39.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.972+0000 7f09837fe640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 0x7f096c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:39.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.972+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f09780cd2b0 con 0x7f09840ff220 2024-03-12T05:10:39.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.972+0000 7f09837fe640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 0x7f096c0a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0974005e10 tx=0x7f0974005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:39.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:39.976+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0978096470 con 0x7f09840ff220 2024-03-12T05:10:40.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.160+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f098402cdd0 con 0x7f09840ff220 2024-03-12T05:10:40.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.161+0000 7f09817fa640 1 -- 172.21.15.45:0/2023348936 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f0978009280 con 0x7f09840ff220 2024-03-12T05:10:40.159 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:40.159 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.163+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 msgr2=0x7f096c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.163+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 0x7f096c0a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0974005e10 tx=0x7f0974005da0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.163+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 msgr2=0x7f0984197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f097800b450 tx=0x7f097800b910 comp rx=0 tx=0).stop 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 shutdown_connections 2024-03-12T05:10:40.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f096c0a1a30 0x7f096c0a3ef0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09841027b0 0x7f098419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09840ffbe0 0x7f0984197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 --2- 172.21.15.45:0/2023348936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09840ff220 0x7f0984197120 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 >> 172.21.15.45:0/2023348936 conn(0x7f09840fada0 msgr2=0x7f09841057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 shutdown_connections 2024-03-12T05:10:40.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.164+0000 7f098a69d640 1 -- 172.21.15.45:0/2023348936 wait complete. 2024-03-12T05:10:40.172 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:40.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 -- 172.21.15.45:0/1129179400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 msgr2=0x7f75c4109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 --2- 172.21.15.45:0/1129179400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4109780 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009ca0 tx=0x7f75bc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 -- 172.21.15.45:0/1129179400 shutdown_connections 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 --2- 172.21.15.45:0/1129179400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4109780 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 --2- 172.21.15.45:0/1129179400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c40ff210 0x7f75c40ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 --2- 172.21.15.45:0/1129179400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c41027a0 0x7f75c4102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 -- 172.21.15.45:0/1129179400 >> 172.21.15.45:0/1129179400 conn(0x7f75c40fadb0 msgr2=0x7f75c40fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 -- 172.21.15.45:0/1129179400 shutdown_connections 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.282+0000 7f75cbd77640 1 -- 172.21.15.45:0/1129179400 wait complete. 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 Processor -- start 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 -- start start 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c40ff210 0x7f75c4197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 0x7f75c419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c419ec10 con 0x7f75c40ffbd0 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c419ed80 con 0x7f75c40ff210 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75cbd77640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c419ef40 con 0x7f75c41027a0 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47522/0 (socket says 172.21.15.45:47522) 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 -- 172.21.15.45:0/3430366725 learned_addr learned my addr 172.21.15.45:0/3430366725 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75ca314640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 0x7f75c419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9b13640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c40ff210 0x7f75c4197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 -- 172.21.15.45:0/3430366725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 msgr2=0x7f75c419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 0x7f75c419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 -- 172.21.15.45:0/3430366725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c40ff210 msgr2=0x7f75c4197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c40ff210 0x7f75c4197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 -- 172.21.15.45:0/3430366725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc009900 con 0x7f75c40ffbd0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75ca314640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 0x7f75c419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.283+0000 7f75c9312640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f75b400d770 tx=0x7f75b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.284+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4009c30 con 0x7f75c40ffbd0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.284+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f75b400de80 con 0x7f75c40ffbd0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.284+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c419f220 con 0x7f75c40ffbd0 2024-03-12T05:10:40.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.284+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4004e00 con 0x7f75c40ffbd0 2024-03-12T05:10:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.284+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c419f6e0 con 0x7f75c40ffbd0 2024-03-12T05:10:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.285+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4108660 con 0x7f75c40ffbd0 2024-03-12T05:10:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.286+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f75b4004f60 con 0x7f75c40ffbd0 2024-03-12T05:10:40.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.286+0000 7f75baffd640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 0x7f75b00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.287+0000 7f75c9b13640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 0x7f75b00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.287+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f75b40ccf30 con 0x7f75c40ffbd0 2024-03-12T05:10:40.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.287+0000 7f75c9b13640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 0x7f75b00a3ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005fd0 tx=0x7f75c000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.291+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f75b4096150 con 0x7f75c40ffbd0 2024-03-12T05:10:40.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.360+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f75b4096520 con 0x7f75c40ffbd0 2024-03-12T05:10:40.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3105015333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3406671457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3340270124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:40 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2023348936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.476+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75c402cda0 con 0x7f75c40ffbd0 2024-03-12T05:10:40.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.477+0000 7f75baffd640 1 -- 172.21.15.45:0/3430366725 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75b405e550 con 0x7f75c40ffbd0 2024-03-12T05:10:40.474 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 msgr2=0x7f75b00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 0x7f75b00a3ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005fd0 tx=0x7f75c000b5b0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 msgr2=0x7f75c4197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f75b400d770 tx=0x7f75b400dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 shutdown_connections 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f75b00a1a30 0x7f75b00a3ef0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75c40ffbd0 0x7f75c4197660 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c41027a0 0x7f75c419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 --2- 172.21.15.45:0/3430366725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75c40ff210 0x7f75c4197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 >> 172.21.15.45:0/3430366725 conn(0x7f75c40fadb0 msgr2=0x7f75c4105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 shutdown_connections 2024-03-12T05:10:40.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.479+0000 7f75cbd77640 1 -- 172.21.15.45:0/3430366725 wait complete. 2024-03-12T05:10:40.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3105015333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3406671457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3340270124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.486 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:40 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2023348936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:40.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:40.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3105015333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3406671457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3340270124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:40.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:40 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2023348936' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1447993713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 msgr2=0x7fb764104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1447993713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764104050 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb760009900 tx=0x7fb76002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1447993713 shutdown_connections 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1447993713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb764106ec0 0x7fb7641092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1447993713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1447993713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764104050 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1447993713 >> 172.21.15.45:0/1447993713 conn(0x7fb7640fb610 msgr2=0x7fb7640fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1447993713 shutdown_connections 2024-03-12T05:10:40.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.595+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1447993713 wait complete. 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 Processor -- start 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 -- start start 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb764106ec0 0x7fb76419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76419ed30 con 0x7fb764101c60 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76419eea0 con 0x7fb764104590 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb76bc9d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76419f060 con 0x7fb764106ec0 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47548/0 (socket says 172.21.15.45:47548) 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 -- 172.21.15.45:0/1152204072 learned_addr learned my addr 172.21.15.45:0/1152204072 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769238640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 -- 172.21.15.45:0/1152204072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb764106ec0 msgr2=0x7fb76419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb764106ec0 0x7fb76419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 -- 172.21.15.45:0/1152204072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 msgr2=0x7fb764197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769a39640 1 -- 172.21.15.45:0/1152204072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760009560 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.596+0000 7fb769238640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb769a39640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb760036040 tx=0x7fb7600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600044e0 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb76002de90 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760005470 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76419f2e0 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.597+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76419f870 con 0x7fb764101c60 2024-03-12T05:10:40.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.598+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7640fd100 con 0x7fb764101c60 2024-03-12T05:10:40.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.599+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb760003800 con 0x7fb764101c60 2024-03-12T05:10:40.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.599+0000 7fb75affd640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 0x7fb7500a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.599+0000 7fb769238640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 0x7fb7500a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.599+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fb7600ef7a0 con 0x7fb764101c60 2024-03-12T05:10:40.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.600+0000 7fb769238640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 0x7fb7500a3f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb754006fd0 tx=0x7fb754008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.604+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb7600b8970 con 0x7fb764101c60 2024-03-12T05:10:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.783+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb76402cea0 con 0x7fb764101c60 2024-03-12T05:10:40.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.784+0000 7fb75affd640 1 -- 172.21.15.45:0/1152204072 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7600bc030 con 0x7fb764101c60 2024-03-12T05:10:40.781 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 msgr2=0x7fb7500a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 0x7fb7500a3f80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb754006fd0 tx=0x7fb754008040 comp rx=0 tx=0).stop 2024-03-12T05:10:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 msgr2=0x7fb764197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb760036040 tx=0x7fb7600042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 shutdown_connections 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fb7500a1ac0 0x7fb7500a3f80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb764106ec0 0x7fb76419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb764104590 0x7fb764197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 --2- 172.21.15.45:0/1152204072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb764101c60 0x7fb764197240 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 >> 172.21.15.45:0/1152204072 conn(0x7fb7640fb610 msgr2=0x7fb764103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 shutdown_connections 2024-03-12T05:10:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.786+0000 7fb76bc9d640 1 -- 172.21.15.45:0/1152204072 wait complete. 2024-03-12T05:10:40.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:40.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 -- 172.21.15.45:0/3461125579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680101190 msgr2=0x7f96801015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 --2- 172.21.15.45:0/3461125579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680101190 0x7f96801015b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f967c009900 tx=0x7f967c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 -- 172.21.15.45:0/3461125579 shutdown_connections 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 --2- 172.21.15.45:0/3461125579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680102d40 0x7f96801095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 --2- 172.21.15.45:0/3461125579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102380 0x7f9680102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 --2- 172.21.15.45:0/3461125579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680101190 0x7f96801015b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 -- 172.21.15.45:0/3461125579 >> 172.21.15.45:0/3461125579 conn(0x7f96800fc720 msgr2=0x7f96800feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 -- 172.21.15.45:0/3461125579 shutdown_connections 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.902+0000 7f968877a640 1 -- 172.21.15.45:0/3461125579 wait complete. 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 Processor -- start 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 -- start start 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680101190 0x7f9680197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102d40 0x7f968019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019ed10 con 0x7f9680102380 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019ee80 con 0x7f9680102d40 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f968877a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019f040 con 0x7f9680101190 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47574/0 (socket says 172.21.15.45:47574) 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 -- 172.21.15.45:0/4270332817 learned_addr learned my addr 172.21.15.45:0/4270332817 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9686516640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680101190 0x7f9680197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9686d17640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102d40 0x7f968019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 -- 172.21.15.45:0/4270332817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680101190 msgr2=0x7f9680197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680101190 0x7f9680197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 -- 172.21.15.45:0/4270332817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102d40 msgr2=0x7f968019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102d40 0x7f968019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 -- 172.21.15.45:0/4270332817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c009560 con 0x7f9680102380 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.903+0000 7f9685d15640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f967000c890 tx=0x7f967000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.904+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670004220 con 0x7f9680102380 2024-03-12T05:10:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.904+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968019f320 con 0x7f9680102380 2024-03-12T05:10:40.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.904+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9670007e60 con 0x7f9680102380 2024-03-12T05:10:40.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.904+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670004a00 con 0x7f9680102380 2024-03-12T05:10:40.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.904+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968019f8b0 con 0x7f9680102380 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.905+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9680108580 con 0x7f9680102380 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.906+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f96700079b0 con 0x7f9680102380 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.906+0000 7f96777fe640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 0x7f966c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.906+0000 7f9686516640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 0x7f966c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.906+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f96700cd280 con 0x7f9680102380 2024-03-12T05:10:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.907+0000 7f9686516640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 0x7f966c0a3f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f967c02c7d0 tx=0x7f967c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:40.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:40.911+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9670096340 con 0x7f9680102380 2024-03-12T05:10:41.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.094+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f968010cce0 con 0x7f9680102380 2024-03-12T05:10:41.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.095+0000 7f96777fe640 1 -- 172.21.15.45:0/4270332817 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f967001d020 con 0x7f9680102380 2024-03-12T05:10:41.092 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:41.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 msgr2=0x7f966c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 0x7f966c0a3f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f967c02c7d0 tx=0x7f967c005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 msgr2=0x7f9680197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f967000c890 tx=0x7f967000cd50 comp rx=0 tx=0).stop 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 shutdown_connections 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f966c0a1ac0 0x7f966c0a3f80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9680102380 0x7f9680197760 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9680102d40 0x7f968019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 --2- 172.21.15.45:0/4270332817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9680101190 0x7f9680197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 >> 172.21.15.45:0/4270332817 conn(0x7f96800fc720 msgr2=0x7f9680105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 shutdown_connections 2024-03-12T05:10:41.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.097+0000 7f968877a640 1 -- 172.21.15.45:0/4270332817 wait complete. 2024-03-12T05:10:41.105 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-12T05:10:41.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/3426566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 msgr2=0x7f2edc1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/3426566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc1004c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009900 tx=0x7f2ecc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/3426566446 shutdown_connections 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/3426566446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101c50 0x7f2edc1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/3426566446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101290 0x7f2edc101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/3426566446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc1004c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.210+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/3426566446 >> 172.21.15.45:0/3426566446 conn(0x7f2edc0fb610 msgr2=0x7f2edc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/3426566446 shutdown_connections 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/3426566446 wait complete. 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 Processor -- start 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- start start 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101290 0x7f2edc1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 0x7f2edc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc19ec50 con 0x7f2edc1000a0 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc19edc0 con 0x7f2edc101290 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2ee0c4e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc19ef80 con 0x7f2edc101c50 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2eda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2eda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47584/0 (socket says 172.21.15.45:47584) 2024-03-12T05:10:41.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.211+0000 7f2eda59c640 1 -- 172.21.15.45:0/1358910136 learned_addr learned my addr 172.21.15.45:0/1358910136 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ed3fff640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101290 0x7f2edc1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2edad9d640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 0x7f2edc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 -- 172.21.15.45:0/1358910136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 msgr2=0x7f2edc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 0x7f2edc19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 -- 172.21.15.45:0/1358910136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101290 msgr2=0x7f2edc1976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101290 0x7f2edc1976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 -- 172.21.15.45:0/1358910136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc009560 con 0x7f2edc1000a0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2edad9d640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 0x7f2edc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2eda59c640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc000c00 tx=0x7f2ecc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0044e0 con 0x7f2edc1000a0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2ecc02de90 con 0x7f2edc1000a0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc19f200 con 0x7f2edc1000a0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc005410 con 0x7f2edc1000a0 2024-03-12T05:10:41.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.212+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc19f710 con 0x7f2edc1000a0 2024-03-12T05:10:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.213+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc107440 con 0x7f2edc1000a0 2024-03-12T05:10:41.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.214+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2ecc005570 con 0x7f2edc1000a0 2024-03-12T05:10:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.215+0000 7f2ed37fe640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 0x7f2ec80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.215+0000 7f2ed3fff640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 0x7f2ec80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.215+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2ecc0ef6b0 con 0x7f2edc1000a0 2024-03-12T05:10:41.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.216+0000 7f2ed3fff640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 0x7f2ec80a3f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2edc198670 tx=0x7f2ec400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.220+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ecc0b8880 con 0x7f2edc1000a0 2024-03-12T05:10:41.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:41 smithi045 ceph-mon[32174]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:10:41.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:41 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3430366725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:41 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1152204072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:41 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4270332817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:41 smithi045 ceph-mon[32174]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-12T05:10:41.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.441+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2edc02cdd0 con 0x7f2edc1000a0 2024-03-12T05:10:41.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.441+0000 7f2ed37fe640 1 -- 172.21.15.45:0/1358910136 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f2ecc0bbf40 con 0x7f2edc1000a0 2024-03-12T05:10:41.439 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:41.439 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:10:37.628388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:41.439 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 25 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 msgr2=0x7f2ec80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 0x7f2ec80a3f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2edc198670 tx=0x7f2ec400b390 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 msgr2=0x7f2edc197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc000c00 tx=0x7f2ecc0042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 shutdown_connections 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2ec80a1ac0 0x7f2ec80a3f80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2edc101c50 0x7f2edc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2edc101290 0x7f2edc1976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 --2- 172.21.15.45:0/1358910136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2edc1000a0 0x7f2edc197160 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.443+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 >> 172.21.15.45:0/1358910136 conn(0x7f2edc0fb610 msgr2=0x7f2edc104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.444+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 shutdown_connections 2024-03-12T05:10:41.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.444+0000 7f2ee0c4e640 1 -- 172.21.15.45:0/1358910136 wait complete. 2024-03-12T05:10:41.452 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-12T05:10:41.452 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-12T05:10:41.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 -- 172.21.15.45:0/104970698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef641000a0 msgr2=0x7fef641004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 --2- 172.21.15.45:0/104970698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef641000a0 0x7fef641004c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fef60009900 tx=0x7fef6002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 -- 172.21.15.45:0/104970698 shutdown_connections 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 --2- 172.21.15.45:0/104970698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef64101c50 0x7fef641084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 --2- 172.21.15.45:0/104970698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.557+0000 7fef6b713640 1 --2- 172.21.15.45:0/104970698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef641000a0 0x7fef641004c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- 172.21.15.45:0/104970698 >> 172.21.15.45:0/104970698 conn(0x7fef640fb610 msgr2=0x7fef640fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- 172.21.15.45:0/104970698 shutdown_connections 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- 172.21.15.45:0/104970698 wait complete. 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 Processor -- start 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- start start 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef641000a0 0x7fef64197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6419ec40 con 0x7fef64101c50 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6419edb0 con 0x7fef641000a0 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef6b713640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6419ef70 con 0x7fef64101290 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef69cb0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef69cb0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47608/0 (socket says 172.21.15.45:47608) 2024-03-12T05:10:41.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.558+0000 7fef69cb0640 1 -- 172.21.15.45:0/1132312960 learned_addr learned my addr 172.21.15.45:0/1132312960 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef694af640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef641000a0 0x7fef64197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef68cae640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 -- 172.21.15.45:0/1132312960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 msgr2=0x7fef64197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 -- 172.21.15.45:0/1132312960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef641000a0 msgr2=0x7fef64197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef641000a0 0x7fef64197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 -- 172.21.15.45:0/1132312960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef60009560 con 0x7fef64101c50 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef68cae640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef69cb0640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00eaa0 tx=0x7fef5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c00cdb0 con 0x7fef64101c50 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef5c005020 con 0x7fef64101c50 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6419f250 con 0x7fef64101c50 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0116b0 con 0x7fef64101c50 2024-03-12T05:10:41.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.559+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6419f710 con 0x7fef64101c50 2024-03-12T05:10:41.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.560+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef64107430 con 0x7fef64101c50 2024-03-12T05:10:41.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.561+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fef5c005190 con 0x7fef64101c50 2024-03-12T05:10:41.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.562+0000 7fef5a7fc640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 0x7fef500a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.562+0000 7fef694af640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 0x7fef500a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.562+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fef5c0cd860 con 0x7fef64101c50 2024-03-12T05:10:41.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.562+0000 7fef694af640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 0x7fef500a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fef60006010 tx=0x7fef60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.566+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef5c096920 con 0x7fef64101c50 2024-03-12T05:10:41.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:41 smithi130 ceph-mon[35398]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:10:41.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:41 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3430366725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:41 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1152204072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:41 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4270332817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:41 smithi130 ceph-mon[35398]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-12T05:10:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.744+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef6402ceb0 con 0x7fef500a1ac0 2024-03-12T05:10:41.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:41 smithi116 ceph-mon[35486]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:10:41.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:41 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3430366725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:41 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1152204072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:41 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4270332817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:41.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:41 smithi116 ceph-mon[35486]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-12T05:10:41.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.752+0000 7fef5a7fc640 1 -- 172.21.15.45:0/1132312960 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259093 (secure 0 0 0) 0x7fef6402ceb0 con 0x7fef500a1ac0 2024-03-12T05:10:41.753 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:41.754 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.754+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 msgr2=0x7fef500a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.754+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 0x7fef500a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fef60006010 tx=0x7fef60000bc0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.754+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 msgr2=0x7fef6419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.754+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00eaa0 tx=0x7fef5c00ef60 comp rx=0 tx=0).stop 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 shutdown_connections 2024-03-12T05:10:41.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fef500a1ac0 0x7fef500a3f80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef64101c50 0x7fef6419e700 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef64101290 0x7fef64197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 --2- 172.21.15.45:0/1132312960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef641000a0 0x7fef64197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 >> 172.21.15.45:0/1132312960 conn(0x7fef640fb610 msgr2=0x7fef64104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 shutdown_connections 2024-03-12T05:10:41.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.755+0000 7fef6b713640 1 -- 172.21.15.45:0/1132312960 wait complete. 2024-03-12T05:10:41.763 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:10:41.129487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665653,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1343,"num_read_kb":802,"num_write":456,"num_write_kb":3104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181408,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373340,"kb_used_data":11708,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568484,"statfs":{"total":1151940427776,"available":1151558127616,"internally_reserved":0,"allocated":11988992,"data_stored":7454604,"data_compressed":113928,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150307+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:10:13.150307+0000","last_peered":"2024-03-12T05:10:13.150307+0000","last_clean":"2024-03-12T05:10:13.150307+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:10:13.150307+0000","last_undegraded":"2024-03-12T05:10:13.150307+0000","last_fullsized":"2024-03-12T05:10:13.150307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145142+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:10:13.145142+0000","last_peered":"2024-03-12T05:10:13.145142+0000","last_clean":"2024-03-12T05:10:13.145142+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:10:13.145142+0000","last_undegraded":"2024-03-12T05:10:13.145142+0000","last_fullsized":"2024-03-12T05:10:13.145142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153971+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:10:13.153971+0000","last_peered":"2024-03-12T05:10:13.153971+0000","last_clean":"2024-03-12T05:10:13.153971+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:10:13.153971+0000","last_undegraded":"2024-03-12T05:10:13.153971+0000","last_fullsized":"2024-03-12T05:10:13.153971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.123843+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:10:13.123843+0000","last_peered":"2024-03-12T05:10:13.123843+0000","last_clean":"2024-03-12T05:10:13.123843+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:10:13.123843+0000","last_undegraded":"2024-03-12T05:10:13.123843+0000","last_fullsized":"2024-03-12T05:10:13.123843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.130074+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:10:13.130074+0000","last_peered":"2024-03-12T05:10:13.130074+0000","last_clean":"2024-03-12T05:10:13.130074+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:10:13.130074+0000","last_undegraded":"2024-03-12T05:10:13.130074+0000","last_fullsized":"2024-03-12T05:10:13.130074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143421+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:10:13.143421+0000","last_peered":"2024-03-12T05:10:13.143421+0000","last_clean":"2024-03-12T05:10:13.143421+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:10:13.143421+0000","last_undegraded":"2024-03-12T05:10:13.143421+0000","last_fullsized":"2024-03-12T05:10:13.143421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138393+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:10:13.138393+0000","last_peered":"2024-03-12T05:10:13.138393+0000","last_clean":"2024-03-12T05:10:13.138393+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:10:13.138393+0000","last_undegraded":"2024-03-12T05:10:13.138393+0000","last_fullsized":"2024-03-12T05:10:13.138393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136869+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:10:13.136869+0000","last_peered":"2024-03-12T05:10:13.136869+0000","last_clean":"2024-03-12T05:10:13.136869+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:10:13.136869+0000","last_undegraded":"2024-03-12T05:10:13.136869+0000","last_fullsized":"2024-03-12T05:10:13.136869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154745+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:10:13.154745+0000","last_peered":"2024-03-12T05:10:13.154745+0000","last_clean":"2024-03-12T05:10:13.154745+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:10:13.154745+0000","last_undegraded":"2024-03-12T05:10:13.154745+0000","last_fullsized":"2024-03-12T05:10:13.154745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143667+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:10:13.143667+0000","last_peered":"2024-03-12T05:10:13.143667+0000","last_clean":"2024-03-12T05:10:13.143667+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:10:13.143667+0000","last_undegraded":"2024-03-12T05:10:13.143667+0000","last_fullsized":"2024-03-12T05:10:13.143667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153713+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:10:13.153713+0000","last_peered":"2024-03-12T05:10:13.153713+0000","last_clean":"2024-03-12T05:10:13.153713+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:10:13.153713+0000","last_undegraded":"2024-03-12T05:10:13.153713+0000","last_fullsized":"2024-03-12T05:10:13.153713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143429+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:10:13.143429+0000","last_peered":"2024-03-12T05:10:13.143429+0000","last_clean":"2024-03-12T05:10:13.143429+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:10:13.143429+0000","last_undegraded":"2024-03-12T05:10:13.143429+0000","last_fullsized":"2024-03-12T05:10:13.143429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138246+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:10:13.138246+0000","last_peered":"2024-03-12T05:10:13.138246+0000","last_clean":"2024-03-12T05:10:13.138246+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:10:13.138246+0000","last_undegraded":"2024-03-12T05:10:13.138246+0000","last_fullsized":"2024-03-12T05:10:13.138246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145379+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:10:13.145379+0000","last_peered":"2024-03-12T05:10:13.145379+0000","last_clean":"2024-03-12T05:10:13.145379+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:10:13.145379+0000","last_undegraded":"2024-03-12T05:10:13.145379+0000","last_fullsized":"2024-03-12T05:10:13.145379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140306+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:10:13.140306+0000","last_peered":"2024-03-12T05:10:13.140306+0000","last_clean":"2024-03-12T05:10:13.140306+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:10:13.140306+0000","last_undegraded":"2024-03-12T05:10:13.140306+0000","last_fullsized":"2024-03-12T05:10:13.140306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154503+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:10:13.154503+0000","last_peered":"2024-03-12T05:10:13.154503+0000","last_clean":"2024-03-12T05:10:13.154503+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:10:13.154503+0000","last_undegraded":"2024-03-12T05:10:13.154503+0000","last_fullsized":"2024-03-12T05:10:13.154503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150237+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:10:13.150237+0000","last_peered":"2024-03-12T05:10:13.150237+0000","last_clean":"2024-03-12T05:10:13.150237+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:10:13.150237+0000","last_undegraded":"2024-03-12T05:10:13.150237+0000","last_fullsized":"2024-03-12T05:10:13.150237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144239+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:10:13.144239+0000","last_peered":"2024-03-12T05:10:13.144239+0000","last_clean":"2024-03-12T05:10:13.144239+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:10:13.144239+0000","last_undegraded":"2024-03-12T05:10:13.144239+0000","last_fullsized":"2024-03-12T05:10:13.144239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.133361+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:10:13.133361+0000","last_peered":"2024-03-12T05:10:13.133361+0000","last_clean":"2024-03-12T05:10:13.133361+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:10:13.133361+0000","last_undegraded":"2024-03-12T05:10:13.133361+0000","last_fullsized":"2024-03-12T05:10:13.133361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.202691+0000","last_change":"2024-03-12T05:09:44.142398+0000","last_active":"2024-03-12T05:10:38.202691+0000","last_peered":"2024-03-12T05:10:38.202691+0000","last_clean":"2024-03-12T05:10:38.202691+0000","last_became_active":"2024-03-12T05:09:44.129576+0000","last_became_peered":"2024-03-12T05:09:44.129576+0000","last_unstale":"2024-03-12T05:10:38.202691+0000","last_undegraded":"2024-03-12T05:10:38.202691+0000","last_fullsized":"2024-03-12T05:10:38.202691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:32.791210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135879+0000","last_change":"2024-03-12T05:09:44.135049+0000","last_active":"2024-03-12T05:10:13.135879+0000","last_peered":"2024-03-12T05:10:13.135879+0000","last_clean":"2024-03-12T05:10:13.135879+0000","last_became_active":"2024-03-12T05:09:44.134638+0000","last_became_peered":"2024-03-12T05:09:44.134638+0000","last_unstale":"2024-03-12T05:10:13.135879+0000","last_undegraded":"2024-03-12T05:10:13.135879+0000","last_fullsized":"2024-03-12T05:10:13.135879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:28:59.785062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381970+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:09:44.381970+0000","last_peered":"2024-03-12T05:09:44.381970+0000","last_clean":"2024-03-12T05:09:44.381970+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:09:44.381970+0000","last_undegraded":"2024-03-12T05:09:44.381970+0000","last_fullsized":"2024-03-12T05:09:44.381970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.155371+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:10:13.155371+0000","last_peered":"2024-03-12T05:10:13.155371+0000","last_clean":"2024-03-12T05:10:13.155371+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:10:13.155371+0000","last_undegraded":"2024-03-12T05:10:13.155371+0000","last_fullsized":"2024-03-12T05:10:13.155371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128279+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:09:44.128279+0000","last_peered":"2024-03-12T05:09:44.128279+0000","last_clean":"2024-03-12T05:09:44.128279+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:09:44.128279+0000","last_undegraded":"2024-03-12T05:09:44.128279+0000","last_fullsized":"2024-03-12T05:09:44.128279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152192+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:10:13.152192+0000","last_peered":"2024-03-12T05:10:13.152192+0000","last_clean":"2024-03-12T05:10:13.152192+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:10:13.152192+0000","last_undegraded":"2024-03-12T05:10:13.152192+0000","last_fullsized":"2024-03-12T05:10:13.152192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128527+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:09:44.128527+0000","last_peered":"2024-03-12T05:09:44.128527+0000","last_clean":"2024-03-12T05:09:44.128527+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:09:44.128527+0000","last_undegraded":"2024-03-12T05:09:44.128527+0000","last_fullsized":"2024-03-12T05:09:44.128527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152370+0000","last_change":"2024-03-12T05:09:44.133756+0000","last_active":"2024-03-12T05:10:13.152370+0000","last_peered":"2024-03-12T05:10:13.152370+0000","last_clean":"2024-03-12T05:10:13.152370+0000","last_became_active":"2024-03-12T05:09:44.133312+0000","last_became_peered":"2024-03-12T05:09:44.133312+0000","last_unstale":"2024-03-12T05:10:13.152370+0000","last_undegraded":"2024-03-12T05:10:13.152370+0000","last_fullsized":"2024-03-12T05:10:13.152370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:54:24.882257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.515180+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:10:37.515180+0000","last_peered":"2024-03-12T05:10:37.515180+0000","last_clean":"2024-03-12T05:10:37.515180+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:10:37.515180+0000","last_undegraded":"2024-03-12T05:10:37.515180+0000","last_fullsized":"2024-03-12T05:10:37.515180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135098+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:10:13.135098+0000","last_peered":"2024-03-12T05:10:13.135098+0000","last_clean":"2024-03-12T05:10:13.135098+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:10:13.135098+0000","last_undegraded":"2024-03-12T05:10:13.135098+0000","last_fullsized":"2024-03-12T05:10:13.135098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381204+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:09:44.381204+0000","last_peered":"2024-03-12T05:09:44.381204+0000","last_clean":"2024-03-12T05:09:44.381204+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:09:44.381204+0000","last_undegraded":"2024-03-12T05:09:44.381204+0000","last_fullsized":"2024-03-12T05:09:44.381204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144231+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:10:13.144231+0000","last_peered":"2024-03-12T05:10:13.144231+0000","last_clean":"2024-03-12T05:10:13.144231+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:10:13.144231+0000","last_undegraded":"2024-03-12T05:10:13.144231+0000","last_fullsized":"2024-03-12T05:10:13.144231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126104+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:09:44.126104+0000","last_peered":"2024-03-12T05:09:44.126104+0000","last_clean":"2024-03-12T05:09:44.126104+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:09:44.126104+0000","last_undegraded":"2024-03-12T05:09:44.126104+0000","last_fullsized":"2024-03-12T05:09:44.126104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.134969+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:10:13.134969+0000","last_peered":"2024-03-12T05:10:13.134969+0000","last_clean":"2024-03-12T05:10:13.134969+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:10:13.134969+0000","last_undegraded":"2024-03-12T05:10:13.134969+0000","last_fullsized":"2024-03-12T05:10:13.134969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128856+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:09:44.128856+0000","last_peered":"2024-03-12T05:09:44.128856+0000","last_clean":"2024-03-12T05:09:44.128856+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:09:44.128856+0000","last_undegraded":"2024-03-12T05:09:44.128856+0000","last_fullsized":"2024-03-12T05:09:44.128856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137840+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:10:13.137840+0000","last_peered":"2024-03-12T05:10:13.137840+0000","last_clean":"2024-03-12T05:10:13.137840+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:10:13.137840+0000","last_undegraded":"2024-03-12T05:10:13.137840+0000","last_fullsized":"2024-03-12T05:10:13.137840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'8","reported_seq":259,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.516014+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:10:38.516014+0000","last_peered":"2024-03-12T05:10:38.516014+0000","last_clean":"2024-03-12T05:10:38.516014+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:10:38.516014+0000","last_undegraded":"2024-03-12T05:10:38.516014+0000","last_fullsized":"2024-03-12T05:10:38.516014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.148250+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:10:13.148250+0000","last_peered":"2024-03-12T05:10:13.148250+0000","last_clean":"2024-03-12T05:10:13.148250+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:10:13.148250+0000","last_undegraded":"2024-03-12T05:10:13.148250+0000","last_fullsized":"2024-03-12T05:10:13.148250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.315313+0000","last_change":"2024-03-12T05:09:44.315313+0000","last_active":"2024-03-12T05:09:44.315313+0000","last_peered":"2024-03-12T05:09:44.315313+0000","last_clean":"2024-03-12T05:09:44.315313+0000","last_became_active":"2024-03-12T05:09:44.311453+0000","last_became_peered":"2024-03-12T05:09:44.311453+0000","last_unstale":"2024-03-12T05:09:44.315313+0000","last_undegraded":"2024-03-12T05:09:44.315313+0000","last_fullsized":"2024-03-12T05:09:44.315313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:21.022485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.142842+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:10:13.142842+0000","last_peered":"2024-03-12T05:10:13.142842+0000","last_clean":"2024-03-12T05:10:13.142842+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:10:13.142842+0000","last_undegraded":"2024-03-12T05:10:13.142842+0000","last_fullsized":"2024-03-12T05:10:13.142842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132030+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:09:44.132030+0000","last_peered":"2024-03-12T05:09:44.132030+0000","last_clean":"2024-03-12T05:09:44.132030+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:09:44.132030+0000","last_undegraded":"2024-03-12T05:09:44.132030+0000","last_fullsized":"2024-03-12T05:09:44.132030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154851+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:10:13.154851+0000","last_peered":"2024-03-12T05:10:13.154851+0000","last_clean":"2024-03-12T05:10:13.154851+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:10:13.154851+0000","last_undegraded":"2024-03-12T05:10:13.154851+0000","last_fullsized":"2024-03-12T05:10:13.154851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'35","reported_seq":295,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.517196+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:10:38.517196+0000","last_peered":"2024-03-12T05:10:38.517196+0000","last_clean":"2024-03-12T05:10:38.517196+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:10:38.517196+0000","last_undegraded":"2024-03-12T05:10:38.517196+0000","last_fullsized":"2024-03-12T05:10:38.517196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64343,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":183,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138155+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:10:13.138155+0000","last_peered":"2024-03-12T05:10:13.138155+0000","last_clean":"2024-03-12T05:10:13.138155+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:10:13.138155+0000","last_undegraded":"2024-03-12T05:10:13.138155+0000","last_fullsized":"2024-03-12T05:10:13.138155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131709+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:09:44.131709+0000","last_peered":"2024-03-12T05:09:44.131709+0000","last_clean":"2024-03-12T05:09:44.131709+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:09:44.131709+0000","last_undegraded":"2024-03-12T05:09:44.131709+0000","last_fullsized":"2024-03-12T05:09:44.131709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139579+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:10:13.139579+0000","last_peered":"2024-03-12T05:10:13.139579+0000","last_clean":"2024-03-12T05:10:13.139579+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:10:13.139579+0000","last_undegraded":"2024-03-12T05:10:13.139579+0000","last_fullsized":"2024-03-12T05:10:13.139579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128625+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:09:44.128625+0000","last_peered":"2024-03-12T05:09:44.128625+0000","last_clean":"2024-03-12T05:09:44.128625+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:09:44.128625+0000","last_undegraded":"2024-03-12T05:09:44.128625+0000","last_fullsized":"2024-03-12T05:09:44.128625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140001+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:10:13.140001+0000","last_peered":"2024-03-12T05:10:13.140001+0000","last_clean":"2024-03-12T05:10:13.140001+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:10:13.140001+0000","last_undegraded":"2024-03-12T05:10:13.140001+0000","last_fullsized":"2024-03-12T05:10:13.140001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.514995+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:10:37.514995+0000","last_peered":"2024-03-12T05:10:37.514995+0000","last_clean":"2024-03-12T05:10:37.514995+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:10:37.514995+0000","last_undegraded":"2024-03-12T05:10:37.514995+0000","last_fullsized":"2024-03-12T05:10:37.514995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141856+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:10:13.141856+0000","last_peered":"2024-03-12T05:10:13.141856+0000","last_clean":"2024-03-12T05:10:13.141856+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:10:13.141856+0000","last_undegraded":"2024-03-12T05:10:13.141856+0000","last_fullsized":"2024-03-12T05:10:13.141856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127773+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:09:44.127773+0000","last_peered":"2024-03-12T05:09:44.127773+0000","last_clean":"2024-03-12T05:09:44.127773+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:09:44.127773+0000","last_undegraded":"2024-03-12T05:09:44.127773+0000","last_fullsized":"2024-03-12T05:09:44.127773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145977+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:10:13.145977+0000","last_peered":"2024-03-12T05:10:13.145977+0000","last_clean":"2024-03-12T05:10:13.145977+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:10:13.145977+0000","last_undegraded":"2024-03-12T05:10:13.145977+0000","last_fullsized":"2024-03-12T05:10:13.145977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128951+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:09:44.128951+0000","last_peered":"2024-03-12T05:09:44.128951+0000","last_clean":"2024-03-12T05:09:44.128951+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:09:44.128951+0000","last_undegraded":"2024-03-12T05:09:44.128951+0000","last_fullsized":"2024-03-12T05:09:44.128951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144510+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:10:13.144510+0000","last_peered":"2024-03-12T05:10:13.144510+0000","last_clean":"2024-03-12T05:10:13.144510+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:10:13.144510+0000","last_undegraded":"2024-03-12T05:10:13.144510+0000","last_fullsized":"2024-03-12T05:10:13.144510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128017+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:09:44.128017+0000","last_peered":"2024-03-12T05:09:44.128017+0000","last_clean":"2024-03-12T05:09:44.128017+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:09:44.128017+0000","last_undegraded":"2024-03-12T05:09:44.128017+0000","last_fullsized":"2024-03-12T05:09:44.128017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.131225+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:10:13.131225+0000","last_peered":"2024-03-12T05:10:13.131225+0000","last_clean":"2024-03-12T05:10:13.131225+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:10:13.131225+0000","last_undegraded":"2024-03-12T05:10:13.131225+0000","last_fullsized":"2024-03-12T05:10:13.131225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'7","reported_seq":145,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.204402+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:10:39.204402+0000","last_peered":"2024-03-12T05:10:39.204402+0000","last_clean":"2024-03-12T05:10:39.204402+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:10:39.204402+0000","last_undegraded":"2024-03-12T05:10:39.204402+0000","last_fullsized":"2024-03-12T05:10:39.204402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.146033+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:10:13.146033+0000","last_peered":"2024-03-12T05:10:13.146033+0000","last_clean":"2024-03-12T05:10:13.146033+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:10:13.146033+0000","last_undegraded":"2024-03-12T05:10:13.146033+0000","last_fullsized":"2024-03-12T05:10:13.146033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.128862+0000","last_change":"2024-03-12T05:09:44.133911+0000","last_active":"2024-03-12T05:10:13.128862+0000","last_peered":"2024-03-12T05:10:13.128862+0000","last_clean":"2024-03-12T05:10:13.128862+0000","last_became_active":"2024-03-12T05:09:44.133488+0000","last_became_peered":"2024-03-12T05:09:44.133488+0000","last_unstale":"2024-03-12T05:10:13.128862+0000","last_undegraded":"2024-03-12T05:10:13.128862+0000","last_fullsized":"2024-03-12T05:10:13.128862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:44.969275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.515921+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:10:38.515921+0000","last_peered":"2024-03-12T05:10:38.515921+0000","last_clean":"2024-03-12T05:10:38.515921+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:10:38.515921+0000","last_undegraded":"2024-03-12T05:10:38.515921+0000","last_fullsized":"2024-03-12T05:10:38.515921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144632+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:10:13.144632+0000","last_peered":"2024-03-12T05:10:13.144632+0000","last_clean":"2024-03-12T05:10:13.144632+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:10:13.144632+0000","last_undegraded":"2024-03-12T05:10:13.144632+0000","last_fullsized":"2024-03-12T05:10:13.144632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310523+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:09:44.310523+0000","last_peered":"2024-03-12T05:09:44.310523+0000","last_clean":"2024-03-12T05:09:44.310523+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:09:44.310523+0000","last_undegraded":"2024-03-12T05:09:44.310523+0000","last_fullsized":"2024-03-12T05:09:44.310523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125472+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:09:44.125472+0000","last_peered":"2024-03-12T05:09:44.125472+0000","last_clean":"2024-03-12T05:09:44.125472+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:09:44.125472+0000","last_undegraded":"2024-03-12T05:09:44.125472+0000","last_fullsized":"2024-03-12T05:09:44.125472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.126630+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:10:13.126630+0000","last_peered":"2024-03-12T05:10:13.126630+0000","last_clean":"2024-03-12T05:10:13.126630+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:10:13.126630+0000","last_undegraded":"2024-03-12T05:10:13.126630+0000","last_fullsized":"2024-03-12T05:10:13.126630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125644+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:09:44.125644+0000","last_peered":"2024-03-12T05:09:44.125644+0000","last_clean":"2024-03-12T05:09:44.125644+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:09:44.125644+0000","last_undegraded":"2024-03-12T05:09:44.125644+0000","last_fullsized":"2024-03-12T05:09:44.125644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128082+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:09:44.128082+0000","last_peered":"2024-03-12T05:09:44.128082+0000","last_clean":"2024-03-12T05:09:44.128082+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:09:44.128082+0000","last_undegraded":"2024-03-12T05:09:44.128082+0000","last_fullsized":"2024-03-12T05:09:44.128082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.127735+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:10:13.127735+0000","last_peered":"2024-03-12T05:10:13.127735+0000","last_clean":"2024-03-12T05:10:13.127735+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:10:13.127735+0000","last_undegraded":"2024-03-12T05:10:13.127735+0000","last_fullsized":"2024-03-12T05:10:13.127735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.799041+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:10:37.799041+0000","last_peered":"2024-03-12T05:10:37.799041+0000","last_clean":"2024-03-12T05:10:37.799041+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:10:37.799041+0000","last_undegraded":"2024-03-12T05:10:37.799041+0000","last_fullsized":"2024-03-12T05:10:37.799041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139019+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:10:13.139019+0000","last_peered":"2024-03-12T05:10:13.139019+0000","last_clean":"2024-03-12T05:10:13.139019+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:10:13.139019+0000","last_undegraded":"2024-03-12T05:10:13.139019+0000","last_fullsized":"2024-03-12T05:10:13.139019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310664+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:09:44.310664+0000","last_peered":"2024-03-12T05:09:44.310664+0000","last_clean":"2024-03-12T05:09:44.310664+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:09:44.310664+0000","last_undegraded":"2024-03-12T05:09:44.310664+0000","last_fullsized":"2024-03-12T05:09:44.310664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141996+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:10:13.141996+0000","last_peered":"2024-03-12T05:10:13.141996+0000","last_clean":"2024-03-12T05:10:13.141996+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:10:13.141996+0000","last_undegraded":"2024-03-12T05:10:13.141996+0000","last_fullsized":"2024-03-12T05:10:13.141996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.136091+0000","last_change":"2024-03-12T05:09:44.136091+0000","last_active":"2024-03-12T05:09:44.136091+0000","last_peered":"2024-03-12T05:09:44.136091+0000","last_clean":"2024-03-12T05:09:44.136091+0000","last_became_active":"2024-03-12T05:09:44.130865+0000","last_became_peered":"2024-03-12T05:09:44.130865+0000","last_unstale":"2024-03-12T05:09:44.136091+0000","last_undegraded":"2024-03-12T05:09:44.136091+0000","last_fullsized":"2024-03-12T05:09:44.136091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:13.588722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.099732+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:10:13.099732+0000","last_peered":"2024-03-12T05:10:13.099732+0000","last_clean":"2024-03-12T05:10:13.099732+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:10:13.099732+0000","last_undegraded":"2024-03-12T05:10:13.099732+0000","last_fullsized":"2024-03-12T05:10:13.099732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381493+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:09:44.381493+0000","last_peered":"2024-03-12T05:09:44.381493+0000","last_clean":"2024-03-12T05:09:44.381493+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:09:44.381493+0000","last_undegraded":"2024-03-12T05:09:44.381493+0000","last_fullsized":"2024-03-12T05:09:44.381493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141751+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:10:13.141751+0000","last_peered":"2024-03-12T05:10:13.141751+0000","last_clean":"2024-03-12T05:10:13.141751+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:10:13.141751+0000","last_undegraded":"2024-03-12T05:10:13.141751+0000","last_fullsized":"2024-03-12T05:10:13.141751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:36.342780+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:10:36.342780+0000","last_peered":"2024-03-12T05:10:36.342780+0000","last_clean":"2024-03-12T05:10:36.342780+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:10:36.342780+0000","last_undegraded":"2024-03-12T05:10:36.342780+0000","last_fullsized":"2024-03-12T05:10:36.342780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137122+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:10:13.137122+0000","last_peered":"2024-03-12T05:10:13.137122+0000","last_clean":"2024-03-12T05:10:13.137122+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:10:13.137122+0000","last_undegraded":"2024-03-12T05:10:13.137122+0000","last_fullsized":"2024-03-12T05:10:13.137122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.145784+0000","last_change":"2024-03-12T05:09:44.145284+0000","last_active":"2024-03-12T05:09:44.145784+0000","last_peered":"2024-03-12T05:09:44.145784+0000","last_clean":"2024-03-12T05:09:44.145784+0000","last_became_active":"2024-03-12T05:09:44.134640+0000","last_became_peered":"2024-03-12T05:09:44.134640+0000","last_unstale":"2024-03-12T05:09:44.145784+0000","last_undegraded":"2024-03-12T05:09:44.145784+0000","last_fullsized":"2024-03-12T05:09:44.145784+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:17.719372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153940+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:10:13.153940+0000","last_peered":"2024-03-12T05:10:13.153940+0000","last_clean":"2024-03-12T05:10:13.153940+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:10:13.153940+0000","last_undegraded":"2024-03-12T05:10:13.153940+0000","last_fullsized":"2024-03-12T05:10:13.153940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'9","reported_seq":151,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.518590+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:10:39.518590+0000","last_peered":"2024-03-12T05:10:39.518590+0000","last_clean":"2024-03-12T05:10:39.518590+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:10:39.518590+0000","last_undegraded":"2024-03-12T05:10:39.518590+0000","last_fullsized":"2024-03-12T05:10:39.518590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153801+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:10:13.153801+0000","last_peered":"2024-03-12T05:10:13.153801+0000","last_clean":"2024-03-12T05:10:13.153801+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:10:13.153801+0000","last_undegraded":"2024-03-12T05:10:13.153801+0000","last_fullsized":"2024-03-12T05:10:13.153801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131890+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:09:44.131890+0000","last_peered":"2024-03-12T05:09:44.131890+0000","last_clean":"2024-03-12T05:09:44.131890+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:09:44.131890+0000","last_undegraded":"2024-03-12T05:09:44.131890+0000","last_fullsized":"2024-03-12T05:09:44.131890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140168+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:10:13.140168+0000","last_peered":"2024-03-12T05:10:13.140168+0000","last_clean":"2024-03-12T05:10:13.140168+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:10:13.140168+0000","last_undegraded":"2024-03-12T05:10:13.140168+0000","last_fullsized":"2024-03-12T05:10:13.140168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126596+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:09:44.126596+0000","last_peered":"2024-03-12T05:09:44.126596+0000","last_clean":"2024-03-12T05:09:44.126596+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:09:44.126596+0000","last_undegraded":"2024-03-12T05:09:44.126596+0000","last_fullsized":"2024-03-12T05:09:44.126596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140750+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:10:13.140750+0000","last_peered":"2024-03-12T05:10:13.140750+0000","last_clean":"2024-03-12T05:10:13.140750+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:10:13.140750+0000","last_undegraded":"2024-03-12T05:10:13.140750+0000","last_fullsized":"2024-03-12T05:10:13.140750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139173+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:10:13.139173+0000","last_peered":"2024-03-12T05:10:13.139173+0000","last_clean":"2024-03-12T05:10:13.139173+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:10:13.139173+0000","last_undegraded":"2024-03-12T05:10:13.139173+0000","last_fullsized":"2024-03-12T05:10:13.139173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136516+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:10:13.136516+0000","last_peered":"2024-03-12T05:10:13.136516+0000","last_clean":"2024-03-12T05:10:13.136516+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:10:13.136516+0000","last_undegraded":"2024-03-12T05:10:13.136516+0000","last_fullsized":"2024-03-12T05:10:13.136516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136016+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:10:13.136016+0000","last_peered":"2024-03-12T05:10:13.136016+0000","last_clean":"2024-03-12T05:10:13.136016+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:10:13.136016+0000","last_undegraded":"2024-03-12T05:10:13.136016+0000","last_fullsized":"2024-03-12T05:10:13.136016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140867+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:10:13.140867+0000","last_peered":"2024-03-12T05:10:13.140867+0000","last_clean":"2024-03-12T05:10:13.140867+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:10:13.140867+0000","last_undegraded":"2024-03-12T05:10:13.140867+0000","last_fullsized":"2024-03-12T05:10:13.140867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136721+0000","last_change":"2024-03-12T05:09:44.129436+0000","last_active":"2024-03-12T05:10:13.136721+0000","last_peered":"2024-03-12T05:10:13.136721+0000","last_clean":"2024-03-12T05:10:13.136721+0000","last_became_active":"2024-03-12T05:09:44.129063+0000","last_became_peered":"2024-03-12T05:09:44.129063+0000","last_unstale":"2024-03-12T05:10:13.136721+0000","last_undegraded":"2024-03-12T05:10:13.136721+0000","last_fullsized":"2024-03-12T05:10:13.136721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:06.724357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138474+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:10:13.138474+0000","last_peered":"2024-03-12T05:10:13.138474+0000","last_clean":"2024-03-12T05:10:13.138474+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:10:13.138474+0000","last_undegraded":"2024-03-12T05:10:13.138474+0000","last_fullsized":"2024-03-12T05:10:13.138474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152412+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:10:13.152412+0000","last_peered":"2024-03-12T05:10:13.152412+0000","last_clean":"2024-03-12T05:10:13.152412+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:10:13.152412+0000","last_undegraded":"2024-03-12T05:10:13.152412+0000","last_fullsized":"2024-03-12T05:10:13.152412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136405+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:10:13.136405+0000","last_peered":"2024-03-12T05:10:13.136405+0000","last_clean":"2024-03-12T05:10:13.136405+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:10:13.136405+0000","last_undegraded":"2024-03-12T05:10:13.136405+0000","last_fullsized":"2024-03-12T05:10:13.136405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136449+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:10:13.136449+0000","last_peered":"2024-03-12T05:10:13.136449+0000","last_clean":"2024-03-12T05:10:13.136449+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:10:13.136449+0000","last_undegraded":"2024-03-12T05:10:13.136449+0000","last_fullsized":"2024-03-12T05:10:13.136449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136850+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:10:13.136850+0000","last_peered":"2024-03-12T05:10:13.136850+0000","last_clean":"2024-03-12T05:10:13.136850+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:10:13.136850+0000","last_undegraded":"2024-03-12T05:10:13.136850+0000","last_fullsized":"2024-03-12T05:10:13.136850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152464+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:10:13.152464+0000","last_peered":"2024-03-12T05:10:13.152464+0000","last_clean":"2024-03-12T05:10:13.152464+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:10:13.152464+0000","last_undegraded":"2024-03-12T05:10:13.152464+0000","last_fullsized":"2024-03-12T05:10:13.152464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140276+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:10:13.140276+0000","last_peered":"2024-03-12T05:10:13.140276+0000","last_clean":"2024-03-12T05:10:13.140276+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:10:13.140276+0000","last_undegraded":"2024-03-12T05:10:13.140276+0000","last_fullsized":"2024-03-12T05:10:13.140276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144685+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:10:13.144685+0000","last_peered":"2024-03-12T05:10:13.144685+0000","last_clean":"2024-03-12T05:10:13.144685+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:10:13.144685+0000","last_undegraded":"2024-03-12T05:10:13.144685+0000","last_fullsized":"2024-03-12T05:10:13.144685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75051,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1207,"num_read_kb":683,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":220057,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1112443,"data_compressed":37976,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":456383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":456089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1051774,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495679,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":517800,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":458839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":523985,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":455925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1049575,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62539,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68724,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67088,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:10:41.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:10:41.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 -- 172.21.15.45:0/2115826478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101c50 msgr2=0x7fe7601084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 --2- 172.21.15.45:0/2115826478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101c50 0x7fe7601084e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe748009900 tx=0x7fe74802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:41.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 -- 172.21.15.45:0/2115826478 shutdown_connections 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 --2- 172.21.15.45:0/2115826478 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101c50 0x7fe7601084e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 --2- 172.21.15.45:0/2115826478 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe760101290 0x7fe760101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.908+0000 7fe766dbc640 1 --2- 172.21.15.45:0/2115826478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7601000a0 0x7fe7601004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- 172.21.15.45:0/2115826478 >> 172.21.15.45:0/2115826478 conn(0x7fe7600fb610 msgr2=0x7fe7600fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- 172.21.15.45:0/2115826478 shutdown_connections 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- 172.21.15.45:0/2115826478 wait complete. 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 Processor -- start 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- start start 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7601000a0 0x7fe760196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe760101c50 0x7fe76019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76019db10 con 0x7fe760101290 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76019dc80 con 0x7fe760101c50 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe766dbc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76019de40 con 0x7fe7601000a0 2024-03-12T05:10:41.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe75ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe75ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47628/0 (socket says 172.21.15.45:47628) 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.909+0000 7fe75ffff640 1 -- 172.21.15.45:0/657803960 learned_addr learned my addr 172.21.15.45:0/657803960 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe764b58640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7601000a0 0x7fe760196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 -- 172.21.15.45:0/657803960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7601000a0 msgr2=0x7fe760196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7601000a0 0x7fe760196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 -- 172.21.15.45:0/657803960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe760101c50 msgr2=0x7fe76019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe760101c50 0x7fe76019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 -- 172.21.15.45:0/657803960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748009560 con 0x7fe760101290 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75ffff640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe75000ea80 tx=0x7fe75000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75000ce40 con 0x7fe760101290 2024-03-12T05:10:41.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe750005020 con 0x7fe760101290 2024-03-12T05:10:41.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76019e120 con 0x7fe760101290 2024-03-12T05:10:41.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500116b0 con 0x7fe760101290 2024-03-12T05:10:41.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.910+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76019e5e0 con 0x7fe760101290 2024-03-12T05:10:41.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.911+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7601073e0 con 0x7fe760101290 2024-03-12T05:10:41.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.912+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe750005190 con 0x7fe760101290 2024-03-12T05:10:41.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.912+0000 7fe75dffb640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 0x7fe74c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:41.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.913+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe7500cd900 con 0x7fe760101290 2024-03-12T05:10:41.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.913+0000 7fe764b58640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 0x7fe74c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:41.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.913+0000 7fe764b58640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 0x7fe74c0a3f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe758005db0 tx=0x7fe758005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:41.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:41.917+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe750096960 con 0x7fe760101290 2024-03-12T05:10:42.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.095+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe76002cf10 con 0x7fe74c0a1ac0 2024-03-12T05:10:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.103+0000 7fe75dffb640 1 -- 172.21.15.45:0/657803960 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259093 (secure 0 0 0) 0x7fe76002cf10 con 0x7fe74c0a1ac0 2024-03-12T05:10:42.103 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 msgr2=0x7fe74c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 0x7fe74c0a3f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe758005db0 tx=0x7fe758005d40 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 msgr2=0x7fe760196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe75000ea80 tx=0x7fe75000ef40 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 shutdown_connections 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe74c0a1ac0 0x7fe74c0a3f80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe760101290 0x7fe760196560 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe760101c50 0x7fe76019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 --2- 172.21.15.45:0/657803960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7601000a0 0x7fe760196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 >> 172.21.15.45:0/657803960 conn(0x7fe7600fb610 msgr2=0x7fe7601045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 shutdown_connections 2024-03-12T05:10:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.106+0000 7fe766dbc640 1 -- 172.21.15.45:0/657803960 wait complete. 2024-03-12T05:10:42.114 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:10:41.129487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665653,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1343,"num_read_kb":802,"num_write":456,"num_write_kb":3104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181408,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373340,"kb_used_data":11708,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568484,"statfs":{"total":1151940427776,"available":1151558127616,"internally_reserved":0,"allocated":11988992,"data_stored":7454604,"data_compressed":113928,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150307+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:10:13.150307+0000","last_peered":"2024-03-12T05:10:13.150307+0000","last_clean":"2024-03-12T05:10:13.150307+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:10:13.150307+0000","last_undegraded":"2024-03-12T05:10:13.150307+0000","last_fullsized":"2024-03-12T05:10:13.150307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145142+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:10:13.145142+0000","last_peered":"2024-03-12T05:10:13.145142+0000","last_clean":"2024-03-12T05:10:13.145142+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:10:13.145142+0000","last_undegraded":"2024-03-12T05:10:13.145142+0000","last_fullsized":"2024-03-12T05:10:13.145142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153971+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:10:13.153971+0000","last_peered":"2024-03-12T05:10:13.153971+0000","last_clean":"2024-03-12T05:10:13.153971+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:10:13.153971+0000","last_undegraded":"2024-03-12T05:10:13.153971+0000","last_fullsized":"2024-03-12T05:10:13.153971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.123843+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:10:13.123843+0000","last_peered":"2024-03-12T05:10:13.123843+0000","last_clean":"2024-03-12T05:10:13.123843+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:10:13.123843+0000","last_undegraded":"2024-03-12T05:10:13.123843+0000","last_fullsized":"2024-03-12T05:10:13.123843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.130074+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:10:13.130074+0000","last_peered":"2024-03-12T05:10:13.130074+0000","last_clean":"2024-03-12T05:10:13.130074+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:10:13.130074+0000","last_undegraded":"2024-03-12T05:10:13.130074+0000","last_fullsized":"2024-03-12T05:10:13.130074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143421+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:10:13.143421+0000","last_peered":"2024-03-12T05:10:13.143421+0000","last_clean":"2024-03-12T05:10:13.143421+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:10:13.143421+0000","last_undegraded":"2024-03-12T05:10:13.143421+0000","last_fullsized":"2024-03-12T05:10:13.143421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138393+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:10:13.138393+0000","last_peered":"2024-03-12T05:10:13.138393+0000","last_clean":"2024-03-12T05:10:13.138393+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:10:13.138393+0000","last_undegraded":"2024-03-12T05:10:13.138393+0000","last_fullsized":"2024-03-12T05:10:13.138393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136869+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:10:13.136869+0000","last_peered":"2024-03-12T05:10:13.136869+0000","last_clean":"2024-03-12T05:10:13.136869+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:10:13.136869+0000","last_undegraded":"2024-03-12T05:10:13.136869+0000","last_fullsized":"2024-03-12T05:10:13.136869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154745+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:10:13.154745+0000","last_peered":"2024-03-12T05:10:13.154745+0000","last_clean":"2024-03-12T05:10:13.154745+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:10:13.154745+0000","last_undegraded":"2024-03-12T05:10:13.154745+0000","last_fullsized":"2024-03-12T05:10:13.154745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143667+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:10:13.143667+0000","last_peered":"2024-03-12T05:10:13.143667+0000","last_clean":"2024-03-12T05:10:13.143667+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:10:13.143667+0000","last_undegraded":"2024-03-12T05:10:13.143667+0000","last_fullsized":"2024-03-12T05:10:13.143667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153713+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:10:13.153713+0000","last_peered":"2024-03-12T05:10:13.153713+0000","last_clean":"2024-03-12T05:10:13.153713+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:10:13.153713+0000","last_undegraded":"2024-03-12T05:10:13.153713+0000","last_fullsized":"2024-03-12T05:10:13.153713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143429+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:10:13.143429+0000","last_peered":"2024-03-12T05:10:13.143429+0000","last_clean":"2024-03-12T05:10:13.143429+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:10:13.143429+0000","last_undegraded":"2024-03-12T05:10:13.143429+0000","last_fullsized":"2024-03-12T05:10:13.143429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138246+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:10:13.138246+0000","last_peered":"2024-03-12T05:10:13.138246+0000","last_clean":"2024-03-12T05:10:13.138246+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:10:13.138246+0000","last_undegraded":"2024-03-12T05:10:13.138246+0000","last_fullsized":"2024-03-12T05:10:13.138246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145379+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:10:13.145379+0000","last_peered":"2024-03-12T05:10:13.145379+0000","last_clean":"2024-03-12T05:10:13.145379+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:10:13.145379+0000","last_undegraded":"2024-03-12T05:10:13.145379+0000","last_fullsized":"2024-03-12T05:10:13.145379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140306+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:10:13.140306+0000","last_peered":"2024-03-12T05:10:13.140306+0000","last_clean":"2024-03-12T05:10:13.140306+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:10:13.140306+0000","last_undegraded":"2024-03-12T05:10:13.140306+0000","last_fullsized":"2024-03-12T05:10:13.140306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154503+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:10:13.154503+0000","last_peered":"2024-03-12T05:10:13.154503+0000","last_clean":"2024-03-12T05:10:13.154503+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:10:13.154503+0000","last_undegraded":"2024-03-12T05:10:13.154503+0000","last_fullsized":"2024-03-12T05:10:13.154503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150237+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:10:13.150237+0000","last_peered":"2024-03-12T05:10:13.150237+0000","last_clean":"2024-03-12T05:10:13.150237+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:10:13.150237+0000","last_undegraded":"2024-03-12T05:10:13.150237+0000","last_fullsized":"2024-03-12T05:10:13.150237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144239+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:10:13.144239+0000","last_peered":"2024-03-12T05:10:13.144239+0000","last_clean":"2024-03-12T05:10:13.144239+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:10:13.144239+0000","last_undegraded":"2024-03-12T05:10:13.144239+0000","last_fullsized":"2024-03-12T05:10:13.144239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.133361+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:10:13.133361+0000","last_peered":"2024-03-12T05:10:13.133361+0000","last_clean":"2024-03-12T05:10:13.133361+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:10:13.133361+0000","last_undegraded":"2024-03-12T05:10:13.133361+0000","last_fullsized":"2024-03-12T05:10:13.133361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.202691+0000","last_change":"2024-03-12T05:09:44.142398+0000","last_active":"2024-03-12T05:10:38.202691+0000","last_peered":"2024-03-12T05:10:38.202691+0000","last_clean":"2024-03-12T05:10:38.202691+0000","last_became_active":"2024-03-12T05:09:44.129576+0000","last_became_peered":"2024-03-12T05:09:44.129576+0000","last_unstale":"2024-03-12T05:10:38.202691+0000","last_undegraded":"2024-03-12T05:10:38.202691+0000","last_fullsized":"2024-03-12T05:10:38.202691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:32.791210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135879+0000","last_change":"2024-03-12T05:09:44.135049+0000","last_active":"2024-03-12T05:10:13.135879+0000","last_peered":"2024-03-12T05:10:13.135879+0000","last_clean":"2024-03-12T05:10:13.135879+0000","last_became_active":"2024-03-12T05:09:44.134638+0000","last_became_peered":"2024-03-12T05:09:44.134638+0000","last_unstale":"2024-03-12T05:10:13.135879+0000","last_undegraded":"2024-03-12T05:10:13.135879+0000","last_fullsized":"2024-03-12T05:10:13.135879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:28:59.785062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381970+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:09:44.381970+0000","last_peered":"2024-03-12T05:09:44.381970+0000","last_clean":"2024-03-12T05:09:44.381970+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:09:44.381970+0000","last_undegraded":"2024-03-12T05:09:44.381970+0000","last_fullsized":"2024-03-12T05:09:44.381970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.155371+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:10:13.155371+0000","last_peered":"2024-03-12T05:10:13.155371+0000","last_clean":"2024-03-12T05:10:13.155371+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:10:13.155371+0000","last_undegraded":"2024-03-12T05:10:13.155371+0000","last_fullsized":"2024-03-12T05:10:13.155371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128279+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:09:44.128279+0000","last_peered":"2024-03-12T05:09:44.128279+0000","last_clean":"2024-03-12T05:09:44.128279+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:09:44.128279+0000","last_undegraded":"2024-03-12T05:09:44.128279+0000","last_fullsized":"2024-03-12T05:09:44.128279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152192+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:10:13.152192+0000","last_peered":"2024-03-12T05:10:13.152192+0000","last_clean":"2024-03-12T05:10:13.152192+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:10:13.152192+0000","last_undegraded":"2024-03-12T05:10:13.152192+0000","last_fullsized":"2024-03-12T05:10:13.152192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128527+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:09:44.128527+0000","last_peered":"2024-03-12T05:09:44.128527+0000","last_clean":"2024-03-12T05:09:44.128527+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:09:44.128527+0000","last_undegraded":"2024-03-12T05:09:44.128527+0000","last_fullsized":"2024-03-12T05:09:44.128527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152370+0000","last_change":"2024-03-12T05:09:44.133756+0000","last_active":"2024-03-12T05:10:13.152370+0000","last_peered":"2024-03-12T05:10:13.152370+0000","last_clean":"2024-03-12T05:10:13.152370+0000","last_became_active":"2024-03-12T05:09:44.133312+0000","last_became_peered":"2024-03-12T05:09:44.133312+0000","last_unstale":"2024-03-12T05:10:13.152370+0000","last_undegraded":"2024-03-12T05:10:13.152370+0000","last_fullsized":"2024-03-12T05:10:13.152370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:54:24.882257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.515180+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:10:37.515180+0000","last_peered":"2024-03-12T05:10:37.515180+0000","last_clean":"2024-03-12T05:10:37.515180+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:10:37.515180+0000","last_undegraded":"2024-03-12T05:10:37.515180+0000","last_fullsized":"2024-03-12T05:10:37.515180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135098+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:10:13.135098+0000","last_peered":"2024-03-12T05:10:13.135098+0000","last_clean":"2024-03-12T05:10:13.135098+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:10:13.135098+0000","last_undegraded":"2024-03-12T05:10:13.135098+0000","last_fullsized":"2024-03-12T05:10:13.135098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381204+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:09:44.381204+0000","last_peered":"2024-03-12T05:09:44.381204+0000","last_clean":"2024-03-12T05:09:44.381204+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:09:44.381204+0000","last_undegraded":"2024-03-12T05:09:44.381204+0000","last_fullsized":"2024-03-12T05:09:44.381204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144231+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:10:13.144231+0000","last_peered":"2024-03-12T05:10:13.144231+0000","last_clean":"2024-03-12T05:10:13.144231+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:10:13.144231+0000","last_undegraded":"2024-03-12T05:10:13.144231+0000","last_fullsized":"2024-03-12T05:10:13.144231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126104+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:09:44.126104+0000","last_peered":"2024-03-12T05:09:44.126104+0000","last_clean":"2024-03-12T05:09:44.126104+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:09:44.126104+0000","last_undegraded":"2024-03-12T05:09:44.126104+0000","last_fullsized":"2024-03-12T05:09:44.126104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.134969+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:10:13.134969+0000","last_peered":"2024-03-12T05:10:13.134969+0000","last_clean":"2024-03-12T05:10:13.134969+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:10:13.134969+0000","last_undegraded":"2024-03-12T05:10:13.134969+0000","last_fullsized":"2024-03-12T05:10:13.134969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128856+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:09:44.128856+0000","last_peered":"2024-03-12T05:09:44.128856+0000","last_clean":"2024-03-12T05:09:44.128856+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:09:44.128856+0000","last_undegraded":"2024-03-12T05:09:44.128856+0000","last_fullsized":"2024-03-12T05:09:44.128856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137840+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:10:13.137840+0000","last_peered":"2024-03-12T05:10:13.137840+0000","last_clean":"2024-03-12T05:10:13.137840+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:10:13.137840+0000","last_undegraded":"2024-03-12T05:10:13.137840+0000","last_fullsized":"2024-03-12T05:10:13.137840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'8","reported_seq":259,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.516014+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:10:38.516014+0000","last_peered":"2024-03-12T05:10:38.516014+0000","last_clean":"2024-03-12T05:10:38.516014+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:10:38.516014+0000","last_undegraded":"2024-03-12T05:10:38.516014+0000","last_fullsized":"2024-03-12T05:10:38.516014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.148250+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:10:13.148250+0000","last_peered":"2024-03-12T05:10:13.148250+0000","last_clean":"2024-03-12T05:10:13.148250+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:10:13.148250+0000","last_undegraded":"2024-03-12T05:10:13.148250+0000","last_fullsized":"2024-03-12T05:10:13.148250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.315313+0000","last_change":"2024-03-12T05:09:44.315313+0000","last_active":"2024-03-12T05:09:44.315313+0000","last_peered":"2024-03-12T05:09:44.315313+0000","last_clean":"2024-03-12T05:09:44.315313+0000","last_became_active":"2024-03-12T05:09:44.311453+0000","last_became_peered":"2024-03-12T05:09:44.311453+0000","last_unstale":"2024-03-12T05:09:44.315313+0000","last_undegraded":"2024-03-12T05:09:44.315313+0000","last_fullsized":"2024-03-12T05:09:44.315313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:21.022485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.142842+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:10:13.142842+0000","last_peered":"2024-03-12T05:10:13.142842+0000","last_clean":"2024-03-12T05:10:13.142842+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:10:13.142842+0000","last_undegraded":"2024-03-12T05:10:13.142842+0000","last_fullsized":"2024-03-12T05:10:13.142842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132030+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:09:44.132030+0000","last_peered":"2024-03-12T05:09:44.132030+0000","last_clean":"2024-03-12T05:09:44.132030+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:09:44.132030+0000","last_undegraded":"2024-03-12T05:09:44.132030+0000","last_fullsized":"2024-03-12T05:09:44.132030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154851+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:10:13.154851+0000","last_peered":"2024-03-12T05:10:13.154851+0000","last_clean":"2024-03-12T05:10:13.154851+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:10:13.154851+0000","last_undegraded":"2024-03-12T05:10:13.154851+0000","last_fullsized":"2024-03-12T05:10:13.154851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'35","reported_seq":295,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.517196+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:10:38.517196+0000","last_peered":"2024-03-12T05:10:38.517196+0000","last_clean":"2024-03-12T05:10:38.517196+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:10:38.517196+0000","last_undegraded":"2024-03-12T05:10:38.517196+0000","last_fullsized":"2024-03-12T05:10:38.517196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64343,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":183,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138155+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:10:13.138155+0000","last_peered":"2024-03-12T05:10:13.138155+0000","last_clean":"2024-03-12T05:10:13.138155+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:10:13.138155+0000","last_undegraded":"2024-03-12T05:10:13.138155+0000","last_fullsized":"2024-03-12T05:10:13.138155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131709+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:09:44.131709+0000","last_peered":"2024-03-12T05:09:44.131709+0000","last_clean":"2024-03-12T05:09:44.131709+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:09:44.131709+0000","last_undegraded":"2024-03-12T05:09:44.131709+0000","last_fullsized":"2024-03-12T05:09:44.131709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139579+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:10:13.139579+0000","last_peered":"2024-03-12T05:10:13.139579+0000","last_clean":"2024-03-12T05:10:13.139579+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:10:13.139579+0000","last_undegraded":"2024-03-12T05:10:13.139579+0000","last_fullsized":"2024-03-12T05:10:13.139579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128625+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:09:44.128625+0000","last_peered":"2024-03-12T05:09:44.128625+0000","last_clean":"2024-03-12T05:09:44.128625+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:09:44.128625+0000","last_undegraded":"2024-03-12T05:09:44.128625+0000","last_fullsized":"2024-03-12T05:09:44.128625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140001+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:10:13.140001+0000","last_peered":"2024-03-12T05:10:13.140001+0000","last_clean":"2024-03-12T05:10:13.140001+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:10:13.140001+0000","last_undegraded":"2024-03-12T05:10:13.140001+0000","last_fullsized":"2024-03-12T05:10:13.140001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.514995+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:10:37.514995+0000","last_peered":"2024-03-12T05:10:37.514995+0000","last_clean":"2024-03-12T05:10:37.514995+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:10:37.514995+0000","last_undegraded":"2024-03-12T05:10:37.514995+0000","last_fullsized":"2024-03-12T05:10:37.514995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141856+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:10:13.141856+0000","last_peered":"2024-03-12T05:10:13.141856+0000","last_clean":"2024-03-12T05:10:13.141856+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:10:13.141856+0000","last_undegraded":"2024-03-12T05:10:13.141856+0000","last_fullsized":"2024-03-12T05:10:13.141856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127773+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:09:44.127773+0000","last_peered":"2024-03-12T05:09:44.127773+0000","last_clean":"2024-03-12T05:09:44.127773+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:09:44.127773+0000","last_undegraded":"2024-03-12T05:09:44.127773+0000","last_fullsized":"2024-03-12T05:09:44.127773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145977+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:10:13.145977+0000","last_peered":"2024-03-12T05:10:13.145977+0000","last_clean":"2024-03-12T05:10:13.145977+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:10:13.145977+0000","last_undegraded":"2024-03-12T05:10:13.145977+0000","last_fullsized":"2024-03-12T05:10:13.145977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128951+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:09:44.128951+0000","last_peered":"2024-03-12T05:09:44.128951+0000","last_clean":"2024-03-12T05:09:44.128951+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:09:44.128951+0000","last_undegraded":"2024-03-12T05:09:44.128951+0000","last_fullsized":"2024-03-12T05:09:44.128951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144510+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:10:13.144510+0000","last_peered":"2024-03-12T05:10:13.144510+0000","last_clean":"2024-03-12T05:10:13.144510+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:10:13.144510+0000","last_undegraded":"2024-03-12T05:10:13.144510+0000","last_fullsized":"2024-03-12T05:10:13.144510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128017+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:09:44.128017+0000","last_peered":"2024-03-12T05:09:44.128017+0000","last_clean":"2024-03-12T05:09:44.128017+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:09:44.128017+0000","last_undegraded":"2024-03-12T05:09:44.128017+0000","last_fullsized":"2024-03-12T05:09:44.128017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.131225+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:10:13.131225+0000","last_peered":"2024-03-12T05:10:13.131225+0000","last_clean":"2024-03-12T05:10:13.131225+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:10:13.131225+0000","last_undegraded":"2024-03-12T05:10:13.131225+0000","last_fullsized":"2024-03-12T05:10:13.131225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'7","reported_seq":145,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.204402+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:10:39.204402+0000","last_peered":"2024-03-12T05:10:39.204402+0000","last_clean":"2024-03-12T05:10:39.204402+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:10:39.204402+0000","last_undegraded":"2024-03-12T05:10:39.204402+0000","last_fullsized":"2024-03-12T05:10:39.204402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.146033+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:10:13.146033+0000","last_peered":"2024-03-12T05:10:13.146033+0000","last_clean":"2024-03-12T05:10:13.146033+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:10:13.146033+0000","last_undegraded":"2024-03-12T05:10:13.146033+0000","last_fullsized":"2024-03-12T05:10:13.146033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.128862+0000","last_change":"2024-03-12T05:09:44.133911+0000","last_active":"2024-03-12T05:10:13.128862+0000","last_peered":"2024-03-12T05:10:13.128862+0000","last_clean":"2024-03-12T05:10:13.128862+0000","last_became_active":"2024-03-12T05:09:44.133488+0000","last_became_peered":"2024-03-12T05:09:44.133488+0000","last_unstale":"2024-03-12T05:10:13.128862+0000","last_undegraded":"2024-03-12T05:10:13.128862+0000","last_fullsized":"2024-03-12T05:10:13.128862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:44.969275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.515921+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:10:38.515921+0000","last_peered":"2024-03-12T05:10:38.515921+0000","last_clean":"2024-03-12T05:10:38.515921+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:10:38.515921+0000","last_undegraded":"2024-03-12T05:10:38.515921+0000","last_fullsized":"2024-03-12T05:10:38.515921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144632+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:10:13.144632+0000","last_peered":"2024-03-12T05:10:13.144632+0000","last_clean":"2024-03-12T05:10:13.144632+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:10:13.144632+0000","last_undegraded":"2024-03-12T05:10:13.144632+0000","last_fullsized":"2024-03-12T05:10:13.144632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310523+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:09:44.310523+0000","last_peered":"2024-03-12T05:09:44.310523+0000","last_clean":"2024-03-12T05:09:44.310523+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:09:44.310523+0000","last_undegraded":"2024-03-12T05:09:44.310523+0000","last_fullsized":"2024-03-12T05:09:44.310523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125472+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:09:44.125472+0000","last_peered":"2024-03-12T05:09:44.125472+0000","last_clean":"2024-03-12T05:09:44.125472+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:09:44.125472+0000","last_undegraded":"2024-03-12T05:09:44.125472+0000","last_fullsized":"2024-03-12T05:09:44.125472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.126630+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:10:13.126630+0000","last_peered":"2024-03-12T05:10:13.126630+0000","last_clean":"2024-03-12T05:10:13.126630+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:10:13.126630+0000","last_undegraded":"2024-03-12T05:10:13.126630+0000","last_fullsized":"2024-03-12T05:10:13.126630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125644+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:09:44.125644+0000","last_peered":"2024-03-12T05:09:44.125644+0000","last_clean":"2024-03-12T05:09:44.125644+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:09:44.125644+0000","last_undegraded":"2024-03-12T05:09:44.125644+0000","last_fullsized":"2024-03-12T05:09:44.125644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128082+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:09:44.128082+0000","last_peered":"2024-03-12T05:09:44.128082+0000","last_clean":"2024-03-12T05:09:44.128082+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:09:44.128082+0000","last_undegraded":"2024-03-12T05:09:44.128082+0000","last_fullsized":"2024-03-12T05:09:44.128082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.127735+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:10:13.127735+0000","last_peered":"2024-03-12T05:10:13.127735+0000","last_clean":"2024-03-12T05:10:13.127735+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:10:13.127735+0000","last_undegraded":"2024-03-12T05:10:13.127735+0000","last_fullsized":"2024-03-12T05:10:13.127735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.799041+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:10:37.799041+0000","last_peered":"2024-03-12T05:10:37.799041+0000","last_clean":"2024-03-12T05:10:37.799041+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:10:37.799041+0000","last_undegraded":"2024-03-12T05:10:37.799041+0000","last_fullsized":"2024-03-12T05:10:37.799041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139019+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:10:13.139019+0000","last_peered":"2024-03-12T05:10:13.139019+0000","last_clean":"2024-03-12T05:10:13.139019+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:10:13.139019+0000","last_undegraded":"2024-03-12T05:10:13.139019+0000","last_fullsized":"2024-03-12T05:10:13.139019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310664+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:09:44.310664+0000","last_peered":"2024-03-12T05:09:44.310664+0000","last_clean":"2024-03-12T05:09:44.310664+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:09:44.310664+0000","last_undegraded":"2024-03-12T05:09:44.310664+0000","last_fullsized":"2024-03-12T05:09:44.310664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141996+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:10:13.141996+0000","last_peered":"2024-03-12T05:10:13.141996+0000","last_clean":"2024-03-12T05:10:13.141996+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:10:13.141996+0000","last_undegraded":"2024-03-12T05:10:13.141996+0000","last_fullsized":"2024-03-12T05:10:13.141996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.136091+0000","last_change":"2024-03-12T05:09:44.136091+0000","last_active":"2024-03-12T05:09:44.136091+0000","last_peered":"2024-03-12T05:09:44.136091+0000","last_clean":"2024-03-12T05:09:44.136091+0000","last_became_active":"2024-03-12T05:09:44.130865+0000","last_became_peered":"2024-03-12T05:09:44.130865+0000","last_unstale":"2024-03-12T05:09:44.136091+0000","last_undegraded":"2024-03-12T05:09:44.136091+0000","last_fullsized":"2024-03-12T05:09:44.136091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:13.588722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.099732+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:10:13.099732+0000","last_peered":"2024-03-12T05:10:13.099732+0000","last_clean":"2024-03-12T05:10:13.099732+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:10:13.099732+0000","last_undegraded":"2024-03-12T05:10:13.099732+0000","last_fullsized":"2024-03-12T05:10:13.099732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381493+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:09:44.381493+0000","last_peered":"2024-03-12T05:09:44.381493+0000","last_clean":"2024-03-12T05:09:44.381493+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:09:44.381493+0000","last_undegraded":"2024-03-12T05:09:44.381493+0000","last_fullsized":"2024-03-12T05:09:44.381493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141751+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:10:13.141751+0000","last_peered":"2024-03-12T05:10:13.141751+0000","last_clean":"2024-03-12T05:10:13.141751+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:10:13.141751+0000","last_undegraded":"2024-03-12T05:10:13.141751+0000","last_fullsized":"2024-03-12T05:10:13.141751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:36.342780+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:10:36.342780+0000","last_peered":"2024-03-12T05:10:36.342780+0000","last_clean":"2024-03-12T05:10:36.342780+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:10:36.342780+0000","last_undegraded":"2024-03-12T05:10:36.342780+0000","last_fullsized":"2024-03-12T05:10:36.342780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137122+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:10:13.137122+0000","last_peered":"2024-03-12T05:10:13.137122+0000","last_clean":"2024-03-12T05:10:13.137122+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:10:13.137122+0000","last_undegraded":"2024-03-12T05:10:13.137122+0000","last_fullsized":"2024-03-12T05:10:13.137122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.145784+0000","last_change":"2024-03-12T05:09:44.145284+0000","last_active":"2024-03-12T05:09:44.145784+0000","last_peered":"2024-03-12T05:09:44.145784+0000","last_clean":"2024-03-12T05:09:44.145784+0000","last_became_active":"2024-03-12T05:09:44.134640+0000","last_became_peered":"2024-03-12T05:09:44.134640+0000","last_unstale":"2024-03-12T05:09:44.145784+0000","last_undegraded":"2024-03-12T05:09:44.145784+0000","last_fullsized":"2024-03-12T05:09:44.145784+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:17.719372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153940+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:10:13.153940+0000","last_peered":"2024-03-12T05:10:13.153940+0000","last_clean":"2024-03-12T05:10:13.153940+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:10:13.153940+0000","last_undegraded":"2024-03-12T05:10:13.153940+0000","last_fullsized":"2024-03-12T05:10:13.153940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'9","reported_seq":151,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.518590+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:10:39.518590+0000","last_peered":"2024-03-12T05:10:39.518590+0000","last_clean":"2024-03-12T05:10:39.518590+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:10:39.518590+0000","last_undegraded":"2024-03-12T05:10:39.518590+0000","last_fullsized":"2024-03-12T05:10:39.518590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153801+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:10:13.153801+0000","last_peered":"2024-03-12T05:10:13.153801+0000","last_clean":"2024-03-12T05:10:13.153801+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:10:13.153801+0000","last_undegraded":"2024-03-12T05:10:13.153801+0000","last_fullsized":"2024-03-12T05:10:13.153801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131890+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:09:44.131890+0000","last_peered":"2024-03-12T05:09:44.131890+0000","last_clean":"2024-03-12T05:09:44.131890+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:09:44.131890+0000","last_undegraded":"2024-03-12T05:09:44.131890+0000","last_fullsized":"2024-03-12T05:09:44.131890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140168+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:10:13.140168+0000","last_peered":"2024-03-12T05:10:13.140168+0000","last_clean":"2024-03-12T05:10:13.140168+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:10:13.140168+0000","last_undegraded":"2024-03-12T05:10:13.140168+0000","last_fullsized":"2024-03-12T05:10:13.140168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126596+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:09:44.126596+0000","last_peered":"2024-03-12T05:09:44.126596+0000","last_clean":"2024-03-12T05:09:44.126596+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:09:44.126596+0000","last_undegraded":"2024-03-12T05:09:44.126596+0000","last_fullsized":"2024-03-12T05:09:44.126596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140750+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:10:13.140750+0000","last_peered":"2024-03-12T05:10:13.140750+0000","last_clean":"2024-03-12T05:10:13.140750+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:10:13.140750+0000","last_undegraded":"2024-03-12T05:10:13.140750+0000","last_fullsized":"2024-03-12T05:10:13.140750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139173+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:10:13.139173+0000","last_peered":"2024-03-12T05:10:13.139173+0000","last_clean":"2024-03-12T05:10:13.139173+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:10:13.139173+0000","last_undegraded":"2024-03-12T05:10:13.139173+0000","last_fullsized":"2024-03-12T05:10:13.139173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136516+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:10:13.136516+0000","last_peered":"2024-03-12T05:10:13.136516+0000","last_clean":"2024-03-12T05:10:13.136516+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:10:13.136516+0000","last_undegraded":"2024-03-12T05:10:13.136516+0000","last_fullsized":"2024-03-12T05:10:13.136516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136016+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:10:13.136016+0000","last_peered":"2024-03-12T05:10:13.136016+0000","last_clean":"2024-03-12T05:10:13.136016+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:10:13.136016+0000","last_undegraded":"2024-03-12T05:10:13.136016+0000","last_fullsized":"2024-03-12T05:10:13.136016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140867+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:10:13.140867+0000","last_peered":"2024-03-12T05:10:13.140867+0000","last_clean":"2024-03-12T05:10:13.140867+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:10:13.140867+0000","last_undegraded":"2024-03-12T05:10:13.140867+0000","last_fullsized":"2024-03-12T05:10:13.140867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136721+0000","last_change":"2024-03-12T05:09:44.129436+0000","last_active":"2024-03-12T05:10:13.136721+0000","last_peered":"2024-03-12T05:10:13.136721+0000","last_clean":"2024-03-12T05:10:13.136721+0000","last_became_active":"2024-03-12T05:09:44.129063+0000","last_became_peered":"2024-03-12T05:09:44.129063+0000","last_unstale":"2024-03-12T05:10:13.136721+0000","last_undegraded":"2024-03-12T05:10:13.136721+0000","last_fullsized":"2024-03-12T05:10:13.136721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:06.724357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138474+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:10:13.138474+0000","last_peered":"2024-03-12T05:10:13.138474+0000","last_clean":"2024-03-12T05:10:13.138474+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:10:13.138474+0000","last_undegraded":"2024-03-12T05:10:13.138474+0000","last_fullsized":"2024-03-12T05:10:13.138474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152412+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:10:13.152412+0000","last_peered":"2024-03-12T05:10:13.152412+0000","last_clean":"2024-03-12T05:10:13.152412+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:10:13.152412+0000","last_undegraded":"2024-03-12T05:10:13.152412+0000","last_fullsized":"2024-03-12T05:10:13.152412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136405+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:10:13.136405+0000","last_peered":"2024-03-12T05:10:13.136405+0000","last_clean":"2024-03-12T05:10:13.136405+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:10:13.136405+0000","last_undegraded":"2024-03-12T05:10:13.136405+0000","last_fullsized":"2024-03-12T05:10:13.136405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136449+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:10:13.136449+0000","last_peered":"2024-03-12T05:10:13.136449+0000","last_clean":"2024-03-12T05:10:13.136449+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:10:13.136449+0000","last_undegraded":"2024-03-12T05:10:13.136449+0000","last_fullsized":"2024-03-12T05:10:13.136449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136850+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:10:13.136850+0000","last_peered":"2024-03-12T05:10:13.136850+0000","last_clean":"2024-03-12T05:10:13.136850+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:10:13.136850+0000","last_undegraded":"2024-03-12T05:10:13.136850+0000","last_fullsized":"2024-03-12T05:10:13.136850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152464+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:10:13.152464+0000","last_peered":"2024-03-12T05:10:13.152464+0000","last_clean":"2024-03-12T05:10:13.152464+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:10:13.152464+0000","last_undegraded":"2024-03-12T05:10:13.152464+0000","last_fullsized":"2024-03-12T05:10:13.152464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140276+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:10:13.140276+0000","last_peered":"2024-03-12T05:10:13.140276+0000","last_clean":"2024-03-12T05:10:13.140276+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:10:13.140276+0000","last_undegraded":"2024-03-12T05:10:13.140276+0000","last_fullsized":"2024-03-12T05:10:13.140276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144685+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:10:13.144685+0000","last_peered":"2024-03-12T05:10:13.144685+0000","last_clean":"2024-03-12T05:10:13.144685+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:10:13.144685+0000","last_undegraded":"2024-03-12T05:10:13.144685+0000","last_fullsized":"2024-03-12T05:10:13.144685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75051,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1207,"num_read_kb":683,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":220057,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1112443,"data_compressed":37976,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":456383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":456089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1051774,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495679,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":517800,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":458839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":523985,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":455925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1049575,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62539,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68724,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67088,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:10:42.118 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-12T05:10:42.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:10:42.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 -- 172.21.15.45:0/3710739345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8101c20 msgr2=0x7fdab8104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 --2- 172.21.15.45:0/3710739345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8101c20 0x7fdab8104010 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdab0009900 tx=0x7fdab002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 -- 172.21.15.45:0/3710739345 shutdown_connections 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 --2- 172.21.15.45:0/3710739345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 0x7fdab81092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 --2- 172.21.15.45:0/3710739345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8104550 0x7fdab8106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 --2- 172.21.15.45:0/3710739345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8101c20 0x7fdab8104010 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 -- 172.21.15.45:0/3710739345 >> 172.21.15.45:0/3710739345 conn(0x7fdab80fb610 msgr2=0x7fdab80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 -- 172.21.15.45:0/3710739345 shutdown_connections 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 -- 172.21.15.45:0/3710739345 wait complete. 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.260+0000 7fdabd64d640 1 Processor -- start 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- start start 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 0x7fdab81971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 0x7fdab819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab819ecc0 con 0x7fdab8104550 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab819ee30 con 0x7fdab8101c20 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab819eff0 con 0x7fdab8106e80 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47638/0 (socket says 172.21.15.45:47638) 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 -- 172.21.15.45:0/104911061 learned_addr learned my addr 172.21.15.45:0/104911061 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab6ffd640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 0x7fdab81971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab77fe640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 0x7fdab819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 -- 172.21.15.45:0/104911061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 msgr2=0x7fdab819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 0x7fdab819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 -- 172.21.15.45:0/104911061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 msgr2=0x7fdab81971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 0x7fdab81971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 -- 172.21.15.45:0/104911061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab0009560 con 0x7fdab8104550 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab6ffd640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 0x7fdab81971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:42.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdab67fc640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e9c0 tx=0x7fdaa000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0009700 con 0x7fdab8104550 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdaa000cd80 con 0x7fdab8104550 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab819f2d0 con 0x7fdab8104550 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0010430 con 0x7fdab8104550 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.261+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab819f7e0 con 0x7fdab8104550 2024-03-12T05:10:42.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.262+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab80fd100 con 0x7fdab8104550 2024-03-12T05:10:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.264+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdaa0004f50 con 0x7fdab8104550 2024-03-12T05:10:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.264+0000 7fdaaffff640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 0x7fdaa40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.264+0000 7fdab6ffd640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 0x7fdaa40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.264+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fdaa0014070 con 0x7fdab8104550 2024-03-12T05:10:42.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.265+0000 7fdab6ffd640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 0x7fdaa40a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdab0000c00 tx=0x7fdab0005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.268+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdaa0096b00 con 0x7fdab8104550 2024-03-12T05:10:42.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:42 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1358910136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:42.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:42 smithi045 ceph-mon[32174]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:42 smithi045 ceph-mon[32174]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.446+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdab802ceb0 con 0x7fdaa40a1ac0 2024-03-12T05:10:42.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.453+0000 7fdaaffff640 1 -- 172.21.15.45:0/104911061 <== mgr.14152 v2:172.21.15.45:6800/1134390880 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259093 (secure 0 0 0) 0x7fdab802ceb0 con 0x7fdaa40a1ac0 2024-03-12T05:10:42.453 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:42.454 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 msgr2=0x7fdaa40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 0x7fdaa40a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdab0000c00 tx=0x7fdab0005d50 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 msgr2=0x7fdab8197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e9c0 tx=0x7fdaa000ee80 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 shutdown_connections 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fdaa40a1ac0 0x7fdaa40a3f80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdab8104550 0x7fdab8197710 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdab8106e80 0x7fdab819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 --2- 172.21.15.45:0/104911061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101c20 0x7fdab81971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 >> 172.21.15.45:0/104911061 conn(0x7fdab80fb610 msgr2=0x7fdab8102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 shutdown_connections 2024-03-12T05:10:42.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.456+0000 7fdabd64d640 1 -- 172.21.15.45:0/104911061 wait complete. 2024-03-12T05:10:42.464 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:10:41.129487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665653,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1343,"num_read_kb":802,"num_write":456,"num_write_kb":3104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1181408,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373340,"kb_used_data":11708,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568484,"statfs":{"total":1151940427776,"available":1151558127616,"internally_reserved":0,"allocated":11988992,"data_stored":7454604,"data_compressed":113928,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7872,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150307+0000","last_change":"2024-03-12T05:08:37.778346+0000","last_active":"2024-03-12T05:10:13.150307+0000","last_peered":"2024-03-12T05:10:13.150307+0000","last_clean":"2024-03-12T05:10:13.150307+0000","last_became_active":"2024-03-12T05:08:37.777920+0000","last_became_peered":"2024-03-12T05:08:37.777920+0000","last_unstale":"2024-03-12T05:10:13.150307+0000","last_undegraded":"2024-03-12T05:10:13.150307+0000","last_fullsized":"2024-03-12T05:10:13.150307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:58.998305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145142+0000","last_change":"2024-03-12T05:08:37.785143+0000","last_active":"2024-03-12T05:10:13.145142+0000","last_peered":"2024-03-12T05:10:13.145142+0000","last_clean":"2024-03-12T05:10:13.145142+0000","last_became_active":"2024-03-12T05:08:37.784755+0000","last_became_peered":"2024-03-12T05:08:37.784755+0000","last_unstale":"2024-03-12T05:10:13.145142+0000","last_undegraded":"2024-03-12T05:10:13.145142+0000","last_fullsized":"2024-03-12T05:10:13.145142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:30.248453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153971+0000","last_change":"2024-03-12T05:08:37.788916+0000","last_active":"2024-03-12T05:10:13.153971+0000","last_peered":"2024-03-12T05:10:13.153971+0000","last_clean":"2024-03-12T05:10:13.153971+0000","last_became_active":"2024-03-12T05:08:37.788562+0000","last_became_peered":"2024-03-12T05:08:37.788562+0000","last_unstale":"2024-03-12T05:10:13.153971+0000","last_undegraded":"2024-03-12T05:10:13.153971+0000","last_fullsized":"2024-03-12T05:10:13.153971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:22:38.669520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.123843+0000","last_change":"2024-03-12T05:08:37.783425+0000","last_active":"2024-03-12T05:10:13.123843+0000","last_peered":"2024-03-12T05:10:13.123843+0000","last_clean":"2024-03-12T05:10:13.123843+0000","last_became_active":"2024-03-12T05:08:37.782938+0000","last_became_peered":"2024-03-12T05:08:37.782938+0000","last_unstale":"2024-03-12T05:10:13.123843+0000","last_undegraded":"2024-03-12T05:10:13.123843+0000","last_fullsized":"2024-03-12T05:10:13.123843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:50.417643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.130074+0000","last_change":"2024-03-12T05:08:37.771177+0000","last_active":"2024-03-12T05:10:13.130074+0000","last_peered":"2024-03-12T05:10:13.130074+0000","last_clean":"2024-03-12T05:10:13.130074+0000","last_became_active":"2024-03-12T05:08:37.770696+0000","last_became_peered":"2024-03-12T05:08:37.770696+0000","last_unstale":"2024-03-12T05:10:13.130074+0000","last_undegraded":"2024-03-12T05:10:13.130074+0000","last_fullsized":"2024-03-12T05:10:13.130074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:58.509686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143421+0000","last_change":"2024-03-12T05:08:37.777161+0000","last_active":"2024-03-12T05:10:13.143421+0000","last_peered":"2024-03-12T05:10:13.143421+0000","last_clean":"2024-03-12T05:10:13.143421+0000","last_became_active":"2024-03-12T05:08:37.776843+0000","last_became_peered":"2024-03-12T05:08:37.776843+0000","last_unstale":"2024-03-12T05:10:13.143421+0000","last_undegraded":"2024-03-12T05:10:13.143421+0000","last_fullsized":"2024-03-12T05:10:13.143421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:42.119704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138393+0000","last_change":"2024-03-12T05:08:37.769411+0000","last_active":"2024-03-12T05:10:13.138393+0000","last_peered":"2024-03-12T05:10:13.138393+0000","last_clean":"2024-03-12T05:10:13.138393+0000","last_became_active":"2024-03-12T05:08:37.769110+0000","last_became_peered":"2024-03-12T05:08:37.769110+0000","last_unstale":"2024-03-12T05:10:13.138393+0000","last_undegraded":"2024-03-12T05:10:13.138393+0000","last_fullsized":"2024-03-12T05:10:13.138393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:08:00.821218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136869+0000","last_change":"2024-03-12T05:08:37.783124+0000","last_active":"2024-03-12T05:10:13.136869+0000","last_peered":"2024-03-12T05:10:13.136869+0000","last_clean":"2024-03-12T05:10:13.136869+0000","last_became_active":"2024-03-12T05:08:37.776606+0000","last_became_peered":"2024-03-12T05:08:37.776606+0000","last_unstale":"2024-03-12T05:10:13.136869+0000","last_undegraded":"2024-03-12T05:10:13.136869+0000","last_fullsized":"2024-03-12T05:10:13.136869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:45.776297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154745+0000","last_change":"2024-03-12T05:08:37.776783+0000","last_active":"2024-03-12T05:10:13.154745+0000","last_peered":"2024-03-12T05:10:13.154745+0000","last_clean":"2024-03-12T05:10:13.154745+0000","last_became_active":"2024-03-12T05:08:37.776465+0000","last_became_peered":"2024-03-12T05:08:37.776465+0000","last_unstale":"2024-03-12T05:10:13.154745+0000","last_undegraded":"2024-03-12T05:10:13.154745+0000","last_fullsized":"2024-03-12T05:10:13.154745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:08.066978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143667+0000","last_change":"2024-03-12T05:08:37.776304+0000","last_active":"2024-03-12T05:10:13.143667+0000","last_peered":"2024-03-12T05:10:13.143667+0000","last_clean":"2024-03-12T05:10:13.143667+0000","last_became_active":"2024-03-12T05:08:37.775921+0000","last_became_peered":"2024-03-12T05:08:37.775921+0000","last_unstale":"2024-03-12T05:10:13.143667+0000","last_undegraded":"2024-03-12T05:10:13.143667+0000","last_fullsized":"2024-03-12T05:10:13.143667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:49.003425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153713+0000","last_change":"2024-03-12T05:08:37.780735+0000","last_active":"2024-03-12T05:10:13.153713+0000","last_peered":"2024-03-12T05:10:13.153713+0000","last_clean":"2024-03-12T05:10:13.153713+0000","last_became_active":"2024-03-12T05:08:37.780439+0000","last_became_peered":"2024-03-12T05:08:37.780439+0000","last_unstale":"2024-03-12T05:10:13.153713+0000","last_undegraded":"2024-03-12T05:10:13.153713+0000","last_fullsized":"2024-03-12T05:10:13.153713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:58.538585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.143429+0000","last_change":"2024-03-12T05:08:37.785570+0000","last_active":"2024-03-12T05:10:13.143429+0000","last_peered":"2024-03-12T05:10:13.143429+0000","last_clean":"2024-03-12T05:10:13.143429+0000","last_became_active":"2024-03-12T05:08:37.784036+0000","last_became_peered":"2024-03-12T05:08:37.784036+0000","last_unstale":"2024-03-12T05:10:13.143429+0000","last_undegraded":"2024-03-12T05:10:13.143429+0000","last_fullsized":"2024-03-12T05:10:13.143429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:29.594403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138246+0000","last_change":"2024-03-12T05:08:37.760284+0000","last_active":"2024-03-12T05:10:13.138246+0000","last_peered":"2024-03-12T05:10:13.138246+0000","last_clean":"2024-03-12T05:10:13.138246+0000","last_became_active":"2024-03-12T05:08:37.759838+0000","last_became_peered":"2024-03-12T05:08:37.759838+0000","last_unstale":"2024-03-12T05:10:13.138246+0000","last_undegraded":"2024-03-12T05:10:13.138246+0000","last_fullsized":"2024-03-12T05:10:13.138246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:13.830223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145379+0000","last_change":"2024-03-12T05:08:37.783582+0000","last_active":"2024-03-12T05:10:13.145379+0000","last_peered":"2024-03-12T05:10:13.145379+0000","last_clean":"2024-03-12T05:10:13.145379+0000","last_became_active":"2024-03-12T05:08:37.783239+0000","last_became_peered":"2024-03-12T05:08:37.783239+0000","last_unstale":"2024-03-12T05:10:13.145379+0000","last_undegraded":"2024-03-12T05:10:13.145379+0000","last_fullsized":"2024-03-12T05:10:13.145379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:43.598881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140306+0000","last_change":"2024-03-12T05:08:37.781319+0000","last_active":"2024-03-12T05:10:13.140306+0000","last_peered":"2024-03-12T05:10:13.140306+0000","last_clean":"2024-03-12T05:10:13.140306+0000","last_became_active":"2024-03-12T05:08:37.780972+0000","last_became_peered":"2024-03-12T05:08:37.780972+0000","last_unstale":"2024-03-12T05:10:13.140306+0000","last_undegraded":"2024-03-12T05:10:13.140306+0000","last_fullsized":"2024-03-12T05:10:13.140306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:17.249200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154503+0000","last_change":"2024-03-12T05:08:37.777465+0000","last_active":"2024-03-12T05:10:13.154503+0000","last_peered":"2024-03-12T05:10:13.154503+0000","last_clean":"2024-03-12T05:10:13.154503+0000","last_became_active":"2024-03-12T05:08:37.774943+0000","last_became_peered":"2024-03-12T05:08:37.774943+0000","last_unstale":"2024-03-12T05:10:13.154503+0000","last_undegraded":"2024-03-12T05:10:13.154503+0000","last_fullsized":"2024-03-12T05:10:13.154503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:44.071883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.150237+0000","last_change":"2024-03-12T05:08:37.780127+0000","last_active":"2024-03-12T05:10:13.150237+0000","last_peered":"2024-03-12T05:10:13.150237+0000","last_clean":"2024-03-12T05:10:13.150237+0000","last_became_active":"2024-03-12T05:08:37.779801+0000","last_became_peered":"2024-03-12T05:08:37.779801+0000","last_unstale":"2024-03-12T05:10:13.150237+0000","last_undegraded":"2024-03-12T05:10:13.150237+0000","last_fullsized":"2024-03-12T05:10:13.150237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:22:45.622198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144239+0000","last_change":"2024-03-12T05:08:37.774317+0000","last_active":"2024-03-12T05:10:13.144239+0000","last_peered":"2024-03-12T05:10:13.144239+0000","last_clean":"2024-03-12T05:10:13.144239+0000","last_became_active":"2024-03-12T05:08:37.773903+0000","last_became_peered":"2024-03-12T05:08:37.773903+0000","last_unstale":"2024-03-12T05:10:13.144239+0000","last_undegraded":"2024-03-12T05:10:13.144239+0000","last_fullsized":"2024-03-12T05:10:13.144239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:37.921060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.133361+0000","last_change":"2024-03-12T05:08:37.781642+0000","last_active":"2024-03-12T05:10:13.133361+0000","last_peered":"2024-03-12T05:10:13.133361+0000","last_clean":"2024-03-12T05:10:13.133361+0000","last_became_active":"2024-03-12T05:08:37.781314+0000","last_became_peered":"2024-03-12T05:08:37.781314+0000","last_unstale":"2024-03-12T05:10:13.133361+0000","last_undegraded":"2024-03-12T05:10:13.133361+0000","last_fullsized":"2024-03-12T05:10:13.133361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:39.349302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'14","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.202691+0000","last_change":"2024-03-12T05:09:44.142398+0000","last_active":"2024-03-12T05:10:38.202691+0000","last_peered":"2024-03-12T05:10:38.202691+0000","last_clean":"2024-03-12T05:10:38.202691+0000","last_became_active":"2024-03-12T05:09:44.129576+0000","last_became_peered":"2024-03-12T05:09:44.129576+0000","last_unstale":"2024-03-12T05:10:38.202691+0000","last_undegraded":"2024-03-12T05:10:38.202691+0000","last_fullsized":"2024-03-12T05:10:38.202691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:51:32.791210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135879+0000","last_change":"2024-03-12T05:09:44.135049+0000","last_active":"2024-03-12T05:10:13.135879+0000","last_peered":"2024-03-12T05:10:13.135879+0000","last_clean":"2024-03-12T05:10:13.135879+0000","last_became_active":"2024-03-12T05:09:44.134638+0000","last_became_peered":"2024-03-12T05:09:44.134638+0000","last_unstale":"2024-03-12T05:10:13.135879+0000","last_undegraded":"2024-03-12T05:10:13.135879+0000","last_fullsized":"2024-03-12T05:10:13.135879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:28:59.785062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381970+0000","last_change":"2024-03-12T05:08:36.776461+0000","last_active":"2024-03-12T05:09:44.381970+0000","last_peered":"2024-03-12T05:09:44.381970+0000","last_clean":"2024-03-12T05:09:44.381970+0000","last_became_active":"2024-03-12T05:08:36.776108+0000","last_became_peered":"2024-03-12T05:08:36.776108+0000","last_unstale":"2024-03-12T05:09:44.381970+0000","last_undegraded":"2024-03-12T05:09:44.381970+0000","last_fullsized":"2024-03-12T05:09:44.381970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:57.274716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.155371+0000","last_change":"2024-03-12T05:08:37.780537+0000","last_active":"2024-03-12T05:10:13.155371+0000","last_peered":"2024-03-12T05:10:13.155371+0000","last_clean":"2024-03-12T05:10:13.155371+0000","last_became_active":"2024-03-12T05:08:37.780213+0000","last_became_peered":"2024-03-12T05:08:37.780213+0000","last_unstale":"2024-03-12T05:10:13.155371+0000","last_undegraded":"2024-03-12T05:10:13.155371+0000","last_fullsized":"2024-03-12T05:10:13.155371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:17.603921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128279+0000","last_change":"2024-03-12T05:08:36.774205+0000","last_active":"2024-03-12T05:09:44.128279+0000","last_peered":"2024-03-12T05:09:44.128279+0000","last_clean":"2024-03-12T05:09:44.128279+0000","last_became_active":"2024-03-12T05:08:36.773624+0000","last_became_peered":"2024-03-12T05:08:36.773624+0000","last_unstale":"2024-03-12T05:09:44.128279+0000","last_undegraded":"2024-03-12T05:09:44.128279+0000","last_fullsized":"2024-03-12T05:09:44.128279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:11.842997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152192+0000","last_change":"2024-03-12T05:08:37.783734+0000","last_active":"2024-03-12T05:10:13.152192+0000","last_peered":"2024-03-12T05:10:13.152192+0000","last_clean":"2024-03-12T05:10:13.152192+0000","last_became_active":"2024-03-12T05:08:37.783415+0000","last_became_peered":"2024-03-12T05:08:37.783415+0000","last_unstale":"2024-03-12T05:10:13.152192+0000","last_undegraded":"2024-03-12T05:10:13.152192+0000","last_fullsized":"2024-03-12T05:10:13.152192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:58:07.362800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128527+0000","last_change":"2024-03-12T05:08:36.770290+0000","last_active":"2024-03-12T05:09:44.128527+0000","last_peered":"2024-03-12T05:09:44.128527+0000","last_clean":"2024-03-12T05:09:44.128527+0000","last_became_active":"2024-03-12T05:08:36.769949+0000","last_became_peered":"2024-03-12T05:08:36.769949+0000","last_unstale":"2024-03-12T05:09:44.128527+0000","last_undegraded":"2024-03-12T05:09:44.128527+0000","last_fullsized":"2024-03-12T05:09:44.128527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:04:50.190188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152370+0000","last_change":"2024-03-12T05:09:44.133756+0000","last_active":"2024-03-12T05:10:13.152370+0000","last_peered":"2024-03-12T05:10:13.152370+0000","last_clean":"2024-03-12T05:10:13.152370+0000","last_became_active":"2024-03-12T05:09:44.133312+0000","last_became_peered":"2024-03-12T05:09:44.133312+0000","last_unstale":"2024-03-12T05:10:13.152370+0000","last_undegraded":"2024-03-12T05:10:13.152370+0000","last_fullsized":"2024-03-12T05:10:13.152370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:54:24.882257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.515180+0000","last_change":"2024-03-12T05:08:36.774049+0000","last_active":"2024-03-12T05:10:37.515180+0000","last_peered":"2024-03-12T05:10:37.515180+0000","last_clean":"2024-03-12T05:10:37.515180+0000","last_became_active":"2024-03-12T05:08:36.773713+0000","last_became_peered":"2024-03-12T05:08:36.773713+0000","last_unstale":"2024-03-12T05:10:37.515180+0000","last_undegraded":"2024-03-12T05:10:37.515180+0000","last_fullsized":"2024-03-12T05:10:37.515180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:18:38.281175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.135098+0000","last_change":"2024-03-12T05:08:37.768981+0000","last_active":"2024-03-12T05:10:13.135098+0000","last_peered":"2024-03-12T05:10:13.135098+0000","last_clean":"2024-03-12T05:10:13.135098+0000","last_became_active":"2024-03-12T05:08:37.768528+0000","last_became_peered":"2024-03-12T05:08:37.768528+0000","last_unstale":"2024-03-12T05:10:13.135098+0000","last_undegraded":"2024-03-12T05:10:13.135098+0000","last_fullsized":"2024-03-12T05:10:13.135098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:35.310163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381204+0000","last_change":"2024-03-12T05:08:36.772779+0000","last_active":"2024-03-12T05:09:44.381204+0000","last_peered":"2024-03-12T05:09:44.381204+0000","last_clean":"2024-03-12T05:09:44.381204+0000","last_became_active":"2024-03-12T05:08:36.772455+0000","last_became_peered":"2024-03-12T05:08:36.772455+0000","last_unstale":"2024-03-12T05:09:44.381204+0000","last_undegraded":"2024-03-12T05:09:44.381204+0000","last_fullsized":"2024-03-12T05:09:44.381204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:49.856091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144231+0000","last_change":"2024-03-12T05:08:37.778044+0000","last_active":"2024-03-12T05:10:13.144231+0000","last_peered":"2024-03-12T05:10:13.144231+0000","last_clean":"2024-03-12T05:10:13.144231+0000","last_became_active":"2024-03-12T05:08:37.777673+0000","last_became_peered":"2024-03-12T05:08:37.777673+0000","last_unstale":"2024-03-12T05:10:13.144231+0000","last_undegraded":"2024-03-12T05:10:13.144231+0000","last_fullsized":"2024-03-12T05:10:13.144231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:39:56.026432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126104+0000","last_change":"2024-03-12T05:08:36.771172+0000","last_active":"2024-03-12T05:09:44.126104+0000","last_peered":"2024-03-12T05:09:44.126104+0000","last_clean":"2024-03-12T05:09:44.126104+0000","last_became_active":"2024-03-12T05:08:36.770844+0000","last_became_peered":"2024-03-12T05:08:36.770844+0000","last_unstale":"2024-03-12T05:09:44.126104+0000","last_undegraded":"2024-03-12T05:09:44.126104+0000","last_fullsized":"2024-03-12T05:09:44.126104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:14.199219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.134969+0000","last_change":"2024-03-12T05:08:37.784196+0000","last_active":"2024-03-12T05:10:13.134969+0000","last_peered":"2024-03-12T05:10:13.134969+0000","last_clean":"2024-03-12T05:10:13.134969+0000","last_became_active":"2024-03-12T05:08:37.783853+0000","last_became_peered":"2024-03-12T05:08:37.783853+0000","last_unstale":"2024-03-12T05:10:13.134969+0000","last_undegraded":"2024-03-12T05:10:13.134969+0000","last_fullsized":"2024-03-12T05:10:13.134969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.744519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128856+0000","last_change":"2024-03-12T05:08:36.768923+0000","last_active":"2024-03-12T05:09:44.128856+0000","last_peered":"2024-03-12T05:09:44.128856+0000","last_clean":"2024-03-12T05:09:44.128856+0000","last_became_active":"2024-03-12T05:08:36.768568+0000","last_became_peered":"2024-03-12T05:08:36.768568+0000","last_unstale":"2024-03-12T05:09:44.128856+0000","last_undegraded":"2024-03-12T05:09:44.128856+0000","last_fullsized":"2024-03-12T05:09:44.128856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:35:49.999225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137840+0000","last_change":"2024-03-12T05:08:37.775816+0000","last_active":"2024-03-12T05:10:13.137840+0000","last_peered":"2024-03-12T05:10:13.137840+0000","last_clean":"2024-03-12T05:10:13.137840+0000","last_became_active":"2024-03-12T05:08:37.775497+0000","last_became_peered":"2024-03-12T05:08:37.775497+0000","last_unstale":"2024-03-12T05:10:13.137840+0000","last_undegraded":"2024-03-12T05:10:13.137840+0000","last_fullsized":"2024-03-12T05:10:13.137840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:52:23.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'8","reported_seq":259,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.516014+0000","last_change":"2024-03-12T05:08:36.769624+0000","last_active":"2024-03-12T05:10:38.516014+0000","last_peered":"2024-03-12T05:10:38.516014+0000","last_clean":"2024-03-12T05:10:38.516014+0000","last_became_active":"2024-03-12T05:08:36.769334+0000","last_became_peered":"2024-03-12T05:08:36.769334+0000","last_unstale":"2024-03-12T05:10:38.516014+0000","last_undegraded":"2024-03-12T05:10:38.516014+0000","last_fullsized":"2024-03-12T05:10:38.516014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:16.947305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.148250+0000","last_change":"2024-03-12T05:08:37.771783+0000","last_active":"2024-03-12T05:10:13.148250+0000","last_peered":"2024-03-12T05:10:13.148250+0000","last_clean":"2024-03-12T05:10:13.148250+0000","last_became_active":"2024-03-12T05:08:37.770837+0000","last_became_peered":"2024-03-12T05:08:37.770837+0000","last_unstale":"2024-03-12T05:10:13.148250+0000","last_undegraded":"2024-03-12T05:10:13.148250+0000","last_fullsized":"2024-03-12T05:10:13.148250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:36.572534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.315313+0000","last_change":"2024-03-12T05:09:44.315313+0000","last_active":"2024-03-12T05:09:44.315313+0000","last_peered":"2024-03-12T05:09:44.315313+0000","last_clean":"2024-03-12T05:09:44.315313+0000","last_became_active":"2024-03-12T05:09:44.311453+0000","last_became_peered":"2024-03-12T05:09:44.311453+0000","last_unstale":"2024-03-12T05:09:44.315313+0000","last_undegraded":"2024-03-12T05:09:44.315313+0000","last_fullsized":"2024-03-12T05:09:44.315313+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:21.022485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.142842+0000","last_change":"2024-03-12T05:08:37.782277+0000","last_active":"2024-03-12T05:10:13.142842+0000","last_peered":"2024-03-12T05:10:13.142842+0000","last_clean":"2024-03-12T05:10:13.142842+0000","last_became_active":"2024-03-12T05:08:37.781939+0000","last_became_peered":"2024-03-12T05:08:37.781939+0000","last_unstale":"2024-03-12T05:10:13.142842+0000","last_undegraded":"2024-03-12T05:10:13.142842+0000","last_fullsized":"2024-03-12T05:10:13.142842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:01.457407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.132030+0000","last_change":"2024-03-12T05:08:36.775446+0000","last_active":"2024-03-12T05:09:44.132030+0000","last_peered":"2024-03-12T05:09:44.132030+0000","last_clean":"2024-03-12T05:09:44.132030+0000","last_became_active":"2024-03-12T05:08:36.775145+0000","last_became_peered":"2024-03-12T05:08:36.775145+0000","last_unstale":"2024-03-12T05:09:44.132030+0000","last_undegraded":"2024-03-12T05:09:44.132030+0000","last_fullsized":"2024-03-12T05:09:44.132030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:37.381105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.154851+0000","last_change":"2024-03-12T05:08:37.791737+0000","last_active":"2024-03-12T05:10:13.154851+0000","last_peered":"2024-03-12T05:10:13.154851+0000","last_clean":"2024-03-12T05:10:13.154851+0000","last_became_active":"2024-03-12T05:08:37.791408+0000","last_became_peered":"2024-03-12T05:08:37.791408+0000","last_unstale":"2024-03-12T05:10:13.154851+0000","last_undegraded":"2024-03-12T05:10:13.154851+0000","last_fullsized":"2024-03-12T05:10:13.154851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:58.020548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'35","reported_seq":295,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.517196+0000","last_change":"2024-03-12T05:08:36.769309+0000","last_active":"2024-03-12T05:10:38.517196+0000","last_peered":"2024-03-12T05:10:38.517196+0000","last_clean":"2024-03-12T05:10:38.517196+0000","last_became_active":"2024-03-12T05:08:36.768982+0000","last_became_peered":"2024-03-12T05:08:36.768982+0000","last_unstale":"2024-03-12T05:10:38.517196+0000","last_undegraded":"2024-03-12T05:10:38.517196+0000","last_fullsized":"2024-03-12T05:10:38.517196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:52:40.120894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64343,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":183,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138155+0000","last_change":"2024-03-12T05:08:37.773820+0000","last_active":"2024-03-12T05:10:13.138155+0000","last_peered":"2024-03-12T05:10:13.138155+0000","last_clean":"2024-03-12T05:10:13.138155+0000","last_became_active":"2024-03-12T05:08:37.773447+0000","last_became_peered":"2024-03-12T05:08:37.773447+0000","last_unstale":"2024-03-12T05:10:13.138155+0000","last_undegraded":"2024-03-12T05:10:13.138155+0000","last_fullsized":"2024-03-12T05:10:13.138155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:05:42.586306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131709+0000","last_change":"2024-03-12T05:08:36.766483+0000","last_active":"2024-03-12T05:09:44.131709+0000","last_peered":"2024-03-12T05:09:44.131709+0000","last_clean":"2024-03-12T05:09:44.131709+0000","last_became_active":"2024-03-12T05:08:36.766130+0000","last_became_peered":"2024-03-12T05:08:36.766130+0000","last_unstale":"2024-03-12T05:09:44.131709+0000","last_undegraded":"2024-03-12T05:09:44.131709+0000","last_fullsized":"2024-03-12T05:09:44.131709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.897452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139579+0000","last_change":"2024-03-12T05:08:37.779959+0000","last_active":"2024-03-12T05:10:13.139579+0000","last_peered":"2024-03-12T05:10:13.139579+0000","last_clean":"2024-03-12T05:10:13.139579+0000","last_became_active":"2024-03-12T05:08:37.779074+0000","last_became_peered":"2024-03-12T05:08:37.779074+0000","last_unstale":"2024-03-12T05:10:13.139579+0000","last_undegraded":"2024-03-12T05:10:13.139579+0000","last_fullsized":"2024-03-12T05:10:13.139579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:57.731528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128625+0000","last_change":"2024-03-12T05:08:36.774586+0000","last_active":"2024-03-12T05:09:44.128625+0000","last_peered":"2024-03-12T05:09:44.128625+0000","last_clean":"2024-03-12T05:09:44.128625+0000","last_became_active":"2024-03-12T05:08:36.773646+0000","last_became_peered":"2024-03-12T05:08:36.773646+0000","last_unstale":"2024-03-12T05:09:44.128625+0000","last_undegraded":"2024-03-12T05:09:44.128625+0000","last_fullsized":"2024-03-12T05:09:44.128625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:09.241627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140001+0000","last_change":"2024-03-12T05:08:37.776342+0000","last_active":"2024-03-12T05:10:13.140001+0000","last_peered":"2024-03-12T05:10:13.140001+0000","last_clean":"2024-03-12T05:10:13.140001+0000","last_became_active":"2024-03-12T05:08:37.774634+0000","last_became_peered":"2024-03-12T05:08:37.774634+0000","last_unstale":"2024-03-12T05:10:13.140001+0000","last_undegraded":"2024-03-12T05:10:13.140001+0000","last_fullsized":"2024-03-12T05:10:13.140001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:35.200956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'8","reported_seq":147,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.514995+0000","last_change":"2024-03-12T05:08:36.772126+0000","last_active":"2024-03-12T05:10:37.514995+0000","last_peered":"2024-03-12T05:10:37.514995+0000","last_clean":"2024-03-12T05:10:37.514995+0000","last_became_active":"2024-03-12T05:08:36.767964+0000","last_became_peered":"2024-03-12T05:08:36.767964+0000","last_unstale":"2024-03-12T05:10:37.514995+0000","last_undegraded":"2024-03-12T05:10:37.514995+0000","last_fullsized":"2024-03-12T05:10:37.514995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:52.295993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141856+0000","last_change":"2024-03-12T05:08:37.772284+0000","last_active":"2024-03-12T05:10:13.141856+0000","last_peered":"2024-03-12T05:10:13.141856+0000","last_clean":"2024-03-12T05:10:13.141856+0000","last_became_active":"2024-03-12T05:08:37.771973+0000","last_became_peered":"2024-03-12T05:08:37.771973+0000","last_unstale":"2024-03-12T05:10:13.141856+0000","last_undegraded":"2024-03-12T05:10:13.141856+0000","last_fullsized":"2024-03-12T05:10:13.141856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:46.406989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.127773+0000","last_change":"2024-03-12T05:08:36.776707+0000","last_active":"2024-03-12T05:09:44.127773+0000","last_peered":"2024-03-12T05:09:44.127773+0000","last_clean":"2024-03-12T05:09:44.127773+0000","last_became_active":"2024-03-12T05:08:36.776310+0000","last_became_peered":"2024-03-12T05:08:36.776310+0000","last_unstale":"2024-03-12T05:09:44.127773+0000","last_undegraded":"2024-03-12T05:09:44.127773+0000","last_fullsized":"2024-03-12T05:09:44.127773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:44.991141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.145977+0000","last_change":"2024-03-12T05:08:37.776426+0000","last_active":"2024-03-12T05:10:13.145977+0000","last_peered":"2024-03-12T05:10:13.145977+0000","last_clean":"2024-03-12T05:10:13.145977+0000","last_became_active":"2024-03-12T05:08:37.776082+0000","last_became_peered":"2024-03-12T05:08:37.776082+0000","last_unstale":"2024-03-12T05:10:13.145977+0000","last_undegraded":"2024-03-12T05:10:13.145977+0000","last_fullsized":"2024-03-12T05:10:13.145977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:59:04.208899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128951+0000","last_change":"2024-03-12T05:08:36.766534+0000","last_active":"2024-03-12T05:09:44.128951+0000","last_peered":"2024-03-12T05:09:44.128951+0000","last_clean":"2024-03-12T05:09:44.128951+0000","last_became_active":"2024-03-12T05:08:36.766223+0000","last_became_peered":"2024-03-12T05:08:36.766223+0000","last_unstale":"2024-03-12T05:09:44.128951+0000","last_undegraded":"2024-03-12T05:09:44.128951+0000","last_fullsized":"2024-03-12T05:09:44.128951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:39:47.159962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144510+0000","last_change":"2024-03-12T05:08:37.787947+0000","last_active":"2024-03-12T05:10:13.144510+0000","last_peered":"2024-03-12T05:10:13.144510+0000","last_clean":"2024-03-12T05:10:13.144510+0000","last_became_active":"2024-03-12T05:08:37.787615+0000","last_became_peered":"2024-03-12T05:08:37.787615+0000","last_unstale":"2024-03-12T05:10:13.144510+0000","last_undegraded":"2024-03-12T05:10:13.144510+0000","last_fullsized":"2024-03-12T05:10:13.144510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:11.882813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128017+0000","last_change":"2024-03-12T05:08:36.769273+0000","last_active":"2024-03-12T05:09:44.128017+0000","last_peered":"2024-03-12T05:09:44.128017+0000","last_clean":"2024-03-12T05:09:44.128017+0000","last_became_active":"2024-03-12T05:08:36.768938+0000","last_became_peered":"2024-03-12T05:08:36.768938+0000","last_unstale":"2024-03-12T05:09:44.128017+0000","last_undegraded":"2024-03-12T05:09:44.128017+0000","last_fullsized":"2024-03-12T05:09:44.128017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:15:31.433171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.131225+0000","last_change":"2024-03-12T05:08:37.770457+0000","last_active":"2024-03-12T05:10:13.131225+0000","last_peered":"2024-03-12T05:10:13.131225+0000","last_clean":"2024-03-12T05:10:13.131225+0000","last_became_active":"2024-03-12T05:08:37.767980+0000","last_became_peered":"2024-03-12T05:08:37.767980+0000","last_unstale":"2024-03-12T05:10:13.131225+0000","last_undegraded":"2024-03-12T05:10:13.131225+0000","last_fullsized":"2024-03-12T05:10:13.131225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:18.591323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'7","reported_seq":145,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.204402+0000","last_change":"2024-03-12T05:08:36.774143+0000","last_active":"2024-03-12T05:10:39.204402+0000","last_peered":"2024-03-12T05:10:39.204402+0000","last_clean":"2024-03-12T05:10:39.204402+0000","last_became_active":"2024-03-12T05:08:36.773828+0000","last_became_peered":"2024-03-12T05:08:36.773828+0000","last_unstale":"2024-03-12T05:10:39.204402+0000","last_undegraded":"2024-03-12T05:10:39.204402+0000","last_fullsized":"2024-03-12T05:10:39.204402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:06.035100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.146033+0000","last_change":"2024-03-12T05:08:37.776362+0000","last_active":"2024-03-12T05:10:13.146033+0000","last_peered":"2024-03-12T05:10:13.146033+0000","last_clean":"2024-03-12T05:10:13.146033+0000","last_became_active":"2024-03-12T05:08:37.775982+0000","last_became_peered":"2024-03-12T05:08:37.775982+0000","last_unstale":"2024-03-12T05:10:13.146033+0000","last_undegraded":"2024-03-12T05:10:13.146033+0000","last_fullsized":"2024-03-12T05:10:13.146033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:09.554114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.128862+0000","last_change":"2024-03-12T05:09:44.133911+0000","last_active":"2024-03-12T05:10:13.128862+0000","last_peered":"2024-03-12T05:10:13.128862+0000","last_clean":"2024-03-12T05:10:13.128862+0000","last_became_active":"2024-03-12T05:09:44.133488+0000","last_became_peered":"2024-03-12T05:09:44.133488+0000","last_unstale":"2024-03-12T05:10:13.128862+0000","last_undegraded":"2024-03-12T05:10:13.128862+0000","last_fullsized":"2024-03-12T05:10:13.128862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:44.969275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"87'7","reported_seq":144,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:38.515921+0000","last_change":"2024-03-12T05:08:36.770349+0000","last_active":"2024-03-12T05:10:38.515921+0000","last_peered":"2024-03-12T05:10:38.515921+0000","last_clean":"2024-03-12T05:10:38.515921+0000","last_became_active":"2024-03-12T05:08:36.770022+0000","last_became_peered":"2024-03-12T05:08:36.770022+0000","last_unstale":"2024-03-12T05:10:38.515921+0000","last_undegraded":"2024-03-12T05:10:38.515921+0000","last_fullsized":"2024-03-12T05:10:38.515921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:50:49.795416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144632+0000","last_change":"2024-03-12T05:08:37.790664+0000","last_active":"2024-03-12T05:10:13.144632+0000","last_peered":"2024-03-12T05:10:13.144632+0000","last_clean":"2024-03-12T05:10:13.144632+0000","last_became_active":"2024-03-12T05:08:37.790220+0000","last_became_peered":"2024-03-12T05:08:37.790220+0000","last_unstale":"2024-03-12T05:10:13.144632+0000","last_undegraded":"2024-03-12T05:10:13.144632+0000","last_fullsized":"2024-03-12T05:10:13.144632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:12:00.262594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310523+0000","last_change":"2024-03-12T05:08:36.771899+0000","last_active":"2024-03-12T05:09:44.310523+0000","last_peered":"2024-03-12T05:09:44.310523+0000","last_clean":"2024-03-12T05:09:44.310523+0000","last_became_active":"2024-03-12T05:08:36.771548+0000","last_became_peered":"2024-03-12T05:08:36.771548+0000","last_unstale":"2024-03-12T05:09:44.310523+0000","last_undegraded":"2024-03-12T05:09:44.310523+0000","last_fullsized":"2024-03-12T05:09:44.310523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:14.660309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125472+0000","last_change":"2024-03-12T05:07:27.330025+0000","last_active":"2024-03-12T05:09:44.125472+0000","last_peered":"2024-03-12T05:09:44.125472+0000","last_clean":"2024-03-12T05:09:44.125472+0000","last_became_active":"2024-03-12T05:07:27.329627+0000","last_became_peered":"2024-03-12T05:07:27.329627+0000","last_unstale":"2024-03-12T05:09:44.125472+0000","last_undegraded":"2024-03-12T05:09:44.125472+0000","last_fullsized":"2024-03-12T05:09:44.125472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:04:07.692505+0000","last_clean_scrub_stamp":"2024-03-12T05:04:07.692505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:26:02.645390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.126630+0000","last_change":"2024-03-12T05:08:37.776243+0000","last_active":"2024-03-12T05:10:13.126630+0000","last_peered":"2024-03-12T05:10:13.126630+0000","last_clean":"2024-03-12T05:10:13.126630+0000","last_became_active":"2024-03-12T05:08:37.775925+0000","last_became_peered":"2024-03-12T05:08:37.775925+0000","last_unstale":"2024-03-12T05:10:13.126630+0000","last_undegraded":"2024-03-12T05:10:13.126630+0000","last_fullsized":"2024-03-12T05:10:13.126630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:25.883185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.125644+0000","last_change":"2024-03-12T05:08:36.776138+0000","last_active":"2024-03-12T05:09:44.125644+0000","last_peered":"2024-03-12T05:09:44.125644+0000","last_clean":"2024-03-12T05:09:44.125644+0000","last_became_active":"2024-03-12T05:08:36.774483+0000","last_became_peered":"2024-03-12T05:08:36.774483+0000","last_unstale":"2024-03-12T05:09:44.125644+0000","last_undegraded":"2024-03-12T05:09:44.125644+0000","last_fullsized":"2024-03-12T05:09:44.125644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:17:22.297508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.128082+0000","last_change":"2024-03-12T05:08:36.776691+0000","last_active":"2024-03-12T05:09:44.128082+0000","last_peered":"2024-03-12T05:09:44.128082+0000","last_clean":"2024-03-12T05:09:44.128082+0000","last_became_active":"2024-03-12T05:08:36.775967+0000","last_became_peered":"2024-03-12T05:08:36.775967+0000","last_unstale":"2024-03-12T05:09:44.128082+0000","last_undegraded":"2024-03-12T05:09:44.128082+0000","last_fullsized":"2024-03-12T05:09:44.128082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:22:41.338397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.127735+0000","last_change":"2024-03-12T05:08:37.772360+0000","last_active":"2024-03-12T05:10:13.127735+0000","last_peered":"2024-03-12T05:10:13.127735+0000","last_clean":"2024-03-12T05:10:13.127735+0000","last_became_active":"2024-03-12T05:08:37.771989+0000","last_became_peered":"2024-03-12T05:08:37.771989+0000","last_unstale":"2024-03-12T05:10:13.127735+0000","last_undegraded":"2024-03-12T05:10:13.127735+0000","last_fullsized":"2024-03-12T05:10:13.127735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:28:38.376424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:37.799041+0000","last_change":"2024-03-12T05:08:36.767547+0000","last_active":"2024-03-12T05:10:37.799041+0000","last_peered":"2024-03-12T05:10:37.799041+0000","last_clean":"2024-03-12T05:10:37.799041+0000","last_became_active":"2024-03-12T05:08:36.767184+0000","last_became_peered":"2024-03-12T05:08:36.767184+0000","last_unstale":"2024-03-12T05:10:37.799041+0000","last_undegraded":"2024-03-12T05:10:37.799041+0000","last_fullsized":"2024-03-12T05:10:37.799041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:23:03.754705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139019+0000","last_change":"2024-03-12T05:08:37.772221+0000","last_active":"2024-03-12T05:10:13.139019+0000","last_peered":"2024-03-12T05:10:13.139019+0000","last_clean":"2024-03-12T05:10:13.139019+0000","last_became_active":"2024-03-12T05:08:37.770943+0000","last_became_peered":"2024-03-12T05:08:37.770943+0000","last_unstale":"2024-03-12T05:10:13.139019+0000","last_undegraded":"2024-03-12T05:10:13.139019+0000","last_fullsized":"2024-03-12T05:10:13.139019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:27:10.533613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.310664+0000","last_change":"2024-03-12T05:08:36.773595+0000","last_active":"2024-03-12T05:09:44.310664+0000","last_peered":"2024-03-12T05:09:44.310664+0000","last_clean":"2024-03-12T05:09:44.310664+0000","last_became_active":"2024-03-12T05:08:36.772932+0000","last_became_peered":"2024-03-12T05:08:36.772932+0000","last_unstale":"2024-03-12T05:09:44.310664+0000","last_undegraded":"2024-03-12T05:09:44.310664+0000","last_fullsized":"2024-03-12T05:09:44.310664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:12:11.129029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141996+0000","last_change":"2024-03-12T05:08:37.775341+0000","last_active":"2024-03-12T05:10:13.141996+0000","last_peered":"2024-03-12T05:10:13.141996+0000","last_clean":"2024-03-12T05:10:13.141996+0000","last_became_active":"2024-03-12T05:08:37.774989+0000","last_became_peered":"2024-03-12T05:08:37.774989+0000","last_unstale":"2024-03-12T05:10:13.141996+0000","last_undegraded":"2024-03-12T05:10:13.141996+0000","last_fullsized":"2024-03-12T05:10:13.141996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:46.936181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.136091+0000","last_change":"2024-03-12T05:09:44.136091+0000","last_active":"2024-03-12T05:09:44.136091+0000","last_peered":"2024-03-12T05:09:44.136091+0000","last_clean":"2024-03-12T05:09:44.136091+0000","last_became_active":"2024-03-12T05:09:44.130865+0000","last_became_peered":"2024-03-12T05:09:44.130865+0000","last_unstale":"2024-03-12T05:09:44.136091+0000","last_undegraded":"2024-03-12T05:09:44.136091+0000","last_fullsized":"2024-03-12T05:09:44.136091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:26:13.588722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.099732+0000","last_change":"2024-03-12T05:08:37.774862+0000","last_active":"2024-03-12T05:10:13.099732+0000","last_peered":"2024-03-12T05:10:13.099732+0000","last_clean":"2024-03-12T05:10:13.099732+0000","last_became_active":"2024-03-12T05:08:37.774505+0000","last_became_peered":"2024-03-12T05:08:37.774505+0000","last_unstale":"2024-03-12T05:10:13.099732+0000","last_undegraded":"2024-03-12T05:10:13.099732+0000","last_fullsized":"2024-03-12T05:10:13.099732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:54:36.958020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.381493+0000","last_change":"2024-03-12T05:08:36.772467+0000","last_active":"2024-03-12T05:09:44.381493+0000","last_peered":"2024-03-12T05:09:44.381493+0000","last_clean":"2024-03-12T05:09:44.381493+0000","last_became_active":"2024-03-12T05:08:36.772116+0000","last_became_peered":"2024-03-12T05:08:36.772116+0000","last_unstale":"2024-03-12T05:09:44.381493+0000","last_undegraded":"2024-03-12T05:09:44.381493+0000","last_fullsized":"2024-03-12T05:09:44.381493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:53.803906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.141751+0000","last_change":"2024-03-12T05:08:37.775189+0000","last_active":"2024-03-12T05:10:13.141751+0000","last_peered":"2024-03-12T05:10:13.141751+0000","last_clean":"2024-03-12T05:10:13.141751+0000","last_became_active":"2024-03-12T05:08:37.774549+0000","last_became_peered":"2024-03-12T05:08:37.774549+0000","last_unstale":"2024-03-12T05:10:13.141751+0000","last_undegraded":"2024-03-12T05:10:13.141751+0000","last_fullsized":"2024-03-12T05:10:13.141751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:39:49.354324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:36.342780+0000","last_change":"2024-03-12T05:08:36.771738+0000","last_active":"2024-03-12T05:10:36.342780+0000","last_peered":"2024-03-12T05:10:36.342780+0000","last_clean":"2024-03-12T05:10:36.342780+0000","last_became_active":"2024-03-12T05:08:36.771424+0000","last_became_peered":"2024-03-12T05:08:36.771424+0000","last_unstale":"2024-03-12T05:10:36.342780+0000","last_undegraded":"2024-03-12T05:10:36.342780+0000","last_fullsized":"2024-03-12T05:10:36.342780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:22:41.673490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.137122+0000","last_change":"2024-03-12T05:08:37.782364+0000","last_active":"2024-03-12T05:10:13.137122+0000","last_peered":"2024-03-12T05:10:13.137122+0000","last_clean":"2024-03-12T05:10:13.137122+0000","last_became_active":"2024-03-12T05:08:37.782023+0000","last_became_peered":"2024-03-12T05:08:37.782023+0000","last_unstale":"2024-03-12T05:10:13.137122+0000","last_undegraded":"2024-03-12T05:10:13.137122+0000","last_fullsized":"2024-03-12T05:10:13.137122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:56.520054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.145784+0000","last_change":"2024-03-12T05:09:44.145284+0000","last_active":"2024-03-12T05:09:44.145784+0000","last_peered":"2024-03-12T05:09:44.145784+0000","last_clean":"2024-03-12T05:09:44.145784+0000","last_became_active":"2024-03-12T05:09:44.134640+0000","last_became_peered":"2024-03-12T05:09:44.134640+0000","last_unstale":"2024-03-12T05:09:44.145784+0000","last_undegraded":"2024-03-12T05:09:44.145784+0000","last_fullsized":"2024-03-12T05:09:44.145784+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:17.719372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153940+0000","last_change":"2024-03-12T05:08:37.783055+0000","last_active":"2024-03-12T05:10:13.153940+0000","last_peered":"2024-03-12T05:10:13.153940+0000","last_clean":"2024-03-12T05:10:13.153940+0000","last_became_active":"2024-03-12T05:08:37.782739+0000","last_became_peered":"2024-03-12T05:08:37.782739+0000","last_unstale":"2024-03-12T05:10:13.153940+0000","last_undegraded":"2024-03-12T05:10:13.153940+0000","last_fullsized":"2024-03-12T05:10:13.153940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:43:55.269560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'9","reported_seq":151,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:39.518590+0000","last_change":"2024-03-12T05:08:36.770638+0000","last_active":"2024-03-12T05:10:39.518590+0000","last_peered":"2024-03-12T05:10:39.518590+0000","last_clean":"2024-03-12T05:10:39.518590+0000","last_became_active":"2024-03-12T05:08:36.770239+0000","last_became_peered":"2024-03-12T05:08:36.770239+0000","last_unstale":"2024-03-12T05:10:39.518590+0000","last_undegraded":"2024-03-12T05:10:39.518590+0000","last_fullsized":"2024-03-12T05:10:39.518590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:14:42.828109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2570,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.153801+0000","last_change":"2024-03-12T05:08:37.778179+0000","last_active":"2024-03-12T05:10:13.153801+0000","last_peered":"2024-03-12T05:10:13.153801+0000","last_clean":"2024-03-12T05:10:13.153801+0000","last_became_active":"2024-03-12T05:08:37.777877+0000","last_became_peered":"2024-03-12T05:08:37.777877+0000","last_unstale":"2024-03-12T05:10:13.153801+0000","last_undegraded":"2024-03-12T05:10:13.153801+0000","last_fullsized":"2024-03-12T05:10:13.153801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:55.313905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.131890+0000","last_change":"2024-03-12T05:08:36.767590+0000","last_active":"2024-03-12T05:09:44.131890+0000","last_peered":"2024-03-12T05:09:44.131890+0000","last_clean":"2024-03-12T05:09:44.131890+0000","last_became_active":"2024-03-12T05:08:36.767275+0000","last_became_peered":"2024-03-12T05:08:36.767275+0000","last_unstale":"2024-03-12T05:09:44.131890+0000","last_undegraded":"2024-03-12T05:09:44.131890+0000","last_fullsized":"2024-03-12T05:09:44.131890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:54:32.277729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140168+0000","last_change":"2024-03-12T05:08:37.779284+0000","last_active":"2024-03-12T05:10:13.140168+0000","last_peered":"2024-03-12T05:10:13.140168+0000","last_clean":"2024-03-12T05:10:13.140168+0000","last_became_active":"2024-03-12T05:08:37.778959+0000","last_became_peered":"2024-03-12T05:08:37.778959+0000","last_unstale":"2024-03-12T05:10:13.140168+0000","last_undegraded":"2024-03-12T05:10:13.140168+0000","last_fullsized":"2024-03-12T05:10:13.140168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:22.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:09:44.126596+0000","last_change":"2024-03-12T05:08:36.773184+0000","last_active":"2024-03-12T05:09:44.126596+0000","last_peered":"2024-03-12T05:09:44.126596+0000","last_clean":"2024-03-12T05:09:44.126596+0000","last_became_active":"2024-03-12T05:08:36.772777+0000","last_became_peered":"2024-03-12T05:08:36.772777+0000","last_unstale":"2024-03-12T05:09:44.126596+0000","last_undegraded":"2024-03-12T05:09:44.126596+0000","last_fullsized":"2024-03-12T05:09:44.126596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:35.741499+0000","last_clean_scrub_stamp":"2024-03-12T05:08:35.741499+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:02:50.437541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140750+0000","last_change":"2024-03-12T05:08:37.770632+0000","last_active":"2024-03-12T05:10:13.140750+0000","last_peered":"2024-03-12T05:10:13.140750+0000","last_clean":"2024-03-12T05:10:13.140750+0000","last_became_active":"2024-03-12T05:08:37.770117+0000","last_became_peered":"2024-03-12T05:08:37.770117+0000","last_unstale":"2024-03-12T05:10:13.140750+0000","last_undegraded":"2024-03-12T05:10:13.140750+0000","last_fullsized":"2024-03-12T05:10:13.140750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:11.402921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.139173+0000","last_change":"2024-03-12T05:08:37.774881+0000","last_active":"2024-03-12T05:10:13.139173+0000","last_peered":"2024-03-12T05:10:13.139173+0000","last_clean":"2024-03-12T05:10:13.139173+0000","last_became_active":"2024-03-12T05:08:37.774020+0000","last_became_peered":"2024-03-12T05:08:37.774020+0000","last_unstale":"2024-03-12T05:10:13.139173+0000","last_undegraded":"2024-03-12T05:10:13.139173+0000","last_fullsized":"2024-03-12T05:10:13.139173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:15:48.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136516+0000","last_change":"2024-03-12T05:08:37.780199+0000","last_active":"2024-03-12T05:10:13.136516+0000","last_peered":"2024-03-12T05:10:13.136516+0000","last_clean":"2024-03-12T05:10:13.136516+0000","last_became_active":"2024-03-12T05:08:37.779376+0000","last_became_peered":"2024-03-12T05:08:37.779376+0000","last_unstale":"2024-03-12T05:10:13.136516+0000","last_undegraded":"2024-03-12T05:10:13.136516+0000","last_fullsized":"2024-03-12T05:10:13.136516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:22.293373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136016+0000","last_change":"2024-03-12T05:08:37.785567+0000","last_active":"2024-03-12T05:10:13.136016+0000","last_peered":"2024-03-12T05:10:13.136016+0000","last_clean":"2024-03-12T05:10:13.136016+0000","last_became_active":"2024-03-12T05:08:37.785217+0000","last_became_peered":"2024-03-12T05:08:37.785217+0000","last_unstale":"2024-03-12T05:10:13.136016+0000","last_undegraded":"2024-03-12T05:10:13.136016+0000","last_fullsized":"2024-03-12T05:10:13.136016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:28:21.173024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140867+0000","last_change":"2024-03-12T05:08:37.789310+0000","last_active":"2024-03-12T05:10:13.140867+0000","last_peered":"2024-03-12T05:10:13.140867+0000","last_clean":"2024-03-12T05:10:13.140867+0000","last_became_active":"2024-03-12T05:08:37.788467+0000","last_became_peered":"2024-03-12T05:08:37.788467+0000","last_unstale":"2024-03-12T05:10:13.140867+0000","last_undegraded":"2024-03-12T05:10:13.140867+0000","last_fullsized":"2024-03-12T05:10:13.140867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:00:40.532815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136721+0000","last_change":"2024-03-12T05:09:44.129436+0000","last_active":"2024-03-12T05:10:13.136721+0000","last_peered":"2024-03-12T05:10:13.136721+0000","last_clean":"2024-03-12T05:10:13.136721+0000","last_became_active":"2024-03-12T05:09:44.129063+0000","last_became_peered":"2024-03-12T05:09:44.129063+0000","last_unstale":"2024-03-12T05:10:13.136721+0000","last_undegraded":"2024-03-12T05:10:13.136721+0000","last_fullsized":"2024-03-12T05:10:13.136721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:06.724357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.138474+0000","last_change":"2024-03-12T05:08:37.780086+0000","last_active":"2024-03-12T05:10:13.138474+0000","last_peered":"2024-03-12T05:10:13.138474+0000","last_clean":"2024-03-12T05:10:13.138474+0000","last_became_active":"2024-03-12T05:08:37.779767+0000","last_became_peered":"2024-03-12T05:08:37.779767+0000","last_unstale":"2024-03-12T05:10:13.138474+0000","last_undegraded":"2024-03-12T05:10:13.138474+0000","last_fullsized":"2024-03-12T05:10:13.138474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:09.356636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152412+0000","last_change":"2024-03-12T05:08:37.775717+0000","last_active":"2024-03-12T05:10:13.152412+0000","last_peered":"2024-03-12T05:10:13.152412+0000","last_clean":"2024-03-12T05:10:13.152412+0000","last_became_active":"2024-03-12T05:08:37.775387+0000","last_became_peered":"2024-03-12T05:08:37.775387+0000","last_unstale":"2024-03-12T05:10:13.152412+0000","last_undegraded":"2024-03-12T05:10:13.152412+0000","last_fullsized":"2024-03-12T05:10:13.152412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:38:34.285153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136405+0000","last_change":"2024-03-12T05:08:37.781746+0000","last_active":"2024-03-12T05:10:13.136405+0000","last_peered":"2024-03-12T05:10:13.136405+0000","last_clean":"2024-03-12T05:10:13.136405+0000","last_became_active":"2024-03-12T05:08:37.781382+0000","last_became_peered":"2024-03-12T05:08:37.781382+0000","last_unstale":"2024-03-12T05:10:13.136405+0000","last_undegraded":"2024-03-12T05:10:13.136405+0000","last_fullsized":"2024-03-12T05:10:13.136405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:12:57.530055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136449+0000","last_change":"2024-03-12T05:08:37.771608+0000","last_active":"2024-03-12T05:10:13.136449+0000","last_peered":"2024-03-12T05:10:13.136449+0000","last_clean":"2024-03-12T05:10:13.136449+0000","last_became_active":"2024-03-12T05:08:37.770335+0000","last_became_peered":"2024-03-12T05:08:37.770335+0000","last_unstale":"2024-03-12T05:10:13.136449+0000","last_undegraded":"2024-03-12T05:10:13.136449+0000","last_fullsized":"2024-03-12T05:10:13.136449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:55.577987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.136850+0000","last_change":"2024-03-12T05:08:37.788615+0000","last_active":"2024-03-12T05:10:13.136850+0000","last_peered":"2024-03-12T05:10:13.136850+0000","last_clean":"2024-03-12T05:10:13.136850+0000","last_became_active":"2024-03-12T05:08:37.788248+0000","last_became_peered":"2024-03-12T05:08:37.788248+0000","last_unstale":"2024-03-12T05:10:13.136850+0000","last_undegraded":"2024-03-12T05:10:13.136850+0000","last_fullsized":"2024-03-12T05:10:13.136850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:28.802049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.152464+0000","last_change":"2024-03-12T05:08:37.790512+0000","last_active":"2024-03-12T05:10:13.152464+0000","last_peered":"2024-03-12T05:10:13.152464+0000","last_clean":"2024-03-12T05:10:13.152464+0000","last_became_active":"2024-03-12T05:08:37.789811+0000","last_became_peered":"2024-03-12T05:08:37.789811+0000","last_unstale":"2024-03-12T05:10:13.152464+0000","last_undegraded":"2024-03-12T05:10:13.152464+0000","last_fullsized":"2024-03-12T05:10:13.152464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:35:13.860390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.140276+0000","last_change":"2024-03-12T05:08:37.775139+0000","last_active":"2024-03-12T05:10:13.140276+0000","last_peered":"2024-03-12T05:10:13.140276+0000","last_clean":"2024-03-12T05:10:13.140276+0000","last_became_active":"2024-03-12T05:08:37.774469+0000","last_became_peered":"2024-03-12T05:08:37.774469+0000","last_unstale":"2024-03-12T05:10:13.140276+0000","last_undegraded":"2024-03-12T05:10:13.140276+0000","last_fullsized":"2024-03-12T05:10:13.140276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:42.991694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-12T05:10:13.144685+0000","last_change":"2024-03-12T05:08:37.787116+0000","last_active":"2024-03-12T05:10:13.144685+0000","last_peered":"2024-03-12T05:10:13.144685+0000","last_clean":"2024-03-12T05:10:13.144685+0000","last_became_active":"2024-03-12T05:08:37.786786+0000","last_became_peered":"2024-03-12T05:08:37.786786+0000","last_unstale":"2024-03-12T05:10:13.144685+0000","last_undegraded":"2024-03-12T05:10:13.144685+0000","last_fullsized":"2024-03-12T05:10:13.144685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:08:36.746003+0000","last_clean_scrub_stamp":"2024-03-12T05:08:36.746003+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:31:14.298370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":247898,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75051,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1207,"num_read_kb":683,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":220057,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1112443,"data_compressed":37976,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":456383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":456089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1051774,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495679,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":517800,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":458839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":523985,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":455925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":456970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1049575,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34478,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36116,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62539,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":68724,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67088,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:10:42.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 -- 172.21.15.45:0/512551163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 msgr2=0x7ff0cc1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 --2- 172.21.15.45:0/512551163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1015d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009900 tx=0x7ff0c802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 -- 172.21.15.45:0/512551163 shutdown_connections 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 --2- 172.21.15.45:0/512551163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc102d60 0x7ff0cc1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 --2- 172.21.15.45:0/512551163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 --2- 172.21.15.45:0/512551163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1015d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 -- 172.21.15.45:0/512551163 >> 172.21.15.45:0/512551163 conn(0x7ff0cc0fc720 msgr2=0x7ff0cc0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 -- 172.21.15.45:0/512551163 shutdown_connections 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.610+0000 7ff0d4696640 1 -- 172.21.15.45:0/512551163 wait complete. 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 Processor -- start 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 -- start start 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 0x7ff0cc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19ebe0 con 0x7ff0cc1011b0 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19ed50 con 0x7ff0cc102d60 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d4696640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19ef10 con 0x7ff0cc1023a0 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47664/0 (socket says 172.21.15.45:47664) 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 -- 172.21.15.45:0/223339966 learned_addr learned my addr 172.21.15.45:0/223339966 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:42.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2c33640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 0x7ff0cc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d1c31640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 -- 172.21.15.45:0/223339966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 msgr2=0x7ff0cc197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 -- 172.21.15.45:0/223339966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 msgr2=0x7ff0cc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 0x7ff0cc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2432640 1 -- 172.21.15.45:0/223339966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8009560 con 0x7ff0cc1011b0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d2c33640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 0x7ff0cc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.611+0000 7ff0d1c31640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0d2432640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000c00 tx=0x7ff0c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80044e0 con 0x7ff0cc1011b0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0c802de90 con 0x7ff0cc1011b0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0cc19f190 con 0x7ff0cc1011b0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8005410 con 0x7ff0cc1011b0 2024-03-12T05:10:42.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.612+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc19f6a0 con 0x7ff0cc1011b0 2024-03-12T05:10:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.613+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc1084d0 con 0x7ff0cc1011b0 2024-03-12T05:10:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.614+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff0c8005570 con 0x7ff0cc1011b0 2024-03-12T05:10:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.614+0000 7ff0c37fe640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 0x7ff0b80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.614+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7ff0c80ef770 con 0x7ff0cc1011b0 2024-03-12T05:10:42.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.614+0000 7ff0d1c31640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 0x7ff0b80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.615+0000 7ff0d1c31640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 0x7ff0b80a3f80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc198600 tx=0x7ff0bc005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.619+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0c80b8940 con 0x7ff0cc1011b0 2024-03-12T05:10:42.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:42 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1358910136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:42.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:42 smithi130 ceph-mon[35398]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:42 smithi130 ceph-mon[35398]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:42 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1358910136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:42.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:42 smithi116 ceph-mon[35486]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:42 smithi116 ceph-mon[35486]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:42.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.800+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0cc107f40 con 0x7ff0cc1011b0 2024-03-12T05:10:42.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.801+0000 7ff0c37fe640 1 -- 172.21.15.45:0/223339966 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7ff0c80bc000 con 0x7ff0cc1011b0 2024-03-12T05:10:42.799 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:42.799 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 msgr2=0x7ff0b80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 0x7ff0b80a3f80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc198600 tx=0x7ff0bc005d90 comp rx=0 tx=0).stop 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 msgr2=0x7ff0cc1970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8000c00 tx=0x7ff0c80042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 shutdown_connections 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7ff0b80a1ac0 0x7ff0b80a3f80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0cc102d60 0x7ff0cc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0cc1023a0 0x7ff0cc197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 --2- 172.21.15.45:0/223339966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0cc1011b0 0x7ff0cc1970f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.803+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 >> 172.21.15.45:0/223339966 conn(0x7ff0cc0fc720 msgr2=0x7ff0cc1056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.804+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 shutdown_connections 2024-03-12T05:10:42.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.804+0000 7ff0d4696640 1 -- 172.21.15.45:0/223339966 wait complete. 2024-03-12T05:10:42.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:10:42.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 -- 172.21.15.45:0/317296547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c106e60 msgr2=0x7f837c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 --2- 172.21.15.45:0/317296547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c106e60 0x7f837c1092b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f836c009ca0 tx=0x7f836c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 -- 172.21.15.45:0/317296547 shutdown_connections 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 --2- 172.21.15.45:0/317296547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c106e60 0x7f837c1092b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 --2- 172.21.15.45:0/317296547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 0x7f837c106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 --2- 172.21.15.45:0/317296547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c101c00 0x7f837c103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.920+0000 7f838192c640 1 -- 172.21.15.45:0/317296547 >> 172.21.15.45:0/317296547 conn(0x7f837c0fb610 msgr2=0x7f837c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- 172.21.15.45:0/317296547 shutdown_connections 2024-03-12T05:10:42.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- 172.21.15.45:0/317296547 wait complete. 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 Processor -- start 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- start start 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 0x7f837c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 0x7f837c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c1a0da0 con 0x7f837c101c00 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c1a0f10 con 0x7f837c106e60 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f838192c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c1a10d0 con 0x7f837c104530 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.921+0000 7f837affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47676/0 (socket says 172.21.15.45:47676) 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 -- 172.21.15.45:0/201164614 learned_addr learned my addr 172.21.15.45:0/201164614 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837a7fc640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 0x7f837c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837b7fe640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 0x7f837c1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 -- 172.21.15.45:0/201164614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 msgr2=0x7f837c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 0x7f837c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 -- 172.21.15.45:0/201164614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 msgr2=0x7f837c1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 0x7f837c1a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 -- 172.21.15.45:0/201164614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c009900 con 0x7f837c101c00 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837b7fe640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 0x7f837c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f837affd640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f837400c920 tx=0x7f837400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374007c70 con 0x7f837c101c00 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c1a13b0 con 0x7f837c101c00 2024-03-12T05:10:42.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8374007dd0 con 0x7f837c101c00 2024-03-12T05:10:42.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83740187c0 con 0x7f837c101c00 2024-03-12T05:10:42.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.922+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c1a1910 con 0x7f837c101c00 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.923+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c108950 con 0x7f837c101c00 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.924+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f837400f450 con 0x7f837c101c00 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.925+0000 7f838092a640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 0x7f83680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.925+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f83740cda80 con 0x7f837c101c00 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.925+0000 7f837a7fc640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 0x7f83680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:42.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.925+0000 7f837a7fc640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 0x7f83680a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f837c19a7c0 tx=0x7f836400b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:42.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:42.928+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8374096b70 con 0x7f837c101c00 2024-03-12T05:10:43.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.109+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f837c10cbf0 con 0x7f837c101c00 2024-03-12T05:10:43.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.109+0000 7f838092a640 1 -- 172.21.15.45:0/201164614 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f837409a230 con 0x7f837c101c00 2024-03-12T05:10:43.107 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 msgr2=0x7f83680a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 0x7f83680a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f837c19a7c0 tx=0x7f836400b590 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 msgr2=0x7f837c1992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f837400c920 tx=0x7f837400cde0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 shutdown_connections 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f83680a1ac0 0x7f83680a3f80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f837c106e60 0x7f837c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f837c104530 0x7f837c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 --2- 172.21.15.45:0/201164614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f837c101c00 0x7f837c1992b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 >> 172.21.15.45:0/201164614 conn(0x7f837c0fb610 msgr2=0x7f837c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 shutdown_connections 2024-03-12T05:10:43.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.111+0000 7f838192c640 1 -- 172.21.15.45:0/201164614 wait complete. 2024-03-12T05:10:43.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:10:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- 172.21.15.45:0/1210210041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324101190 msgr2=0x7f03241015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- 172.21.15.45:0/1210210041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324101190 0x7f03241015b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0320009900 tx=0x7f032002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- 172.21.15.45:0/1210210041 shutdown_connections 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- 172.21.15.45:0/1210210041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f03241095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- 172.21.15.45:0/1210210041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324102380 0x7f0324102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- 172.21.15.45:0/1210210041 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324101190 0x7f03241015b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- 172.21.15.45:0/1210210041 >> 172.21.15.45:0/1210210041 conn(0x7f03240fc720 msgr2=0x7f03240feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- 172.21.15.45:0/1210210041 shutdown_connections 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- 172.21.15.45:0/1210210041 wait complete. 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 Processor -- start 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- start start 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324101190 0x7f0324197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f032419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032419ec80 con 0x7f0324102380 2024-03-12T05:10:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032419edf0 con 0x7f0324101190 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.229+0000 7f032bd77640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032419efb0 con 0x7f0324102d40 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47694/0 (socket says 172.21.15.45:47694) 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 -- 172.21.15.45:0/2524244117 learned_addr learned my addr 172.21.15.45:0/2524244117 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f032a314640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f032419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 -- 172.21.15.45:0/2524244117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 msgr2=0x7f032419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f032419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 -- 172.21.15.45:0/2524244117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324101190 msgr2=0x7f0324197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329b13640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324101190 0x7f0324197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324101190 0x7f0324197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 -- 172.21.15.45:0/2524244117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0320009560 con 0x7f0324102380 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f032a314640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f032419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:43.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f0329312640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f031400c890 tx=0x7f031400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314004220 con 0x7f0324102380 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0314007e60 con 0x7f0324102380 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032419f290 con 0x7f0324102380 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314004a00 con 0x7f0324102380 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.230+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032419f7a0 con 0x7f0324102380 2024-03-12T05:10:43.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.231+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324108570 con 0x7f0324102380 2024-03-12T05:10:43.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.232+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f03140079b0 con 0x7f0324102380 2024-03-12T05:10:43.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.233+0000 7f031affd640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 0x7f03100a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.233+0000 7f0329b13640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 0x7f03100a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.233+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f03140cd330 con 0x7f0324102380 2024-03-12T05:10:43.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.233+0000 7f0329b13640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 0x7f03100a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0320036040 tx=0x7f0320005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.237+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f03140963f0 con 0x7f0324102380 2024-03-12T05:10:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.421+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f032402cda0 con 0x7f0324102380 2024-03-12T05:10:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.421+0000 7f031affd640 1 -- 172.21.15.45:0/2524244117 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0314099ab0 con 0x7f0324102380 2024-03-12T05:10:43.419 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 msgr2=0x7f03100a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 0x7f03100a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0320036040 tx=0x7f0320005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 msgr2=0x7f03241976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f031400c890 tx=0x7f031400cd50 comp rx=0 tx=0).stop 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 shutdown_connections 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f03100a1ac0 0x7f03100a3f80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0324102380 0x7f03241976d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0324102d40 0x7f032419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 --2- 172.21.15.45:0/2524244117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0324101190 0x7f0324197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 >> 172.21.15.45:0/2524244117 conn(0x7f03240fc720 msgr2=0x7f0324105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 shutdown_connections 2024-03-12T05:10:43.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.423+0000 7f032bd77640 1 -- 172.21.15.45:0/2524244117 wait complete. 2024-03-12T05:10:43.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:43 smithi045 ceph-mon[32174]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:43.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:43 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/223339966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:43.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:43 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/201164614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:43.431 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:43 smithi045 ceph-mon[32174]: pgmap v347: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-12T05:10:43.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 -- 172.21.15.45:0/500137654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44106ef0 msgr2=0x7f2e441092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/500137654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44106ef0 0x7f2e441092e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009ca0 tx=0x7f2e3c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 -- 172.21.15.45:0/500137654 shutdown_connections 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/500137654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44106ef0 0x7f2e441092e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/500137654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e441045c0 0x7f2e441069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/500137654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e44101c90 0x7f2e44104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 -- 172.21.15.45:0/500137654 >> 172.21.15.45:0/500137654 conn(0x7f2e440fb620 msgr2=0x7f2e440fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 -- 172.21.15.45:0/500137654 shutdown_connections 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.540+0000 7f2e4b73a640 1 -- 172.21.15.45:0/500137654 wait complete. 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 Processor -- start 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 -- start start 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e441045c0 0x7f2e44197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e44106ef0 0x7f2e4419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4419ec30 con 0x7f2e44101c90 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4419eda0 con 0x7f2e441045c0 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e4b73a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4419ef60 con 0x7f2e44106ef0 2024-03-12T05:10:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47710/0 (socket says 172.21.15.45:47710) 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 -- 172.21.15.45:0/3991008639 learned_addr learned my addr 172.21.15.45:0/3991008639 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e48cd5640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e441045c0 0x7f2e44197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e49cd7640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e44106ef0 0x7f2e4419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 -- 172.21.15.45:0/3991008639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e44106ef0 msgr2=0x7f2e4419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e44106ef0 0x7f2e4419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 -- 172.21.15.45:0/3991008639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e441045c0 msgr2=0x7f2e44197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e441045c0 0x7f2e44197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.541+0000 7f2e494d6640 1 -- 172.21.15.45:0/3991008639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c009900 con 0x7f2e44101c90 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e494d6640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e9f0 tx=0x7f2e4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40009700 con 0x7f2e44101c90 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e4000cd80 con 0x7f2e44101c90 2024-03-12T05:10:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4419f240 con 0x7f2e44101c90 2024-03-12T05:10:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40018840 con 0x7f2e44101c90 2024-03-12T05:10:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.542+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4419f7a0 con 0x7f2e44101c90 2024-03-12T05:10:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.543+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e440fd110 con 0x7f2e44101c90 2024-03-12T05:10:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.544+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2e400189a0 con 0x7f2e44101c90 2024-03-12T05:10:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.545+0000 7f2e3a7fc640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 0x7f2e300a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.545+0000 7f2e48cd5640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 0x7f2e300a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.545+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f2e40014070 con 0x7f2e44101c90 2024-03-12T05:10:43.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.545+0000 7f2e48cd5640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 0x7f2e300a4000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2e44198650 tx=0x7f2e3400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.548+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2e40096e00 con 0x7f2e44101c90 2024-03-12T05:10:43.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:43 smithi130 ceph-mon[35398]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:43.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:43 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/223339966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:43.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:43 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/201164614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:43.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:43 smithi130 ceph-mon[35398]: pgmap v347: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-12T05:10:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.729+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2e4410cb70 con 0x7f2e44101c90 2024-03-12T05:10:43.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.730+0000 7f2e3a7fc640 1 -- 172.21.15.45:0/3991008639 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2e400260a0 con 0x7f2e44101c90 2024-03-12T05:10:43.727 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:10:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 msgr2=0x7f2e300a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 0x7f2e300a4000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2e44198650 tx=0x7f2e3400b390 comp rx=0 tx=0).stop 2024-03-12T05:10:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 msgr2=0x7f2e44197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000e9f0 tx=0x7f2e4000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 shutdown_connections 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f2e300a1b40 0x7f2e300a4000 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e44106ef0 0x7f2e4419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e441045c0 0x7f2e44197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 --2- 172.21.15.45:0/3991008639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e44101c90 0x7f2e44197140 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 >> 172.21.15.45:0/3991008639 conn(0x7f2e440fb620 msgr2=0x7f2e44102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 shutdown_connections 2024-03-12T05:10:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.732+0000 7f2e4b73a640 1 -- 172.21.15.45:0/3991008639 wait complete. 2024-03-12T05:10:43.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:10:43.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:43 smithi116 ceph-mon[35486]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:10:43.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:43 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/223339966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:43.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:43 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/201164614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:10:43.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:43 smithi116 ceph-mon[35486]: pgmap v347: 97 pgs: 97 active+clean; 650 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 -- 172.21.15.45:0/2442298997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68100120 msgr2=0x7f1e68100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2442298997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68100120 0x7f1e68100540 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009900 tx=0x7f1e5802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 -- 172.21.15.45:0/2442298997 shutdown_connections 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2442298997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68101cd0 0x7f1e68108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2442298997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101310 0x7f1e68101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2442298997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68100120 0x7f1e68100540 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 -- 172.21.15.45:0/2442298997 >> 172.21.15.45:0/2442298997 conn(0x7f1e680fb6b0 msgr2=0x7f1e680fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 -- 172.21.15.45:0/2442298997 shutdown_connections 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.845+0000 7f1e6d177640 1 -- 172.21.15.45:0/2442298997 wait complete. 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 Processor -- start 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 -- start start 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68100120 0x7f1e68197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101cd0 0x7f1e6819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6819ec70 con 0x7f1e68101310 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6819ede0 con 0x7f1e68100120 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6d177640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6819efa0 con 0x7f1e68101cd0 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47736/0 (socket says 172.21.15.45:47736) 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 -- 172.21.15.45:0/2653332522 learned_addr learned my addr 172.21.15.45:0/2653332522 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e6759e640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101cd0 0x7f1e6819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e66d9d640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68100120 0x7f1e68197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 -- 172.21.15.45:0/2653332522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101cd0 msgr2=0x7f1e6819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101cd0 0x7f1e6819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 -- 172.21.15.45:0/2653332522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68100120 msgr2=0x7f1e68197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68100120 0x7f1e68197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:43.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 -- 172.21.15.45:0/2653332522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e58009560 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.846+0000 7f1e5ffff640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000d770 tx=0x7f1e5000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.847+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50009c30 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.847+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1e5000de80 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.847+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6819f280 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.847+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50004e00 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.847+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6819f740 con 0x7f1e68101310 2024-03-12T05:10:43.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.848+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e68107510 con 0x7f1e68101310 2024-03-12T05:10:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.849+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1e50004f60 con 0x7f1e68101310 2024-03-12T05:10:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.849+0000 7f1e64d99640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 0x7f1e540a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.849+0000 7f1e66d9d640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 0x7f1e540a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.849+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f1e500cd0d0 con 0x7f1e68101310 2024-03-12T05:10:43.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.850+0000 7f1e66d9d640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 0x7f1e540a3f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1e58006010 tx=0x7f1e58000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:43.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:43.854+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e50096190 con 0x7f1e68101310 2024-03-12T05:10:44.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.072+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e6810ccc0 con 0x7f1e68101310 2024-03-12T05:10:44.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.073+0000 7f1e64d99640 1 -- 172.21.15.45:0/2653332522 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f1e50017090 con 0x7f1e68101310 2024-03-12T05:10:44.071 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:44.071 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:10:37.628388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:10:44.071 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 25 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 msgr2=0x7f1e540a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 0x7f1e540a3f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1e58006010 tx=0x7f1e58000bc0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 msgr2=0x7f1e681976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000d770 tx=0x7f1e5000dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 shutdown_connections 2024-03-12T05:10:44.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f1e540a1ac0 0x7f1e540a3f80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e68101310 0x7f1e681976c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e68101cd0 0x7f1e6819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 --2- 172.21.15.45:0/2653332522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e68100120 0x7f1e68197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 >> 172.21.15.45:0/2653332522 conn(0x7f1e680fb6b0 msgr2=0x7f1e681046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 shutdown_connections 2024-03-12T05:10:44.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.075+0000 7f1e6d177640 1 -- 172.21.15.45:0/2653332522 wait complete. 2024-03-12T05:10:44.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:10:44.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- 172.21.15.45:0/1165718743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db41000b0 msgr2=0x7f4db41004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1165718743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db41000b0 0x7f4db41004d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009900 tx=0x7f4db002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- 172.21.15.45:0/1165718743 shutdown_connections 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1165718743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db4101c60 0x7f4db41084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1165718743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41012a0 0x7f4db4101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1165718743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db41000b0 0x7f4db41004d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- 172.21.15.45:0/1165718743 >> 172.21.15.45:0/1165718743 conn(0x7f4db40fb660 msgr2=0x7f4db40fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- 172.21.15.45:0/1165718743 shutdown_connections 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- 172.21.15.45:0/1165718743 wait complete. 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 Processor -- start 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- start start 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 0x7f4db4197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db41012a0 0x7f4db41976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db419ec50 con 0x7f4db4101c60 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db419edc0 con 0x7f4db41000b0 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.190+0000 7f4dbb955640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db419ef80 con 0x7f4db41012a0 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47752/0 (socket says 172.21.15.45:47752) 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 -- 172.21.15.45:0/1894281347 learned_addr learned my addr 172.21.15.45:0/1894281347 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db96f1640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 0x7f4db4197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db8ef0640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db41012a0 0x7f4db41976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 -- 172.21.15.45:0/1894281347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db41012a0 msgr2=0x7f4db41976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db41012a0 0x7f4db41976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 -- 172.21.15.45:0/1894281347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 msgr2=0x7f4db4197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 0x7f4db4197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 -- 172.21.15.45:0/1894281347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0009560 con 0x7f4db4101c60 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db96f1640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 0x7f4db4197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:44.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4db9ef2640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00eaa0 tx=0x7f4dac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac00cdb0 con 0x7f4db4101c60 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4dac005020 con 0x7f4db4101c60 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db419f260 con 0x7f4db4101c60 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0116b0 con 0x7f4db4101c60 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.191+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db419f720 con 0x7f4db4101c60 2024-03-12T05:10:44.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.192+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db41073f0 con 0x7f4db4101c60 2024-03-12T05:10:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.193+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f4dac005190 con 0x7f4db4101c60 2024-03-12T05:10:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.194+0000 7f4daa7fc640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 0x7f4da00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.194+0000 7f4db96f1640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 0x7f4da00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.194+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4dac0cd820 con 0x7f4db4101c60 2024-03-12T05:10:44.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.194+0000 7f4db96f1640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 0x7f4da00a3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006010 tx=0x7f4db0005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.198+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4dac096880 con 0x7f4db4101c60 2024-03-12T05:10:44.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.381+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db402ce40 con 0x7f4db4101c60 2024-03-12T05:10:44.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.383+0000 7f4daa7fc640 1 -- 172.21.15.45:0/1894281347 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22635 (secure 0 0 0) 0x7f4dac099f40 con 0x7f4db4101c60 2024-03-12T05:10:44.381 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:44.381 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":87,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:09:44.121222+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:10:44.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 msgr2=0x7f4da00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 0x7f4da00a3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006010 tx=0x7f4db0005c00 comp rx=0 tx=0).stop 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 msgr2=0x7f4db419e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00eaa0 tx=0x7f4dac00ef60 comp rx=0 tx=0).stop 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 shutdown_connections 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f4da00a1ac0 0x7f4da00a3f80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4db4101c60 0x7f4db419e710 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4db41012a0 0x7f4db41976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 --2- 172.21.15.45:0/1894281347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4db41000b0 0x7f4db4197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 >> 172.21.15.45:0/1894281347 conn(0x7f4db40fb660 msgr2=0x7f4db41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 shutdown_connections 2024-03-12T05:10:44.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.385+0000 7f4dbb955640 1 -- 172.21.15.45:0/1894281347 wait complete. 2024-03-12T05:10:44.396 DEBUG:tasks.fwd_scrub:Yielding 2024-03-12T05:10:44.396 INFO:teuthology.run_tasks:Running task exec... 2024-03-12T05:10:44.411 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:10:44.411 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:10:44.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-12T05:10:44.412 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-12T05:10:44.412 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:10:44.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:10:44.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:10:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 -- 172.21.15.45:0/147164564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 msgr2=0x7fe8a0104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/147164564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a0104010 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe890009900 tx=0x7fe89002bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 -- 172.21.15.45:0/147164564 shutdown_connections 2024-03-12T05:10:44.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/147164564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0106e80 0x7fe8a01092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/147164564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0104550 0x7fe8a0106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/147164564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a0104010 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 -- 172.21.15.45:0/147164564 >> 172.21.15.45:0/147164564 conn(0x7fe8a00fb610 msgr2=0x7fe8a00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 -- 172.21.15.45:0/147164564 shutdown_connections 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 -- 172.21.15.45:0/147164564 wait complete. 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.520+0000 7fe8a5c81640 1 Processor -- start 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 -- start start 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0104550 0x7fe8a0197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0106e80 0x7fe8a019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a019ebd0 con 0x7fe8a0101c20 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a019ed40 con 0x7fe8a0104550 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a019ef00 con 0x7fe8a0106e80 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47792/0 (socket says 172.21.15.45:47792) 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 -- 172.21.15.45:0/517239415 learned_addr learned my addr 172.21.15.45:0/517239415 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89effd640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0104550 0x7fe8a0197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89ffff640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0106e80 0x7fe8a019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 -- 172.21.15.45:0/517239415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0106e80 msgr2=0x7fe8a019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0106e80 0x7fe8a019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 -- 172.21.15.45:0/517239415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0104550 msgr2=0x7fe8a0197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0104550 0x7fe8a0197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.521+0000 7fe89f7fe640 1 -- 172.21.15.45:0/517239415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890009560 con 0x7fe8a0101c20 2024-03-12T05:10:44.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe89f7fe640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe890036040 tx=0x7fe8900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900044e0 con 0x7fe8a0101c20 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a019f180 con 0x7fe8a0101c20 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe89002de90 con 0x7fe8a0101c20 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890005470 con 0x7fe8a0101c20 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.522+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a019f6e0 con 0x7fe8a0101c20 2024-03-12T05:10:44.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.523+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a00fd100 con 0x7fe8a0101c20 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 -- 172.21.15.45:0/831317530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba98102410 msgr2=0x7fba98102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/831317530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba98102410 0x7fba98102890 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 -- 172.21.15.45:0/831317530 shutdown_connections 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/831317530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba98102410 0x7fba98102890 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/831317530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102dd0 0x7fba98109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/831317530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba980ff440 0x7fba980ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 -- 172.21.15.45:0/831317530 >> 172.21.15.45:0/831317530 conn(0x7fba980fad70 msgr2=0x7fba980fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 -- 172.21.15.45:0/831317530 shutdown_connections 2024-03-12T05:10:44.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fbaa03e2640 1 -- 172.21.15.45:0/831317530 wait complete. 2024-03-12T05:10:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.524+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe890003800 con 0x7fe8a0101c20 2024-03-12T05:10:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 Processor -- start 2024-03-12T05:10:44.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 -- start start 2024-03-12T05:10:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fe89cff9640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 0x7fe8880a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fe89effd640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 0x7fe8880a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fe8900ef620 con 0x7fe8a0101c20 2024-03-12T05:10:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102410 0x7fba98197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba98102dd0 0x7fba9819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819ec30 con 0x7fba980ff440 2024-03-12T05:10:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819eda0 con 0x7fba98102dd0 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fbaa03e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819ef60 con 0x7fba98102410 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47800/0 (socket says 172.21.15.45:47800) 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 -- 172.21.15.45:0/1268773189 learned_addr learned my addr 172.21.15.45:0/1268773189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9d97d640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102410 0x7fba98197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e97f640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba98102dd0 0x7fba9819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fe89effd640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 0x7fe8880a4030 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe8a01985f0 tx=0x7fe89400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 -- 172.21.15.45:0/1268773189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102410 msgr2=0x7fba98197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102410 0x7fba98197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 -- 172.21.15.45:0/1268773189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba98102dd0 msgr2=0x7fba9819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba98102dd0 0x7fba9819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.525+0000 7fba9e17e640 1 -- 172.21.15.45:0/1268773189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88009560 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.526+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fba9400d770 tx=0x7fba9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.529+0000 7fba8f7fe640 1 -- 172.21.15.45:0/1268773189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94009c30 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.530+0000 7fba8f7fe640 1 -- 172.21.15.45:0/1268773189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba940044d0 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.530+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9819f240 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.530+0000 7fba8f7fe640 1 -- 172.21.15.45:0/1268773189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94003de0 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.530+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9819f770 con 0x7fba980ff440 2024-03-12T05:10:44.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.530+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe8900b87f0 con 0x7fe8a0101c20 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.532+0000 7fba8f7fe640 1 -- 172.21.15.45:0/1268773189 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fba94003f40 con 0x7fba980ff440 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.532+0000 7fba8f7fe640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 0x7fba840a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.532+0000 7fba8f7fe640 1 -- 172.21.15.45:0/1268773189 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba940ccdd0 con 0x7fba980ff440 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.532+0000 7fba9d97d640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 0x7fba840a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.533+0000 7fba9d97d640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 0x7fba840a4000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba88009230 tx=0x7fba88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.533+0000 7fbaa2fb3740 1 Processor -- start 2024-03-12T05:10:44.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.533+0000 7fbaa2fb3740 1 -- start start 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.533+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2cd78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cd8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2c46f90 con 0x5640a2cd9050 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2c444c0 con 0x5640a2cd74c0 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2c3fca0 con 0x5640a2cd8690 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47808/0 (socket says 172.21.15.45:47808) 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 -- 172.21.15.45:0/13748989 learned_addr learned my addr 172.21.15.45:0/13748989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9d97d640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cd8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 -- 172.21.15.45:0/13748989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 msgr2=0x5640a2cd8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e17e640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2cd78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cd8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 -- 172.21.15.45:0/13748989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 msgr2=0x5640a2cd78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2cd78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 -- 172.21.15.45:0/13748989 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640a2c020b0 con 0x5640a2cd9050 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9d97d640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cd8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba9e97f640 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fd0 tx=0x7fba900118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c0d0b4208b359a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba8cbfe640 1 -- 172.21.15.45:0/13748989 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90004d20 con 0x5640a2cd9050 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba8cbfe640 1 -- 172.21.15.45:0/13748989 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba90004e80 con 0x5640a2cd9050 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.534+0000 7fba8cbfe640 1 -- 172.21.15.45:0/13748989 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90005280 con 0x5640a2cd9050 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/13748989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 msgr2=0x5640a2cdf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fd0 tx=0x7fba900118d0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/13748989 shutdown_connections 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2cdf8e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cd8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/13748989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2cd78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/13748989 >> 172.21.15.45:0/13748989 conn(0x5640a2cd26a0 msgr2=0x5640a2cd2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/13748989 shutdown_connections 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.535+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/13748989 wait complete. 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 -- start start 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 Processor -- start 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2be4f40 con 0x5640a2cd9050 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a29134b0 con 0x5640a2cd74c0 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2bc5670 con 0x5640a2cd8690 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fbaa2fb3740 10 client.0 _ll_register_callbacks cb 0x5640a2c32f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47814/0 (socket says 172.21.15.45:47814) 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 -- 172.21.15.45:0/1041904011 learned_addr learned my addr 172.21.15.45:0/1041904011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9d97d640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 -- 172.21.15.45:0/1041904011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 msgr2=0x5640a2ce1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 -- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 msgr2=0x5640a2ce1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.536+0000 7fba9e97f640 1 -- 172.21.15.45:0/1041904011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90009c80 con 0x5640a2cd9050 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba9e97f640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fa0 tx=0x7fba90004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f44bfc93803ee79 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba90003e40 con 0x5640a2cd9050 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba9000f4e0 con 0x5640a2cd9050 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640a2c19f40 con 0x5640a2cd9050 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba9000f6a0 con 0x5640a2cd9050 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 10 client.15480 fetch_fsmap learned FSMap version 25 2024-03-12T05:10:44.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5640a2c3e320 con 0x5640a2cd9050 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5640a2c19f40 con 0x5640a2cd9050 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fba90021040 con 0x5640a2cd9050 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 10 client.15480 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 10 client.15480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 20 client.15480 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2cefc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 4 client.15480 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.537+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640a2c36310 con 0x5640a2ced810 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.538+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2cefc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.538+0000 7fba81ffb640 10 client.15480 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.538+0000 7fba9e17e640 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2cefc30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf99b97a8c5c6048 server_cookie=3dc42400b91fadf2 in_seq=0 out_seq=0 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fba81ffb640 1 -- 172.21.15.45:0/1041904011 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5640a2c36310 con 0x5640a2ced810 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fba81ffb640 10 client.15480 handle_command_reply: tid=0 2024-03-12T05:10:44.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 client.15480 shutdown 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 msgr2=0x5640a2ce8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fba90009fa0 tx=0x7fba90004c70 comp rx=0 tx=0).stop 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 shutdown_connections 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2cefc30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd9050 0x5640a2ce8ce0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fba81ffb640 0 client.15480 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/1041904011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 >> 172.21.15.45:0/1041904011 conn(0x5640a2cd26a0 msgr2=0x5640a2cd18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 shutdown_connections 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/1041904011 wait complete. 2024-03-12T05:10:44.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.539+0000 7fbaa2fb3740 20 client.15480 trim_cache size 0 max 16384 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.554+0000 7fbaa2fb3740 1 Processor -- start 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.554+0000 7fbaa2fb3740 1 -- start start 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd9050 0x5640a2cf4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2cc7350 con 0x5640a2cd8690 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2cd6f00 con 0x5640a2cd74c0 2024-03-12T05:10:44.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2c19f40 con 0x5640a2cd9050 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47828/0 (socket says 172.21.15.45:47828) 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 -- 172.21.15.45:0/750498155 learned_addr learned my addr 172.21.15.45:0/750498155 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9e97f640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd9050 0x5640a2cf4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9e17e640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 -- 172.21.15.45:0/750498155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd9050 msgr2=0x5640a2cf4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd9050 0x5640a2cf4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 -- 172.21.15.45:0/750498155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 msgr2=0x5640a2ce1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 -- 172.21.15.45:0/750498155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640a2bdac00 con 0x5640a2cd8690 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9e17e640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba9d97d640 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9495533a5cc3625d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba83fff640 1 -- 172.21.15.45:0/750498155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88004ac0 con 0x5640a2cd8690 2024-03-12T05:10:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba83fff640 1 -- 172.21.15.45:0/750498155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba880044f0 con 0x5640a2cd8690 2024-03-12T05:10:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.555+0000 7fba83fff640 1 -- 172.21.15.45:0/750498155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88004dc0 con 0x5640a2cd8690 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/750498155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 msgr2=0x5640a2ce1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fba88009900 tx=0x7fba8803f000 comp rx=0 tx=0).stop 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/750498155 shutdown_connections 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd8690 0x5640a2ce1ca0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd9050 0x5640a2cf4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/750498155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd74c0 0x5640a2ce1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/750498155 >> 172.21.15.45:0/750498155 conn(0x5640a2cd26a0 msgr2=0x5640a2cdc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.556+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/750498155 shutdown_connections 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/750498155 wait complete. 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 -- start start 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 Processor -- start 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cf3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd9050 0x5640a2cf3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2cf3d80 con 0x5640a2cd74c0 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2cf3ef0 con 0x5640a2cd9050 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a2d43c30 con 0x5640a2cd8690 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fbaa2fb3740 10 client.0 _ll_register_callbacks cb 0x7fba90009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fba9e17e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fba9e17e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47840/0 (socket says 172.21.15.45:47840) 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.557+0000 7fba9e17e640 1 -- 172.21.15.45:0/3038915049 learned_addr learned my addr 172.21.15.45:0/3038915049 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e97f640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd9050 0x5640a2cf3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9d97d640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cf3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 -- 172.21.15.45:0/3038915049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 msgr2=0x5640a2cf3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cf3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 -- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd9050 msgr2=0x5640a2cf3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd9050 0x5640a2cf3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 -- 172.21.15.45:0/3038915049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba98109ea0 con 0x5640a2cd74c0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba9e17e640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fba940ca7c0 tx=0x7fba94095f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efcf47a263b256a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9409dd40 con 0x5640a2cd74c0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7fba940a2390 con 0x5640a2cd74c0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640a2c65900 con 0x5640a2cd74c0 2024-03-12T05:10:44.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba940041f0 con 0x5640a2cd74c0 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fbaa2fb3740 10 client.15486 fetch_fsmap learned FSMap version 25 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5640a2c00910 con 0x5640a2cd74c0 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5640a2c65900 con 0x5640a2cd74c0 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.558+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fba940959f0 con 0x5640a2cd74c0 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 10 client.15486 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 10 client.15486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 20 client.15486 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2d476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 4 client.15486 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5640a293f860 con 0x5640a2ced810 2024-03-12T05:10:44.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fba9d97d640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2d476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fba9d97d640 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2d476b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285ca1e2919e5fac server_cookie=4f8d9a84f50211a5 in_seq=0 out_seq=0 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.559+0000 7fba81ffb640 10 client.15486 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.560+0000 7fba81ffb640 1 -- 172.21.15.45:0/3038915049 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5640a293f860 con 0x5640a2ced810 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.560+0000 7fba81ffb640 10 client.15486 handle_command_reply: tid=0 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.560+0000 7fbaa2fb3740 1 client.15486 shutdown 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.560+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 msgr2=0x5640a2cf4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.560+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fba940ca7c0 tx=0x7fba94095f40 comp rx=0 tx=0).stop 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 shutdown_connections 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5640a2ced810 0x5640a2d476b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640a2cd9050 0x5640a2cf3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a2cd8690 0x5640a2cf3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 --2- 172.21.15.45:0/3038915049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5640a2cd74c0 0x5640a2cf4950 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 >> 172.21.15.45:0/3038915049 conn(0x5640a2cd26a0 msgr2=0x5640a2cdc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fba81ffb640 0 client.15486 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 shutdown_connections 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 1 -- 172.21.15.45:0/3038915049 wait complete. 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa2fb3740 20 client.15486 trim_cache size 0 max 16384 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "b5af6ed2-357e-4c46-a143-e05802222bef", 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:10:44.564 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 msgr2=0x7fba840a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 0x7fba840a4000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fba88009230 tx=0x7fba88005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 msgr2=0x7fba98197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fba9400d770 tx=0x7fba9400dc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 shutdown_connections 2024-03-12T05:10:44.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fba840a1b40 0x7fba840a4000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba98102dd0 0x7fba9819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba98102410 0x7fba98197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 --2- 172.21.15.45:0/1268773189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba980ff440 0x7fba98197140 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 >> 172.21.15.45:0/1268773189 conn(0x7fba980fad70 msgr2=0x7fba981057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 shutdown_connections 2024-03-12T05:10:44.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.561+0000 7fbaa03e2640 1 -- 172.21.15.45:0/1268773189 wait complete. 2024-03-12T05:10:44.569 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:10:44.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:10:44.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2524244117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3991008639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:44 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2653332522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:44.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- 172.21.15.45:0/1423685604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068102400 msgr2=0x7f4068102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 --2- 172.21.15.45:0/1423685604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068102400 0x7f4068102880 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4058009900 tx=0x7f405802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- 172.21.15.45:0/1423685604 shutdown_connections 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 --2- 172.21.15.45:0/1423685604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068102400 0x7f4068102880 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 --2- 172.21.15.45:0/1423685604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102dc0 0x7f4068109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 --2- 172.21.15.45:0/1423685604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068101210 0x7f4068101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- 172.21.15.45:0/1423685604 >> 172.21.15.45:0/1423685604 conn(0x7f40680fc7a0 msgr2=0x7f40680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- 172.21.15.45:0/1423685604 shutdown_connections 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- 172.21.15.45:0/1423685604 wait complete. 2024-03-12T05:10:44.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 Processor -- start 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.690+0000 7f406d379640 1 -- start start 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 0x7f4068073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068102dc0 0x7f4068073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068074210 con 0x7f4068101210 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4068074380 con 0x7f4068102400 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f406d379640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681a1180 con 0x7f4068102dc0 2024-03-12T05:10:44.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47850/0 (socket says 172.21.15.45:47850) 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 -- 172.21.15.45:0/2680133736 learned_addr learned my addr 172.21.15.45:0/2680133736 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f40677fe640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068102dc0 0x7f4068073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f40667fc640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 0x7f4068073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 -- 172.21.15.45:0/2680133736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068102dc0 msgr2=0x7f4068073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068102dc0 0x7f4068073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 -- 172.21.15.45:0/2680133736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 msgr2=0x7f4068073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 0x7f4068073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 -- 172.21.15.45:0/2680133736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058009560 con 0x7f4068101210 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f40667fc640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 0x7f4068073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f4066ffd640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f405c00d980 tx=0x7f405c00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.691+0000 7f404ffff640 1 -- 172.21.15.45:0/2680133736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c009c30 con 0x7f4068101210 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.692+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40681a1430 con 0x7f4068101210 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.692+0000 7f404ffff640 1 -- 172.21.15.45:0/2680133736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f405c009d90 con 0x7f4068101210 2024-03-12T05:10:44.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.692+0000 7f404ffff640 1 -- 172.21.15.45:0/2680133736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c00f620 con 0x7f4068101210 2024-03-12T05:10:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.692+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40681a1940 con 0x7f4068101210 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 Processor -- start 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 -- start start 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f32806e30 0x559f328092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f32771280 con 0x559f32801cc0 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f3276e7b0 con 0x559f328012d0 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f407001f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f32769f90 con 0x559f32806e30 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.694+0000 7f404ffff640 1 -- 172.21.15.45:0/2680133736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f405c010460 con 0x7f4068101210 2024-03-12T05:10:44.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59488/0 (socket says 172.21.15.45:59488) 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 -- 172.21.15.45:0/1155357979 learned_addr learned my addr 172.21.15.45:0/1155357979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40667fc640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f32806e30 0x559f328092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f4066ffd640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 -- 172.21.15.45:0/1155357979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f32806e30 msgr2=0x559f328092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f32806e30 0x559f328092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 -- 172.21.15.45:0/1155357979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 msgr2=0x559f328067c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328067c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 -- 172.21.15.45:0/1155357979 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f32728570 con 0x559f328012d0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f4066ffd640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f40677fe640 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4060010370 tx=0x7f4060010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8e202907c675 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f404ffff640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 0x7f40500a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f404d7fa640 1 -- 172.21.15.45:0/1155357979 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060004d20 con 0x559f328012d0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f4066ffd640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 0x7f40500a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f404d7fa640 1 -- 172.21.15.45:0/1155357979 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4060004e80 con 0x559f328012d0 2024-03-12T05:10:44.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f404d7fa640 1 -- 172.21.15.45:0/1155357979 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060005400 con 0x559f328012d0 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.695+0000 7f404ffff640 1 -- 172.21.15.45:0/2680133736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f405c0cd300 con 0x7f4068101210 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- 172.21.15.45:0/1155357979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 msgr2=0x559f328016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4060010370 tx=0x7f4060010830 comp rx=0 tx=0).stop 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f4066ffd640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 0x7f40500a3f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x559f32802730 tx=0x7f405c0cd580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- 172.21.15.45:0/1155357979 shutdown_connections 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f32806e30 0x559f328092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 --2- 172.21.15.45:0/1155357979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f328012d0 0x559f328016f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- 172.21.15.45:0/1155357979 >> 172.21.15.45:0/1155357979 conn(0x559f327fc7f0 msgr2=0x559f327fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- 172.21.15.45:0/1155357979 shutdown_connections 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- 172.21.15.45:0/1155357979 wait complete. 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.696+0000 7f407001f740 1 -- start start 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 Processor -- start 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f3280b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32806e30 0x559f328127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f3243d430 con 0x559f32801cc0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f323f9e00 con 0x559f32806e30 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f326eefd0 con 0x559f328012d0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f407001f740 10 client.0 _ll_register_callbacks cb 0x559f32744fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47862/0 (socket says 172.21.15.45:47862) 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 -- 172.21.15.45:0/64641062 learned_addr learned my addr 172.21.15.45:0/64641062 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40677fe640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32806e30 0x559f328127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40677fe640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f3280b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 -- 172.21.15.45:0/64641062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 msgr2=0x559f3280b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f3280b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 -- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32806e30 msgr2=0x559f328127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32806e30 0x559f328127f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 -- 172.21.15.45:0/64641062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4060009c50 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f40667fc640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4058009230 tx=0x7f40580048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5804c704633a35af server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.697+0000 7f404aff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058004250 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f324a6720 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f40580043b0 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40580035f0 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 10 client.15498 fetch_fsmap learned FSMap version 25 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559f32768610 con 0x559f32801cc0 2024-03-12T05:10:44.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559f324a6720 con 0x559f32801cc0 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4058051070 con 0x559f32801cc0 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 10 client.15498 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 20 client.15498 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32817520 0x559f32819940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 4 client.15498 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f324f5890 con 0x559f32817520 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.698+0000 7f40677fe640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32817520 0x559f32819940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f40677fe640 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32817520 0x559f32819940 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e39663873e6a959b server_cookie=a5780c2fc2f28b26 in_seq=0 out_seq=0 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f404aff5640 10 client.15498 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f404aff5640 1 -- 172.21.15.45:0/64641062 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559f324f5890 con 0x559f32817520 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f404aff5640 10 client.15498 handle_command_reply: tid=0 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f407001f740 1 client.15498 shutdown 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 msgr2=0x559f3280b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.699+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4058009230 tx=0x7f40580048f0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 shutdown_connections 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32817520 0x559f32819940 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f3280b7b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32806e30 0x559f328127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f404aff5640 0 client.15498 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 --2- 172.21.15.45:0/64641062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f3280b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 >> 172.21.15.45:0/64641062 conn(0x559f327fc7f0 msgr2=0x559f327fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 shutdown_connections 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 1 -- 172.21.15.45:0/64641062 wait complete. 2024-03-12T05:10:44.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.700+0000 7f407001f740 20 client.15498 trim_cache size 0 max 16384 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.714+0000 7f407001f740 1 Processor -- start 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 -- start start 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f327fc7f0 0x559f32802f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f32803480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f327f8120 con 0x559f32801cc0 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f327f8320 con 0x559f327fc7f0 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f407001f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f32813ce0 con 0x559f328012d0 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47874/0 (socket says 172.21.15.45:47874) 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 -- 172.21.15.45:0/4082505162 learned_addr learned my addr 172.21.15.45:0/4082505162 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40667fc640 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f32803480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 -- 172.21.15.45:0/4082505162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 msgr2=0x559f32803480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f32803480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 -- 172.21.15.45:0/4082505162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f327fc7f0 msgr2=0x559f32802f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f327fc7f0 0x559f32802f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 -- 172.21.15.45:0/4082505162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f32704ac0 con 0x559f32801cc0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f40677fe640 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4060005fd0 tx=0x7f4060010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36380916590b5d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f404cff9640 1 -- 172.21.15.45:0/4082505162 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060004d20 con 0x559f32801cc0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f404cff9640 1 -- 172.21.15.45:0/4082505162 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4060005000 con 0x559f32801cc0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.715+0000 7f404cff9640 1 -- 172.21.15.45:0/4082505162 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406002b5d0 con 0x559f32801cc0 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 -- 172.21.15.45:0/4082505162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 msgr2=0x559f328127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4060005fd0 tx=0x7f4060010830 comp rx=0 tx=0).stop 2024-03-12T05:10:44.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 -- 172.21.15.45:0/4082505162 shutdown_connections 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f32801cc0 0x559f328127b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f32803480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 --2- 172.21.15.45:0/4082505162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f327fc7f0 0x559f32802f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 -- 172.21.15.45:0/4082505162 >> 172.21.15.45:0/4082505162 conn(0x559f32806900 msgr2=0x559f3280b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 -- 172.21.15.45:0/4082505162 shutdown_connections 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.716+0000 7f407001f740 1 -- 172.21.15.45:0/4082505162 wait complete. 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 -- start start 2024-03-12T05:10:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 Processor -- start 2024-03-12T05:10:44.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f328115c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32801cc0 0x559f32811b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f3286da10 con 0x559f327fc7f0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f3286db50 con 0x559f32801cc0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f3286dc90 con 0x559f328012d0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f407001f740 10 client.0 _ll_register_callbacks cb 0x7f4060009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47884/0 (socket says 172.21.15.45:47884) 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 -- 172.21.15.45:0/587283164 learned_addr learned my addr 172.21.15.45:0/587283164 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40667fc640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f328115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32801cc0 0x559f32811b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 -- 172.21.15.45:0/587283164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 msgr2=0x559f328115c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f328115c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 -- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32801cc0 msgr2=0x559f32811b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32801cc0 0x559f32811b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.717+0000 7f40677fe640 1 -- 172.21.15.45:0/587283164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f32728570 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f40677fe640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4060010d30 tx=0x7f4060010e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b11547a113f3375 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060003bd0 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f4060005460 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f324697e0 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4060005620 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 10 client.15504 fetch_fsmap learned FSMap version 25 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559f3272a6a0 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559f324697e0 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4060035040 con 0x559f327fc7f0 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 10 client.15504 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:10:44.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 10 client.15504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 20 client.15504 populate_metadata read hostname 'smithi045' 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32806e30 0x559f32871290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 4 client.15504 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.718+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559f324584e0 con 0x559f32806e30 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f40667fc640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32806e30 0x559f32871290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f40667fc640 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32806e30 0x559f32871290 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88a3c8444ed5075b server_cookie=b3b28e4bd070cb7c in_seq=0 out_seq=0 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f404aff5640 10 client.15504 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f404aff5640 1 -- 172.21.15.45:0/587283164 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559f324584e0 con 0x559f32806e30 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f404aff5640 10 client.15504 handle_command_reply: tid=0 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 client.15504 shutdown 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 msgr2=0x559f32811080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4060010d30 tx=0x7f4060010e60 comp rx=0 tx=0).stop 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 shutdown_connections 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559f32806e30 0x559f32871290 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f32801cc0 0x559f32811b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f328012d0 0x559f328115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f404aff5640 0 client.15504 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 --2- 172.21.15.45:0/587283164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f327fc7f0 0x559f32811080 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 >> 172.21.15.45:0/587283164 conn(0x559f32806900 msgr2=0x559f3280b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 shutdown_connections 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.719+0000 7f407001f740 1 -- 172.21.15.45:0/587283164 wait complete. 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f407001f740 20 client.15504 trim_cache size 0 max 16384 2024-03-12T05:10:44.723 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "b5af6ed2-357e-4c46-a143-e05802222bef": { 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "b5af6ed2-357e-4c46-a143-e05802222bef", 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:10:44.724 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 msgr2=0x7f40500a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 0x7f40500a3f80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x559f32802730 tx=0x7f405c0cd580 comp rx=0 tx=0).stop 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 msgr2=0x7f4068074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f405c00d980 tx=0x7f405c00de40 comp rx=0 tx=0).stop 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 shutdown_connections 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f40500a1ac0 0x7f40500a3f80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4068102dc0 0x7f4068073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4068102400 0x7f4068073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 --2- 172.21.15.45:0/2680133736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4068101210 0x7f4068074f30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 >> 172.21.15.45:0/2680133736 conn(0x7f40680fc7a0 msgr2=0x7f4068105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 shutdown_connections 2024-03-12T05:10:44.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.720+0000 7f406d379640 1 -- 172.21.15.45:0/2680133736 wait complete. 2024-03-12T05:10:44.726 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2524244117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.726 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3991008639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.726 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:44 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2653332522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:44.728 INFO:tasks.cephfs.filesystem:scrub status for tag:b5af6ed2-357e-4c46-a143-e05802222bef - {'path': '/', 'tag': 'b5af6ed2-357e-4c46-a143-e05802222bef', 'options': 'recursive,force'} 2024-03-12T05:10:44.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2524244117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3991008639' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:10:44.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:44 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2653332522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:10:44.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:44.780+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe8a019f310 con 0x7fe8a0101c20 2024-03-12T05:10:45.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.370+0000 7fe89cff9640 1 -- 172.21.15.45:0/517239415 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe8a019f310 con 0x7fe8a0101c20 2024-03-12T05:10:45.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.372+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 msgr2=0x7fe8880a4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.372+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 0x7fe8880a4030 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe8a01985f0 tx=0x7fe89400b390 comp rx=0 tx=0).stop 2024-03-12T05:10:45.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 msgr2=0x7fe8a01970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe890036040 tx=0x7fe8900042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 shutdown_connections 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7fe8880a1b70 0x7fe8880a4030 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe8a0106e80 0x7fe8a019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a0104550 0x7fe8a0197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 --2- 172.21.15.45:0/517239415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8a0101c20 0x7fe8a01970e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 >> 172.21.15.45:0/517239415 conn(0x7fe8a00fb610 msgr2=0x7fe8a01026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 shutdown_connections 2024-03-12T05:10:45.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.373+0000 7fe8a5c81640 1 -- 172.21.15.45:0/517239415 wait complete. 2024-03-12T05:10:45.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 -- 172.21.15.45:0/1004846634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 msgr2=0x7f42381084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 --2- 172.21.15.45:0/1004846634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f42381084e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4230009ca0 tx=0x7f423002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 -- 172.21.15.45:0/1004846634 shutdown_connections 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 --2- 172.21.15.45:0/1004846634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f42381084e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 --2- 172.21.15.45:0/1004846634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4238101290 0x7f4238101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 --2- 172.21.15.45:0/1004846634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42381000a0 0x7f42381004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 -- 172.21.15.45:0/1004846634 >> 172.21.15.45:0/1004846634 conn(0x7f42380fb610 msgr2=0x7f42380fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 -- 172.21.15.45:0/1004846634 shutdown_connections 2024-03-12T05:10:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.502+0000 7f423cb4b640 1 -- 172.21.15.45:0/1004846634 wait complete. 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 Processor -- start 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 -- start start 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42381000a0 0x7f4238197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4238101290 0x7f42381975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423819eb60 con 0x7f4238101c50 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423819ecd0 con 0x7f4238101290 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423cb4b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423819ee90 con 0x7f42381000a0 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f4236d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f4236d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47930/0 (socket says 172.21.15.45:47930) 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f4236d9d640 1 -- 172.21.15.45:0/2852945104 learned_addr learned my addr 172.21.15.45:0/2852945104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f423659c640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42381000a0 0x7f4238197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.503+0000 7f422ffff640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4238101290 0x7f42381975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 -- 172.21.15.45:0/2852945104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42381000a0 msgr2=0x7f4238197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42381000a0 0x7f4238197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 -- 172.21.15.45:0/2852945104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4238101290 msgr2=0x7f42381975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4238101290 0x7f42381975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 -- 172.21.15.45:0/2852945104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230009900 con 0x7f4238101c50 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f4236d9d640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4230037040 tx=0x7f4230032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230034e70 con 0x7f4238101c50 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423819f110 con 0x7f4238101c50 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4230003e20 con 0x7f4238101c50 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230004710 con 0x7f4238101c50 2024-03-12T05:10:45.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.504+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423819f5c0 con 0x7f4238101c50 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.505+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423810b960 con 0x7f4238101c50 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.507+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) v1 ==== 136347+0+0 (secure 0 0 0) 0x7f423003b040 con 0x7f4238101c50 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.507+0000 7f422f7fe640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 0x7f42240a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.507+0000 7f423659c640 1 -- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 msgr2=0x7f42240a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.508+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f42300efdd0 con 0x7f4238101c50 2024-03-12T05:10:45.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.508+0000 7f423659c640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 0x7f42240a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:10:45.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.511+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f42300b9050 con 0x7f4238101c50 2024-03-12T05:10:45.617 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setuser ceph since I am not root 2024-03-12T05:10:45.617 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: ignoring --setgroup ceph since I am not root 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1894281347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:10:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-mon[32174]: pgmap v348: 97 pgs: 97 active+clean; 653 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.3 KiB/s wr, 15 op/s 2024-03-12T05:10:45.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.697+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f423810baf0 con 0x7f4238101c50 2024-03-12T05:10:45.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.701+0000 7f422f7fe640 1 -- 172.21.15.45:0/2852945104 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7f42300bc710 con 0x7f4238101c50 2024-03-12T05:10:45.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 msgr2=0x7f42240a4080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:10:45.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 0x7f42240a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 msgr2=0x7f423819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4230037040 tx=0x7f4230032980 comp rx=0 tx=0).stop 2024-03-12T05:10:45.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 shutdown_connections 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f42240a1bc0 0x7f42240a4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4238101c50 0x7f423819e620 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4238101290 0x7f42381975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 --2- 172.21.15.45:0/2852945104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42381000a0 0x7f4238197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 >> 172.21.15.45:0/2852945104 conn(0x7f42380fb610 msgr2=0x7f42380fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 shutdown_connections 2024-03-12T05:10:45.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.704+0000 7f423cb4b640 1 -- 172.21.15.45:0/2852945104 wait complete. 2024-03-12T05:10:45.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-12T05:10:45.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: ignoring --setuser ceph since I am not root 2024-03-12T05:10:45.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: ignoring --setgroup ceph since I am not root 2024-03-12T05:10:45.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mgr[37286]: -- 172.21.15.130:0/2796617090 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568794a41a0 con 0x55687947a800 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1894281347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:10:45.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-mon[35398]: pgmap v348: 97 pgs: 97 active+clean; 653 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.3 KiB/s wr, 15 op/s 2024-03-12T05:10:45.744 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: ignoring --setuser ceph since I am not root 2024-03-12T05:10:45.744 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: ignoring --setgroup ceph since I am not root 2024-03-12T05:10:45.744 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mgr[36875]: -- 172.21.15.116:0/940689251 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff00b0c1a0 con 0x55ff00ae3000 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1894281347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:10:45.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-mon[35486]: pgmap v348: 97 pgs: 97 active+clean; 653 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.3 KiB/s wr, 15 op/s 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2937448130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc102d80 msgr2=0x7f58dc109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2937448130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc102d80 0x7f58dc109610 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ca0 tx=0x7f58d402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2937448130 shutdown_connections 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2937448130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc102d80 0x7f58dc109610 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2937448130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 0x7f58dc102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2937448130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc1011d0 0x7f58dc1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2937448130 >> 172.21.15.45:0/2937448130 conn(0x7f58dc0fc740 msgr2=0x7f58dc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2937448130 shutdown_connections 2024-03-12T05:10:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.823+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2937448130 wait complete. 2024-03-12T05:10:45.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.824+0000 7f58e1c2a640 1 Processor -- start 2024-03-12T05:10:45.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.824+0000 7f58e1c2a640 1 -- start start 2024-03-12T05:10:45.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.824+0000 7f58e1c2a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.824+0000 7f58e1c2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 0x7f58dc1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc102d80 0x7f58dc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc19eb70 con 0x7f58dc1011d0 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc19ece0 con 0x7f58dc1023c0 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc19eea0 con 0x7f58dc102d80 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47950/0 (socket says 172.21.15.45:47950) 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 -- 172.21.15.45:0/2552578961 learned_addr learned my addr 172.21.15.45:0/2552578961 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58daffd640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 0x7f58dc1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58dbfff640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc102d80 0x7f58dc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 -- 172.21.15.45:0/2552578961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc102d80 msgr2=0x7f58dc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc102d80 0x7f58dc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 -- 172.21.15.45:0/2552578961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 msgr2=0x7f58dc1975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:45.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 0x7f58dc1975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 -- 172.21.15.45:0/2552578961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4009900 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58db7fe640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00e9c0 tx=0x7f58cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0096e0 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f58cc0044e0 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc19f180 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc010430 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.825+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc19f690 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.827+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc1084f0 con 0x7f58dc1011d0 2024-03-12T05:10:45.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.827+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) v1 ==== 136347+0+0 (secure 0 0 0) 0x7f58cc003f80 con 0x7f58dc1011d0 2024-03-12T05:10:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.828+0000 7f58d8ff9640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 0x7f58c40a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.828+0000 7f58daffd640 1 -- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 msgr2=0x7f58c40a4100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.828+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f58cc014070 con 0x7f58dc1011d0 2024-03-12T05:10:45.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.828+0000 7f58daffd640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 0x7f58c40a4100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:10:45.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:45.832+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58cc096980 con 0x7f58dc1011d0 2024-03-12T05:10:46.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.015+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f58dc10ca90 con 0x7f58dc1011d0 2024-03-12T05:10:46.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.019+0000 7f58d8ff9640 1 -- 172.21.15.45:0/2552578961 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f58cc09a040 con 0x7f58dc1011d0 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.021+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 msgr2=0x7f58c40a4100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.021+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 0x7f58c40a4100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.021+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 msgr2=0x7f58dc197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.021+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00e9c0 tx=0x7f58cc00ee80 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 shutdown_connections 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f58c40a1c40 0x7f58c40a4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58dc102d80 0x7f58dc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58dc1023c0 0x7f58dc1975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 --2- 172.21.15.45:0/2552578961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58dc1011d0 0x7f58dc197080 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 >> 172.21.15.45:0/2552578961 conn(0x7f58dc0fc740 msgr2=0x7f58dc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:46.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 shutdown_connections 2024-03-12T05:10:46.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.022+0000 7f58e1c2a640 1 -- 172.21.15.45:0/2552578961 wait complete. 2024-03-12T05:10:46.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-12T05:10:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- 172.21.15.45:0/826846231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 msgr2=0x7f5fe81029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/826846231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81029a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009900 tx=0x7f5fd802bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- 172.21.15.45:0/826846231 shutdown_connections 2024-03-12T05:10:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/826846231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81029a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/826846231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe8102ee0 0x7f5fe8109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/826846231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fe80ff4b0 0x7f5fe80ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- 172.21.15.45:0/826846231 >> 172.21.15.45:0/826846231 conn(0x7f5fe80fada0 msgr2=0x7f5fe80fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- 172.21.15.45:0/826846231 shutdown_connections 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- 172.21.15.45:0/826846231 wait complete. 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 Processor -- start 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 -- start start 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.139+0000 7f5fed2c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe80ff4b0 0x7f5fe8101e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81004d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fe8102ee0 0x7f5fe8100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8101170 con 0x7f5fe8102520 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe81012e0 con 0x7f5fe80ff4b0 2024-03-12T05:10:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe81a1160 con 0x7f5fe8102ee0 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81004d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47970/0 (socket says 172.21.15.45:47970) 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 -- 172.21.15.45:0/2629487747 learned_addr learned my addr 172.21.15.45:0/2629487747 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe80ff4b0 0x7f5fe8101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe77fe640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fe8102ee0 0x7f5fe8100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fe8102ee0 msgr2=0x7f5fe8100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5fe8102ee0 0x7f5fe8100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe80ff4b0 msgr2=0x7f5fe8101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fe80ff4b0 0x7f5fe8101e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8009560 con 0x7f5fe8102520 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fe67fc640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81004d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009430 tx=0x7f5fd80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8004510 con 0x7f5fe8102520 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5fd802de90 con 0x7f5fe8102520 2024-03-12T05:10:46.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe81a13b0 con 0x7f5fe8102520 2024-03-12T05:10:46.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8003cc0 con 0x7f5fe8102520 2024-03-12T05:10:46.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.140+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe81a18c0 con 0x7f5fe8102520 2024-03-12T05:10:46.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.141+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe8108720 con 0x7f5fe8102520 2024-03-12T05:10:46.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.142+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 20) v1 ==== 136347+0+0 (secure 0 0 0) 0x7f5fd8004fd0 con 0x7f5fe8102520 2024-03-12T05:10:46.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.143+0000 7f5fcffff640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:46.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.143+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:46.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.143+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8219+0+0 (secure 0 0 0) 0x7f5fd80ef6d0 con 0x7f5fe8102520 2024-03-12T05:10:46.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.143+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:10:46.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.147+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5fd80b88a0 con 0x7f5fe8102520 2024-03-12T05:10:46.219 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:45.938+0000 7f19b188b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:10:46.219 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:46.032+0000 7f19b188b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:10:46.244 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:45.942+0000 7f6c366c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:10:46.244 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:46.036+0000 7f6c366c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:10:46.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.344+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:46.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.344+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:10:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.352+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe802ce40 con 0x7f5fd00a1bc0 2024-03-12T05:10:46.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:45.974+0000 7f5280c1d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:10:46.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:46.076+0000 7f5280c1d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:10:46.666 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:10:46.666 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-mon[35398]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:10:46.666 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2852945104' entity='client.admin' 2024-03-12T05:10:46.666 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2552578961' entity='client.admin' 2024-03-12T05:10:46.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:10:46.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-mon[35486]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:10:46.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2852945104' entity='client.admin' 2024-03-12T05:10:46.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2552578961' entity='client.admin' 2024-03-12T05:10:46.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/517239415' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:10:46.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-mon[32174]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:10:46.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2852945104' entity='client.admin' 2024-03-12T05:10:46.740 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2552578961' entity='client.admin' 2024-03-12T05:10:46.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.745+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:46.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:46.745+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:10:46.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:46.667+0000 7f19b188b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:10:46.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:46.760+0000 7f19b188b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:10:46.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:46.901+0000 7f19b188b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:10:46.994 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:46.670+0000 7f6c366c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:10:46.994 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:46.762+0000 7f6c366c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:10:46.994 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:46.902+0000 7f6c366c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:10:47.078 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:46.742+0000 7f5280c1d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:10:47.078 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:46.838+0000 7f5280c1d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:10:47.078 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:46.982+0000 7f5280c1d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:10:47.078 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.080+0000 7f5280c1d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:10:47.336 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:46.997+0000 7f6c366c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:10:47.336 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.079+0000 7f6c366c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:10:47.338 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:46.996+0000 7f19b188b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:10:47.338 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.079+0000 7f19b188b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:10:47.338 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.339+0000 7f19b188b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:10:47.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.166+0000 7f5280c1d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:10:47.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:47.546+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:47.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:47.546+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:10:47.625 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.338+0000 7f6c366c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:10:47.625 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.447+0000 7f6c366c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:47.625 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.533+0000 7f6c366c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:10:47.625 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.627+0000 7f6c366c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:10:47.628 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.449+0000 7f19b188b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:47.629 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.535+0000 7f19b188b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:10:47.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:10:47.733 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.436+0000 7f5280c1d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:10:47.733 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.548+0000 7f5280c1d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:47.733 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.636+0000 7f5280c1d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:10:47.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:10:47.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.630+0000 7f19b188b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:10:47.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:47.807+0000 7f19b188b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:10:47.994 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:47.802+0000 7f6c366c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:10:47.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:47 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:10:48.117 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.735+0000 7f5280c1d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:10:48.117 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:47.914+0000 7f5280c1d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:10:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:49.147+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:49.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:49.147+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:10:49.208 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:48.751+0000 7f6c366c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:10:49.208 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:48.881+0000 7f6c366c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:10:49.208 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:49.210+0000 7f6c366c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:10:49.217 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:48.758+0000 7f19b188b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:10:49.218 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:48.887+0000 7f19b188b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:10:49.218 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:49.219+0000 7f19b188b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:10:49.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:48.901+0000 7f5280c1d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:10:49.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:49.036+0000 7f5280c1d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:10:49.469 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:49.307+0000 7f19b188b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:10:49.490 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:49.299+0000 7f6c366c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:10:49.671 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:49.376+0000 7f5280c1d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:10:49.671 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:49.470+0000 7f5280c1d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:10:49.744 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:49.493+0000 7f6c366c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:49.744 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:49.677+0000 7f6c366c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:10:49.969 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:49.503+0000 7f19b188b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:49.970 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:49.686+0000 7f19b188b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:10:50.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:49.674+0000 7f5280c1d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:50.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:49.865+0000 7f5280c1d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:10:50.210 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:49.957+0000 7f6c366c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:50.236 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:49.970+0000 7f19b188b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:50.420 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:50.159+0000 7f5280c1d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:10:50.494 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:50.212+0000 7f6c366c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:50.494 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:50.447+0000 7f6c366c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:10:50.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:50.231+0000 7f19b188b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:50.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:50.465+0000 7f19b188b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:10:50.867 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:50.422+0000 7f5280c1d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:10:50.867 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:50.667+0000 7f5280c1d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:10:51.494 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:51.239+0000 7f6c366c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:10:51.719 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:51.250+0000 7f19b188b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:10:51.867 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:51.501+0000 7f5280c1d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:10:52.119 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:51.810+0000 7f6c366c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:10:52.119 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:51.908+0000 7f6c366c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:10:52.123 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:51.819+0000 7f19b188b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:10:52.124 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:51.916+0000 7f19b188b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:10:52.124 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z[37282]: 2024-03-12T05:10:52.126+0000 7f19b188b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:10:52.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.197+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 21) v1 ==== 136347+0+0 (secure 0 0 0) 0x7f5fd80b8c70 con 0x7f5fe8102520 2024-03-12T05:10:52.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.350+0000 7f5fe6ffd640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1134390880 2024-03-12T05:10:52.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.350+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-12T05:10:52.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.088+0000 7f5280c1d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:10:52.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.191+0000 7f5280c1d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: Standby manager daemon y restarted 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: Standby manager daemon y started 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: Standby manager daemon z restarted 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: Standby manager daemon z started 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.368 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:10:52.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-mon[32174]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.446+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 22) v1 ==== 135964+0+0 (secure 0 0 0) 0x7f5fd80ed220 con 0x7f5fe8102520 2024-03-12T05:10:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.447+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 msgr2=0x7f5fd00a4080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:10:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:52.447+0000 7f5fcffff640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/1134390880,v1:172.21.15.45:6801/1134390880] conn(0x7f5fd00a1bc0 0x7f5fd00a4080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: Standby manager daemon y restarted 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: Standby manager daemon y started 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: Standby manager daemon z restarted 2024-03-12T05:10:52.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: Standby manager daemon z started 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:10:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:52 smithi130 ceph-mon[35398]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.493 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y[36871]: 2024-03-12T05:10:52.121+0000 7f6c366c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: Standby manager daemon y restarted 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: Standby manager daemon y started 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: Standby manager daemon z restarted 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: Standby manager daemon z started 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.116:0/2262647935' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:10:52.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:10:52.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:10:52.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:52 smithi116 ceph-mon[35486]: from='mgr.? 172.21.15.130:0/2172193463' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:10:52.679 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.420+0000 7f5280c1d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51df607700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51df607700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51df607700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51df607700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51df607700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51e26cd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51e26cd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.118 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51e26cd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51e26cd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.119 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:10:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:10:52.681+0000 7f51e26cd700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:10:53.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.452+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mon.0 v2:172.21.15.45:3300/0 9 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f5fd8005280 con 0x7f5fe8102520 2024-03-12T05:10:53.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.454+0000 7f5fcffff640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5fd0034c50 0x7f5fd0035030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:53.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.454+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe802ce40 con 0x7f5fd0034c50 2024-03-12T05:10:53.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.455+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5fd0034c50 0x7f5fd0035030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:53.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.458+0000 7f5fe6ffd640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5fd0034c50 0x7f5fd0035030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8069740 tx=0x7f5fdc00b390 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: Active manager daemon x restarted 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: Activating manager daemon x 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: mgrmap e22: x(active, starting, since 0.0118913s), standbys: z, y 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:10:53.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:10:53.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:10:53.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: Manager daemon x is now available 2024-03-12T05:10:53.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:10:53.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:53 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: Active manager daemon x restarted 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: Activating manager daemon x 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: mgrmap e22: x(active, starting, since 0.0118913s), standbys: z, y 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:10:53.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:10:53.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:10:53.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: Manager daemon x is now available 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:10:53.497 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:53 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:10:53.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.520+0000 7f5fcffff640 1 -- 172.21.15.45:0/2629487747 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5fe802ce40 con 0x7f5fd0034c50 2024-03-12T05:10:53.517 INFO:teuthology.orchestra.run.smithi045.stdout:Schedule set for path / 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5fd0034c50 msgr2=0x7f5fd0035030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5fd0034c50 0x7f5fd0035030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8069740 tx=0x7f5fdc00b390 comp rx=0 tx=0).stop 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 msgr2=0x7f5fe81004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 --2- 172.21.15.45:0/2629487747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5fe8102520 0x7f5fe81004d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009430 tx=0x7f5fd80042f0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fe77fe640 1 -- 172.21.15.45:0/2629487747 reap_dead start 2024-03-12T05:10:53.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 shutdown_connections 2024-03-12T05:10:53.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 >> 172.21.15.45:0/2629487747 conn(0x7f5fe80fada0 msgr2=0x7f5fe81058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:53.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 shutdown_connections 2024-03-12T05:10:53.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.522+0000 7f5fed2c0640 1 -- 172.21.15.45:0/2629487747 wait complete. 2024-03-12T05:10:53.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-03-12T05:10:53.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: Active manager daemon x restarted 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: Activating manager daemon x 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: mgrmap e22: x(active, starting, since 0.0118913s), standbys: z, y 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:10:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:10:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: Manager daemon x is now available 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:10:53.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:53 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:10:53.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-12T05:10:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 -- 172.21.15.45:0/2966624634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc1000a0 msgr2=0x7f1abc1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 --2- 172.21.15.45:0/2966624634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc1000a0 0x7f1abc1004c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009900 tx=0x7f1aac02bc30 comp rx=0 tx=0).stop 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 -- 172.21.15.45:0/2966624634 shutdown_connections 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 --2- 172.21.15.45:0/2966624634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 0x7f1abc1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 --2- 172.21.15.45:0/2966624634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc101290 0x7f1abc101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 --2- 172.21.15.45:0/2966624634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc1000a0 0x7f1abc1004c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 -- 172.21.15.45:0/2966624634 >> 172.21.15.45:0/2966624634 conn(0x7f1abc0fb610 msgr2=0x7f1abc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.642+0000 7f1ac2913640 1 -- 172.21.15.45:0/2966624634 shutdown_connections 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 -- 172.21.15.45:0/2966624634 wait complete. 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 Processor -- start 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 -- start start 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc1000a0 0x7f1abc104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 0x7f1abc102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc103310 con 0x7f1abc101290 2024-03-12T05:10:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc103480 con 0x7f1abc101c50 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.643+0000 7f1ac2913640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc1a1110 con 0x7f1abc1000a0 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41636/0 (socket says 172.21.15.45:41636) 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 -- 172.21.15.45:0/1908091207 learned_addr learned my addr 172.21.15.45:0/1908091207 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abbfff640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc1000a0 0x7f1abc104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ac0eb0640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 0x7f1abc102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 -- 172.21.15.45:0/1908091207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc1000a0 msgr2=0x7f1abc104030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc1000a0 0x7f1abc104030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 -- 172.21.15.45:0/1908091207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 msgr2=0x7f1abc102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 0x7f1abc102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 -- 172.21.15.45:0/1908091207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac009560 con 0x7f1abc101290 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1abb7fe640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c950 tx=0x7f1ab000ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0007c70 con 0x7f1abc101290 2024-03-12T05:10:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1ab0007dd0 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc1a13c0 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0018820 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.644+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc1a1920 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.645+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc10b960 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.646+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f1ab0018980 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.647+0000 7f1ab97fa640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 0x7f1aa40a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.647+0000 7f1abbfff640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 0x7f1aa40a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.647+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f1ab00cd0c0 con 0x7f1abc101290 2024-03-12T05:10:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.647+0000 7f1abbfff640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 0x7f1aa40a3e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1aac02c7d0 tx=0x7f1aac005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:53.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.651+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ab0096570 con 0x7f1abc101290 2024-03-12T05:10:53.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.862+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f1abc0fd790 con 0x7f1aa40a1960 2024-03-12T05:10:53.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.920+0000 7f1ab97fa640 1 -- 172.21.15.45:0/1908091207 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1abc0fd790 con 0x7f1aa40a1960 2024-03-12T05:10:53.917 INFO:teuthology.orchestra.run.smithi045.stdout:Retention added to path / 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 msgr2=0x7f1aa40a3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 0x7f1aa40a3e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1aac02c7d0 tx=0x7f1aac005fd0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 msgr2=0x7f1abc102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c950 tx=0x7f1ab000ce10 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 shutdown_connections 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1aa40a1960 0x7f1aa40a3e20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1abc101290 0x7f1abc102670 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1abc101c50 0x7f1abc102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 --2- 172.21.15.45:0/1908091207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1abc1000a0 0x7f1abc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 >> 172.21.15.45:0/1908091207 conn(0x7f1abc0fb610 msgr2=0x7f1abc0fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 shutdown_connections 2024-03-12T05:10:53.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:53.922+0000 7f1ac2913640 1 -- 172.21.15.45:0/1908091207 wait complete. 2024-03-12T05:10:53.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 -- 172.21.15.45:0/1323691531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100d10 msgr2=0x7f5a20101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 --2- 172.21.15.45:0/1323691531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100d10 0x7f5a20101190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009ca0 tx=0x7f5a1802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 -- 172.21.15.45:0/1323691531 shutdown_connections 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 --2- 172.21.15.45:0/1323691531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100d10 0x7f5a20101190 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 --2- 172.21.15.45:0/1323691531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a20100370 0x7f5a201007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 --2- 172.21.15.45:0/1323691531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a200ffa10 0x7f5a200ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.040+0000 7f5a2839a640 1 -- 172.21.15.45:0/1323691531 >> 172.21.15.45:0/1323691531 conn(0x7f5a200fb610 msgr2=0x7f5a200fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:54.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- 172.21.15.45:0/1323691531 shutdown_connections 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- 172.21.15.45:0/1323691531 wait complete. 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 Processor -- start 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- start start 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 0x7f5a20199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 0x7f5a201a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201a0cb0 con 0x7f5a20100370 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201a0e20 con 0x7f5a20100d10 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a2839a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a201a0fe0 con 0x7f5a200ffa10 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a25935640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a25935640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41654/0 (socket says 172.21.15.45:41654) 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.041+0000 7f5a25935640 1 -- 172.21.15.45:0/2554683706 learned_addr learned my addr 172.21.15.45:0/2554683706 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a26937640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 0x7f5a201a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a26136640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 0x7f5a20199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 -- 172.21.15.45:0/2554683706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 msgr2=0x7f5a20199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 0x7f5a20199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 -- 172.21.15.45:0/2554683706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 msgr2=0x7f5a201a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 0x7f5a201a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 -- 172.21.15.45:0/2554683706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009900 con 0x7f5a20100370 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a26937640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 0x7f5a201a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a26136640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 0x7f5a20199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a25935640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c920 tx=0x7f5a1000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10007c70 con 0x7f5a20100370 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a10007dd0 con 0x7f5a20100370 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a201a12c0 con 0x7f5a20100370 2024-03-12T05:10:54.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a100187c0 con 0x7f5a20100370 2024-03-12T05:10:54.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.042+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a201a17d0 con 0x7f5a20100370 2024-03-12T05:10:54.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.043+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2010a760 con 0x7f5a20100370 2024-03-12T05:10:54.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.044+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f5a1000f450 con 0x7f5a20100370 2024-03-12T05:10:54.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.045+0000 7f5a177fe640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 0x7f5a0c0a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.045+0000 7f5a26136640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 0x7f5a0c0a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.045+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5a100cd350 con 0x7f5a20100370 2024-03-12T05:10:54.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.045+0000 7f5a26136640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 0x7f5a0c0a3da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009b50 tx=0x7f5a1c005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:54.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.049+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a100966f0 con 0x7f5a20100370 2024-03-12T05:10:54.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.251+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a2002ceb0 con 0x7f5a0c0a18e0 2024-03-12T05:10:54.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.258+0000 7f5a177fe640 1 -- 172.21.15.45:0/2554683706 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f5a2002ceb0 con 0x7f5a0c0a18e0 2024-03-12T05:10:54.258 INFO:teuthology.orchestra.run.smithi045.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-12T00:00:00", "created": "2024-03-12T05:10:53", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-12T05:10:54.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 msgr2=0x7f5a0c0a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 0x7f5a0c0a3da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009b50 tx=0x7f5a1c005f90 comp rx=0 tx=0).stop 2024-03-12T05:10:54.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 msgr2=0x7f5a201997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000c920 tx=0x7f5a1000cde0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 shutdown_connections 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a0c0a18e0 0x7f5a0c0a3da0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a20100370 0x7f5a201997a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a20100d10 0x7f5a201a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 --2- 172.21.15.45:0/2554683706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a200ffa10 0x7f5a20199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 >> 172.21.15.45:0/2554683706 conn(0x7f5a200fb610 msgr2=0x7f5a200fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 shutdown_connections 2024-03-12T05:10:54.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.260+0000 7f5a2839a640 1 -- 172.21.15.45:0/2554683706 wait complete. 2024-03-12T05:10:54.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.384+0000 7fd4c4c76640 1 -- 172.21.15.45:0/2934008430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 msgr2=0x7fd4c01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.384+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/2934008430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01011f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009ca0 tx=0x7fd4b802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.384+0000 7fd4c4c76640 1 -- 172.21.15.45:0/2934008430 shutdown_connections 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.384+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/2934008430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01011f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.384+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/2934008430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c01003d0 0x7fd4c0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.385+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/2934008430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c00ffa70 0x7fd4c00ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.385+0000 7fd4c4c76640 1 -- 172.21.15.45:0/2934008430 >> 172.21.15.45:0/2934008430 conn(0x7fd4c00fb610 msgr2=0x7fd4c00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:54.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.385+0000 7fd4c4c76640 1 -- 172.21.15.45:0/2934008430 shutdown_connections 2024-03-12T05:10:54.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.385+0000 7fd4c4c76640 1 -- 172.21.15.45:0/2934008430 wait complete. 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.385+0000 7fd4c4c76640 1 Processor -- start 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 -- start start 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c00ffa70 0x7fd4c0199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c01003d0 0x7fd4c0199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01a0d10 con 0x7fd4c0100d70 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01a0e80 con 0x7fd4c00ffa70 2024-03-12T05:10:54.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4c4c76640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01a1040 con 0x7fd4c01003d0 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41676/0 (socket says 172.21.15.45:41676) 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 -- 172.21.15.45:0/1206634457 learned_addr learned my addr 172.21.15.45:0/1206634457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bdd9b640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c01003d0 0x7fd4c0199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4be59c640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c00ffa70 0x7fd4c0199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 -- 172.21.15.45:0/1206634457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c01003d0 msgr2=0x7fd4c0199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c01003d0 0x7fd4c0199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 -- 172.21.15.45:0/1206634457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c00ffa70 msgr2=0x7fd4c0199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c00ffa70 0x7fd4c0199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 -- 172.21.15.45:0/1206634457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b4009490 con 0x7fd4c0100d70 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.386+0000 7fd4bed9d640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8037040 tx=0x7fd4b8032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.387+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8034e70 con 0x7fd4c0100d70 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.387+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b8009900 con 0x7fd4c0100d70 2024-03-12T05:10:54.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.387+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd4b8003e20 con 0x7fd4c0100d70 2024-03-12T05:10:54.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.387+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8004710 con 0x7fd4c0100d70 2024-03-12T05:10:54.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.387+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c01a15e0 con 0x7fd4c0100d70 2024-03-12T05:10:54.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.388+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c010a8d0 con 0x7fd4c0100d70 2024-03-12T05:10:54.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.389+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7fd4b803b040 con 0x7fd4c0100d70 2024-03-12T05:10:54.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.389+0000 7fd4af7fe640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 0x7fd4a80a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:10:54.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.389+0000 7fd4be59c640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 0x7fd4a80a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:10:54.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.389+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fd4b80ef620 con 0x7fd4c0100d70 2024-03-12T05:10:54.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.390+0000 7fd4be59c640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 0x7fd4a80a3da0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009bd0 tx=0x7fd4b0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:10:54.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.395+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd4b80b8ab0 con 0x7fd4c0100d70 2024-03-12T05:10:54.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:54 smithi130 ceph-mon[35398]: [12/Mar/2024:05:10:52] ENGINE Bus STARTING 2024-03-12T05:10:54.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:54 smithi130 ceph-mon[35398]: [12/Mar/2024:05:10:53] ENGINE Serving on https://172.21.15.45:7150 2024-03-12T05:10:54.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:54 smithi130 ceph-mon[35398]: [12/Mar/2024:05:10:53] ENGINE Serving on http://172.21.15.45:8765 2024-03-12T05:10:54.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:54 smithi130 ceph-mon[35398]: [12/Mar/2024:05:10:53] ENGINE Bus STARTED 2024-03-12T05:10:54.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:54 smithi130 ceph-mon[35398]: mgrmap e23: x(active, since 1.01907s), standbys: z, y 2024-03-12T05:10:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:54 smithi116 ceph-mon[35486]: [12/Mar/2024:05:10:52] ENGINE Bus STARTING 2024-03-12T05:10:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:54 smithi116 ceph-mon[35486]: [12/Mar/2024:05:10:53] ENGINE Serving on https://172.21.15.45:7150 2024-03-12T05:10:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:54 smithi116 ceph-mon[35486]: [12/Mar/2024:05:10:53] ENGINE Serving on http://172.21.15.45:8765 2024-03-12T05:10:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:54 smithi116 ceph-mon[35486]: [12/Mar/2024:05:10:53] ENGINE Bus STARTED 2024-03-12T05:10:54.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:54 smithi116 ceph-mon[35486]: mgrmap e23: x(active, since 1.01907s), standbys: z, y 2024-03-12T05:10:54.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:54 smithi045 ceph-mon[32174]: [12/Mar/2024:05:10:52] ENGINE Bus STARTING 2024-03-12T05:10:54.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:54 smithi045 ceph-mon[32174]: [12/Mar/2024:05:10:53] ENGINE Serving on https://172.21.15.45:7150 2024-03-12T05:10:54.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:54 smithi045 ceph-mon[32174]: [12/Mar/2024:05:10:53] ENGINE Serving on http://172.21.15.45:8765 2024-03-12T05:10:54.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:54 smithi045 ceph-mon[32174]: [12/Mar/2024:05:10:53] ENGINE Bus STARTED 2024-03-12T05:10:54.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:54 smithi045 ceph-mon[32174]: mgrmap e23: x(active, since 1.01907s), standbys: z, y 2024-03-12T05:10:54.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.643+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd4c002ce10 con 0x7fd4a80a18e0 2024-03-12T05:10:54.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.650+0000 7fd4af7fe640 1 -- 172.21.15.45:0/1206634457 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fd4c002ce10 con 0x7fd4a80a18e0 2024-03-12T05:10:54.647 INFO:teuthology.orchestra.run.smithi045.stdout:/ 1m 6m3h 2024-03-12T05:10:54.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 msgr2=0x7fd4a80a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 0x7fd4a80a3da0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009bd0 tx=0x7fd4b0005f90 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 msgr2=0x7fd4c01a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8037040 tx=0x7fd4b8032980 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 shutdown_connections 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd4a80a18e0 0x7fd4a80a3da0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4c0100d70 0x7fd4c01a07d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4c01003d0 0x7fd4c0199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 --2- 172.21.15.45:0/1206634457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4c00ffa70 0x7fd4c0199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:10:54.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 >> 172.21.15.45:0/1206634457 conn(0x7fd4c00fb610 msgr2=0x7fd4c00fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:10:54.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 shutdown_connections 2024-03-12T05:10:54.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:10:54.652+0000 7fd4c4c76640 1 -- 172.21.15.45:0/1206634457 wait complete. 2024-03-12T05:10:54.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-12T05:10:54.692 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-12T05:10:54.704 INFO:teuthology.run_tasks:Running task exec... 2024-03-12T05:10:54.715 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:10:54.715 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:10:54.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-03-12T05:10:54.771 INFO:teuthology.orchestra.run.smithi045.stdout:total 0 2024-03-12T05:10:54.785 INFO:teuthology.orchestra.run.smithi045.stdout:Filesystem Size Used Avail Use% Mounted on 2024-03-12T05:10:54.786 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-fuse 340G 0 340G 0% /tmp/cephfs 2024-03-12T05:10:54.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-03-12T05:10:54.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-03-12T05:10:54.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:55 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:55 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:55.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:55 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: mgrmap e24: x(active, since 3s), standbys: z, y 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:56 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: mgrmap e24: x(active, since 3s), standbys: z, y 2024-03-12T05:10:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:56.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:56 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:56.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: mgrmap e24: x(active, since 3s), standbys: z, y 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:56.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:56 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:56.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-03-12T05:10:56.805 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-03-12T05:10:56.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-03-12T05:10:56.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-12T05:10:56.938 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/: 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:total 2 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 .. 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 data 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data: 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:10:56.939 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 .. 2024-03-12T05:10:56.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-03-12T05:10:56.993 INFO:teuthology.orchestra.run.smithi045.stdout:[Service] 2024-03-12T05:10:56.993 INFO:teuthology.orchestra.run.smithi045.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-03-12T05:10:56.994 INFO:teuthology.orchestra.run.smithi045.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-03-12T05:10:56.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-03-12T05:10:57.077 INFO:teuthology.orchestra.run.smithi045.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-03-12T05:10:57.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:10:57.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:57.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:57 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:57.495 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:57 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi130:/etc/ceph/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:10:57.619 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:57 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:10:58.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:58.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:58 smithi130 ceph-mon[35398]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:58 smithi116 ceph-mon[35486]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-12T05:10:58.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: Updating smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:10:58.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:58 smithi045 ceph-mon[32174]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-12T05:10:59.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:10:59 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:10:59.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:10:59 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:10:59.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:10:59 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:00.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:00 smithi116 ceph-mon[35486]: pgmap v6: 97 pgs: 97 active+clean; 14 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 3 op/s 2024-03-12T05:11:00.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:00 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:00.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-mon[32174]: pgmap v6: 97 pgs: 97 active+clean; 14 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 3 op/s 2024-03-12T05:11:00.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:00 smithi130 ceph-mon[35398]: pgmap v6: 97 pgs: 97 active+clean; 14 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 3 op/s 2024-03-12T05:11:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:00 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:01.367 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:11:00.929+0000 7f51c124b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:11:01.368 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:11:00.929+0000 7f51c124b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:11:01.368 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:11:00.929+0000 7f51c124b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:11:01.368 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:11:00.929+0000 7f51c124b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:11:01.368 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:11:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:11:00.929+0000 7f51c124b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:11:02.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:02 smithi116 ceph-mon[35486]: pgmap v7: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.8 MiB/s wr, 100 op/s 2024-03-12T05:11:02.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:02 smithi045 ceph-mon[32174]: pgmap v7: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.8 MiB/s wr, 100 op/s 2024-03-12T05:11:02.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:02 smithi130 ceph-mon[35398]: pgmap v7: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 2.8 MiB/s wr, 100 op/s 2024-03-12T05:11:04.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:04 smithi116 ceph-mon[35486]: pgmap v8: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-03-12T05:11:04.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:04 smithi116 ceph-mon[35486]: mgrmap e26: x(active, since 10s), standbys: z, y 2024-03-12T05:11:04.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:04 smithi045 ceph-mon[32174]: pgmap v8: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-03-12T05:11:04.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:04 smithi045 ceph-mon[32174]: mgrmap e26: x(active, since 10s), standbys: z, y 2024-03-12T05:11:04.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:04 smithi130 ceph-mon[35398]: pgmap v8: 97 pgs: 97 active+clean; 20 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 2.2 MiB/s wr, 77 op/s 2024-03-12T05:11:04.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:04 smithi130 ceph-mon[35398]: mgrmap e26: x(active, since 10s), standbys: z, y 2024-03-12T05:11:06.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:06 smithi116 ceph-mon[35486]: pgmap v9: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 4.1 MiB/s wr, 153 op/s 2024-03-12T05:11:06.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:06 smithi045 ceph-mon[32174]: pgmap v9: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 4.1 MiB/s wr, 153 op/s 2024-03-12T05:11:06.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:06 smithi130 ceph-mon[35398]: pgmap v9: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 4.1 MiB/s wr, 153 op/s 2024-03-12T05:11:08.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:08 smithi116 ceph-mon[35486]: pgmap v10: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.7 MiB/s wr, 141 op/s 2024-03-12T05:11:08.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:08 smithi045 ceph-mon[32174]: pgmap v10: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.7 MiB/s wr, 141 op/s 2024-03-12T05:11:08.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:08 smithi130 ceph-mon[35398]: pgmap v10: 97 pgs: 97 active+clean; 37 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.7 MiB/s wr, 141 op/s 2024-03-12T05:11:10.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:10 smithi116 ceph-mon[35486]: pgmap v11: 97 pgs: 97 active+clean; 38 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-03-12T05:11:10.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:10 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:10.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:10 smithi045 ceph-mon[32174]: pgmap v11: 97 pgs: 97 active+clean; 38 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-03-12T05:11:10.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:10 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:10.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:10 smithi130 ceph-mon[35398]: pgmap v11: 97 pgs: 97 active+clean; 38 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-03-12T05:11:10.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:10 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:12.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:12 smithi116 ceph-mon[35486]: pgmap v12: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-12T05:11:12.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:12 smithi045 ceph-mon[32174]: pgmap v12: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-12T05:11:12.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:12 smithi130 ceph-mon[35398]: pgmap v12: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-12T05:11:14.494 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:14 smithi116 ceph-mon[35486]: pgmap v13: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-03-12T05:11:14.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:14 smithi045 ceph-mon[32174]: pgmap v13: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-03-12T05:11:14.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:14 smithi130 ceph-mon[35398]: pgmap v13: 97 pgs: 97 active+clean; 46 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-03-12T05:11:14.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:11:14.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 -- 172.21.15.45:0/1876633638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974100270 msgr2=0x7fb974100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 --2- 172.21.15.45:0/1876633638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974100270 0x7fb974100650 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb964009930 tx=0x7fb96402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 -- 172.21.15.45:0/1876633638 shutdown_connections 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 --2- 172.21.15.45:0/1876633638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb974105340 0x7fb97410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 --2- 172.21.15.45:0/1876633638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100c20 0x7fb974104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 --2- 172.21.15.45:0/1876633638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974100270 0x7fb974100650 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 -- 172.21.15.45:0/1876633638 >> 172.21.15.45:0/1876633638 conn(0x7fb974077230 msgr2=0x7fb974077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.851+0000 7fb97990f640 1 -- 172.21.15.45:0/1876633638 shutdown_connections 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 -- 172.21.15.45:0/1876633638 wait complete. 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 Processor -- start 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 -- start start 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100270 0x7fb97419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9741996e0 0x7fb974199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97419a1b0 con 0x7fb974105340 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97419a320 con 0x7fb974100270 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb97990f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97419a4e0 con 0x7fb9741996e0 2024-03-12T05:11:14.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54724/0 (socket says 172.21.15.45:54724) 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.852+0000 7fb96bfff640 1 -- 172.21.15.45:0/272674867 learned_addr learned my addr 172.21.15.45:0/272674867 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb9737fe640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9741996e0 0x7fb974199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb972ffd640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100270 0x7fb97419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 -- 172.21.15.45:0/272674867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9741996e0 msgr2=0x7fb974199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9741996e0 0x7fb974199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 -- 172.21.15.45:0/272674867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100270 msgr2=0x7fb97419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100270 0x7fb97419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 -- 172.21.15.45:0/272674867 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964009560 con 0x7fb974105340 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb96bfff640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb95c00e9c0 tx=0x7fb95c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb97890d640 1 -- 172.21.15.45:0/272674867 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c009700 con 0x7fb974105340 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb97890d640 1 -- 172.21.15.45:0/272674867 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb95c00cd80 con 0x7fb974105340 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9741a6480 con 0x7fb974105340 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb97890d640 1 -- 172.21.15.45:0/272674867 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c010430 con 0x7fb974105340 2024-03-12T05:11:14.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.853+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9741a6960 con 0x7fb974105340 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.855+0000 7fb97890d640 1 -- 172.21.15.45:0/272674867 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fb95c004f30 con 0x7fb974105340 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97890d640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 0x7fb9600a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 Processor -- start 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 -- start start 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb972ffd640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 0x7fb9600a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97890d640 1 -- 172.21.15.45:0/272674867 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb95c014070 con 0x7fb974105340 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x55687275e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568727677b0 0x556872769ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568726cd280 con 0x55687275ebd0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568726ca7b0 con 0x5568727677b0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568726c5f90 con 0x55687275e2b0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54730/0 (socket says 172.21.15.45:54730) 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb96bfff640 1 -- 172.21.15.45:0/910220829 learned_addr learned my addr 172.21.15.45:0/910220829 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.856+0000 7fb972ffd640 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x55687275e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb9737fe640 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568727677b0 0x556872769ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 -- 172.21.15.45:0/910220829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 msgr2=0x55687275e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x55687275e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 -- 172.21.15.45:0/910220829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568727677b0 msgr2=0x556872769ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568727677b0 0x556872769ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 -- 172.21.15.45:0/910220829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556872684570 con 0x55687275ebd0 2024-03-12T05:11:14.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb972ffd640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 0x7fb9600a4180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb964009900 tx=0x7fb964005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb96bfff640 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb95c097190 tx=0x7fb95c09a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaa1b42bf22aea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb969ffb640 1 -- 172.21.15.45:0/910220829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c09aec0 con 0x55687275ebd0 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb969ffb640 1 -- 172.21.15.45:0/910220829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb95c09a7c0 con 0x55687275ebd0 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.857+0000 7fb969ffb640 1 -- 172.21.15.45:0/910220829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c0a2ad0 con 0x55687275ebd0 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- 172.21.15.45:0/910220829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 msgr2=0x556872767270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb95c097190 tx=0x7fb95c09a590 comp rx=0 tx=0).stop 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- 172.21.15.45:0/910220829 shutdown_connections 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x556872767270 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568727677b0 0x556872769ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/910220829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x55687275e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- 172.21.15.45:0/910220829 >> 172.21.15.45:0/910220829 conn(0x556872758990 msgr2=0x556872758d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- 172.21.15.45:0/910220829 shutdown_connections 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- 172.21.15.45:0/910220829 wait complete. 2024-03-12T05:11:14.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.858+0000 7fb97c5b7740 1 -- start start 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 Processor -- start 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x556872767260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x5568727645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556872399430 con 0x5568727677b0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556872355e00 con 0x55687275ebd0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55687264afd0 con 0x55687275e2b0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb97c5b7740 10 client.0 _ll_register_callbacks cb 0x5568726a0fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb9737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x556872767260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb9737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54738/0 (socket says 172.21.15.45:54738) 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb9737fe640 1 -- 172.21.15.45:0/973919697 learned_addr learned my addr 172.21.15.45:0/973919697 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb96bfff640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x5568727645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 -- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 msgr2=0x5568727645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x5568727645f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 -- 172.21.15.45:0/973919697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 msgr2=0x556872764d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb972ffd640 1 -- 172.21.15.45:0/973919697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9741a6960 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.859+0000 7fb9737fe640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb972ffd640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x556872767260 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb964036040 tx=0x7fb964009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd76e6b51c94caa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640035f0 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568723c57e0 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb96402de40 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb964003750 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 10 client.25111 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568726c4610 con 0x55687275e2b0 2024-03-12T05:11:14.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5568723c57e0 con 0x55687275e2b0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb9640052f0 con 0x55687275e2b0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 10 client.25111 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 10 client.25111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 20 client.25111 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727779d0 0x556872779df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 4 client.25111 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.860+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556872402720 con 0x5568727779d0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.861+0000 7fb96bfff640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727779d0 0x556872779df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.861+0000 7fb952ffd640 10 client.25111 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.861+0000 7fb96bfff640 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727779d0 0x556872779df0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25512edd54734c75 server_cookie=ca1e6a05fd0f178c in_seq=0 out_seq=0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb952ffd640 1 -- 172.21.15.45:0/973919697 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556872402720 con 0x5568727779d0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb952ffd640 10 client.25111 handle_command_reply: tid=0 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 client.25111 shutdown 2024-03-12T05:11:14.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 msgr2=0x556872767260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x556872767260 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb964036040 tx=0x7fb964009260 comp rx=0 tx=0).stop 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 shutdown_connections 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727779d0 0x556872779df0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568727677b0 0x556872764d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x5568727645f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/973919697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55687275e2b0 0x556872767260 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 >> 172.21.15.45:0/973919697 conn(0x556872758990 msgr2=0x55687275a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb952ffd640 0 client.25111 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 shutdown_connections 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 1 -- 172.21.15.45:0/973919697 wait complete. 2024-03-12T05:11:14.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.862+0000 7fb97c5b7740 20 client.25111 trim_cache size 0 max 16384 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 Processor -- start 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 -- start start 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x55687276ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x556872766d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556872765a50 con 0x55687275e2b0 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568727540f0 con 0x55687275ebd0 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55687275ad20 con 0x5568727677b0 2024-03-12T05:11:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54740/0 (socket says 172.21.15.45:54740) 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 -- 172.21.15.45:0/1931059559 learned_addr learned my addr 172.21.15.45:0/1931059559 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb9737fe640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x556872766d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 -- 172.21.15.45:0/1931059559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 msgr2=0x556872766d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x556872766d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 -- 172.21.15.45:0/1931059559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 msgr2=0x55687276ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb96bfff640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x55687276ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x55687276ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb972ffd640 1 -- 172.21.15.45:0/1931059559 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556872660ac0 con 0x55687275e2b0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.879+0000 7fb9737fe640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x556872766d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb972ffd640 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb96402c170 tx=0x7fb96402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9299f5e52c34ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb9697fa640 1 -- 172.21.15.45:0/1931059559 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640057f0 con 0x55687275e2b0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb9697fa640 1 -- 172.21.15.45:0/1931059559 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb964005270 con 0x55687275e2b0 2024-03-12T05:11:14.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb9697fa640 1 -- 172.21.15.45:0/1931059559 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96405dba0 con 0x55687275e2b0 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 -- 172.21.15.45:0/1931059559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 msgr2=0x55687276e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb96402c170 tx=0x7fb96402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 -- 172.21.15.45:0/1931059559 shutdown_connections 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x556872766d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275ebd0 0x55687276ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/1931059559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275e2b0 0x55687276e690 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 -- 172.21.15.45:0/1931059559 >> 172.21.15.45:0/1931059559 conn(0x556872758990 msgr2=0x55687275f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 -- 172.21.15.45:0/1931059559 shutdown_connections 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.880+0000 7fb97c5b7740 1 -- 172.21.15.45:0/1931059559 wait complete. 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 -- start start 2024-03-12T05:11:14.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 Processor -- start 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 0x55687276e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x55687276d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55687276d580 con 0x55687275ebd0 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55687276d6f0 con 0x55687275e2b0 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55687276d8b0 con 0x5568727677b0 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb97c5b7740 10 client.0 _ll_register_callbacks cb 0x7fb95c0963a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb96bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54750/0 (socket says 172.21.15.45:54750) 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb96bfff640 1 -- 172.21.15.45:0/3602445193 learned_addr learned my addr 172.21.15.45:0/3602445193 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:14.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb972ffd640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 0x55687276e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.881+0000 7fb9737fe640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x55687276d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 -- 172.21.15.45:0/3602445193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 msgr2=0x55687276d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x55687276d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 -- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 msgr2=0x55687276e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 0x55687276e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 -- 172.21.15.45:0/3602445193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568726c4610 con 0x55687275ebd0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb972ffd640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 0x55687276e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb96bfff640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb95c09a4c0 tx=0x7fb95c005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b56b45fe568b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c09aec0 con 0x55687275ebd0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb95c096e30 con 0x55687275ebd0 2024-03-12T05:11:14.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb95c09a7c0 con 0x55687275ebd0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb95c096ff0 con 0x55687275ebd0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb97c5b7740 10 client.15624 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568726866a0 con 0x55687275ebd0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb95c09a7c0 con 0x55687275ebd0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.882+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb95c01f060 con 0x55687275ebd0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 10 client.15624 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 10 client.15624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 20 client.15624 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727cf1a0 0x5568727d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 4 client.15624 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556872451890 con 0x5568727cf1a0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb972ffd640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727cf1a0 0x5568727d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:14.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb952ffd640 10 client.15624 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.883+0000 7fb972ffd640 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727cf1a0 0x5568727d15c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f56b0390b65d8 server_cookie=899d1f432e6cab11 in_seq=0 out_seq=0 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb952ffd640 1 -- 172.21.15.45:0/3602445193 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556872451890 con 0x5568727cf1a0 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb952ffd640 10 client.15624 handle_command_reply: tid=0 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 client.15624 shutdown 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 msgr2=0x55687276cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb95c09a4c0 tx=0x7fb95c005660 comp rx=0 tx=0).stop 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 shutdown_connections 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568727cf1a0 0x5568727d15c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55687275ebd0 0x55687276cb00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568727677b0 0x55687276d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 --2- 172.21.15.45:0/3602445193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55687275e2b0 0x55687276e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb952ffd640 0 client.15624 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:14.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 >> 172.21.15.45:0/3602445193 conn(0x556872758990 msgr2=0x556872763e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 shutdown_connections 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 1 -- 172.21.15.45:0/3602445193 wait complete. 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97c5b7740 20 client.15624 trim_cache size 0 max 16384 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:11:14.890 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 msgr2=0x7fb9600a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 0x7fb9600a4180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb964009900 tx=0x7fb964005a20 comp rx=0 tx=0).stop 2024-03-12T05:11:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 msgr2=0x7fb97419fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb95c00e9c0 tx=0x7fb95c00ee80 comp rx=0 tx=0).stop 2024-03-12T05:11:14.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 shutdown_connections 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb9600a1cc0 0x7fb9600a4180 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb974105340 0x7fb97419fb60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9741996e0 0x7fb974199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.884+0000 7fb97990f640 1 --2- 172.21.15.45:0/272674867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb974100270 0x7fb97419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.885+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 >> 172.21.15.45:0/272674867 conn(0x7fb974077230 msgr2=0x7fb9740ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.885+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 shutdown_connections 2024-03-12T05:11:14.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:14.885+0000 7fb97990f640 1 -- 172.21.15.45:0/272674867 wait complete. 2024-03-12T05:11:14.893 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:11:14.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:11:15.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 -- 172.21.15.45:0/2081162810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 msgr2=0x7f97d410b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 --2- 172.21.15.45:0/2081162810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d410b1f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009900 tx=0x7f97c402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:15.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 -- 172.21.15.45:0/2081162810 shutdown_connections 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 --2- 172.21.15.45:0/2081162810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d410b1f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 --2- 172.21.15.45:0/2081162810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d410b7c0 0x7f97d410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 --2- 172.21.15.45:0/2081162810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d41023f0 0x7f97d41027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 -- 172.21.15.45:0/2081162810 >> 172.21.15.45:0/2081162810 conn(0x7f97d4077180 msgr2=0x7f97d4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 -- 172.21.15.45:0/2081162810 shutdown_connections 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.010+0000 7f97dc18c640 1 -- 172.21.15.45:0/2081162810 wait complete. 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 Processor -- start 2024-03-12T05:11:15.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 -- start start 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 0x7f97d4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 0x7f97d419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419c600 con 0x7f97d4102d10 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419c770 con 0x7f97d41023f0 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97dc18c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d419c930 con 0x7f97d410b7c0 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97d9727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97d9727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54772/0 (socket says 172.21.15.45:54772) 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97d9727640 1 -- 172.21.15.45:0/550380497 learned_addr learned my addr 172.21.15.45:0/550380497 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97da729640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 0x7f97d419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.011+0000 7f97d9f28640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 0x7f97d4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 -- 172.21.15.45:0/550380497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 msgr2=0x7f97d419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 0x7f97d419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 -- 172.21.15.45:0/550380497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 msgr2=0x7f97d4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 0x7f97d4197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 -- 172.21.15.45:0/550380497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c4009560 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97da729640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 0x7f97d419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9f28640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 0x7f97d4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97d9727640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004490 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f97c4003aa0 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d419cbb0 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4005410 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.012+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d41a4490 con 0x7f97d4102d10 2024-03-12T05:11:15.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.013+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d4067f70 con 0x7f97d4102d10 2024-03-12T05:11:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.014+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f97c4003c10 con 0x7f97d4102d10 2024-03-12T05:11:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.015+0000 7f97caffd640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 0x7f97c00a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.015+0000 7f97d9f28640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 0x7f97c00a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.015+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f97c402de40 con 0x7f97d4102d10 2024-03-12T05:11:15.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.015+0000 7f97d9f28640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 0x7f97c00a4180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009b50 tx=0x7f97d0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.019+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97c40b9390 con 0x7f97d4102d10 2024-03-12T05:11:15.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.287+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97d4198a90 con 0x7f97d4102d10 2024-03-12T05:11:15.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.288+0000 7f97caffd640 1 -- 172.21.15.45:0/550380497 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f97c4043090 con 0x7f97d4102d10 2024-03-12T05:11:15.285 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:15.285 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:11:08.409834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:11:15.285 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 28 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.289+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 msgr2=0x7f97c00a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.289+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 0x7f97c00a4180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009b50 tx=0x7f97d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.289+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 msgr2=0x7f97d4197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.289+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f97c4000c00 tx=0x7f97c4005e60 comp rx=0 tx=0).stop 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 shutdown_connections 2024-03-12T05:11:15.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f97c00a1cc0 0x7f97c00a4180 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97d4102d10 0x7f97d4197c80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d410b7c0 0x7f97d419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 --2- 172.21.15.45:0/550380497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d41023f0 0x7f97d4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 >> 172.21.15.45:0/550380497 conn(0x7f97d4077180 msgr2=0x7f97d40fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 shutdown_connections 2024-03-12T05:11:15.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.290+0000 7f97dc18c640 1 -- 172.21.15.45:0/550380497 wait complete. 2024-03-12T05:11:15.299 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:15.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:11:15.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 -- 172.21.15.45:0/3207298296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e680ff890 msgr2=0x7f6e680ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/3207298296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e680ff890 0x7f6e680ffc70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009930 tx=0x7f6e5c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 -- 172.21.15.45:0/3207298296 shutdown_connections 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/3207298296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/3207298296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e68100240 0x7f6e68105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/3207298296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e680ff890 0x7f6e680ffc70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 -- 172.21.15.45:0/3207298296 >> 172.21.15.45:0/3207298296 conn(0x7f6e680765a0 msgr2=0x7f6e680fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 -- 172.21.15.45:0/3207298296 shutdown_connections 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 -- 172.21.15.45:0/3207298296 wait complete. 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.417+0000 7f6e6f89d640 1 Processor -- start 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 -- start start 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 0x7f6e6819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 0x7f6e68197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68197f90 con 0x7f6e681975a0 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68198100 con 0x7f6e68106450 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6f89d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e681982c0 con 0x7f6e680ff890 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 0x7f6e68197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 0x7f6e6819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 0x7f6e6819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57612/0 (socket says 172.21.15.45:57612) 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6d639640 1 -- 172.21.15.45:0/552756250 learned_addr learned my addr 172.21.15.45:0/552756250 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 0x7f6e68197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54788/0 (socket says 172.21.15.45:54788) 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 -- 172.21.15.45:0/552756250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 msgr2=0x7f6e6819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 0x7f6e6819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 -- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 msgr2=0x7f6e68197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 0x7f6e68197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 -- 172.21.15.45:0/552756250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c009560 con 0x7f6e68106450 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.418+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6819d930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400e9c0 tx=0x7f6e6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.419+0000 7f6e5a7fc640 1 -- 172.21.15.45:0/552756250 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6400ccc0 con 0x7f6e68106450 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.419+0000 7f6e5a7fc640 1 -- 172.21.15.45:0/552756250 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e640044d0 con 0x7f6e68106450 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.419+0000 7f6e5a7fc640 1 -- 172.21.15.45:0/552756250 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e64005690 con 0x7f6e68106450 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.419+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e681a42b0 con 0x7f6e68106450 2024-03-12T05:11:15.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.419+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e681a4770 con 0x7f6e68106450 2024-03-12T05:11:15.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e5a7fc640 1 -- 172.21.15.45:0/552756250 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f6e64004a40 con 0x7f6e68106450 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 Processor -- start 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 -- start start 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97a92960 con 0x557a97ae0750 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97a46310 con 0x557a97ad7b70 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e72475740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97a43840 con 0x557a97ad7250 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:47236/0 (socket says 172.21.15.45:47236) 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57624/0 (socket says 172.21.15.45:57624) 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e6d639640 1 -- 172.21.15.45:0/49276574 learned_addr learned my addr 172.21.15.45:0/49276574 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.422+0000 7f6e5a7fc640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 0x7f6e540a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 -- 172.21.15.45:0/49276574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 msgr2=0x557a97ad7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 -- 172.21.15.45:0/49276574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 msgr2=0x557a97ae2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 -- 172.21.15.45:0/49276574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a97a01060 con 0x557a97ad7b70 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 0x7f6e540a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e5a7fc640 1 -- 172.21.15.45:0/552756250 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e64014070 con 0x7f6e68106450 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e6d639640 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009430 tx=0x7f6e5c005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d8219fe1aea2595 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e537fe640 1 -- 172.21.15.45:0/49276574 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c004880 con 0x557a97ad7b70 2024-03-12T05:11:15.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e537fe640 1 -- 172.21.15.45:0/49276574 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e5c02de90 con 0x557a97ad7b70 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.423+0000 7f6e537fe640 1 -- 172.21.15.45:0/49276574 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c003850 con 0x557a97ad7b70 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- 172.21.15.45:0/49276574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 msgr2=0x557a97ae0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 0x7f6e540a4180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6e60005c40 tx=0x7f6e60015040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009430 tx=0x7f6e5c005de0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- 172.21.15.45:0/49276574 shutdown_connections 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae0210 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 --2- 172.21.15.45:0/49276574 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- 172.21.15.45:0/49276574 >> 172.21.15.45:0/49276574 conn(0x557a97ad1980 msgr2=0x557a97ad1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- 172.21.15.45:0/49276574 shutdown_connections 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- 172.21.15.45:0/49276574 wait complete. 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 -- start start 2024-03-12T05:11:15.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.424+0000 7f6e72475740 1 Processor -- start 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97addc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ade1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97a3f020 con 0x557a97ad7b70 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97712430 con 0x557a97ae0750 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a976cee00 con 0x557a97ad7250 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e72475740 10 client.0 _ll_register_callbacks cb 0x557a979e6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97addc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ade1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57638/0 (socket says 172.21.15.45:57638) 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 -- 172.21.15.45:0/3904838407 learned_addr learned my addr 172.21.15.45:0/3904838407 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 -- 172.21.15.45:0/3904838407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 msgr2=0x557a97ade1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ade1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 -- 172.21.15.45:0/3904838407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 msgr2=0x557a97addc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97addc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 -- 172.21.15.45:0/3904838407 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6810e830 con 0x557a97ad7250 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97addc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e6d639640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009900 tx=0x7f6e5c005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=164535005a20ec24 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.425+0000 7f6e50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c004fc0 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a9773e7e0 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e5c04d070 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e5c0044f0 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 10 client.25132 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557a97a3d6a0 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557a9773e7e0 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f6e5c03c050 con 0x557a97ad7250 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 10 client.25132 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 10 client.25132 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 20 client.25132 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97af0700 0x557a97af2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 4 client.25132 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:15.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.426+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a9777b720 con 0x557a97af0700 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.427+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97af0700 0x557a97af2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.427+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97af0700 0x557a97af2b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9255ee9445ef5613 server_cookie=568a8fe76bb26ec9 in_seq=0 out_seq=0 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.427+0000 7f6e50ff9640 10 client.25132 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e50ff9640 1 -- 172.21.15.45:0/3904838407 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557a9777b720 con 0x557a97af0700 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e50ff9640 10 client.25132 handle_command_reply: tid=0 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 client.25132 shutdown 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 msgr2=0x557a97add740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009900 tx=0x7f6e5c005b50 comp rx=0 tx=0).stop 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 shutdown_connections 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97af0700 0x557a97af2b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97addc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ade1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 --2- 172.21.15.45:0/3904838407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97add740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e50ff9640 0 client.25132 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 >> 172.21.15.45:0/3904838407 conn(0x557a97ad1980 msgr2=0x557a97ad36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 shutdown_connections 2024-03-12T05:11:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 1 -- 172.21.15.45:0/3904838407 wait complete. 2024-03-12T05:11:15.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.428+0000 7f6e72475740 20 client.25132 trim_cache size 0 max 16384 2024-03-12T05:11:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.443+0000 7f6e72475740 1 Processor -- start 2024-03-12T05:11:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 -- start start 2024-03-12T05:11:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ae7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a979c4130 con 0x557a97ad7b70 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97ad2770 con 0x557a97ae0750 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e72475740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97ad6c90 con 0x557a97ad7250 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ae7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54826/0 (socket says 172.21.15.45:54826) 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 -- 172.21.15.45:0/2854256493 learned_addr learned my addr 172.21.15.45:0/2854256493 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6d639640 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 -- 172.21.15.45:0/2854256493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 msgr2=0x557a97ad0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 -- 172.21.15.45:0/2854256493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 msgr2=0x557a97ae7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ae7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 -- 172.21.15.45:0/2854256493 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a979d9c20 con 0x557a97ad7b70 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ae7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.444+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6e6409a110 tx=0x7f6e64085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ea7914e5be1fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e52ffd640 1 -- 172.21.15.45:0/2854256493 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6409a1e0 con 0x557a97ad7b70 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e52ffd640 1 -- 172.21.15.45:0/2854256493 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e640d0e90 con 0x557a97ad7b70 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e52ffd640 1 -- 172.21.15.45:0/2854256493 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6409e560 con 0x557a97ad7b70 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 -- 172.21.15.45:0/2854256493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 msgr2=0x557a97ae75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6e6409a110 tx=0x7f6e64085040 comp rx=0 tx=0).stop 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 -- 172.21.15.45:0/2854256493 shutdown_connections 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ad7b70 0x557a97ae75a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ae0750 0x557a97ae7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 --2- 172.21.15.45:0/2854256493 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ad0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 -- 172.21.15.45:0/2854256493 >> 172.21.15.45:0/2854256493 conn(0x557a97ad1980 msgr2=0x557a97ad02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 -- 172.21.15.45:0/2854256493 shutdown_connections 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.445+0000 7f6e72475740 1 -- 172.21.15.45:0/2854256493 wait complete. 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 -- start start 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 Processor -- start 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ae6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97b46c10 con 0x557a97ae0750 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97b46d50 con 0x557a97ad7b70 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a97b46f10 con 0x557a97ad7250 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e72475740 10 client.0 _ll_register_callbacks cb 0x7f6e64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e6d639640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ae6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e6de3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54828/0 (socket says 172.21.15.45:54828) 2024-03-12T05:11:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e6ce38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.446+0000 7f6e6de3a640 1 -- 172.21.15.45:0/251428934 learned_addr learned my addr 172.21.15.45:0/251428934 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 -- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 msgr2=0x557a97ae6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ae6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 -- 172.21.15.45:0/251428934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 msgr2=0x557a97ae6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 -- 172.21.15.45:0/251428934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6000e960 con 0x557a97ae0750 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6ce38640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6d639640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ae6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e6de3a640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000f010 tx=0x7f6e60009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29744b927e4ccc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60031070 con 0x557a97ae0750 2024-03-12T05:11:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e60004b20 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e5c02de90 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e6001ee70 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 10 client.15648 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f6e681a4770 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e5c02de90 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f6e60034020 con 0x557a97ae0750 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 10 client.15648 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 10 client.15648 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 20 client.15648 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.447+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97b48840 0x557a97b4ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 4 client.15648 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557a977ca890 con 0x557a97b48840 2024-03-12T05:11:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e6d639640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97b48840 0x557a97b4ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e6d639640 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97b48840 0x557a97b4ac60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=15df58a81cedf0a6 server_cookie=657379d06ab89814 in_seq=0 out_seq=0 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e50ff9640 10 client.15648 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e50ff9640 1 -- 172.21.15.45:0/251428934 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557a977ca890 con 0x557a97b48840 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e50ff9640 10 client.15648 handle_command_reply: tid=0 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 client.15648 shutdown 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 msgr2=0x557a97ae6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000f010 tx=0x7f6e60009500 comp rx=0 tx=0).stop 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 shutdown_connections 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557a97b48840 0x557a97b4ac60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557a97ae0750 0x557a97ae6ad0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557a97ad7b70 0x557a97ae6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e50ff9640 0 client.15648 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.448+0000 7f6e72475740 1 --2- 172.21.15.45:0/251428934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557a97ad7250 0x557a97ae6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 >> 172.21.15.45:0/251428934 conn(0x557a97ad1980 msgr2=0x557a97ad0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 shutdown_connections 2024-03-12T05:11:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e72475740 1 -- 172.21.15.45:0/251428934 wait complete. 2024-03-12T05:11:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e72475740 20 client.15648 trim_cache size 0 max 16384 2024-03-12T05:11:15.453 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 msgr2=0x7f6e540a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 0x7f6e540a4180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6e60005c40 tx=0x7f6e60015040 comp rx=0 tx=0).stop 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 msgr2=0x7f6e6819d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6819d930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400e9c0 tx=0x7f6e6400ee80 comp rx=0 tx=0).stop 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 shutdown_connections 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e540a1cc0 0x7f6e540a4180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e681975a0 0x7f6e68197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e68106450 0x7f6e6819d930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 --2- 172.21.15.45:0/552756250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e680ff890 0x7f6e6819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 >> 172.21.15.45:0/552756250 conn(0x7f6e680765a0 msgr2=0x7f6e68104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 shutdown_connections 2024-03-12T05:11:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.449+0000 7f6e6f89d640 1 -- 172.21.15.45:0/552756250 wait complete. 2024-03-12T05:11:15.458 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:15.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:11:15.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/460554100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74100260 msgr2=0x7f0a74100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/460554100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74100260 0x7f0a74100640 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009930 tx=0x7f0a6402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/460554100 shutdown_connections 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/460554100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 0x7f0a7410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/460554100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100c10 0x7f0a74104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/460554100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74100260 0x7f0a74100640 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/460554100 >> 172.21.15.45:0/460554100 conn(0x7f0a74077180 msgr2=0x7f0a74077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/460554100 shutdown_connections 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.573+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/460554100 wait complete. 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 Processor -- start 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 -- start start 2024-03-12T05:11:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100260 0x7f0a7419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 0x7f0a7419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419a220 con 0x7f0a74199750 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419a390 con 0x7f0a74105330 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a7a8c9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419a550 con 0x7f0a74100260 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a78e66640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a78e66640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54840/0 (socket says 172.21.15.45:54840) 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a78e66640 1 -- 172.21.15.45:0/1009990455 learned_addr learned my addr 172.21.15.45:0/1009990455 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a737fe640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 0x7f0a7419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.574+0000 7f0a73fff640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100260 0x7f0a7419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 -- 172.21.15.45:0/1009990455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100260 msgr2=0x7f0a7419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100260 0x7f0a7419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 -- 172.21.15.45:0/1009990455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 msgr2=0x7f0a7419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 0x7f0a7419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 -- 172.21.15.45:0/1009990455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a64009560 con 0x7f0a74199750 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a78e66640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009fd0 tx=0x7f0a6c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a717fa640 1 -- 172.21.15.45:0/1009990455 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00bea0 con 0x7f0a74199750 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a741a64f0 con 0x7f0a74199750 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a717fa640 1 -- 172.21.15.45:0/1009990455 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a6c009940 con 0x7f0a74199750 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a717fa640 1 -- 172.21.15.45:0/1009990455 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0106c0 con 0x7f0a74199750 2024-03-12T05:11:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.575+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a741a69d0 con 0x7f0a74199750 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.577+0000 7f0a717fa640 1 -- 172.21.15.45:0/1009990455 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f0a6c003970 con 0x7f0a74199750 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 Processor -- start 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 -- start start 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2c98960 con 0x5595d2cdd250 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2c4c310 con 0x5595d2cddb70 2024-03-12T05:11:15.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a7d570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2c49840 con 0x5595d2ce6750 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a717fa640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 0x7f0a5c0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54852/0 (socket says 172.21.15.45:54852) 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a737fe640 1 -- 172.21.15.45:0/4073819112 learned_addr learned my addr 172.21.15.45:0/4073819112 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a78e66640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 0x7f0a5c0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a73fff640 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a737fe640 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.578+0000 7f0a717fa640 1 -- 172.21.15.45:0/1009990455 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0a6c0cd6a0 con 0x7f0a74199750 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 -- 172.21.15.45:0/4073819112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 msgr2=0x5595d2ce8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 -- 172.21.15.45:0/4073819112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 msgr2=0x5595d2ce6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 -- 172.21.15.45:0/4073819112 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595d2c07060 con 0x5595d2cdd250 2024-03-12T05:11:15.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a737fe640 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0a680124f0 tx=0x7f0a6800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c697c1f890239787 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a78e66640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 0x7f0a5c0a4180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0cacd0 tx=0x7f0a6c095d60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a5a7fc640 1 -- 172.21.15.45:0/4073819112 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68004020 con 0x5595d2cdd250 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a5a7fc640 1 -- 172.21.15.45:0/4073819112 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a680047a0 con 0x5595d2cdd250 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.579+0000 7f0a5a7fc640 1 -- 172.21.15.45:0/4073819112 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005240 con 0x5595d2cdd250 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- 172.21.15.45:0/4073819112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 msgr2=0x5595d2cdd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0a680124f0 tx=0x7f0a6800b650 comp rx=0 tx=0).stop 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- 172.21.15.45:0/4073819112 shutdown_connections 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 --2- 172.21.15.45:0/4073819112 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2cdd630 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- 172.21.15.45:0/4073819112 >> 172.21.15.45:0/4073819112 conn(0x5595d2cd7980 msgr2=0x5595d2cd7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- 172.21.15.45:0/4073819112 shutdown_connections 2024-03-12T05:11:15.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- 172.21.15.45:0/4073819112 wait complete. 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 -- start start 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.580+0000 7f0a7d570740 1 Processor -- start 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cdd250 0x5595d2ce3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2c45020 con 0x5595d2cddb70 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2918430 con 0x5595d2cdd250 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d28d4e00 con 0x5595d2ce6750 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a7d570740 10 client.0 _ll_register_callbacks cb 0x5595d2beccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54858/0 (socket says 172.21.15.45:54858) 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 -- 172.21.15.45:0/2953618109 learned_addr learned my addr 172.21.15.45:0/2953618109 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cdd250 0x5595d2ce3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 -- 172.21.15.45:0/2953618109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 msgr2=0x5595d2ce4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 -- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cdd250 msgr2=0x5595d2ce3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cdd250 0x5595d2ce3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 -- 172.21.15.45:0/2953618109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6800bd30 con 0x5595d2cddb70 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009930 tx=0x7f0a6402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd775b277828f32a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.581+0000 7f0a537fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6402cbc0 con 0x5595d2cddb70 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0a64004260 con 0x5595d2cddb70 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595d29447e0 con 0x5595d2cddb70 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a64004440 con 0x5595d2cddb70 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 10 client.15660 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5595d2c436a0 con 0x5595d2cddb70 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5595d29447e0 con 0x5595d2cddb70 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0a6403c040 con 0x5595d2cddb70 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 10 client.15660 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 10 client.15660 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 20 client.15660 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2cf6a70 0x5595d2cf8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 4 client.15660 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595d2981720 con 0x5595d2cf6a70 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.582+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2cf6a70 0x5595d2cf8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.583+0000 7f0a537fe640 10 client.15660 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.583+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2cf6a70 0x5595d2cf8e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=17f9345d73d555c5 server_cookie=a604fa038962c6ff in_seq=0 out_seq=0 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.583+0000 7f0a537fe640 1 -- 172.21.15.45:0/2953618109 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5595d2981720 con 0x5595d2cf6a70 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.583+0000 7f0a537fe640 10 client.15660 handle_command_reply: tid=0 2024-03-12T05:11:15.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.583+0000 7f0a7d570740 1 client.15660 shutdown 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 msgr2=0x5595d2ce3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009930 tx=0x7f0a6402ca10 comp rx=0 tx=0).stop 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 shutdown_connections 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2cf6a70 0x5595d2cf8e90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cddb70 0x5595d2ce3cc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2ce6750 0x5595d2ce4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2953618109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cdd250 0x5595d2ce3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 >> 172.21.15.45:0/2953618109 conn(0x5595d2cd7980 msgr2=0x5595d2cd96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a537fe640 0 client.15660 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 shutdown_connections 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 1 -- 172.21.15.45:0/2953618109 wait complete. 2024-03-12T05:11:15.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.584+0000 7f0a7d570740 20 client.15660 trim_cache size 0 max 16384 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 Processor -- start 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 -- start start 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cdd250 0x5595d2ce5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2bca130 con 0x5595d2ce6750 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2cf3660 con 0x5595d2cddb70 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a7d570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2cdc620 con 0x5595d2cdd250 2024-03-12T05:11:15.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54872/0 (socket says 172.21.15.45:54872) 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 -- 172.21.15.45:0/3441970729 learned_addr learned my addr 172.21.15.45:0/3441970729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a73fff640 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cdd250 0x5595d2ce5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 -- 172.21.15.45:0/3441970729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cdd250 msgr2=0x5595d2ce5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cdd250 0x5595d2ce5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 -- 172.21.15.45:0/3441970729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 msgr2=0x5595d2ce4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 -- 172.21.15.45:0/3441970729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595d2bdfc20 con 0x5595d2ce6750 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.597+0000 7f0a737fe640 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0a68005fd0 tx=0x7f0a6800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8bcc1877d9b125a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a59ffb640 1 -- 172.21.15.45:0/3441970729 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68022040 con 0x5595d2ce6750 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a59ffb640 1 -- 172.21.15.45:0/3441970729 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a6802f8e0 con 0x5595d2ce6750 2024-03-12T05:11:15.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a59ffb640 1 -- 172.21.15.45:0/3441970729 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68003ab0 con 0x5595d2ce6750 2024-03-12T05:11:15.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 -- 172.21.15.45:0/3441970729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 msgr2=0x5595d2cdea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0a68005fd0 tx=0x7f0a6800bf50 comp rx=0 tx=0).stop 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 -- 172.21.15.45:0/3441970729 shutdown_connections 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2ce6750 0x5595d2cdea90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2cddb70 0x5595d2ce4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 --2- 172.21.15.45:0/3441970729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cdd250 0x5595d2ce5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 -- 172.21.15.45:0/3441970729 >> 172.21.15.45:0/3441970729 conn(0x5595d2cd7980 msgr2=0x5595d2cfd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 -- 172.21.15.45:0/3441970729 shutdown_connections 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.598+0000 7f0a7d570740 1 -- 172.21.15.45:0/3441970729 wait complete. 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 -- start start 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 Processor -- start 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cddb70 0x5595d2ceba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2ce6750 0x5595d2cebf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2cedf20 con 0x5595d2cdd250 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2cec4c0 con 0x5595d2ce6750 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d2cec680 con 0x5595d2cddb70 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a7d570740 10 client.0 _ll_register_callbacks cb 0x7f0a6800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a737fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54886/0 (socket says 172.21.15.45:54886) 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a737fe640 1 -- 172.21.15.45:0/2872986446 learned_addr learned my addr 172.21.15.45:0/2872986446 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cddb70 0x5595d2ceba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.599+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2ce6750 0x5595d2cebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 -- 172.21.15.45:0/2872986446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cddb70 msgr2=0x5595d2ceba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cddb70 0x5595d2ceba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 -- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2ce6750 msgr2=0x5595d2cebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2ce6750 0x5595d2cebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 -- 172.21.15.45:0/2872986446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595d2c07060 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a737fe640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800b690 tx=0x7f0a6802f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea48d20a7c8bd8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68010460 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0a68000e40 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a680047a0 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a680105c0 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a7d570740 10 client.15666 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5595d2c058c0 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0a680047a0 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.600+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0a6802fd20 con 0x5595d2cdd250 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 10 client.15666 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 10 client.15666 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:11:15.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 20 client.15666 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2d4dff0 0x5595d2d503e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 4 client.15666 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5595d29d0890 con 0x5595d2d4dff0 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2d4dff0 0x5595d2d503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a537fe640 10 client.15666 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a73fff640 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2d4dff0 0x5595d2d503e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b9ca15d71d9a628f server_cookie=2dd8108250e83f8c in_seq=0 out_seq=0 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a537fe640 1 -- 172.21.15.45:0/2872986446 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5595d29d0890 con 0x5595d2d4dff0 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a537fe640 10 client.15666 handle_command_reply: tid=0 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 client.15666 shutdown 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 msgr2=0x5595d2ced650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800b690 tx=0x7f0a6802f5f0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 shutdown_connections 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5595d2d4dff0 0x5595d2d503e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595d2ce6750 0x5595d2cebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a537fe640 0 client.15666 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595d2cddb70 0x5595d2ceba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.601+0000 7f0a7d570740 1 --2- 172.21.15.45:0/2872986446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5595d2cdd250 0x5595d2ced650 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 >> 172.21.15.45:0/2872986446 conn(0x5595d2cd7980 msgr2=0x5595d2ce1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 shutdown_connections 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7d570740 1 -- 172.21.15.45:0/2872986446 wait complete. 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7d570740 20 client.15666 trim_cache size 0 max 16384 2024-03-12T05:11:15.605 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 msgr2=0x7f0a5c0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 0x7f0a5c0a4180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0cacd0 tx=0x7f0a6c095d60 comp rx=0 tx=0).stop 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 msgr2=0x7f0a74199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009fd0 tx=0x7f0a6c00b260 comp rx=0 tx=0).stop 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 shutdown_connections 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0a5c0a1cc0 0x7f0a5c0a4180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a74199750 0x7f0a74199bd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a74105330 0x7f0a7419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 --2- 172.21.15.45:0/1009990455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a74100260 0x7f0a7419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 >> 172.21.15.45:0/1009990455 conn(0x7f0a74077180 msgr2=0x7f0a740fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 shutdown_connections 2024-03-12T05:11:15.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.602+0000 7f0a7a8c9640 1 -- 172.21.15.45:0/1009990455 wait complete. 2024-03-12T05:11:15.610 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:15.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:11:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 -- 172.21.15.45:0/3157730872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc0ff890 msgr2=0x7f95cc0ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/3157730872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc0ff890 0x7f95cc0ffc70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009930 tx=0x7f95c802bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 -- 172.21.15.45:0/3157730872 shutdown_connections 2024-03-12T05:11:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/3157730872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc106450 0x7f95cc10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/3157730872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc100240 0x7f95cc105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/3157730872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc0ff890 0x7f95cc0ffc70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 -- 172.21.15.45:0/3157730872 >> 172.21.15.45:0/3157730872 conn(0x7f95cc0765a0 msgr2=0x7f95cc0fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 -- 172.21.15.45:0/3157730872 shutdown_connections 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.753+0000 7f95d3f0c640 1 -- 172.21.15.45:0/3157730872 wait complete. 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 Processor -- start 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 -- start start 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 0x7f95cc197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc19bfe0 0x7f95cc19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc19c9d0 con 0x7f95cc106450 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc19cb40 con 0x7f95cc19bfe0 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d3f0c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc19ccb0 con 0x7f95cc0ff890 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d14a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 0x7f95cc197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57716/0 (socket says 172.21.15.45:57716) 2024-03-12T05:11:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d14a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 0x7f95cc197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54898/0 (socket says 172.21.15.45:54898) 2024-03-12T05:11:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d1ca8640 1 -- 172.21.15.45:0/926752122 learned_addr learned my addr 172.21.15.45:0/926752122 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.754+0000 7f95d24a9640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc19bfe0 0x7f95cc19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 -- 172.21.15.45:0/926752122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc19bfe0 msgr2=0x7f95cc19c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc19bfe0 0x7f95cc19c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 -- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 msgr2=0x7f95cc197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 0x7f95cc197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 -- 172.21.15.45:0/926752122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c8009560 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009230 tx=0x7f95c8005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95c2ffd640 1 -- 172.21.15.45:0/926752122 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8004350 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc100730 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95c2ffd640 1 -- 172.21.15.45:0/926752122 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95c802de90 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc100c90 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.755+0000 7f95c2ffd640 1 -- 172.21.15.45:0/926752122 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8005220 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.758+0000 7f95c2ffd640 1 -- 172.21.15.45:0/926752122 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f95c8005380 con 0x7f95cc0ff890 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.758+0000 7f95d6bbe740 1 Processor -- start 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 -- start start 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da59b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77da09960 con 0x55e77da4eb70 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d9bd310 con 0x55e77da4e250 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d9ba840 con 0x55e77da57750 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d24a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d24a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:47332/0 (socket says 172.21.15.45:47332) 2024-03-12T05:11:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54908/0 (socket says 172.21.15.45:54908) 2024-03-12T05:11:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d24a9640 1 -- 172.21.15.45:0/4268486018 learned_addr learned my addr 172.21.15.45:0/4268486018 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da59b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95c2ffd640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 0x7f95b80a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4268486018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 msgr2=0x55e77da59b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da59b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4268486018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 msgr2=0x55e77da4e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.759+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4268486018 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e77d978060 con 0x55e77da4eb70 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95c2ffd640 1 -- 172.21.15.45:0/926752122 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95c80ef870 con 0x7f95cc0ff890 2024-03-12T05:11:15.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d24a9640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da59b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d24a9640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 0x7f95b80a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f95c80ecf50 tx=0x7f95c8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0aae7b7261fd64 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95d24a9640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 0x7f95b80a4180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95c40107d0 tx=0x7f95c4004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95b7fff640 1 -- 172.21.15.45:0/4268486018 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80bc130 con 0x55e77da4eb70 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95b7fff640 1 -- 172.21.15.45:0/4268486018 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95c80f2e90 con 0x55e77da4eb70 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.760+0000 7f95b7fff640 1 -- 172.21.15.45:0/4268486018 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80c04b0 con 0x55e77da4eb70 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4268486018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 msgr2=0x55e77da57210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f95c80ecf50 tx=0x7f95c8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4268486018 shutdown_connections 2024-03-12T05:11:15.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4eb70 0x55e77da57210 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da59b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4268486018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4e250 0x55e77da4e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.761+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4268486018 >> 172.21.15.45:0/4268486018 conn(0x55e77da48980 msgr2=0x55e77da48d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4268486018 shutdown_connections 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4268486018 wait complete. 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 -- start start 2024-03-12T05:11:15.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 Processor -- start 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54916/0 (socket says 172.21.15.45:54916) 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57740/0 (socket says 172.21.15.45:57740) 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.762+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 0x55e77da550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d9b6020 con 0x55e77da4e250 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d689430 con 0x55e77da57750 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d645e00 con 0x55e77da4eb70 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d1ca8640 1 -- 172.21.15.45:0/3396899443 learned_addr learned my addr 172.21.15.45:0/3396899443 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d6bbe740 10 client.0 _ll_register_callbacks cb 0x55e77d95dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d24a9640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 0x55e77da550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 -- 172.21.15.45:0/3396899443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 msgr2=0x55e77da550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 0x55e77da550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 -- 172.21.15.45:0/3396899443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 msgr2=0x55e77da54650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d14a7640 1 -- 172.21.15.45:0/3396899443 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc100c90 con 0x55e77da4eb70 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.763+0000 7f95d24a9640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 0x55e77da550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d14a7640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0fed20 tx=0x7f95bc009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7d406ff82e2844a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc015050 con 0x55e77da4eb70 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95bc00b7f0 con 0x55e77da4eb70 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e77d6b57e0 con 0x55e77da4eb70 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95bc004500 con 0x55e77da4eb70 2024-03-12T05:11:15.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 10 client.25168 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e77d9b46a0 con 0x55e77da4eb70 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e77d6b57e0 con 0x55e77da4eb70 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f95bc003c50 con 0x55e77da4eb70 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 10 client.25168 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 10 client.25168 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 20 client.25168 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77da69cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 4 client.25168 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.764+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e77d6f2720 con 0x55e77da67880 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.765+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77da69cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.765+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77da69cc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da944f3a936d471b server_cookie=1c78d560901eafe6 in_seq=0 out_seq=0 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.765+0000 7f95b57fa640 10 client.25168 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95b57fa640 1 -- 172.21.15.45:0/3396899443 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e77d6f2720 con 0x55e77da67880 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95b57fa640 10 client.25168 handle_command_reply: tid=0 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 client.25168 shutdown 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 msgr2=0x55e77da54b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0fed20 tx=0x7f95bc009d80 comp rx=0 tx=0).stop 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 shutdown_connections 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77da69cc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95b57fa640 0 client.25168 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da57750 0x55e77da550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4eb70 0x55e77da54b90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/3396899443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da54650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.766+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 >> 172.21.15.45:0/3396899443 conn(0x55e77da48980 msgr2=0x55e77da4a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.767+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 shutdown_connections 2024-03-12T05:11:15.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.767+0000 7f95d6bbe740 1 -- 172.21.15.45:0/3396899443 wait complete. 2024-03-12T05:11:15.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.767+0000 7f95d6bbe740 20 client.25168 trim_cache size 0 max 16384 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 Processor -- start 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 -- start start 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4e250 0x55e77da56d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da53630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77d93b130 con 0x55e77da57750 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77da59eb0 con 0x55e77da4eb70 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77da44080 con 0x55e77da4e250 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d24a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d24a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54926/0 (socket says 172.21.15.45:54926) 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d24a9640 1 -- 172.21.15.45:0/4043663266 learned_addr learned my addr 172.21.15.45:0/4043663266 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4e250 0x55e77da56d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.783+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da53630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 -- 172.21.15.45:0/4043663266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4e250 msgr2=0x55e77da56d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4e250 0x55e77da56d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 -- 172.21.15.45:0/4043663266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 msgr2=0x55e77da518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 -- 172.21.15.45:0/4043663266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e77d950c20 con 0x55e77da4eb70 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d24a9640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da53630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95bc01b440 tx=0x7f95bc00b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d480899a339540a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95b77fe640 1 -- 172.21.15.45:0/4043663266 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc015030 con 0x55e77da4eb70 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95b77fe640 1 -- 172.21.15.45:0/4043663266 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95bc016040 con 0x55e77da4eb70 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.784+0000 7f95b77fe640 1 -- 172.21.15.45:0/4043663266 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00ba50 con 0x55e77da4eb70 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4043663266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 msgr2=0x55e77da53630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da53630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95bc01b440 tx=0x7f95bc00b710 comp rx=0 tx=0).stop 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4043663266 shutdown_connections 2024-03-12T05:11:15.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da57750 0x55e77da518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da53630 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4043663266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da4e250 0x55e77da56d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4043663266 >> 172.21.15.45:0/4043663266 conn(0x55e77da48980 msgr2=0x55e77da52f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4043663266 shutdown_connections 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.785+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4043663266 wait complete. 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 -- start start 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 Processor -- start 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da5d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da5d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77da5ed00 con 0x55e77da4e250 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77dabddf0 con 0x55e77da4eb70 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e77dabdf60 con 0x55e77da57750 2024-03-12T05:11:15.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d6bbe740 10 client.0 _ll_register_callbacks cb 0x7f95c80f53e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d24a9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da5d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d1ca8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54928/0 (socket says 172.21.15.45:54928) 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4052057868 learned_addr learned my addr 172.21.15.45:0/4052057868 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.786+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da5d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 msgr2=0x55e77da5d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da5d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4052057868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 msgr2=0x55e77da5d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da5d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 -- 172.21.15.45:0/4052057868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c400f9c0 con 0x55e77da4e250 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da5d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d1ca8640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95c80b87c0 tx=0x7f95c80bc360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e51e170aa32ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c80bc3d0 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95c80bca60 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95c80f2e90 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95c80bbcc0 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.787+0000 7f95d6bbe740 10 client.15684 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e77d9b46a0 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95c80f2e90 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f95c803c070 con 0x55e77da4e250 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 10 client.15684 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 10 client.15684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 20 client.15684 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77dac14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 4 client.15684 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e77d741890 con 0x55e77da67880 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.788+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77dac14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.789+0000 7f95b57fa640 10 client.15684 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.789+0000 7f95d14a7640 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77dac14c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3e1d6edd04754a0 server_cookie=c0005cdc2fa2c107 in_seq=0 out_seq=0 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95b57fa640 1 -- 172.21.15.45:0/4052057868 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e77d741890 con 0x55e77da67880 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95b57fa640 10 client.15684 handle_command_reply: tid=0 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 client.15684 shutdown 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 msgr2=0x55e77da5cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95c80b87c0 tx=0x7f95c80bc360 comp rx=0 tx=0).stop 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 shutdown_connections 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e77da67880 0x55e77dac14c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e77da57750 0x55e77da5d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e77da4eb70 0x55e77da5d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 --2- 172.21.15.45:0/4052057868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e77da4e250 0x55e77da5cd60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 >> 172.21.15.45:0/4052057868 conn(0x55e77da48980 msgr2=0x55e77da472f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95b57fa640 0 client.15684 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 shutdown_connections 2024-03-12T05:11:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 1 -- 172.21.15.45:0/4052057868 wait complete. 2024-03-12T05:11:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.790+0000 7f95d6bbe740 20 client.15684 trim_cache size 0 max 16384 2024-03-12T05:11:15.794 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 msgr2=0x7f95b80a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 0x7f95b80a4180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f95c40107d0 tx=0x7f95c4004a40 comp rx=0 tx=0).stop 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 msgr2=0x7f95cc197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009230 tx=0x7f95c8005dd0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 shutdown_connections 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95b80a1cc0 0x7f95b80a4180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95cc106450 0x7f95cc197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95cc19bfe0 0x7f95cc19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 --2- 172.21.15.45:0/926752122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95cc0ff890 0x7f95cc197720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 >> 172.21.15.45:0/926752122 conn(0x7f95cc0765a0 msgr2=0x7f95cc104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 shutdown_connections 2024-03-12T05:11:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.791+0000 7f95d3f0c640 1 -- 172.21.15.45:0/926752122 wait complete. 2024-03-12T05:11:15.801 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:15.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:11:15.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.934+0000 7f708cadf640 1 -- 172.21.15.45:0/2582350937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100c10 msgr2=0x7f7088104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.934+0000 7f708cadf640 1 --2- 172.21.15.45:0/2582350937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100c10 0x7f7088104c50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7070009860 tx=0x7f707002bb70 comp rx=0 tx=0).stop 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 -- 172.21.15.45:0/2582350937 shutdown_connections 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 --2- 172.21.15.45:0/2582350937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f708810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 --2- 172.21.15.45:0/2582350937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100c10 0x7f7088104c50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 --2- 172.21.15.45:0/2582350937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088100260 0x7f7088100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 -- 172.21.15.45:0/2582350937 >> 172.21.15.45:0/2582350937 conn(0x7f7088077180 msgr2=0x7f7088077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 -- 172.21.15.45:0/2582350937 shutdown_connections 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.935+0000 7f708cadf640 1 -- 172.21.15.45:0/2582350937 wait complete. 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 Processor -- start 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 -- start start 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088199dc0 0x7f708819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708819a7b0 con 0x7f7088105330 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708819a920 con 0x7f7088199dc0 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708cadf640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708819aa90 con 0x7f7088100260 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54944/0 (socket says 172.21.15.45:54944) 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 -- 172.21.15.45:0/4023905538 learned_addr learned my addr 172.21.15.45:0/4023905538 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57770/0 (socket says 172.21.15.45:57770) 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f7086d9d640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088199dc0 0x7f708819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 -- 172.21.15.45:0/4023905538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 msgr2=0x7f7088195500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 -- 172.21.15.45:0/4023905538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088199dc0 msgr2=0x7f708819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088199dc0 0x7f708819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f707ffff640 1 -- 172.21.15.45:0/4023905538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080009900 con 0x7f7088105330 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.936+0000 7f708659c640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f707ffff640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f70700045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f707f7fe640 1 -- 172.21.15.45:0/4023905538 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707002c510 con 0x7f7088105330 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7070009490 con 0x7f7088105330 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70881a2360 con 0x7f7088105330 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f707f7fe640 1 -- 172.21.15.45:0/4023905538 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f707002ce90 con 0x7f7088105330 2024-03-12T05:11:15.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.937+0000 7f707f7fe640 1 -- 172.21.15.45:0/4023905538 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070003e30 con 0x7f7088105330 2024-03-12T05:11:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.940+0000 7f707f7fe640 1 -- 172.21.15.45:0/4023905538 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f707003b040 con 0x7f7088105330 2024-03-12T05:11:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.940+0000 7f708f78c740 1 Processor -- start 2024-03-12T05:11:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.940+0000 7f708f78c740 1 -- start start 2024-03-12T05:11:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.940+0000 7f708f78c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e912e250 0x55d2e912e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708f78c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708f78c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e9137750 0x55d2e9139b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708f78c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e90e9960 con 0x55d2e9137750 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708f78c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e909d310 con 0x55d2e912e250 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708f78c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e909a840 con 0x55d2e912eb70 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f707f7fe640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 0x7f70740a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57784/0 (socket says 172.21.15.45:57784) 2024-03-12T05:11:15.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f707ffff640 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e9137750 0x55d2e9139b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 -- 172.21.15.45:0/1178223771 learned_addr learned my addr 172.21.15.45:0/1178223771 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f7086d9d640 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e912e250 0x55d2e912e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f7086d9d640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 0x7f70740a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 -- 172.21.15.45:0/1178223771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e912e250 msgr2=0x55d2e912e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e912e250 0x55d2e912e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 -- 172.21.15.45:0/1178223771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e9137750 msgr2=0x55d2e9139b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e9137750 0x55d2e9139b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f708659c640 1 -- 172.21.15.45:0/1178223771 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2e9058060 con 0x55d2e912eb70 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.941+0000 7f707f7fe640 1 -- 172.21.15.45:0/4023905538 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f70700ef2b0 con 0x7f7088105330 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708659c640 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70780115f0 tx=0x7f7078011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3676816ae27be518 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f7086d9d640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 0x7f70740a8610 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7080005fd0 tx=0x7f708000c420 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f707cff9640 1 -- 172.21.15.45:0/1178223771 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800f640 con 0x55d2e912eb70 2024-03-12T05:11:15.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f707cff9640 1 -- 172.21.15.45:0/1178223771 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f707800fc50 con 0x55d2e912eb70 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f707cff9640 1 -- 172.21.15.45:0/1178223771 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078019cb0 con 0x55d2e912eb70 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 -- 172.21.15.45:0/1178223771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 msgr2=0x55d2e9137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70780115f0 tx=0x7f7078011ab0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 -- 172.21.15.45:0/1178223771 shutdown_connections 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e9137750 0x55d2e9139b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912eb70 0x55d2e9137210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.942+0000 7f708f78c740 1 --2- 172.21.15.45:0/1178223771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e912e250 0x55d2e912e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.943+0000 7f708f78c740 1 -- 172.21.15.45:0/1178223771 >> 172.21.15.45:0/1178223771 conn(0x55d2e9128980 msgr2=0x55d2e9128d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.943+0000 7f708f78c740 1 -- 172.21.15.45:0/1178223771 shutdown_connections 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.943+0000 7f708f78c740 1 -- 172.21.15.45:0/1178223771 wait complete. 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.943+0000 7f708f78c740 1 -- start start 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.943+0000 7f708f78c740 1 Processor -- start 2024-03-12T05:11:15.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e9134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9135240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e9096020 con 0x55d2e912eb70 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e8d69430 con 0x55d2e9137750 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e8d25e00 con 0x55d2e912e250 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57796/0 (socket says 172.21.15.45:57796) 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 -- 172.21.15.45:0/3324839036 learned_addr learned my addr 172.21.15.45:0/3324839036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f707ffff640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e9134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708f78c740 10 client.0 _ll_register_callbacks cb 0x55d2e903dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f7086d9d640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 -- 172.21.15.45:0/3324839036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 msgr2=0x55d2e9135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9135240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 -- 172.21.15.45:0/3324839036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 msgr2=0x55d2e9134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e9134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.944+0000 7f708659c640 1 -- 172.21.15.45:0/3324839036 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7078009d40 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f708659c640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7078009250 tx=0x7f7078019a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db3feb4d95e65fef server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707800f850 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d2e8d957e0 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f707800f9b0 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7078004030 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f708f78c740 10 client.25186 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d2e90946a0 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f707802e020 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.945+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7078020070 con 0x55d2e912e250 2024-03-12T05:11:15.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 10 client.25186 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 10 client.25186 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 20 client.25186 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e9147b70 0x55d2e9149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 4 client.25186 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d2e8dd2720 con 0x55d2e9147b70 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f707ffff640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e9147b70 0x55d2e9149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f707ffff640 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e9147b70 0x55d2e9149fa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8cb580578d6136d server_cookie=bdfa17fae4932f9d in_seq=0 out_seq=0 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.946+0000 7f706dffb640 10 client.25186 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f706dffb640 1 -- 172.21.15.45:0/3324839036 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d2e8dd2720 con 0x55d2e9147b70 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f706dffb640 10 client.25186 handle_command_reply: tid=0 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 client.25186 shutdown 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 msgr2=0x55d2e91347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7078009250 tx=0x7f7078019a50 comp rx=0 tx=0).stop 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 shutdown_connections 2024-03-12T05:11:15.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e9134d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e9147b70 0x55d2e9149fa0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9135240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 --2- 172.21.15.45:0/3324839036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e91347c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 >> 172.21.15.45:0/3324839036 conn(0x55d2e9128980 msgr2=0x55d2e912a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f706dffb640 0 client.25186 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 shutdown_connections 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.947+0000 7f708f78c740 1 -- 172.21.15.45:0/3324839036 wait complete. 2024-03-12T05:11:15.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.948+0000 7f708f78c740 20 client.25186 trim_cache size 0 max 16384 2024-03-12T05:11:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 Processor -- start 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 -- start start 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e9136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e912faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9130030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e901b130 con 0x55d2e912eb70 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e9132d10 con 0x55d2e9137750 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708f78c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e9129300 con 0x55d2e912e250 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f707ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e912faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e9136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f7086d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f707ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e912faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54976/0 (socket says 172.21.15.45:54976) 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.963+0000 7f707ffff640 1 -- 172.21.15.45:0/2192543002 learned_addr learned my addr 172.21.15.45:0/2192543002 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 -- 172.21.15.45:0/2192543002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 msgr2=0x55d2e9130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9130030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 -- 172.21.15.45:0/2192543002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 msgr2=0x55d2e912faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e912faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 -- 172.21.15.45:0/2192543002 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2e9030c20 con 0x55d2e912e250 2024-03-12T05:11:15.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f708659c640 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e9136ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7078010010 tx=0x7f7078011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4048400e8dab383a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f706ffff640 1 -- 172.21.15.45:0/2192543002 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078019ac0 con 0x55d2e912e250 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f706ffff640 1 -- 172.21.15.45:0/2192543002 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f707802e030 con 0x55d2e912e250 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.964+0000 7f706ffff640 1 -- 172.21.15.45:0/2192543002 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7078003850 con 0x55d2e912e250 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- 172.21.15.45:0/2192543002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 msgr2=0x55d2e9136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e9136ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7078010010 tx=0x7f7078011ab0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- 172.21.15.45:0/2192543002 shutdown_connections 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e912faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e9130030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 --2- 172.21.15.45:0/2192543002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e9136ef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- 172.21.15.45:0/2192543002 >> 172.21.15.45:0/2192543002 conn(0x55d2e9128980 msgr2=0x55d2e914e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- 172.21.15.45:0/2192543002 shutdown_connections 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- 172.21.15.45:0/2192543002 wait complete. 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 -- start start 2024-03-12T05:11:15.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.965+0000 7f708f78c740 1 Processor -- start 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e913cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e919d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e919d600 con 0x55d2e912eb70 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e919d740 con 0x55d2e9137750 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2e919d8b0 con 0x55d2e912e250 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708f78c740 10 client.0 _ll_register_callbacks cb 0x7f7078009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e913cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57814/0 (socket says 172.21.15.45:57814) 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f708659c640 1 -- 172.21.15.45:0/196313629 learned_addr learned my addr 172.21.15.45:0/196313629 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f7086d9d640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e919d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 -- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 msgr2=0x55d2e913c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 -- 172.21.15.45:0/196313629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 msgr2=0x55d2e919d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e919d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.966+0000 7f707ffff640 1 -- 172.21.15.45:0/196313629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2e90946a0 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f707ffff640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e913cd70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f70700bbde0 tx=0x7f7070005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342138b811a145f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f708659c640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f706dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70700c4540 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f70700035f0 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f707800fc50 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70700037b0 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f708f78c740 10 client.15702 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.967+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d2e90568c0 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f707800fc50 con 0x55d2e912eb70 2024-03-12T05:11:15.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f70700ecb60 con 0x55d2e912eb70 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 10 client.15702 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 10 client.15702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 20 client.15702 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e919eed0 0x55d2e91a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 4 client.15702 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d2e8e21890 con 0x55d2e919eed0 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708659c640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e919eed0 0x55d2e91a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f708659c640 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e919eed0 0x55d2e91a12f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b7659277fd66055 server_cookie=a7f2d93ad15e73df in_seq=0 out_seq=0 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.968+0000 7f706dffb640 10 client.15702 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f706dffb640 1 -- 172.21.15.45:0/196313629 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d2e8e21890 con 0x55d2e919eed0 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f706dffb640 10 client.15702 handle_command_reply: tid=0 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 client.15702 shutdown 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 msgr2=0x55d2e913cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e913cd70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f70700bbde0 tx=0x7f7070005f90 comp rx=0 tx=0).stop 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 shutdown_connections 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d2e912eb70 0x55d2e913cd70 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55d2e919eed0 0x55d2e91a12f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2e9137750 0x55d2e919d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 --2- 172.21.15.45:0/196313629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d2e912e250 0x55d2e913c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 >> 172.21.15.45:0/196313629 conn(0x55d2e9128980 msgr2=0x55d2e912faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f706dffb640 0 client.15702 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:15.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 shutdown_connections 2024-03-12T05:11:15.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 1 -- 172.21.15.45:0/196313629 wait complete. 2024-03-12T05:11:15.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.969+0000 7f708f78c740 20 client.15702 trim_cache size 0 max 16384 2024-03-12T05:11:15.972 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:15.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 msgr2=0x7f70740a8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 0x7f70740a8610 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7080005fd0 tx=0x7f708000c420 comp rx=0 tx=0).stop 2024-03-12T05:11:15.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 msgr2=0x7f7088195a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:15.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f70700045a0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 shutdown_connections 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70740a6150 0x7f70740a8610 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7088105330 0x7f7088195a40 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7088199dc0 0x7f708819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 --2- 172.21.15.45:0/4023905538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7088100260 0x7f7088195500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 >> 172.21.15.45:0/4023905538 conn(0x7f7088077180 msgr2=0x7f70880fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 shutdown_connections 2024-03-12T05:11:15.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:15.970+0000 7f708cadf640 1 -- 172.21.15.45:0/4023905538 wait complete. 2024-03-12T05:11:15.980 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:15.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 -- 172.21.15.45:0/1380555807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 msgr2=0x7fe9a410e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/1380555807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a410e070 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009cd0 tx=0x7fe99c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 -- 172.21.15.45:0/1380555807 shutdown_connections 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/1380555807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a410e070 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/1380555807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a4068fa0 0x7fe9a4105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/1380555807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a40685f0 0x7fe9a40689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 -- 172.21.15.45:0/1380555807 >> 172.21.15.45:0/1380555807 conn(0x7fe9a4077180 msgr2=0x7fe9a4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 -- 172.21.15.45:0/1380555807 shutdown_connections 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.112+0000 7fe9abe8d640 1 -- 172.21.15.45:0/1380555807 wait complete. 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 Processor -- start 2024-03-12T05:11:16.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 -- start start 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a40685f0 0x7fe9a419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a4199790 0x7fe9a4199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419a240 con 0x7fe9a41064c0 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419a3b0 con 0x7fe9a4199790 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9abe8d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419a520 con 0x7fe9a40685f0 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54994/0 (socket says 172.21.15.45:54994) 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 -- 172.21.15.45:0/4144593079 learned_addr learned my addr 172.21.15.45:0/4144593079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a40685f0 0x7fe9a419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a4199790 0x7fe9a4199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 -- 172.21.15.45:0/4144593079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a40685f0 msgr2=0x7fe9a419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a40685f0 0x7fe9a419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 -- 172.21.15.45:0/4144593079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a4199790 msgr2=0x7fe9a4199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a4199790 0x7fe9a4199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.113+0000 7fe9a9428640 1 -- 172.21.15.45:0/4144593079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c009900 con 0x7fe9a41064c0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe9a9428640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9c0 tx=0x7fe9a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe99affd640 1 -- 172.21.15.45:0/4144593079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00096c0 con 0x7fe9a41064c0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe99affd640 1 -- 172.21.15.45:0/4144593079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe9a00054d0 con 0x7fe9a41064c0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe99affd640 1 -- 172.21.15.45:0/4144593079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0010430 con 0x7fe9a41064c0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a41a64d0 con 0x7fe9a41064c0 2024-03-12T05:11:16.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.114+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a41a6960 con 0x7fe9a41064c0 2024-03-12T05:11:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.117+0000 7fe99affd640 1 -- 172.21.15.45:0/4144593079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fe9a0004f90 con 0x7fe9a41064c0 2024-03-12T05:11:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.117+0000 7fe9aea5c740 1 Processor -- start 2024-03-12T05:11:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 -- start start 2024-03-12T05:11:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe99affd640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 0x7fe9900a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e75e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e7ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e81350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 0x7fe9900a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0e28960 con 0x5604e0e75a60 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0ddc310 con 0x5604e0e76380 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0dd9840 con 0x5604e0e7ef60 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe99affd640 1 -- 172.21.15.45:0/4144593079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe9a0014070 con 0x7fe9a41064c0 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e75e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9428640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e7ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e75e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54996/0 (socket says 172.21.15.45:54996) 2024-03-12T05:11:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9c29640 1 -- 172.21.15.45:0/2261346808 learned_addr learned my addr 172.21.15.45:0/2261346808 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e81350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9428640 1 -- 172.21.15.45:0/2261346808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 msgr2=0x5604e0e81350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e81350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9428640 1 -- 172.21.15.45:0/2261346808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 msgr2=0x5604e0e75e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.118+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e75e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe9a9428640 1 -- 172.21.15.45:0/2261346808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e0d97060 con 0x5604e0e76380 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 0x7fe9900a42d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c630 tx=0x7fe99c032f00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e7ea20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00cb320 tx=0x7fe9a00969d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f241bfe3ada24f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe98ffff640 1 -- 172.21.15.45:0/2261346808 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00b8ea0 con 0x5604e0e76380 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe98ffff640 1 -- 172.21.15.45:0/2261346808 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe9a009a090 con 0x5604e0e76380 2024-03-12T05:11:16.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.119+0000 7fe98ffff640 1 -- 172.21.15.45:0/2261346808 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0096da0 con 0x5604e0e76380 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2261346808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 msgr2=0x5604e0e7ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e7ea20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00cb320 tx=0x7fe9a00969d0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2261346808 shutdown_connections 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e81350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e7ea20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2261346808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e75e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2261346808 >> 172.21.15.45:0/2261346808 conn(0x5604e0e70190 msgr2=0x5604e0e70570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2261346808 shutdown_connections 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2261346808 wait complete. 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.120+0000 7fe9aea5c740 1 -- start start 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 Processor -- start 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e75a60 0x5604e0e7bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0e7ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0dd5020 con 0x5604e0e7ef60 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0aa8430 con 0x5604e0e75a60 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0a64e00 con 0x5604e0e76380 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aea5c740 10 client.0 _ll_register_callbacks cb 0x5604e0d7ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9a9428640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9a9428640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57856/0 (socket says 172.21.15.45:57856) 2024-03-12T05:11:16.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9a9428640 1 -- 172.21.15.45:0/2404043754 learned_addr learned my addr 172.21.15.45:0/2404043754 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e75a60 0x5604e0e7bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.121+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0e7ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 -- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e75a60 msgr2=0x5604e0e7bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e75a60 0x5604e0e7bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 -- 172.21.15.45:0/2404043754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 msgr2=0x5604e0e7ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0e7ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 -- 172.21.15.45:0/2404043754 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a41a6960 con 0x5604e0e76380 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9a9428640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00989e0 tx=0x7fe9a00a2ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9710479aff5b6a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe98d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00b8ea0 con 0x5604e0e76380 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe9a00b3230 con 0x5604e0e76380 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604e0ad47e0 con 0x5604e0e76380 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9a00cb260 con 0x5604e0e76380 2024-03-12T05:11:16.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9aea5c740 10 client.25204 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.122+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5604e0dd36a0 con 0x5604e0e76380 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5604e0ad47e0 con 0x5604e0e76380 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe9a001f060 con 0x5604e0e76380 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 10 client.25204 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 10 client.25204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 20 client.25204 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0e8f110 0x5604e0e91550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 4 client.25204 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604e0b11720 con 0x5604e0e8f110 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.123+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0e8f110 0x5604e0e91550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.124+0000 7fe98d7fa640 10 client.25204 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.124+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0e8f110 0x5604e0e91550 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d01bdc4269d7c3e server_cookie=2a93b0ade21acb45 in_seq=0 out_seq=0 2024-03-12T05:11:16.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe98d7fa640 1 -- 172.21.15.45:0/2404043754 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5604e0b11720 con 0x5604e0e8f110 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe98d7fa640 10 client.25204 handle_command_reply: tid=0 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 client.25204 shutdown 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 msgr2=0x5604e0e7c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00989e0 tx=0x7fe9a00a2ed0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 shutdown_connections 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0e8f110 0x5604e0e91550 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0e7ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e76380 0x5604e0e7c520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/2404043754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e75a60 0x5604e0e7bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe98d7fa640 0 client.25204 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 >> 172.21.15.45:0/2404043754 conn(0x5604e0e70190 msgr2=0x5604e0e71650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 shutdown_connections 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 1 -- 172.21.15.45:0/2404043754 wait complete. 2024-03-12T05:11:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.125+0000 7fe9aea5c740 20 client.25204 trim_cache size 0 max 16384 2024-03-12T05:11:16.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.139+0000 7fe9aea5c740 1 Processor -- start 2024-03-12T05:11:16.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 -- start start 2024-03-12T05:11:16.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e66a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e85e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0d5a130 con 0x5604e0e75a60 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0e75260 con 0x5604e0e76380 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0e756c0 con 0x5604e0e7ef60 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55018/0 (socket says 172.21.15.45:55018) 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 -- 172.21.15.45:0/3598480932 learned_addr learned my addr 172.21.15.45:0/3598480932 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9428640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e66a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e85e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 -- 172.21.15.45:0/3598480932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 msgr2=0x5604e0e85e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e85e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 -- 172.21.15.45:0/3598480932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 msgr2=0x5604e0e66a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e66a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.140+0000 7fe9a9c29640 1 -- 172.21.15.45:0/3598480932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604e0d6fc20 con 0x5604e0e75a60 2024-03-12T05:11:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.141+0000 7fe9a9428640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e66a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.141+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x5604e0e91e30 tx=0x7fe994005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0775fe0d895405d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.141+0000 7fe98f7fe640 1 -- 172.21.15.45:0/3598480932 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994017070 con 0x5604e0e75a60 2024-03-12T05:11:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.141+0000 7fe98f7fe640 1 -- 172.21.15.45:0/3598480932 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe994018040 con 0x5604e0e75a60 2024-03-12T05:11:16.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.141+0000 7fe98f7fe640 1 -- 172.21.15.45:0/3598480932 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940107b0 con 0x5604e0e75a60 2024-03-12T05:11:16.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- 172.21.15.45:0/3598480932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 msgr2=0x5604e0e7c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x5604e0e91e30 tx=0x7fe994005f90 comp rx=0 tx=0).stop 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- 172.21.15.45:0/3598480932 shutdown_connections 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e7ef60 0x5604e0e85e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e66a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/3598480932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e75a60 0x5604e0e7c880 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- 172.21.15.45:0/3598480932 >> 172.21.15.45:0/3598480932 conn(0x5604e0e70190 msgr2=0x5604e0e662f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- 172.21.15.45:0/3598480932 shutdown_connections 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- 172.21.15.45:0/3598480932 wait complete. 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 -- start start 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.142+0000 7fe9aea5c740 1 Processor -- start 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 0x5604e0e850b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e855f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0edf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0edfb50 con 0x5604e0e7ef60 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0edfc90 con 0x5604e0e76380 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604e0edfe50 con 0x5604e0e75a60 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 0x5604e0e850b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9a9c29640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 0x5604e0e850b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57884/0 (socket says 172.21.15.45:57884) 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9a9c29640 1 -- 172.21.15.45:0/4201494842 learned_addr learned my addr 172.21.15.45:0/4201494842 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0edf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aea5c740 10 client.0 _ll_register_callbacks cb 0x7fe9a0096310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 -- 172.21.15.45:0/4201494842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 msgr2=0x5604e0e850b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 0x5604e0e850b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 -- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 msgr2=0x5604e0e855f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9a9428640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e855f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9aa42a640 1 -- 172.21.15.45:0/4201494842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe994004040 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.143+0000 7fe9a9428640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e855f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aa42a640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0edf610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe99c037040 tx=0x7fe99c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db70af0a18286a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c0329e0 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe99c003970 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9a009a090 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe99c032360 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 10 client.15720 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5604e0d958c0 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe9a009a090 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe99c041410 con 0x5604e0e7ef60 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 10 client.15720 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:16.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.144+0000 7fe9aea5c740 20 client.15720 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0ee76c0 0x5604e0ee9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 4 client.15720 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5604e0b60890 con 0x5604e0ee76c0 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0ee76c0 0x5604e0ee9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9a9c29640 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0ee76c0 0x5604e0ee9b00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f60e916a960d6f server_cookie=df2b35789eddd562 in_seq=0 out_seq=0 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe98d7fa640 10 client.15720 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe98d7fa640 1 -- 172.21.15.45:0/4201494842 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5604e0b60890 con 0x5604e0ee76c0 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe98d7fa640 10 client.15720 handle_command_reply: tid=0 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 1 client.15720 shutdown 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 msgr2=0x5604e0edf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.145+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0edf610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe99c037040 tx=0x7fe99c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 shutdown_connections 2024-03-12T05:11:16.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604e0ee76c0 0x5604e0ee9b00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604e0e7ef60 0x5604e0edf610 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604e0e76380 0x5604e0e855f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 --2- 172.21.15.45:0/4201494842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604e0e75a60 0x5604e0e850b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 >> 172.21.15.45:0/4201494842 conn(0x5604e0e70190 msgr2=0x5604e0e71350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe98d7fa640 0 client.15720 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 shutdown_connections 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 1 -- 172.21.15.45:0/4201494842 wait complete. 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.146+0000 7fe9aea5c740 20 client.15720 trim_cache size 0 max 16384 2024-03-12T05:11:16.151 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:16.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 msgr2=0x7fe9900a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 0x7fe9900a42d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c630 tx=0x7fe99c032f00 comp rx=0 tx=0).stop 2024-03-12T05:11:16.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 msgr2=0x7fe9a419fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:16.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9c0 tx=0x7fe9a000ee80 comp rx=0 tx=0).stop 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 shutdown_connections 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9900a1e10 0x7fe9900a42d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9a41064c0 0x7fe9a419fb60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9a4199790 0x7fe9a4199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 --2- 172.21.15.45:0/4144593079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9a40685f0 0x7fe9a419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 >> 172.21.15.45:0/4144593079 conn(0x7fe9a4077180 msgr2=0x7fe9a40fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 shutdown_connections 2024-03-12T05:11:16.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:16.147+0000 7fe9abe8d640 1 -- 172.21.15.45:0/4144593079 wait complete. 2024-03-12T05:11:16.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:16 smithi045 ceph-mon[32174]: pgmap v14: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 3.4 MiB/s wr, 130 op/s 2024-03-12T05:11:16.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:16 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/550380497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:16.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:16 smithi130 ceph-mon[35398]: pgmap v14: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 3.4 MiB/s wr, 130 op/s 2024-03-12T05:11:16.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:16 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/550380497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:16 smithi116 ceph-mon[35486]: pgmap v14: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 3.4 MiB/s wr, 130 op/s 2024-03-12T05:11:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:16 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/550380497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:17.157 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:11:17.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:11:17.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:11:17.266 INFO:teuthology.orchestra.run.smithi045.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-03-12T05:11:17.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-12T05:11:17.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 -- 172.21.15.45:0/2519368061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 msgr2=0x7fec201027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 --2- 172.21.15.45:0/2519368061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec201027f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009ca0 tx=0x7fec1c02c180 comp rx=0 tx=0).stop 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 -- 172.21.15.45:0/2519368061 shutdown_connections 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 --2- 172.21.15.45:0/2519368061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec2010b750 0x7fec2010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 --2- 172.21.15.45:0/2519368061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec20102d30 0x7fec2010b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 --2- 172.21.15.45:0/2519368061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec201027f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 -- 172.21.15.45:0/2519368061 >> 172.21.15.45:0/2519368061 conn(0x7fec200771e0 msgr2=0x7fec200775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 -- 172.21.15.45:0/2519368061 shutdown_connections 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.285+0000 7fec286d0640 1 -- 172.21.15.45:0/2519368061 wait complete. 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 Processor -- start 2024-03-12T05:11:17.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 -- start start 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 0x7fec20197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 0x7fec2019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2019c550 con 0x7fec20102410 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2019c6c0 con 0x7fec2010b750 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec286d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2019c880 con 0x7fec20102d30 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55050/0 (socket says 172.21.15.45:55050) 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.286+0000 7fec2646c640 1 -- 172.21.15.45:0/269941472 learned_addr learned my addr 172.21.15.45:0/269941472 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 0x7fec2019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 0x7fec20197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 -- 172.21.15.45:0/269941472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 msgr2=0x7fec20197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 0x7fec20197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 -- 172.21.15.45:0/269941472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 msgr2=0x7fec2019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 0x7fec2019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 -- 172.21.15.45:0/269941472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c009900 con 0x7fec20102410 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 0x7fec20197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 0x7fec2019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec2646c640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fec1c004af0 tx=0x7fec1c0347e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec177fe640 1 -- 172.21.15.45:0/269941472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c003d80 con 0x7fec20102410 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec177fe640 1 -- 172.21.15.45:0/269941472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fec1c032b30 con 0x7fec20102410 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2019cb00 con 0x7fec20102410 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec177fe640 1 -- 172.21.15.45:0/269941472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0046e0 con 0x7fec20102410 2024-03-12T05:11:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.287+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2019cdd0 con 0x7fec20102410 2024-03-12T05:11:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.289+0000 7fec177fe640 1 -- 172.21.15.45:0/269941472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fec1c03b040 con 0x7fec20102410 2024-03-12T05:11:17.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.289+0000 7fec2b376740 1 Processor -- start 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 -- start start 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e564760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e56fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e4d2f90 con 0x55a38e564d30 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e4d04c0 con 0x55a38e56d8b0 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2b376740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e4cbca0 con 0x55a38e564380 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55058/0 (socket says 172.21.15.45:55058) 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 -- 172.21.15.45:0/4102357392 learned_addr learned my addr 172.21.15.45:0/4102357392 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec26c6d640 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e564760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec25c6b640 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e56fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 -- 172.21.15.45:0/4102357392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 msgr2=0x55a38e564760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e564760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 -- 172.21.15.45:0/4102357392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 msgr2=0x55a38e56fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e56fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec2646c640 1 -- 172.21.15.45:0/4102357392 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a38e48e0b0 con 0x55a38e564d30 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.290+0000 7fec177fe640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 0x7fec0c0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 0x7fec0c0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec2646c640 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0edc50 tx=0x7fec1c0b9270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf1ae127ff089a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec177fe640 1 -- 172.21.15.45:0/269941472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fec1c0f04e0 con 0x7fec20102410 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec14ff9640 1 -- 172.21.15.45:0/4102357392 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0bcda0 con 0x55a38e564d30 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec14ff9640 1 -- 172.21.15.45:0/4102357392 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fec1c0bcb10 con 0x55a38e564d30 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.291+0000 7fec14ff9640 1 -- 172.21.15.45:0/4102357392 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c004040 con 0x55a38e564d30 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- 172.21.15.45:0/4102357392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 msgr2=0x55a38e56d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 0x7fec0c0a4180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec18005fd0 tx=0x7fec18009370 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0edc50 tx=0x7fec1c0b9270 comp rx=0 tx=0).stop 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- 172.21.15.45:0/4102357392 shutdown_connections 2024-03-12T05:11:17.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56d2e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e56fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 --2- 172.21.15.45:0/4102357392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e564760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- 172.21.15.45:0/4102357392 >> 172.21.15.45:0/4102357392 conn(0x55a38e55e840 msgr2=0x55a38e55ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- 172.21.15.45:0/4102357392 shutdown_connections 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- 172.21.15.45:0/4102357392 wait complete. 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 -- start start 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 Processor -- start 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.292+0000 7fec2b376740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e564380 0x55a38e56a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e56d8b0 0x55a38e56b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e470f40 con 0x55a38e564d30 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e19f4b0 con 0x55a38e564380 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e451670 con 0x55a38e56d8b0 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 10 client.0 _ll_register_callbacks cb 0x55a38e4bef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55064/0 (socket says 172.21.15.45:55064) 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 -- 172.21.15.45:0/4120838449 learned_addr learned my addr 172.21.15.45:0/4120838449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec26c6d640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e56d8b0 0x55a38e56b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2646c640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e564380 0x55a38e56a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 -- 172.21.15.45:0/4120838449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e56d8b0 msgr2=0x55a38e56b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e56d8b0 0x55a38e56b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 -- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e564380 msgr2=0x55a38e56a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e564380 0x55a38e56a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 -- 172.21.15.45:0/4120838449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2019cdd0 con 0x55a38e564d30 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec25c6b640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fec10013740 tx=0x7fec10013b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778d2138c6cd885a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10011d70 con 0x55a38e564d30 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fec1001b3e0 con 0x55a38e564d30 2024-03-12T05:11:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a38e4a5f40 con 0x55a38e564d30 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec10011ed0 con 0x55a38e564d30 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 10 client.15732 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.293+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a38e4ca320 con 0x55a38e564d30 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fec10028080 con 0x55a38e564d30 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fec10022030 con 0x55a38e564d30 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 10 client.15732 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 20 client.15732 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e57dce0 0x55a38e580100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 4 client.15732 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a38e4c2310 con 0x55a38e57dce0 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.294+0000 7fec2646c640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e57dce0 0x55a38e580100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2646c640 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e57dce0 0x55a38e580100 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13d58355fded4edb server_cookie=5943a50deb054235 in_seq=0 out_seq=0 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec09ffb640 10 client.15732 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec09ffb640 1 -- 172.21.15.45:0/4120838449 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a38e4c2310 con 0x55a38e57dce0 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec09ffb640 10 client.15732 handle_command_reply: tid=0 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 client.15732 shutdown 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 msgr2=0x55a38e56ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fec10013740 tx=0x7fec10013b40 comp rx=0 tx=0).stop 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 shutdown_connections 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e57dce0 0x55a38e580100 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e56ad20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e56d8b0 0x55a38e56b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec09ffb640 0 client.15732 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 --2- 172.21.15.45:0/4120838449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e564380 0x55a38e56a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 >> 172.21.15.45:0/4120838449 conn(0x55a38e55e840 msgr2=0x55a38e5607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 shutdown_connections 2024-03-12T05:11:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 1 -- 172.21.15.45:0/4120838449 wait complete. 2024-03-12T05:11:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.295+0000 7fec2b376740 20 client.15732 trim_cache size 0 max 16384 2024-03-12T05:11:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 Processor -- start 2024-03-12T05:11:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 -- start start 2024-03-12T05:11:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e56b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e574be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e4c2310 con 0x55a38e564d30 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e55a180 con 0x55a38e56d8b0 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2b376740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e55f410 con 0x55a38e564380 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec25c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec25c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55080/0 (socket says 172.21.15.45:55080) 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec25c6b640 1 -- 172.21.15.45:0/3658259949 learned_addr learned my addr 172.21.15.45:0/3658259949 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec26c6d640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e574be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.311+0000 7fec2646c640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e56b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 -- 172.21.15.45:0/3658259949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 msgr2=0x55a38e56b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e56b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 -- 172.21.15.45:0/3658259949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 msgr2=0x55a38e574be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e574be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 -- 172.21.15.45:0/3658259949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a38e466c00 con 0x55a38e564d30 2024-03-12T05:11:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec26c6d640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e574be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec25c6b640 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fec1001c890 tx=0x7fec10009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3820e7b49015148c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec0bfff640 1 -- 172.21.15.45:0/3658259949 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1001cb00 con 0x55a38e564d30 2024-03-12T05:11:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec0bfff640 1 -- 172.21.15.45:0/3658259949 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a38e4a5f40 con 0x55a38e564d30 2024-03-12T05:11:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.312+0000 7fec0bfff640 1 -- 172.21.15.45:0/3658259949 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec100035f0 con 0x55a38e564d30 2024-03-12T05:11:17.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- 172.21.15.45:0/3658259949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 msgr2=0x55a38e5746a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fec1001c890 tx=0x7fec10009fd0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- 172.21.15.45:0/3658259949 shutdown_connections 2024-03-12T05:11:17.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564d30 0x55a38e5746a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e574be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- 172.21.15.45:0/3658259949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564380 0x55a38e56b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- 172.21.15.45:0/3658259949 >> 172.21.15.45:0/3658259949 conn(0x55a38e55e840 msgr2=0x55a38e55d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- 172.21.15.45:0/3658259949 shutdown_connections 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- 172.21.15.45:0/3658259949 wait complete. 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 -- start start 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 Processor -- start 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 0x55a38e5cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.313+0000 7fec2b376740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e573c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e574360 con 0x55a38e564380 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e5744d0 con 0x55a38e56d8b0 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a38e574690 con 0x55a38e564d30 2024-03-12T05:11:17.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 10 client.0 _ll_register_callbacks cb 0x7fec1c0f3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55084/0 (socket says 172.21.15.45:55084) 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 -- 172.21.15.45:0/269879166 learned_addr learned my addr 172.21.15.45:0/269879166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 0x55a38e5cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e573c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 -- 172.21.15.45:0/269879166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 msgr2=0x55a38e5cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 0x55a38e5cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 -- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 msgr2=0x55a38e573c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e573c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 -- 172.21.15.45:0/269879166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec18011da0 con 0x55a38e564380 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 0x55a38e5cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec26c6d640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e573c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2646c640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0ee230 tx=0x7fec1c005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0509328b2c3067 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:17.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0f6190 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fec1c004840 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a38e4f1900 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec1c0d7080 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 10 client.15738 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.314+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a38e4ca320 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fec1c086020 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fec1c0b9640 con 0x55a38e564380 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 10 client.15738 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 20 client.15738 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e5d64d0 0x55a38e5d88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 4 client.15738 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a38e1cb860 con 0x55a38e5d64d0 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e5d64d0 0x55a38e5d88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec25c6b640 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e5d64d0 0x55a38e5d88f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cba1373e5c0576d server_cookie=9f207d979fab84b5 in_seq=0 out_seq=0 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.315+0000 7fec09ffb640 10 client.15738 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.316+0000 7fec09ffb640 1 -- 172.21.15.45:0/269879166 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a38e1cb860 con 0x55a38e5d64d0 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.316+0000 7fec09ffb640 10 client.15738 handle_command_reply: tid=0 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.316+0000 7fec2b376740 1 client.15738 shutdown 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.316+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 msgr2=0x55a38e5cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.316+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0ee230 tx=0x7fec1c005df0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 shutdown_connections 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a38e5d64d0 0x55a38e5d88f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a38e56d8b0 0x55a38e573c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a38e564d30 0x55a38e5cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec09ffb640 0 client.15738 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 --2- 172.21.15.45:0/269879166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a38e564380 0x55a38e5cd7f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 >> 172.21.15.45:0/269879166 conn(0x55a38e55e840 msgr2=0x55a38e55d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 shutdown_connections 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 1 -- 172.21.15.45:0/269879166 wait complete. 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec2b376740 20 client.15738 trim_cache size 0 max 16384 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "f31bfe34-8ce7-47b3-97cb-6804e79b1c04", 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:11:17.320 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 msgr2=0x7fec0c0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 0x7fec0c0a4180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec18005fd0 tx=0x7fec18009370 comp rx=0 tx=0).stop 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 msgr2=0x7fec20197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fec1c004af0 tx=0x7fec1c0347e0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 shutdown_connections 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fec0c0a1cc0 0x7fec0c0a4180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec2010b750 0x7fec2019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec20102d30 0x7fec20197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 --2- 172.21.15.45:0/269941472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec20102410 0x7fec20197690 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 >> 172.21.15.45:0/269941472 conn(0x7fec200771e0 msgr2=0x7fec200fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 shutdown_connections 2024-03-12T05:11:17.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.317+0000 7fec286d0640 1 -- 172.21.15.45:0/269941472 wait complete. 2024-03-12T05:11:17.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:11:17.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 -- 172.21.15.45:0/1986558290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54100c20 msgr2=0x7f4c54104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/1986558290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54100c20 0x7f4c54104c60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009930 tx=0x7f4c4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 -- 172.21.15.45:0/1986558290 shutdown_connections 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/1986558290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54100c20 0x7f4c54104c60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/1986558290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 0x7f4c5410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/1986558290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 0x7f4c54100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.490+0000 7f4c5c18c640 1 -- 172.21.15.45:0/1986558290 >> 172.21.15.45:0/1986558290 conn(0x7f4c54077230 msgr2=0x7f4c54077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- 172.21.15.45:0/1986558290 shutdown_connections 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- 172.21.15.45:0/1986558290 wait complete. 2024-03-12T05:11:17.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 Processor -- start 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- start start 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 0x7f4c54198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 0x7f4c54198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a3c70 con 0x7f4c54199020 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a3de0 con 0x7f4c54105340 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5c18c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a3fa0 con 0x7f4c54100270 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5a729640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5a729640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56002/0 (socket says 172.21.15.45:56002) 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.491+0000 7f4c5a729640 1 -- 172.21.15.45:0/946147855 learned_addr learned my addr 172.21.15.45:0/946147855 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c59f28640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 0x7f4c54198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c59727640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 0x7f4c54198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 -- 172.21.15.45:0/946147855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 msgr2=0x7f4c54198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 0x7f4c54198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 -- 172.21.15.45:0/946147855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 msgr2=0x7f4c54198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 0x7f4c54198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 -- 172.21.15.45:0/946147855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c44009560 con 0x7f4c54199020 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5a729640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00eb10 tx=0x7f4c4c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c4affd640 1 -- 172.21.15.45:0/946147855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c004b20 con 0x7f4c54199020 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c4affd640 1 -- 172.21.15.45:0/946147855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4c004c80 con 0x7f4c54199020 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c541a4280 con 0x7f4c54199020 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c4affd640 1 -- 172.21.15.45:0/946147855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c010690 con 0x7f4c54199020 2024-03-12T05:11:17.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.492+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541a4760 con 0x7f4c54199020 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c4affd640 1 -- 172.21.15.45:0/946147855 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f4c4c005020 con 0x7f4c54199020 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 Processor -- start 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 -- start start 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f187fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f0e7280 con 0x55e97f178900 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f0e47b0 con 0x55e97f177fe0 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f0dff90 con 0x55e97f185bc0 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59f28640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59727640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59f28640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56004/0 (socket says 172.21.15.45:56004) 2024-03-12T05:11:17.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59f28640 1 -- 172.21.15.45:0/4078188886 learned_addr learned my addr 172.21.15.45:0/4078188886 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59727640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43068/0 (socket says 172.21.15.45:43068) 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c4affd640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 0x7f4c400a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.495+0000 7f4c59727640 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f187fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 -- 172.21.15.45:0/4078188886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 msgr2=0x55e97f187fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f187fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 -- 172.21.15.45:0/4078188886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 msgr2=0x55e97f1783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c5a729640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 0x7f4c400a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 -- 172.21.15.45:0/4078188886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e97f09e570 con 0x55e97f178900 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59727640 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c4affd640 1 -- 172.21.15.45:0/946147855 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4c4c014070 con 0x7f4c54199020 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c59f28640 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009cf0 tx=0x7f4c50011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64b7e263363a758 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c5a729640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 0x7f4c400a4200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0cb5b0 tx=0x7f4c4c06dc10 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c3ffff640 1 -- 172.21.15.45:0/4078188886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50009ea0 con 0x55e97f178900 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c3ffff640 1 -- 172.21.15.45:0/4078188886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c50004230 con 0x55e97f178900 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.496+0000 7f4c3ffff640 1 -- 172.21.15.45:0/4078188886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50019750 con 0x55e97f178900 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- 172.21.15.45:0/4078188886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 msgr2=0x55e97f185680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009cf0 tx=0x7f4c50011810 comp rx=0 tx=0).stop 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- 172.21.15.45:0/4078188886 shutdown_connections 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f178900 0x55e97f185680 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f187fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/4078188886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- 172.21.15.45:0/4078188886 >> 172.21.15.45:0/4078188886 conn(0x55e97f172990 msgr2=0x55e97f172d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- 172.21.15.45:0/4078188886 shutdown_connections 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- 172.21.15.45:0/4078188886 wait complete. 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.497+0000 7f4c5ed65740 1 -- start start 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 Processor -- start 2024-03-12T05:11:17.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f182970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f185bc0 0x55e97f182f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97edb3430 con 0x55e97f185bc0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97ed6fe00 con 0x55e97f177fe0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f064fd0 con 0x55e97f178900 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c5ed65740 10 client.0 _ll_register_callbacks cb 0x55e97f0bafd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f185bc0 0x55e97f182f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43070/0 (socket says 172.21.15.45:43070) 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 -- 172.21.15.45:0/3531755260 learned_addr learned my addr 172.21.15.45:0/3531755260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59f28640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f182970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 -- 172.21.15.45:0/3531755260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 msgr2=0x55e97f182970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f182970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 -- 172.21.15.45:0/3531755260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f185bc0 msgr2=0x55e97f182f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f185bc0 0x55e97f182f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59727640 1 -- 172.21.15.45:0/3531755260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c50009950 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.498+0000 7f4c59f28640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f182970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c59727640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c4402c680 tx=0x7f4c4402c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda4935c88fdbdf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4402cb60 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4c44004480 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e97eddf7e0 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c440035f0 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c5ed65740 10 client.25244 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e97f0de610 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e97eddf7e0 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.499+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4c4404d030 con 0x55e97f177fe0 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 10 client.25244 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 10 client.25244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:17.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 20 client.25244 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:17.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f195d50 0x55e97f198170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 4 client.25244 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:17.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e97ee1c720 con 0x55e97f195d50 2024-03-12T05:11:17.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c59f28640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f195d50 0x55e97f198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c59f28640 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f195d50 0x55e97f198170 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=188c38ae598a08c6 server_cookie=699c2aed711da5ba in_seq=0 out_seq=0 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.500+0000 7f4c3d7fa640 10 client.25244 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/3531755260 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e97ee1c720 con 0x55e97f195d50 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c3d7fa640 10 client.25244 handle_command_reply: tid=0 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 client.25244 shutdown 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 msgr2=0x55e97f1855e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c4402c680 tx=0x7f4c4402c610 comp rx=0 tx=0).stop 2024-03-12T05:11:17.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 shutdown_connections 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f195d50 0x55e97f198170 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f185bc0 0x55e97f182f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f182970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c3d7fa640 0 client.25244 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/3531755260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f177fe0 0x55e97f1855e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 >> 172.21.15.45:0/3531755260 conn(0x55e97f172990 msgr2=0x55e97f174730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 shutdown_connections 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 1 -- 172.21.15.45:0/3531755260 wait complete. 2024-03-12T05:11:17.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.501+0000 7f4c5ed65740 20 client.25244 trim_cache size 0 max 16384 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 Processor -- start 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 -- start start 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 0x55e97f1855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f181240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97eddf7e0 con 0x55e97f177fe0 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f16e0d0 con 0x55e97f178900 2024-03-12T05:11:17.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f1727f0 con 0x55e97f185bc0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56020/0 (socket says 172.21.15.45:56020) 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 0x55e97f1855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 -- 172.21.15.45:0/1840535421 learned_addr learned my addr 172.21.15.45:0/1840535421 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 -- 172.21.15.45:0/1840535421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 msgr2=0x55e97f181240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f181240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 -- 172.21.15.45:0/1840535421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 msgr2=0x55e97f1855e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.519+0000 7f4c59727640 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 0x55e97f1855e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c59727640 1 -- 172.21.15.45:0/1840535421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e97f07aac0 con 0x55e97f177fe0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 0x55e97f1855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c59727640 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000c00 tx=0x7f4c440053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dc78758b62c4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c3f7fe640 1 -- 172.21.15.45:0/1840535421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4402cb60 con 0x55e97f177fe0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c3f7fe640 1 -- 172.21.15.45:0/1840535421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c44049070 con 0x55e97f177fe0 2024-03-12T05:11:17.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.520+0000 7f4c3f7fe640 1 -- 172.21.15.45:0/1840535421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44003aa0 con 0x55e97f177fe0 2024-03-12T05:11:17.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1840535421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 msgr2=0x55e97f1850a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000c00 tx=0x7f4c440053a0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1840535421 shutdown_connections 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f185bc0 0x55e97f181240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f178900 0x55e97f1855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1840535421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f1850a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1840535421 >> 172.21.15.45:0/1840535421 conn(0x55e97f172990 msgr2=0x55e97f19caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1840535421 shutdown_connections 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1840535421 wait complete. 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 -- start start 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.521+0000 7f4c5ed65740 1 Processor -- start 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f189b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f185bc0 0x55e97f18a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f1ec310 con 0x55e97f177fe0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f1ec480 con 0x55e97f185bc0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e97f1ec640 con 0x55e97f178900 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 10 client.0 _ll_register_callbacks cb 0x7f4c500097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56032/0 (socket says 172.21.15.45:56032) 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 -- 172.21.15.45:0/1088428194 learned_addr learned my addr 172.21.15.45:0/1088428194 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f189b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f185bc0 0x55e97f18a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 -- 172.21.15.45:0/1088428194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 msgr2=0x55e97f189b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f189b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 -- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f185bc0 msgr2=0x55e97f18a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f185bc0 0x55e97f18a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 -- 172.21.15.45:0/1088428194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e97f09e570 con 0x55e97f177fe0 2024-03-12T05:11:17.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f189b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c59727640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55e97f183d00 tx=0x7f4c44004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d512e6cc45c100 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44005410 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4c44000e40 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.522+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c50004230 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c44005570 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 10 client.15756 fetch_fsmap learned FSMap version 28 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e97f0de610 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4c4406b080 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 28) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4c4404d040 con 0x55e97f177fe0 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 10 client.15756 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 20 client.15756 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:17.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f1ed970 0x55e97f1efd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 4 client.15756 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e97ee6b890 con 0x55e97f1ed970 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.523+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f1ed970 0x55e97f1efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c3d7fa640 10 client.15756 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c59f28640 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f1ed970 0x55e97f1efd60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa4460f6358eb35 server_cookie=33965185552b994e in_seq=0 out_seq=0 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c3d7fa640 1 -- 172.21.15.45:0/1088428194 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e97ee6b890 con 0x55e97f1ed970 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c3d7fa640 10 client.15756 handle_command_reply: tid=0 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 client.15756 shutdown 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 msgr2=0x55e97f189650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55e97f183d00 tx=0x7f4c44004ef0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 shutdown_connections 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e97f1ed970 0x55e97f1efd60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e97f185bc0 0x55e97f18a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e97f178900 0x55e97f189b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c3d7fa640 0 client.15756 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 --2- 172.21.15.45:0/1088428194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e97f177fe0 0x55e97f189650 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 >> 172.21.15.45:0/1088428194 conn(0x55e97f172990 msgr2=0x55e97f1811c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 shutdown_connections 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 1 -- 172.21.15.45:0/1088428194 wait complete. 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.524+0000 7f4c5ed65740 20 client.15756 trim_cache size 0 max 16384 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:11:17.527 INFO:teuthology.orchestra.run.smithi045.stdout: "f31bfe34-8ce7-47b3-97cb-6804e79b1c04": { 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "f31bfe34-8ce7-47b3-97cb-6804e79b1c04", 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:11:17.528 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 msgr2=0x7f4c400a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 0x7f4c400a4200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0cb5b0 tx=0x7f4c4c06dc10 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 msgr2=0x7f4c541a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00eb10 tx=0x7f4c4c00c2d0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 shutdown_connections 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4c400a1d40 0x7f4c400a4200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c54199020 0x7f4c541a35d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c54105340 0x7f4c54198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 --2- 172.21.15.45:0/946147855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4c54100270 0x7f4c54198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 >> 172.21.15.45:0/946147855 conn(0x7f4c54077230 msgr2=0x7f4c540fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 shutdown_connections 2024-03-12T05:11:17.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:17.525+0000 7f4c5c18c640 1 -- 172.21.15.45:0/946147855 wait complete. 2024-03-12T05:11:17.534 INFO:tasks.cephfs.filesystem:scrub status for tag:f31bfe34-8ce7-47b3-97cb-6804e79b1c04 - {'path': '/', 'tag': 'f31bfe34-8ce7-47b3-97cb-6804e79b1c04', 'options': 'recursive,force'} 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/: 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:total 2 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 25373287 Mar 12 05:10 . 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 25373287 Mar 12 05:10 .. 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 data 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data: 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:total 45 2024-03-12T05:11:17.616 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 . 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 25373287 Mar 12 05:10 .. 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 22087177 Mar 12 05:11 base 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 549376 Mar 12 05:11 global 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 log 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_commit_ts 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_dynshmem 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Mar 12 05:10 pg_hba.conf 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Mar 12 05:10 pg_ident.conf 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:11 pg_logical 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 pg_multixact 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_notify 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_replslot 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_serial 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_snapshots 2024-03-12T05:11:17.617 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_stat 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_stat_tmp 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 pg_subtrans 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_tblspc 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_twophase 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:10 PG_VERSION 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:10 pg_wal 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 pg_xact 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Mar 12 05:10 postgresql.auto.conf 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Mar 12 05:10 postgresql.conf 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base: 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:total 3 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 22087177 Mar 12 05:11 . 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:17.618 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 1 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 6160899 Mar 12 05:11 13493 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 13494 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/1: 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:total 7778 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 . 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Mar 12 05:11 .. 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 112 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 113 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 1247 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1247_vm 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:11 1249 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1249_fsm 2024-03-12T05:11:17.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1249_vm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:11 1259 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1259_vm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:11:17.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 1417 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 1418 2024-03-12T05:11:17.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 174 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 175 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2187 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2224 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2328 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2336 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2579 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:11:17.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2604 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606_vm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:11:17.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:11 2608 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2608_vm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2610 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2610_vm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2611 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2613 2024-03-12T05:11:17.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:11 2619 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2619_fsm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2619_vm 2024-03-12T05:11:17.625 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2620 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:11 2658 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 2659 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2662 2024-03-12T05:11:17.626 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2663 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2664 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2665 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2666 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2667 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2668 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2669 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2670 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:11 2673 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:11 2674 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2678 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2679 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2680 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:11:17.627 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2683 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2686 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2687 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:10 2688 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:10 2689 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2696 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2699 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2701 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2702 2024-03-12T05:11:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2703 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2704 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2754 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2755 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2830 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2832 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:11:17.629 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2834 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2836 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2995 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2996 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:11:17.630 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3118 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3119 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3256 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3257 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3258 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3350 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3351 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3379 2024-03-12T05:11:17.631 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3380 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3381 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3429 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3430 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3433 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3439 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3455 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:11:17.632 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3466 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3467 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3468 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3501 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3502 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3503 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3534 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3542 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3574 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3575 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3576 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3596 2024-03-12T05:11:17.633 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3597 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3598 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:11:17.634 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3606 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3607 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3997 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4143 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:11:17.635 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4145 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4147 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4149 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4151 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4153 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4155 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4157 2024-03-12T05:11:17.636 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:11:17.859 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4159 2024-03-12T05:11:17.859 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:11:17.859 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4161 2024-03-12T05:11:17.859 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4163 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4165 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4167 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4169 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4171 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4173 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 5002 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 548 2024-03-12T05:11:17.860 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 549 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6102 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6104 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6106 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6110 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6111 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6112 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6113 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6117 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 826 2024-03-12T05:11:17.861 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 827 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 828 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:total 7778 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 . 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Mar 12 05:11 .. 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 112 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 113 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 1247 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:11:17.862 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1247_vm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:11 1249 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1249_fsm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1249_vm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:11 1259 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1259_vm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:11:17.863 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1417 2024-03-12T05:11:17.864 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1418 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 174 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 175 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2187 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2224 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2328 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2336 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2579 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:11:17.865 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2604 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606_vm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:11:17.866 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:11 2608 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2608_vm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2610 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2610_vm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2611 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2613 2024-03-12T05:11:17.867 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:11:17.868 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:11 2619 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2619_fsm 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2619_vm 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2620 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:11 2658 2024-03-12T05:11:17.869 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 2659 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2662 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2663 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2664 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2665 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2666 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2667 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2668 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2669 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2670 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:11 2673 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:11 2674 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:11:17.870 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2678 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2679 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2680 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2683 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2686 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2687 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2688 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2689 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:11:17.871 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2696 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2699 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2701 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2702 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2703 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2704 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2754 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2755 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:11:17.872 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2830 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2832 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2834 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2836 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:11:17.873 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2995 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2996 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3118 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3119 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3256 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3257 2024-03-12T05:11:17.874 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3258 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3350 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3351 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3379 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3380 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3381 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:11:17.875 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3429 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3430 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3433 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3439 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:11:17.876 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3455 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3466 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3467 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3468 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3501 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3502 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3503 2024-03-12T05:11:17.877 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3534 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3542 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3574 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3575 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3576 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3596 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3597 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3598 2024-03-12T05:11:17.878 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:11:17.879 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3606 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3607 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3997 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4143 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4145 2024-03-12T05:11:17.880 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4147 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4149 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4151 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4153 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4155 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4157 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4159 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:11:17.881 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4161 2024-03-12T05:11:17.882 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:11:18.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4163 2024-03-12T05:11:18.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:11:18.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4165 2024-03-12T05:11:18.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:11:18.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4167 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4169 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4171 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4173 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 5002 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 548 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 549 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6102 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6104 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6106 2024-03-12T05:11:18.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6110 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6111 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6112 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6113 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6117 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 826 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 827 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 828 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:total 7778 2024-03-12T05:11:18.091 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 . 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Mar 12 05:11 .. 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 112 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 113 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 1247 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1247_vm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:11 1249 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1249_fsm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1249_vm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:11 1259 2024-03-12T05:11:18.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1259_vm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:11:18.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1417 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1418 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 174 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 175 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2187 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2224 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2328 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2336 2024-03-12T05:11:18.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2579 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:11:18.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2604 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606_vm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:11 2608 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2608_vm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:11:18.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2610 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2610_vm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2611 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2613 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:11:18.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:11 2619 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2619_fsm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2619_vm 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2620 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:11:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:11 2658 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 2659 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2662 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2663 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2664 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2665 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2666 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2667 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2668 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2669 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2670 2024-03-12T05:11:18.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:11 2673 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:11 2674 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2678 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2679 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2680 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2683 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2686 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2687 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2688 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2689 2024-03-12T05:11:18.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2696 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2699 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2701 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2702 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2703 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2704 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2754 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2755 2024-03-12T05:11:18.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2830 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2832 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2834 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2836 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:11:18.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2995 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2996 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:11:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3118 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3119 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3256 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3257 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3258 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3350 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3351 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3379 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3380 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3381 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:11:18.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3429 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3430 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3433 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3439 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3455 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3466 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3467 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3468 2024-03-12T05:11:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3501 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3502 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3503 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3534 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3542 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3574 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3575 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3576 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3596 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3597 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3598 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:11:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3606 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3607 2024-03-12T05:11:18.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3997 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4143 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4145 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4147 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4149 2024-03-12T05:11:18.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4151 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4153 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4155 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4157 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4159 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4161 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4163 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:11:18.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4165 2024-03-12T05:11:18.110 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:11:18.110 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4167 2024-03-12T05:11:18.126 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:11:18.126 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4169 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4171 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4173 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 5002 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 548 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 549 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6102 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6104 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6106 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6110 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6111 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6112 2024-03-12T05:11:18.127 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6113 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6117 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 826 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 827 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 828 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/global: 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:total 538 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 549376 Mar 12 05:11 . 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1213 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1213_fsm 2024-03-12T05:11:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1213_vm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1214 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1214_fsm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1214_vm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 1232 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 1233 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1260 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1260_fsm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1260_vm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1261 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1261_fsm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1261_vm 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1262 2024-03-12T05:11:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1262_fsm 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1262_vm 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2396 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2396_fsm 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2396_vm 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2397 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2671 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2672 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2676 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2677 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2694 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2695 2024-03-12T05:11:18.130 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2697 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2698 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2846 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2847 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2964 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2965 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2966 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2967 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3592 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3593 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4060 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4061 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4175 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4176 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4177 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4178 2024-03-12T05:11:18.131 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4181 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4182 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4183 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4184 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4185 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4186 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6000 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6001 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6002 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6100 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6114 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6115 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 pg_control 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/log: 2024-03-12T05:11:18.132 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.133 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 . 2024-03-12T05:11:18.175 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.175 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.175 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-03-12T05:11:18.175 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.175 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:total 3 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:11 . 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 mappings 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:11 replorigin_checkpoint 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 snapshots 2024-03-12T05:11:18.176 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:11 .. 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:11 .. 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:total 2 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 . 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 members 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 offsets 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 .. 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 0000 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 .. 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 0000 2024-03-12T05:11:18.178 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.179 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.180 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-03-12T05:11:18.191 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 0000 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.192 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:total 16386 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:10 . 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:11 000000010000000000000001 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 archive_status 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:10 .. 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:11:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 39480870 Mar 12 05:11 .. 2024-03-12T05:11:18.194 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 0000 2024-03-12T05:11:18.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-03-12T05:11:18.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-03-12T05:11:18.474 INFO:teuthology.orchestra.run.smithi045.stderr:dropping old tables... 2024-03-12T05:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-03-12T05:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-03-12T05:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-03-12T05:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-03-12T05:11:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:creating tables... 2024-03-12T05:11:18.529 INFO:teuthology.orchestra.run.smithi045.stderr:generating data (client-side)... 2024-03-12T05:11:18.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: pgmap v15: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 1.4 MiB/s wr, 48 op/s 2024-03-12T05:11:18.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:18 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: pgmap v15: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 1.4 MiB/s wr, 48 op/s 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:18 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: pgmap v15: 97 pgs: 97 active+clean; 53 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 1.4 MiB/s wr, 48 op/s 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:18.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:18 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.50 s, remaining 15.50 s) 2024-03-12T05:11:20.190 INFO:teuthology.orchestra.run.smithi045.stderr:200000 of 3200000 tuples (6%) done (elapsed 1.64 s, remaining 24.58 s) 2024-03-12T05:11:20.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:20 smithi045 ceph-mon[32174]: pgmap v16: 97 pgs: 97 active+clean; 56 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-03-12T05:11:20.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:20 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:20.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:20 smithi130 ceph-mon[35398]: pgmap v16: 97 pgs: 97 active+clean; 56 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-03-12T05:11:20.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:20 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:20.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:20 smithi116 ceph-mon[35486]: pgmap v16: 97 pgs: 97 active+clean; 56 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-03-12T05:11:20.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:20 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:11:21.336 INFO:teuthology.orchestra.run.smithi045.stderr:300000 of 3200000 tuples (9%) done (elapsed 2.78 s, remaining 26.92 s) 2024-03-12T05:11:22.063 INFO:teuthology.orchestra.run.smithi045.stderr:400000 of 3200000 tuples (12%) done (elapsed 3.51 s, remaining 24.58 s) 2024-03-12T05:11:22.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:22 smithi045 ceph-mon[32174]: pgmap v17: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-03-12T05:11:22.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:22 smithi130 ceph-mon[35398]: pgmap v17: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-03-12T05:11:22.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:22 smithi116 ceph-mon[35486]: pgmap v17: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.7 MiB/s wr, 135 op/s 2024-03-12T05:11:23.165 INFO:teuthology.orchestra.run.smithi045.stderr:500000 of 3200000 tuples (15%) done (elapsed 4.61 s, remaining 24.92 s) 2024-03-12T05:11:24.049 INFO:teuthology.orchestra.run.smithi045.stderr:600000 of 3200000 tuples (18%) done (elapsed 5.50 s, remaining 23.82 s) 2024-03-12T05:11:24.521 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:24 smithi130 ceph-mon[35398]: pgmap v18: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-03-12T05:11:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:24 smithi045 ceph-mon[32174]: pgmap v18: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-03-12T05:11:24.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:24 smithi116 ceph-mon[35486]: pgmap v18: 97 pgs: 97 active+clean; 58 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-03-12T05:11:25.195 INFO:teuthology.orchestra.run.smithi045.stderr:700000 of 3200000 tuples (21%) done (elapsed 6.64 s, remaining 23.73 s) 2024-03-12T05:11:26.297 INFO:teuthology.orchestra.run.smithi045.stderr:800000 of 3200000 tuples (25%) done (elapsed 7.75 s, remaining 23.24 s) 2024-03-12T05:11:26.546 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:26 smithi116 ceph-mon[35486]: pgmap v19: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 19 MiB/s wr, 129 op/s 2024-03-12T05:11:26.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:26 smithi045 ceph-mon[32174]: pgmap v19: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 19 MiB/s wr, 129 op/s 2024-03-12T05:11:26.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:26 smithi130 ceph-mon[35398]: pgmap v19: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 19 MiB/s wr, 129 op/s 2024-03-12T05:11:27.110 INFO:teuthology.orchestra.run.smithi045.stderr:900000 of 3200000 tuples (28%) done (elapsed 8.56 s, remaining 21.87 s) 2024-03-12T05:11:28.241 INFO:teuthology.orchestra.run.smithi045.stderr:1000000 of 3200000 tuples (31%) done (elapsed 9.69 s, remaining 21.32 s) 2024-03-12T05:11:28.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:28 smithi045 ceph-mon[32174]: pgmap v20: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 18 MiB/s wr, 110 op/s 2024-03-12T05:11:28.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:11:28.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:11:28.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:28 smithi130 ceph-mon[35398]: pgmap v20: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 18 MiB/s wr, 110 op/s 2024-03-12T05:11:28.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:11:28.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:11:28.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:28 smithi116 ceph-mon[35486]: pgmap v20: 97 pgs: 97 active+clean; 180 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 18 MiB/s wr, 110 op/s 2024-03-12T05:11:28.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:11:28.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:11:29.380 INFO:teuthology.orchestra.run.smithi045.stderr:1100000 of 3200000 tuples (34%) done (elapsed 10.83 s, remaining 20.67 s) 2024-03-12T05:11:29.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:29 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:11:29.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:29 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:11:29.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:29 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:11:30.172 INFO:teuthology.orchestra.run.smithi045.stderr:1200000 of 3200000 tuples (37%) done (elapsed 11.62 s, remaining 19.37 s) 2024-03-12T05:11:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:30 smithi045 ceph-mon[32174]: pgmap v21: 97 pgs: 97 active+clean; 285 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 35 MiB/s wr, 123 op/s 2024-03-12T05:11:30.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:30 smithi130 ceph-mon[35398]: pgmap v21: 97 pgs: 97 active+clean; 285 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 35 MiB/s wr, 123 op/s 2024-03-12T05:11:30.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:30 smithi116 ceph-mon[35486]: pgmap v21: 97 pgs: 97 active+clean; 285 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 35 MiB/s wr, 123 op/s 2024-03-12T05:11:31.275 INFO:teuthology.orchestra.run.smithi045.stderr:1300000 of 3200000 tuples (40%) done (elapsed 12.72 s, remaining 18.60 s) 2024-03-12T05:11:32.414 INFO:teuthology.orchestra.run.smithi045.stderr:1400000 of 3200000 tuples (43%) done (elapsed 13.86 s, remaining 17.82 s) 2024-03-12T05:11:32.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:32 smithi045 ceph-mon[32174]: pgmap v22: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 37 MiB/s wr, 81 op/s 2024-03-12T05:11:32.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:32 smithi130 ceph-mon[35398]: pgmap v22: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 37 MiB/s wr, 81 op/s 2024-03-12T05:11:32.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:32 smithi116 ceph-mon[35486]: pgmap v22: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 37 MiB/s wr, 81 op/s 2024-03-12T05:11:33.221 INFO:teuthology.orchestra.run.smithi045.stderr:1500000 of 3200000 tuples (46%) done (elapsed 14.67 s, remaining 16.63 s) 2024-03-12T05:11:34.341 INFO:teuthology.orchestra.run.smithi045.stderr:1600000 of 3200000 tuples (50%) done (elapsed 15.79 s, remaining 15.79 s) 2024-03-12T05:11:34.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:34 smithi045 ceph-mon[32174]: pgmap v23: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 MiB/s wr, 36 op/s 2024-03-12T05:11:34.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:34 smithi130 ceph-mon[35398]: pgmap v23: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 MiB/s wr, 36 op/s 2024-03-12T05:11:34.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:34 smithi116 ceph-mon[35486]: pgmap v23: 97 pgs: 97 active+clean; 293 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 MiB/s wr, 36 op/s 2024-03-12T05:11:35.080 INFO:teuthology.orchestra.run.smithi045.stderr:1700000 of 3200000 tuples (53%) done (elapsed 16.53 s, remaining 14.58 s) 2024-03-12T05:11:36.285 INFO:teuthology.orchestra.run.smithi045.stderr:1800000 of 3200000 tuples (56%) done (elapsed 17.73 s, remaining 13.79 s) 2024-03-12T05:11:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:36 smithi045 ceph-mon[32174]: pgmap v24: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 56 MiB/s wr, 50 op/s 2024-03-12T05:11:36.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:36 smithi130 ceph-mon[35398]: pgmap v24: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 56 MiB/s wr, 50 op/s 2024-03-12T05:11:36.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:36 smithi116 ceph-mon[35486]: pgmap v24: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 56 MiB/s wr, 50 op/s 2024-03-12T05:11:37.375 INFO:teuthology.orchestra.run.smithi045.stderr:1900000 of 3200000 tuples (59%) done (elapsed 18.82 s, remaining 12.88 s) 2024-03-12T05:11:38.197 INFO:teuthology.orchestra.run.smithi045.stderr:2000000 of 3200000 tuples (62%) done (elapsed 19.65 s, remaining 11.79 s) 2024-03-12T05:11:38.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:38 smithi045 ceph-mon[32174]: pgmap v25: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:38.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:38 smithi130 ceph-mon[35398]: pgmap v25: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:38.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:38 smithi116 ceph-mon[35486]: pgmap v25: 97 pgs: 97 active+clean; 405 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:39.288 INFO:teuthology.orchestra.run.smithi045.stderr:2100000 of 3200000 tuples (65%) done (elapsed 20.74 s, remaining 10.86 s) 2024-03-12T05:11:40.419 INFO:teuthology.orchestra.run.smithi045.stderr:2200000 of 3200000 tuples (68%) done (elapsed 21.87 s, remaining 9.94 s) 2024-03-12T05:11:40.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:40 smithi045 ceph-mon[32174]: pgmap v26: 97 pgs: 97 active+clean; 509 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 56 MiB/s wr, 42 op/s 2024-03-12T05:11:40.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:40 smithi130 ceph-mon[35398]: pgmap v26: 97 pgs: 97 active+clean; 509 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 56 MiB/s wr, 42 op/s 2024-03-12T05:11:40.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:40 smithi116 ceph-mon[35486]: pgmap v26: 97 pgs: 97 active+clean; 509 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 56 MiB/s wr, 42 op/s 2024-03-12T05:11:41.235 INFO:teuthology.orchestra.run.smithi045.stderr:2300000 of 3200000 tuples (71%) done (elapsed 22.68 s, remaining 8.88 s) 2024-03-12T05:11:42.396 INFO:teuthology.orchestra.run.smithi045.stderr:2400000 of 3200000 tuples (75%) done (elapsed 23.84 s, remaining 7.95 s) 2024-03-12T05:11:42.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:42 smithi045 ceph-mon[32174]: pgmap v27: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 MiB/s wr, 32 op/s 2024-03-12T05:11:42.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:42 smithi130 ceph-mon[35398]: pgmap v27: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 MiB/s wr, 32 op/s 2024-03-12T05:11:42.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:42 smithi116 ceph-mon[35486]: pgmap v27: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 MiB/s wr, 32 op/s 2024-03-12T05:11:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2500000 of 3200000 tuples (78%) done (elapsed 24.97 s, remaining 6.99 s) 2024-03-12T05:11:44.289 INFO:teuthology.orchestra.run.smithi045.stderr:2600000 of 3200000 tuples (81%) done (elapsed 25.74 s, remaining 5.94 s) 2024-03-12T05:11:44.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:44 smithi045 ceph-mon[32174]: pgmap v28: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:44.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:44 smithi130 ceph-mon[35398]: pgmap v28: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:44.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:44 smithi116 ceph-mon[35486]: pgmap v28: 97 pgs: 97 active+clean; 517 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:45.439 INFO:teuthology.orchestra.run.smithi045.stderr:2700000 of 3200000 tuples (84%) done (elapsed 26.89 s, remaining 4.98 s) 2024-03-12T05:11:46.205 INFO:teuthology.orchestra.run.smithi045.stderr:2800000 of 3200000 tuples (87%) done (elapsed 27.65 s, remaining 3.95 s) 2024-03-12T05:11:46.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:46 smithi045 ceph-mon[32174]: pgmap v29: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 58 MiB/s wr, 44 op/s 2024-03-12T05:11:46.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:46 smithi130 ceph-mon[35398]: pgmap v29: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 58 MiB/s wr, 44 op/s 2024-03-12T05:11:46.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:46 smithi116 ceph-mon[35486]: pgmap v29: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 58 MiB/s wr, 44 op/s 2024-03-12T05:11:47.336 INFO:teuthology.orchestra.run.smithi045.stderr:2900000 of 3200000 tuples (90%) done (elapsed 28.78 s, remaining 2.98 s) 2024-03-12T05:11:47.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:11:47.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 -- 172.21.15.45:0/2048577681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38100c20 msgr2=0x7feb38104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2048577681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38100c20 0x7feb38104c60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7feb28009930 tx=0x7feb2802bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 -- 172.21.15.45:0/2048577681 shutdown_connections 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2048577681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38100c20 0x7feb38104c60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2048577681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38105340 0x7feb3810cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2048577681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38100270 0x7feb38100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 -- 172.21.15.45:0/2048577681 >> 172.21.15.45:0/2048577681 conn(0x7feb38077230 msgr2=0x7feb38077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 -- 172.21.15.45:0/2048577681 shutdown_connections 2024-03-12T05:11:47.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.658+0000 7feb3e84d640 1 -- 172.21.15.45:0/2048577681 wait complete. 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 Processor -- start 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 -- start start 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 0x7feb3819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 0x7feb38199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3819a1d0 con 0x7feb38105340 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3819a340 con 0x7feb38199700 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb3e84d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3819a500 con 0x7feb38100270 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.659+0000 7feb377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50990/0 (socket says 172.21.15.45:50990) 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 -- 172.21.15.45:0/2442775068 learned_addr learned my addr 172.21.15.45:0/2442775068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb3cdea640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 0x7feb38199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb37fff640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 0x7feb3819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 -- 172.21.15.45:0/2442775068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 msgr2=0x7feb3819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 0x7feb3819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 -- 172.21.15.45:0/2442775068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 msgr2=0x7feb38199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 0x7feb38199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 -- 172.21.15.45:0/2442775068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb28009560 con 0x7feb38105340 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb3cdea640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 0x7feb38199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb37fff640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 0x7feb3819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb377fe640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7feb28009230 tx=0x7feb28005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb357fa640 1 -- 172.21.15.45:0/2442775068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb280043d0 con 0x7feb38105340 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb357fa640 1 -- 172.21.15.45:0/2442775068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb2802de90 con 0x7feb38105340 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb381a6440 con 0x7feb38105340 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb357fa640 1 -- 172.21.15.45:0/2442775068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb280052e0 con 0x7feb38105340 2024-03-12T05:11:47.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.660+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb381a6970 con 0x7feb38105340 2024-03-12T05:11:47.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.663+0000 7feb357fa640 1 -- 172.21.15.45:0/2442775068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7feb28003cc0 con 0x7feb38105340 2024-03-12T05:11:47.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.663+0000 7feb41419740 1 Processor -- start 2024-03-12T05:11:47.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 -- start start 2024-03-12T05:11:47.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb357fa640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 0x7feb240a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 0x55edf7ff3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 0x55edf7ffeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7f62280 con 0x55edf7ffc7b0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7f5f7b0 con 0x55edf7ff32b0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb41419740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7f5af90 con 0x55edf7ff3bd0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb3cdea640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 0x7feb240a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb357fa640 1 -- 172.21.15.45:0/2442775068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7feb280ef940 con 0x7feb38105340 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb3cdea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 0x55edf7ffeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb3cdea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 0x55edf7ffeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51004/0 (socket says 172.21.15.45:51004) 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb3cdea640 1 -- 172.21.15.45:0/1072273347 learned_addr learned my addr 172.21.15.45:0/1072273347 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 0x55edf7ff3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb377fe640 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 -- 172.21.15.45:0/1072273347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 msgr2=0x55edf7ffc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 -- 172.21.15.45:0/1072273347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 msgr2=0x55edf7ffeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 0x55edf7ffeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.664+0000 7feb37fff640 1 -- 172.21.15.45:0/1072273347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edf7f19570 con 0x55edf7ff32b0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.665+0000 7feb3cdea640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 0x7feb240a4200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb30009950 tx=0x7feb30009e40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.665+0000 7feb37fff640 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 0x55edf7ff3690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb20009c30 tx=0x7feb200117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22eb9ec6aa5c1fff server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.665+0000 7feb2e7fc640 1 -- 172.21.15.45:0/1072273347 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20012450 con 0x55edf7ff32b0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.665+0000 7feb2e7fc640 1 -- 172.21.15.45:0/1072273347 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb20012a60 con 0x55edf7ff32b0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.665+0000 7feb2e7fc640 1 -- 172.21.15.45:0/1072273347 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2000f870 con 0x55edf7ff32b0 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- 172.21.15.45:0/1072273347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 msgr2=0x55edf7ff3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 0x55edf7ff3690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb20009c30 tx=0x7feb200117b0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- 172.21.15.45:0/1072273347 shutdown_connections 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ffc7b0 0x55edf7ffeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 --2- 172.21.15.45:0/1072273347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff32b0 0x55edf7ff3690 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- 172.21.15.45:0/1072273347 >> 172.21.15.45:0/1072273347 conn(0x55edf7fed990 msgr2=0x55edf7fedd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- 172.21.15.45:0/1072273347 shutdown_connections 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- 172.21.15.45:0/1072273347 wait complete. 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 -- start start 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.666+0000 7feb41419740 1 Processor -- start 2024-03-12T05:11:47.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ff94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 0x55edf7ff9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7c2e430 con 0x55edf7ff32b0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7beae00 con 0x55edf7ffc7b0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7edffd0 con 0x55edf7ff3bd0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb41419740 10 client.0 _ll_register_callbacks cb 0x55edf7f35fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51012/0 (socket says 172.21.15.45:51012) 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 -- 172.21.15.45:0/563944583 learned_addr learned my addr 172.21.15.45:0/563944583 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb377fe640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ff94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb3cdea640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 0x55edf7ff9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 -- 172.21.15.45:0/563944583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 msgr2=0x55edf7ff94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ff94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 -- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 msgr2=0x55edf7ff9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 0x55edf7ff9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 -- 172.21.15.45:0/563944583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb20009860 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb377fe640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ff94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb3cdea640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 0x55edf7ff9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb37fff640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7feb20011cb0 tx=0x7feb20012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d386d85ca5b549b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.667+0000 7feb177fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20012760 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7feb20005040 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55edf7c5a7e0 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb200128c0 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 10 client.15768 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55edf7f59610 con 0x55edf7ff32b0 2024-03-12T05:11:47.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55edf7c5a7e0 con 0x55edf7ff32b0 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7feb20022070 con 0x55edf7ff32b0 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 10 client.15768 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 20 client.15768 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf800c920 0x55edf800ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 4 client.15768 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.668+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55edf7c97720 con 0x55edf800c920 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.669+0000 7feb377fe640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf800c920 0x55edf800ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.669+0000 7feb177fe640 10 client.15768 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.669+0000 7feb377fe640 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf800c920 0x55edf800ed40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd826d86c5c331ba server_cookie=757fea63ebecadad in_seq=0 out_seq=0 2024-03-12T05:11:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb177fe640 1 -- 172.21.15.45:0/563944583 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55edf7c97720 con 0x55edf800c920 2024-03-12T05:11:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb177fe640 10 client.15768 handle_command_reply: tid=0 2024-03-12T05:11:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 client.15768 shutdown 2024-03-12T05:11:47.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 msgr2=0x55edf7ffc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7feb20011cb0 tx=0x7feb20012ef0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 shutdown_connections 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf800c920 0x55edf800ed40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ffc7b0 0x55edf7ff9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.670+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff3bd0 0x55edf7ff94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb41419740 1 --2- 172.21.15.45:0/563944583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff32b0 0x55edf7ffc120 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb177fe640 0 client.15768 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 >> 172.21.15.45:0/563944583 conn(0x55edf7fed990 msgr2=0x55edf7fef730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 shutdown_connections 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb41419740 1 -- 172.21.15.45:0/563944583 wait complete. 2024-03-12T05:11:47.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.671+0000 7feb41419740 20 client.15768 trim_cache size 0 max 16384 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 Processor -- start 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 -- start start 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7fed990 0x55edf8008110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7ffbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7fee9e0 con 0x55edf7ff3bd0 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7ff2f10 con 0x55edf7fed990 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb41419740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf7ff2c80 con 0x55edf7ff32b0 2024-03-12T05:11:47.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51028/0 (socket says 172.21.15.45:51028) 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 -- 172.21.15.45:0/1105931474 learned_addr learned my addr 172.21.15.45:0/1105931474 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb377fe640 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7ffbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb37fff640 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7fed990 0x55edf8008110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 -- 172.21.15.45:0/1105931474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 msgr2=0x55edf7ffbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7ffbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 -- 172.21.15.45:0/1105931474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7fed990 msgr2=0x55edf8008110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7fed990 0x55edf8008110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.688+0000 7feb3cdea640 1 -- 172.21.15.45:0/1105931474 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edf7ef5ac0 con 0x55edf7ff3bd0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.689+0000 7feb3cdea640 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feb30009fd0 tx=0x7feb30004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c96b3cb61948e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.689+0000 7feb2dffb640 1 -- 172.21.15.45:0/1105931474 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30025070 con 0x55edf7ff3bd0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.689+0000 7feb2dffb640 1 -- 172.21.15.45:0/1105931474 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb30009af0 con 0x55edf7ff3bd0 2024-03-12T05:11:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.689+0000 7feb2dffb640 1 -- 172.21.15.45:0/1105931474 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3001e560 con 0x55edf7ff3bd0 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- 172.21.15.45:0/1105931474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 msgr2=0x55edf7ffc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7feb30009fd0 tx=0x7feb30004ab0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- 172.21.15.45:0/1105931474 shutdown_connections 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffc120 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7ffbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 --2- 172.21.15.45:0/1105931474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7fed990 0x55edf8008110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- 172.21.15.45:0/1105931474 >> 172.21.15.45:0/1105931474 conn(0x55edf7ffeec0 msgr2=0x55edf80136c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- 172.21.15.45:0/1105931474 shutdown_connections 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- 172.21.15.45:0/1105931474 wait complete. 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 -- start start 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.690+0000 7feb41419740 1 Processor -- start 2024-03-12T05:11:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7fffa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf8003f70 con 0x55edf7fed990 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf80004a0 con 0x55edf7ff3bd0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edf8000660 con 0x55edf7ff32b0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb41419740 10 client.0 _ll_register_callbacks cb 0x7feb200095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51036/0 (socket says 172.21.15.45:51036) 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 -- 172.21.15.45:0/389434977 learned_addr learned my addr 172.21.15.45:0/389434977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb3cdea640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb377fe640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7fffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 -- 172.21.15.45:0/389434977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 msgr2=0x55edf7fffa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7fffa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 -- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff3bd0 msgr2=0x55edf7ffff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 -- 172.21.15.45:0/389434977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3001a040 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb37fff640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feb200117f0 tx=0x7feb20012970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4035e65da477b7fd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.691+0000 7feb177fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2000fd50 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7feb20009a80 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb20012a60 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb2000feb0 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55edf7f1b6a0 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb20012a60 con 0x55edf7fed990 2024-03-12T05:11:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7feb20022040 con 0x55edf7fed990 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 20 client.15774 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf7ffc7b0 0x55edf8066140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 4 client.15774 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55edf7ce6890 con 0x55edf7ffc7b0 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.692+0000 7feb377fe640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf7ffc7b0 0x55edf8066140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb177fe640 10 client.15774 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb377fe640 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf7ffc7b0 0x55edf8066140 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7956272ca8f51eb7 server_cookie=22f89a3b52c96d39 in_seq=0 out_seq=0 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb177fe640 1 -- 172.21.15.45:0/389434977 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55edf7ce6890 con 0x55edf7ffc7b0 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb177fe640 10 client.15774 handle_command_reply: tid=0 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 client.15774 shutdown 2024-03-12T05:11:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 msgr2=0x55edf80036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7feb200117f0 tx=0x7feb20012970 comp rx=0 tx=0).stop 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 shutdown_connections 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55edf7ffc7b0 0x55edf8066140 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edf7ff3bd0 0x55edf7ffff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55edf7ff32b0 0x55edf7fffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb177fe640 0 client.15774 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 --2- 172.21.15.45:0/389434977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55edf7fed990 0x55edf80036a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 >> 172.21.15.45:0/389434977 conn(0x55edf7ffeec0 msgr2=0x55edf7ff7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.693+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 shutdown_connections 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb41419740 1 -- 172.21.15.45:0/389434977 wait complete. 2024-03-12T05:11:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb41419740 20 client.15774 trim_cache size 0 max 16384 2024-03-12T05:11:47.697 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:47.697 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:11:47.697 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:11:47.697 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 msgr2=0x7feb240a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 0x7feb240a4200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feb30009950 tx=0x7feb30009e40 comp rx=0 tx=0).stop 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 msgr2=0x7feb3819fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7feb28009230 tx=0x7feb28005e20 comp rx=0 tx=0).stop 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 shutdown_connections 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb240a1d40 0x7feb240a4200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb38105340 0x7feb3819fb80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb38199700 0x7feb38199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 --2- 172.21.15.45:0/2442775068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb38100270 0x7feb3819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 >> 172.21.15.45:0/2442775068 conn(0x7feb38077230 msgr2=0x7feb380ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 shutdown_connections 2024-03-12T05:11:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.694+0000 7feb3e84d640 1 -- 172.21.15.45:0/2442775068 wait complete. 2024-03-12T05:11:47.703 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:11:47.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:11:47.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 -- 172.21.15.45:0/2554756484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441053c0 msgr2=0x7f384410cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 --2- 172.21.15.45:0/2554756484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441053c0 0x7f384410cf90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f383c009cd0 tx=0x7f383c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 -- 172.21.15.45:0/2554756484 shutdown_connections 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 --2- 172.21.15.45:0/2554756484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441053c0 0x7f384410cf90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 --2- 172.21.15.45:0/2554756484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3844100ca0 0x7f3844104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 --2- 172.21.15.45:0/2554756484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f38441002f0 0x7f38441006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 -- 172.21.15.45:0/2554756484 >> 172.21.15.45:0/2554756484 conn(0x7f3844077180 msgr2=0x7f3844077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 -- 172.21.15.45:0/2554756484 shutdown_connections 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 -- 172.21.15.45:0/2554756484 wait complete. 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.825+0000 7f384a6c3640 1 Processor -- start 2024-03-12T05:11:47.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 -- start start 2024-03-12T05:11:47.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38441053c0 0x7f3844198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 0x7f38441a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441a3c90 con 0x7f38441002f0 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441a3e00 con 0x7f38441053c0 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f384a6c3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441a3fc0 con 0x7f3844199040 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51056/0 (socket says 172.21.15.45:51056) 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 -- 172.21.15.45:0/3675331522 learned_addr learned my addr 172.21.15.45:0/3675331522 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f38437fe640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38441053c0 0x7f3844198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3848c60640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 0x7f38441a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 -- 172.21.15.45:0/3675331522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 msgr2=0x7f38441a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 0x7f38441a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 -- 172.21.15.45:0/3675331522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38441053c0 msgr2=0x7f3844198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.826+0000 7f3843fff640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38441053c0 0x7f3844198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:47.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f3843fff640 1 -- 172.21.15.45:0/3675331522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c009900 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f3848c60640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 0x7f38441a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f3843fff640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9f0 tx=0x7f383400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834009720 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f383400cd80 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38441a42a0 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834010430 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.827+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38441a4760 con 0x7f38441002f0 2024-03-12T05:11:47.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.828+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3844067f70 con 0x7f38441002f0 2024-03-12T05:11:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.829+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f3834005020 con 0x7f38441002f0 2024-03-12T05:11:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.830+0000 7f38417fa640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 0x7f382c0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.830+0000 7f38437fe640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 0x7f382c0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.830+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3834014070 con 0x7f38441002f0 2024-03-12T05:11:47.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.830+0000 7f38437fe640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 0x7f382c0a4180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3844068e90 tx=0x7f3838005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:47.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:47.835+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3834096b00 con 0x7f38441002f0 2024-03-12T05:11:48.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.105+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3844068930 con 0x7f38441002f0 2024-03-12T05:11:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.106+0000 7f38417fa640 1 -- 172.21.15.45:0/3675331522 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11403 (secure 0 0 0) 0x7f383409a1c0 con 0x7f38441002f0 2024-03-12T05:11:48.103 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:11:48.103 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:11:18.408486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:11:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 29 2024-03-12T05:11:48.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 msgr2=0x7f382c0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 0x7f382c0a4180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3844068e90 tx=0x7f3838005f90 comp rx=0 tx=0).stop 2024-03-12T05:11:48.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 msgr2=0x7f3844198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f383400e9f0 tx=0x7f383400edf0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 shutdown_connections 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f382c0a1cc0 0x7f382c0a4180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3844199040 0x7f38441a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38441053c0 0x7f3844198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 --2- 172.21.15.45:0/3675331522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38441002f0 0x7f3844198520 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 >> 172.21.15.45:0/3675331522 conn(0x7f3844077180 msgr2=0x7f38440fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 shutdown_connections 2024-03-12T05:11:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.108+0000 7f384a6c3640 1 -- 172.21.15.45:0/3675331522 wait complete. 2024-03-12T05:11:48.117 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:48.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:11:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 -- 172.21.15.45:0/2732108861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 msgr2=0x7fafc8100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 --2- 172.21.15.45:0/2732108861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8100640 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009930 tx=0x7fafb802bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 -- 172.21.15.45:0/2732108861 shutdown_connections 2024-03-12T05:11:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 --2- 172.21.15.45:0/2732108861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 --2- 172.21.15.45:0/2732108861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8100c10 0x7fafc8104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 --2- 172.21.15.45:0/2732108861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8100640 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.235+0000 7fafcd423640 1 -- 172.21.15.45:0/2732108861 >> 172.21.15.45:0/2732108861 conn(0x7fafc8077180 msgr2=0x7fafc8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.236+0000 7fafcd423640 1 -- 172.21.15.45:0/2732108861 shutdown_connections 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.236+0000 7fafcd423640 1 -- 172.21.15.45:0/2732108861 wait complete. 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.236+0000 7fafcd423640 1 Processor -- start 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.236+0000 7fafcd423640 1 -- start start 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc8102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 0x7fafc81a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81a5ed0 con 0x7fafc8100260 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81a6040 con 0x7fafc8105330 2024-03-12T05:11:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81a6200 con 0x7fafc8102770 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51078/0 (socket says 172.21.15.45:51078) 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 -- 172.21.15.45:0/210722047 learned_addr learned my addr 172.21.15.45:0/210722047 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc67fc640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc8102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc77fe640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 0x7fafc81a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 -- 172.21.15.45:0/210722047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 msgr2=0x7fafc81a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 0x7fafc81a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 -- 172.21.15.45:0/210722047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 msgr2=0x7fafc8102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc8102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 -- 172.21.15.45:0/210722047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8009560 con 0x7fafc8100260 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc77fe640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 0x7fafc81a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc67fc640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005ab0 tx=0x7fafb80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafaffff640 1 -- 172.21.15.45:0/210722047 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80044c0 con 0x7fafc8100260 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafaffff640 1 -- 172.21.15.45:0/210722047 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafb8003aa0 con 0x7fafc8100260 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc81a6480 con 0x7fafc8100260 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafaffff640 1 -- 172.21.15.45:0/210722047 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8005540 con 0x7fafc8100260 2024-03-12T05:11:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.237+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81a6990 con 0x7fafc8100260 2024-03-12T05:11:48.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.240+0000 7fafaffff640 1 -- 172.21.15.45:0/210722047 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fafb8003c10 con 0x7fafc8100260 2024-03-12T05:11:48.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.240+0000 7fafcfff9740 1 Processor -- start 2024-03-12T05:11:48.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.240+0000 7fafcfff9740 1 -- start start 2024-03-12T05:11:48.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafaffff640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 0x7fafb00a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52e0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff5292960 con 0x557ff52d7250 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff5246310 con 0x557ff52d7b70 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafcfff9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff5243840 con 0x557ff52e0750 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc77fe640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 0x7fafb00a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafaffff640 1 -- 172.21.15.45:0/210722047 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fafb802de40 con 0x7fafc8100260 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51084/0 (socket says 172.21.15.45:51084) 2024-03-12T05:11:48.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 -- 172.21.15.45:0/4267723142 learned_addr learned my addr 172.21.15.45:0/4267723142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc67fc640 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52e0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 -- 172.21.15.45:0/4267723142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 msgr2=0x557ff52e2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 -- 172.21.15.45:0/4267723142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 msgr2=0x557ff52e0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52e0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc6ffd640 1 -- 172.21.15.45:0/4267723142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ff5201060 con 0x557ff52d7250 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc77fe640 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.241+0000 7fafc77fe640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 0x7fafb00a4180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafc0005c40 tx=0x7fafc0009980 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006010 tx=0x7fafb8094e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c522aad7b97dfd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafad7fa640 1 -- 172.21.15.45:0/4267723142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80bf070 con 0x557ff52d7250 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafad7fa640 1 -- 172.21.15.45:0/4267723142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafb80056a0 con 0x557ff52d7250 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafad7fa640 1 -- 172.21.15.45:0/4267723142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80c55a0 con 0x557ff52d7250 2024-03-12T05:11:48.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafcfff9740 1 -- 172.21.15.45:0/4267723142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 msgr2=0x557ff52d7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.242+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafb8006010 tx=0x7fafb8094e60 comp rx=0 tx=0).stop 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- 172.21.15.45:0/4267723142 shutdown_connections 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52e0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4267723142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52d7630 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- 172.21.15.45:0/4267723142 >> 172.21.15.45:0/4267723142 conn(0x557ff52d1980 msgr2=0x557ff52d1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- 172.21.15.45:0/4267723142 shutdown_connections 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- 172.21.15.45:0/4267723142 wait complete. 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- start start 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 Processor -- start 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52ddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52de170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff523f020 con 0x557ff52d7250 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff4f12430 con 0x557ff52d7b70 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff4ecee00 con 0x557ff52e0750 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafcfff9740 10 client.0 _ll_register_callbacks cb 0x557ff51e6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51088/0 (socket says 172.21.15.45:51088) 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafc6ffd640 1 -- 172.21.15.45:0/1418250571 learned_addr learned my addr 172.21.15.45:0/1418250571 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafc67fc640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52ddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.243+0000 7fafc77fe640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 -- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 msgr2=0x557ff52de170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52de170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 -- 172.21.15.45:0/1418250571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 msgr2=0x557ff52ddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52ddc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 -- 172.21.15.45:0/1418250571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81a6990 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafb80b9260 tx=0x7fafb80bcd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd6259ce126f70f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafaaff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80bf030 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fafb80c0070 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ff4f3e7e0 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb80bce00 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafcfff9740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557ff523d6a0 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.244+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557ff4f3e7e0 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fafb803c070 con 0x557ff52d7250 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 10 client.15792 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 20 client.15792 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff52f0440 0x557ff52f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 4 client.15792 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ff4f7b720 con 0x557ff52f0440 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafc67fc640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff52f0440 0x557ff52f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafaaff5640 10 client.15792 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.245+0000 7fafc67fc640 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff52f0440 0x557ff52f2830 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2c29ec6e381b5265 server_cookie=bc6190cb614ecbbf in_seq=0 out_seq=0 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafaaff5640 1 -- 172.21.15.45:0/1418250571 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557ff4f7b720 con 0x557ff52f0440 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafaaff5640 10 client.15792 handle_command_reply: tid=0 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 client.15792 shutdown 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 msgr2=0x557ff52dd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fafb80b9260 tx=0x7fafb80bcd00 comp rx=0 tx=0).stop 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 shutdown_connections 2024-03-12T05:11:48.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff52f0440 0x557ff52f2830 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52de170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7b70 0x557ff52ddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 --2- 172.21.15.45:0/1418250571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7250 0x557ff52dd6f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 >> 172.21.15.45:0/1418250571 conn(0x557ff52d1980 msgr2=0x557ff52d36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafaaff5640 0 client.15792 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 shutdown_connections 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 1 -- 172.21.15.45:0/1418250571 wait complete. 2024-03-12T05:11:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.246+0000 7fafcfff9740 20 client.15792 trim_cache size 0 max 16384 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 Processor -- start 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 -- start start 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52d7250 0x557ff52dfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52e0750 0x557ff52da970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff51c4130 con 0x557ff52d7b70 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff52de860 con 0x557ff52e0750 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafcfff9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff4f3e7e0 con 0x557ff52d7250 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51098/0 (socket says 172.21.15.45:51098) 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafc67fc640 1 -- 172.21.15.45:0/2453573520 learned_addr learned my addr 172.21.15.45:0/2453573520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52d7250 0x557ff52dfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.260+0000 7fafc77fe640 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52e0750 0x557ff52da970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 -- 172.21.15.45:0/2453573520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52d7250 msgr2=0x557ff52dfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52d7250 0x557ff52dfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 -- 172.21.15.45:0/2453573520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52e0750 msgr2=0x557ff52da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52e0750 0x557ff52da970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 -- 172.21.15.45:0/2453573520 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ff51d9c20 con 0x557ff52d7b70 2024-03-12T05:11:48.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafc67fc640 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fafbc004700 tx=0x7fafbc01b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b16a6256ba9afc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafacff9640 1 -- 172.21.15.45:0/2453573520 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc016040 con 0x557ff52d7b70 2024-03-12T05:11:48.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafacff9640 1 -- 172.21.15.45:0/2453573520 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafbc00ed90 con 0x557ff52d7b70 2024-03-12T05:11:48.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.261+0000 7fafacff9640 1 -- 172.21.15.45:0/2453573520 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc004a00 con 0x557ff52d7b70 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- 172.21.15.45:0/2453573520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 msgr2=0x557ff52ddf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fafbc004700 tx=0x7fafbc01b680 comp rx=0 tx=0).stop 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- 172.21.15.45:0/2453573520 shutdown_connections 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52ddf90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52e0750 0x557ff52da970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 --2- 172.21.15.45:0/2453573520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52d7250 0x557ff52dfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- 172.21.15.45:0/2453573520 >> 172.21.15.45:0/2453573520 conn(0x557ff52d1980 msgr2=0x557ff52f71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- 172.21.15.45:0/2453573520 shutdown_connections 2024-03-12T05:11:48.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- 172.21.15.45:0/2453573520 wait complete. 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 -- start start 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.262+0000 7fafcfff9740 1 Processor -- start 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 0x557ff52e7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff52e6420 con 0x557ff52d7b70 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff52e6560 con 0x557ff52d7250 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ff52e6720 con 0x557ff52e0750 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51112/0 (socket says 172.21.15.45:51112) 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 -- 172.21.15.45:0/4235948554 learned_addr learned my addr 172.21.15.45:0/4235948554 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafcfff9740 10 client.0 _ll_register_callbacks cb 0x7fafb80b90a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 0x557ff52e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc77fe640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 -- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 msgr2=0x557ff52e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 -- 172.21.15.45:0/4235948554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 msgr2=0x557ff52e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 0x557ff52e7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 -- 172.21.15.45:0/4235948554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc00be50 con 0x557ff52d7b70 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 0x557ff52e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc77fe640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafc67fc640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b820 tx=0x7fafbc01b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116beb5ac2a1ecd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.263+0000 7fafaaff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc016050 con 0x557ff52d7b70 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fafbc00e8e0 con 0x557ff52d7b70 2024-03-12T05:11:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafb80056a0 con 0x557ff52d7b70 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafbc00eaa0 con 0x557ff52d7b70 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 10 client.15798 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557ff51ff8c0 con 0x557ff52d7b70 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fafb80056a0 con 0x557ff52d7b70 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fafbc0256b0 con 0x557ff52d7b70 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 10 client.15798 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 10 client.15798 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 20 client.15798 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff5347e90 0x557ff534a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 4 client.15798 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557ff4fca890 con 0x557ff5347e90 2024-03-12T05:11:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.264+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff5347e90 0x557ff534a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafc6ffd640 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff5347e90 0x557ff534a2b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1223fbfa28581886 server_cookie=5eab285cf322a996 in_seq=0 out_seq=0 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafaaff5640 10 client.15798 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafaaff5640 1 -- 172.21.15.45:0/4235948554 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557ff4fca890 con 0x557ff5347e90 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafaaff5640 10 client.15798 handle_command_reply: tid=0 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 client.15798 shutdown 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 msgr2=0x557ff52e59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00b820 tx=0x7fafbc01b7b0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 shutdown_connections 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557ff5347e90 0x557ff534a2b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557ff52d7b70 0x557ff52e59a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ff52e0750 0x557ff52e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafaaff5640 0 client.15798 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 --2- 172.21.15.45:0/4235948554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557ff52d7250 0x557ff52e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 >> 172.21.15.45:0/4235948554 conn(0x557ff52d1980 msgr2=0x557ff52d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 shutdown_connections 2024-03-12T05:11:48.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 1 -- 172.21.15.45:0/4235948554 wait complete. 2024-03-12T05:11:48.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.265+0000 7fafcfff9740 20 client.15798 trim_cache size 0 max 16384 2024-03-12T05:11:48.270 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 msgr2=0x7fafb00a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 0x7fafb00a4180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafc0005c40 tx=0x7fafc0009980 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 msgr2=0x7fafc8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005ab0 tx=0x7fafb80040c0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 shutdown_connections 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fafb00a1cc0 0x7fafb00a4180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc8102770 0x7fafc81a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fafc8105330 0x7fafc8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 --2- 172.21.15.45:0/210722047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafc8100260 0x7fafc8101cf0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 >> 172.21.15.45:0/210722047 conn(0x7fafc8077180 msgr2=0x7fafc80686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 shutdown_connections 2024-03-12T05:11:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.266+0000 7fafcd423640 1 -- 172.21.15.45:0/210722047 wait complete. 2024-03-12T05:11:48.275 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:48.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:11:48.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:48 smithi045 ceph-mon[32174]: pgmap v30: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 30 op/s 2024-03-12T05:11:48.367 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:48 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3675331522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:48.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/921651584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 msgr2=0x7fa0e0100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/921651584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e0100640 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009930 tx=0x7fa0d402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:48.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/921651584 shutdown_connections 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/921651584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e0105330 0x7fa0e010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/921651584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0100c10 0x7fa0e0104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/921651584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e0100640 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/921651584 >> 172.21.15.45:0/921651584 conn(0x7fa0e0077180 msgr2=0x7fa0e0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/921651584 shutdown_connections 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/921651584 wait complete. 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 Processor -- start 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.401+0000 7fa0e6cb5640 1 -- start start 2024-03-12T05:11:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 0x7fa0e0197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e019bfa0 0x7fa0e019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e019c990 con 0x7fa0e0100260 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e019cb00 con 0x7fa0e019bfa0 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e019cc70 con 0x7fa0e0105330 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51126/0 (socket says 172.21.15.45:51126) 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 -- 172.21.15.45:0/2292710235 learned_addr learned my addr 172.21.15.45:0/2292710235 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0dffff640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 0x7fa0e0197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 -- 172.21.15.45:0/2292710235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 msgr2=0x7fa0e0197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e5252640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e019bfa0 0x7fa0e019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 0x7fa0e0197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 -- 172.21.15.45:0/2292710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e019bfa0 msgr2=0x7fa0e019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e019bfa0 0x7fa0e019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 -- 172.21.15.45:0/2292710235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d4009560 con 0x7fa0e0100260 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0dffff640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 0x7fa0e0197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009230 tx=0x7fa0d4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0ddffb640 1 -- 172.21.15.45:0/2292710235 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004350 con 0x7fa0e0100260 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e01a4250 con 0x7fa0e0100260 2024-03-12T05:11:48.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0ddffb640 1 -- 172.21.15.45:0/2292710235 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0d402de90 con 0x7fa0e0100260 2024-03-12T05:11:48.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.402+0000 7fa0ddffb640 1 -- 172.21.15.45:0/2292710235 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4005220 con 0x7fa0e0100260 2024-03-12T05:11:48.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.403+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e01a46b0 con 0x7fa0e0100260 2024-03-12T05:11:48.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 Processor -- start 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 -- start start 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0ddffb640 1 -- 172.21.15.45:0/2292710235 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fa0d4003c30 con 0x7fa0e0100260 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f46b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f474210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f426960 con 0x55e89f474750 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f3da310 con 0x55e89f46bb70 2024-03-12T05:11:48.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e9967740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f3d7840 con 0x55e89f46b250 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51134/0 (socket says 172.21.15.45:51134) 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0dffff640 1 -- 172.21.15.45:0/1757795563 learned_addr learned my addr 172.21.15.45:0/1757795563 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e5252640 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f46b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.406+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f474210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 -- 172.21.15.45:0/1757795563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 msgr2=0x55e89f46b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f46b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 -- 172.21.15.45:0/1757795563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 msgr2=0x55e89f474210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f474210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 -- 172.21.15.45:0/1757795563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e89f395060 con 0x55e89f474750 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0ddffb640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 0x7fa0d00a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0e5252640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 0x7fa0d00a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0ddffb640 1 -- 172.21.15.45:0/2292710235 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa0d40efa80 con 0x7fa0e0100260 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0dffff640 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009cb0 tx=0x7fa0cc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9694140fcd1462a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0caffd640 1 -- 172.21.15.45:0/1757795563 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc004020 con 0x55e89f474750 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0caffd640 1 -- 172.21.15.45:0/1757795563 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0cc0035f0 con 0x55e89f474750 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.407+0000 7fa0caffd640 1 -- 172.21.15.45:0/1757795563 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0126b0 con 0x55e89f474750 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e5252640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 0x7fa0d00a4180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005fd0 tx=0x7fa0d8009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 -- 172.21.15.45:0/1757795563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 msgr2=0x55e89f476b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009cb0 tx=0x7fa0cc0117b0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 -- 172.21.15.45:0/1757795563 shutdown_connections 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f476b40 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f474210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 --2- 172.21.15.45:0/1757795563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f46b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 -- 172.21.15.45:0/1757795563 >> 172.21.15.45:0/1757795563 conn(0x55e89f465980 msgr2=0x55e89f465d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 -- 172.21.15.45:0/1757795563 shutdown_connections 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.408+0000 7fa0e9967740 1 -- 172.21.15.45:0/1757795563 wait complete. 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 -- start start 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 Processor -- start 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f471760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f471ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f3d3020 con 0x55e89f474750 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f0a6430 con 0x55e89f46bb70 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f062e00 con 0x55e89f46b250 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e9967740 10 client.0 _ll_register_callbacks cb 0x55e89f37acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e5252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e5252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51136/0 (socket says 172.21.15.45:51136) 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e5252640 1 -- 172.21.15.45:0/3005478525 learned_addr learned my addr 172.21.15.45:0/3005478525 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f471760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.409+0000 7fa0dffff640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f471ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 -- 172.21.15.45:0/3005478525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 msgr2=0x55e89f471760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f471760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 -- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 msgr2=0x55e89f471ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f471ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 -- 172.21.15.45:0/3005478525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc0098e0 con 0x55e89f474750 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e5252640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80126d0 tx=0x7fa0d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4a6f903a9dfe93 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0bffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801b870 con 0x55e89f474750 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e89f0d27e0 con 0x55e89f474750 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa0d801b9d0 con 0x55e89f474750 2024-03-12T05:11:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0d801bb90 con 0x55e89f474750 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e9967740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.410+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e89f3d16a0 con 0x55e89f474750 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0d8032080 con 0x55e89f474750 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa0d802c030 con 0x55e89f474750 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 10 client.15810 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 20 client.15810 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f484c30 0x55e89f487050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 4 client.15810 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e89f10f720 con 0x55e89f484c30 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f484c30 0x55e89f487050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0bffff640 10 client.15810 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.411+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f484c30 0x55e89f487050 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=744f2788201dc4dc server_cookie=c7e2aea3c6e78348 in_seq=0 out_seq=0 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0bffff640 1 -- 172.21.15.45:0/3005478525 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e89f10f720 con 0x55e89f484c30 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0bffff640 10 client.15810 handle_command_reply: tid=0 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 client.15810 shutdown 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 msgr2=0x55e89f4721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80126d0 tx=0x7fa0d8004a40 comp rx=0 tx=0).stop 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 shutdown_connections 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f484c30 0x55e89f487050 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4721e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f471ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 --2- 172.21.15.45:0/3005478525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f471760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0bffff640 0 client.15810 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:48.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 >> 172.21.15.45:0/3005478525 conn(0x55e89f465980 msgr2=0x55e89f4676d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 shutdown_connections 2024-03-12T05:11:48.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.412+0000 7fa0e9967740 1 -- 172.21.15.45:0/3005478525 wait complete. 2024-03-12T05:11:48.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.413+0000 7fa0e9967740 20 client.15810 trim_cache size 0 max 16384 2024-03-12T05:11:48.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 Processor -- start 2024-03-12T05:11:48.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 -- start start 2024-03-12T05:11:48.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f472000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f474750 0x55e89f46e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f358130 con 0x55e89f46b250 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f46acc0 con 0x55e89f46bb70 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e9967740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f48c300 con 0x55e89f474750 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e4a51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e4a51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51142/0 (socket says 172.21.15.45:51142) 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.427+0000 7fa0e4a51640 1 -- 172.21.15.45:0/498246428 learned_addr learned my addr 172.21.15.45:0/498246428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e5252640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f474750 0x55e89f46e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0dffff640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f472000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 -- 172.21.15.45:0/498246428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f474750 msgr2=0x55e89f46e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f474750 0x55e89f46e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 -- 172.21.15.45:0/498246428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 msgr2=0x55e89f472000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f472000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 -- 172.21.15.45:0/498246428 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e89f36dc20 con 0x55e89f46b250 2024-03-12T05:11:48.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0dffff640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f472000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40b89d0 tx=0x7fa0d40a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c6e72a9fcb4060 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0ca7fc640 1 -- 172.21.15.45:0/498246428 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40f7040 con 0x55e89f46b250 2024-03-12T05:11:48.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0ca7fc640 1 -- 172.21.15.45:0/498246428 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0d40bc340 con 0x55e89f46b250 2024-03-12T05:11:48.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.428+0000 7fa0ca7fc640 1 -- 172.21.15.45:0/498246428 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40c06c0 con 0x55e89f46b250 2024-03-12T05:11:48.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- 172.21.15.45:0/498246428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 msgr2=0x55e89f473e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40b89d0 tx=0x7fa0d40a7040 comp rx=0 tx=0).stop 2024-03-12T05:11:48.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- 172.21.15.45:0/498246428 shutdown_connections 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f474750 0x55e89f46e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f472000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 --2- 172.21.15.45:0/498246428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f46b250 0x55e89f473e90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- 172.21.15.45:0/498246428 >> 172.21.15.45:0/498246428 conn(0x55e89f465980 msgr2=0x55e89f46ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- 172.21.15.45:0/498246428 shutdown_connections 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- 172.21.15.45:0/498246428 wait complete. 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.429+0000 7fa0e9967740 1 -- start start 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 Processor -- start 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f47a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f47adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f4db6a0 con 0x55e89f474750 2024-03-12T05:11:48.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f4db7e0 con 0x55e89f46bb70 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89f4db9a0 con 0x55e89f46b250 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e9967740 10 client.0 _ll_register_callbacks cb 0x7fa0cc009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51158/0 (socket says 172.21.15.45:51158) 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 -- 172.21.15.45:0/4163101575 learned_addr learned my addr 172.21.15.45:0/4163101575 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0dffff640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f47adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f47a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 -- 172.21.15.45:0/4163101575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 msgr2=0x55e89f47a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f47a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 -- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 msgr2=0x55e89f47adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f47adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e5252640 1 -- 172.21.15.45:0/4163101575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e01a46b0 con 0x55e89f474750 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f47a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.430+0000 7fa0dffff640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f47adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e5252640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80045b0 tx=0x7fa0d8009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb26cec7f50aeaf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d801bb40 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa0d801bca0 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0cc0035f0 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0d801bea0 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 10 client.15816 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e89f3938c0 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0cc0035f0 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa0d802e800 con 0x55e89f474750 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 10 client.15816 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 10 client.15816 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:11:48.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 20 client.15816 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f4dd160 0x55e89f4df580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 4 client.15816 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.431+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e89f15e890 con 0x55e89f4dd160 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f4dd160 0x55e89f4df580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e4a51640 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f4dd160 0x55e89f4df580 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=127c79730a2397c server_cookie=ddfb99ce6a729561 in_seq=0 out_seq=0 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0bffff640 10 client.15816 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0bffff640 1 -- 172.21.15.45:0/4163101575 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e89f15e890 con 0x55e89f4dd160 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0bffff640 10 client.15816 handle_command_reply: tid=0 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 client.15816 shutdown 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 msgr2=0x55e89f4db160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80045b0 tx=0x7fa0d8009cd0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 shutdown_connections 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e89f4dd160 0x55e89f4df580 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e89f474750 0x55e89f4db160 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89f46bb70 0x55e89f47adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 --2- 172.21.15.45:0/4163101575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e89f46b250 0x55e89f47a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 >> 172.21.15.45:0/4163101575 conn(0x55e89f465980 msgr2=0x55e89f4642f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.432+0000 7fa0bffff640 0 client.15816 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 shutdown_connections 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e9967740 1 -- 172.21.15.45:0/4163101575 wait complete. 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e9967740 20 client.15816 trim_cache size 0 max 16384 2024-03-12T05:11:48.436 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 msgr2=0x7fa0d00a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 0x7fa0d00a4180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005fd0 tx=0x7fa0d8009500 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 msgr2=0x7fa0e01976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009230 tx=0x7fa0d4005dd0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 shutdown_connections 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0d00a1cc0 0x7fa0d00a4180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e019bfa0 0x7fa0e019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e0105330 0x7fa0e0197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 --2- 172.21.15.45:0/2292710235 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0e0100260 0x7fa0e01976e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 >> 172.21.15.45:0/2292710235 conn(0x7fa0e0077180 msgr2=0x7fa0e0068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.433+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 shutdown_connections 2024-03-12T05:11:48.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.434+0000 7fa0e6cb5640 1 -- 172.21.15.45:0/2292710235 wait complete. 2024-03-12T05:11:48.442 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:48.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:11:48.503 INFO:teuthology.orchestra.run.smithi045.stderr:3000000 of 3200000 tuples (93%) done (elapsed 29.95 s, remaining 2.00 s) 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- 172.21.15.45:0/2394787706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 msgr2=0x7f3aec10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- 172.21.15.45:0/2394787706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec10e000 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009cd0 tx=0x7f3ae402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- 172.21.15.45:0/2394787706 shutdown_connections 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- 172.21.15.45:0/2394787706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec10e000 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- 172.21.15.45:0/2394787706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec068f80 0x7f3aec105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- 172.21.15.45:0/2394787706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 0x7f3aec0689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- 172.21.15.45:0/2394787706 >> 172.21.15.45:0/2394787706 conn(0x7f3aec077180 msgr2=0x7f3aec077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- 172.21.15.45:0/2394787706 shutdown_connections 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- 172.21.15.45:0/2394787706 wait complete. 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 Processor -- start 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 -- start start 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 0x7f3aec100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.567+0000 7f3af4081640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec100ac0 0x7f3aec1a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1a5e30 con 0x7f3aec106430 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1a5fa0 con 0x7f3aec100ac0 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1a6160 con 0x7f3aec0685d0 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51178/0 (socket says 172.21.15.45:51178) 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 -- 172.21.15.45:0/4062174176 learned_addr learned my addr 172.21.15.45:0/4062174176 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af261e640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec100ac0 0x7f3aec1a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 0x7f3aec100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 -- 172.21.15.45:0/4062174176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 msgr2=0x7f3aec100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 0x7f3aec100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 -- 172.21.15.45:0/4062174176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec100ac0 msgr2=0x7f3aec1a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec100ac0 0x7f3aec1a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 -- 172.21.15.45:0/4062174176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4009900 con 0x7f3aec106430 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af161c640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00d770 tx=0x7f3adc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3ae2ffd640 1 -- 172.21.15.45:0/4062174176 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc009c30 con 0x7f3aec106430 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3ae2ffd640 1 -- 172.21.15.45:0/4062174176 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3adc00de80 con 0x7f3aec106430 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec1a6440 con 0x7f3aec106430 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3ae2ffd640 1 -- 172.21.15.45:0/4062174176 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc00f600 con 0x7f3aec106430 2024-03-12T05:11:48.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.568+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec1a6950 con 0x7f3aec106430 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3ae2ffd640 1 -- 172.21.15.45:0/4062174176 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f3adc010460 con 0x7f3aec106430 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 Processor -- start 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 -- start start 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0effb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0eaf960 con 0x55c2c0ef4250 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0e63310 con 0x55c2c0efd750 2024-03-12T05:11:48.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0e60840 con 0x55c2c0ef4b70 2024-03-12T05:11:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3ae2ffd640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 0x7f3ad80a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.571+0000 7f3af261e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51188/0 (socket says 172.21.15.45:51188) 2024-03-12T05:11:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 -- 172.21.15.45:0/27917457 learned_addr learned my addr 172.21.15.45:0/27917457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af161c640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0effb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 0x7f3ad80a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3ae2ffd640 1 -- 172.21.15.45:0/4062174176 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3adc0cd770 con 0x7f3aec106430 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 -- 172.21.15.45:0/27917457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4b70 msgr2=0x55c2c0efd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 -- 172.21.15.45:0/27917457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 msgr2=0x55c2c0effb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0effb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 -- 172.21.15.45:0/27917457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2c0e1e060 con 0x55c2c0ef4250 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af161c640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0effb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4004af0 tx=0x7f3ae4034a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fd891eb074f4e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3af261e640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 0x7f3ad80a41b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c700 tx=0x7f3ae4049000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3ad7fff640 1 -- 172.21.15.45:0/27917457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4048070 con 0x55c2c0ef4250 2024-03-12T05:11:48.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.572+0000 7f3ad7fff640 1 -- 172.21.15.45:0/27917457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3ae4032380 con 0x55c2c0ef4250 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3ad7fff640 1 -- 172.21.15.45:0/27917457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4004040 con 0x55c2c0ef4250 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 -- 172.21.15.45:0/27917457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 msgr2=0x55c2c0ef4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4004af0 tx=0x7f3ae4034a60 comp rx=0 tx=0).stop 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 -- 172.21.15.45:0/27917457 shutdown_connections 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0effb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/27917457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0ef4630 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 -- 172.21.15.45:0/27917457 >> 172.21.15.45:0/27917457 conn(0x55c2c0eee980 msgr2=0x55c2c0eeed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 -- 172.21.15.45:0/27917457 shutdown_connections 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.573+0000 7f3af6c4d740 1 -- 172.21.15.45:0/27917457 wait complete. 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 -- start start 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 Processor -- start 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0efb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0e5c020 con 0x55c2c0ef4b70 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0b2f430 con 0x55c2c0efd750 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0aebe00 con 0x55c2c0ef4250 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.574+0000 7f3af6c4d740 10 client.0 _ll_register_callbacks cb 0x55c2c0e03cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51204/0 (socket says 172.21.15.45:51204) 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af161c640 1 -- 172.21.15.45:0/3297026849 learned_addr learned my addr 172.21.15.45:0/3297026849 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0efb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 -- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 msgr2=0x55c2c0efa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efa7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 -- 172.21.15.45:0/3297026849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 msgr2=0x55c2c0efad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 -- 172.21.15.45:0/3297026849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec10e810 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af161c640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3af261e640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0efb250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c630 tx=0x7f3ae402c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=688716ec57c917ae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.575+0000 7f3ad57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4048050 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2c0b98720 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3ae4004040 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ae4004200 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3af6c4d740 10 client.25310 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c2c0e5a6a0 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.576+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c2c0b98720 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3ae405d070 con 0x55c2c0efd750 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 10 client.25310 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 10 client.25310 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:11:48.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 20 client.25310 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f0d7a0 0x55c2c0f0fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 4 client.25310 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2c0be7890 con 0x55c2c0f0d7a0 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.577+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f0d7a0 0x55c2c0f0fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.578+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f0d7a0 0x55c2c0f0fbc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6894ab084eab0f2 server_cookie=4bb089b73781c822 in_seq=0 out_seq=0 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.578+0000 7f3ad57fa640 10 client.25310 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3ad57fa640 1 -- 172.21.15.45:0/3297026849 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c2c0be7890 con 0x55c2c0f0d7a0 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3ad57fa640 10 client.25310 handle_command_reply: tid=0 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 client.25310 shutdown 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 msgr2=0x55c2c0efb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0efb250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c630 tx=0x7f3ae402c770 comp rx=0 tx=0).stop 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 shutdown_connections 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f0d7a0 0x55c2c0f0fbc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0efad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0efb250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3ad57fa640 0 client.25310 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3297026849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efa7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 >> 172.21.15.45:0/3297026849 conn(0x55c2c0eee980 msgr2=0x55c2c0ef06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 shutdown_connections 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3297026849 wait complete. 2024-03-12T05:11:48.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.579+0000 7f3af6c4d740 20 client.25310 trim_cache size 0 max 16384 2024-03-12T05:11:48.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 Processor -- start 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 -- start start 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0ef7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0de1130 con 0x55c2c0ef4b70 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0ee0a40 con 0x55c2c0efd750 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.596+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0f09550 con 0x55c2c0ef4250 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51218/0 (socket says 172.21.15.45:51218) 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 -- 172.21.15.45:0/3177153934 learned_addr learned my addr 172.21.15.45:0/3177153934 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af261e640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0ef7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 -- 172.21.15.45:0/3177153934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 msgr2=0x55c2c0efcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efcf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 -- 172.21.15.45:0/3177153934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 msgr2=0x55c2c0ef7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0ef7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 -- 172.21.15.45:0/3177153934 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2c0df6c20 con 0x55c2c0ef4b70 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af261e640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0ef7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3af161c640 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3adc09a0e0 tx=0x7f3adc065930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4908532c5ef921e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3ad77fe640 1 -- 172.21.15.45:0/3177153934 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc09a850 con 0x55c2c0ef4b70 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3ad77fe640 1 -- 172.21.15.45:0/3177153934 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3adc09e2a0 con 0x55c2c0ef4b70 2024-03-12T05:11:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.597+0000 7f3ad77fe640 1 -- 172.21.15.45:0/3177153934 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0967c0 con 0x55c2c0ef4b70 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3177153934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 msgr2=0x55c2c0ef7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3adc09a0e0 tx=0x7f3adc065930 comp rx=0 tx=0).stop 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3177153934 shutdown_connections 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4b70 0x55c2c0ef7a70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0efd750 0x55c2c0ef7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/3177153934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0ef4250 0x55c2c0efcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3177153934 >> 172.21.15.45:0/3177153934 conn(0x55c2c0eee980 msgr2=0x55c2c0ef90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3177153934 shutdown_connections 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.598+0000 7f3af6c4d740 1 -- 172.21.15.45:0/3177153934 wait complete. 2024-03-12T05:11:48.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 -- start start 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 Processor -- start 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0ef4b70 0x55c2c0f03c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0efd750 0x55c2c0f63cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0f641f0 con 0x55c2c0ef4250 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0f64330 con 0x55c2c0ef4b70 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2c0f644c0 con 0x55c2c0efd750 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af6c4d740 10 client.0 _ll_register_callbacks cb 0x7f3ae4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af1e1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af1e1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51220/0 (socket says 172.21.15.45:51220) 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af1e1d640 1 -- 172.21.15.45:0/874528607 learned_addr learned my addr 172.21.15.45:0/874528607 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.599+0000 7f3af161c640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0ef4b70 0x55c2c0f03c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af261e640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0efd750 0x55c2c0f63cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 -- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0efd750 msgr2=0x55c2c0f63cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0efd750 0x55c2c0f63cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 -- 172.21.15.45:0/874528607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0ef4b70 msgr2=0x55c2c0f03c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0ef4b70 0x55c2c0f03c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 -- 172.21.15.45:0/874528607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2c0e5a6a0 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af1e1d640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x55c2c0f10470 tx=0x7f3ae801b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a239d5a37d0d1f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3ad57fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8017040 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3ae8004230 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2c0b5b7e0 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ae801ba90 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af6c4d740 10 client.15834 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f3aec1a6950 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.600+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c2c0b5b7e0 con 0x55c2c0ef4250 2024-03-12T05:11:48.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3ae8003a20 con 0x55c2c0ef4250 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 10 client.15834 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 10 client.15834 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 20 client.15834 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f65d80 0x55c2c0f68170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 4 client.15834 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c2c0b07700 con 0x55c2c0f65d80 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af161c640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f65d80 0x55c2c0f68170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3af161c640 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f65d80 0x55c2c0f68170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c45b116adc3fe22f server_cookie=8ad55e3050e223d3 in_seq=0 out_seq=0 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.601+0000 7f3ad57fa640 10 client.15834 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:48.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3ad57fa640 1 -- 172.21.15.45:0/874528607 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c2c0b07700 con 0x55c2c0f65d80 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3ad57fa640 10 client.15834 handle_command_reply: tid=0 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 client.15834 shutdown 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 msgr2=0x55c2c0f03750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x55c2c0f10470 tx=0x7f3ae801b670 comp rx=0 tx=0).stop 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 shutdown_connections 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55c2c0f65d80 0x55c2c0f68170 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c2c0efd750 0x55c2c0f63cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2c0ef4b70 0x55c2c0f03c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 --2- 172.21.15.45:0/874528607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c2c0ef4250 0x55c2c0f03750 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3ad57fa640 0 client.15834 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 >> 172.21.15.45:0/874528607 conn(0x55c2c0eee980 msgr2=0x55c2c0efb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 shutdown_connections 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 1 -- 172.21.15.45:0/874528607 wait complete. 2024-03-12T05:11:48.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.602+0000 7f3af6c4d740 20 client.15834 trim_cache size 0 max 16384 2024-03-12T05:11:48.606 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 msgr2=0x7f3ad80a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 0x7f3ad80a41b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c700 tx=0x7f3ae4049000 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 msgr2=0x7f3aec100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00d770 tx=0x7f3adc00dc30 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 shutdown_connections 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ad80a1cf0 0x7f3ad80a41b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3aec106430 0x7f3aec100580 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aec100ac0 0x7f3aec1a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 --2- 172.21.15.45:0/4062174176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aec0685d0 0x7f3aec100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 >> 172.21.15.45:0/4062174176 conn(0x7f3aec077180 msgr2=0x7f3aec0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 shutdown_connections 2024-03-12T05:11:48.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.603+0000 7f3af4081640 1 -- 172.21.15.45:0/4062174176 wait complete. 2024-03-12T05:11:48.611 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:48.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:11:48.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:48 smithi130 ceph-mon[35398]: pgmap v30: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 30 op/s 2024-03-12T05:11:48.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:48 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3675331522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:48.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 -- 172.21.15.45:0/3976082009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70081009e0 msgr2=0x7f700810cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 --2- 172.21.15.45:0/3976082009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70081009e0 0x7f700810cea0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7000009ca0 tx=0x7f700002c180 comp rx=0 tx=0).stop 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 -- 172.21.15.45:0/3976082009 shutdown_connections 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 --2- 172.21.15.45:0/3976082009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70081009e0 0x7f700810cea0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 --2- 172.21.15.45:0/3976082009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 0x7f70081004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 --2- 172.21.15.45:0/3976082009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7008106100 0x7f70081064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 -- 172.21.15.45:0/3976082009 >> 172.21.15.45:0/3976082009 conn(0x7f7008077110 msgr2=0x7f7008077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 -- 172.21.15.45:0/3976082009 shutdown_connections 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.733+0000 7f700d320640 1 -- 172.21.15.45:0/3976082009 wait complete. 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 Processor -- start 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 -- start start 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 0x7f70081975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70081009e0 0x7f7008197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700819c460 con 0x7f7008106100 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700819c5d0 con 0x7f70081009e0 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f700d320640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700819c790 con 0x7f7008100040 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f70077fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f70077fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51234/0 (socket says 172.21.15.45:51234) 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f70077fe640 1 -- 172.21.15.45:0/370787674 learned_addr learned my addr 172.21.15.45:0/370787674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f7006ffd640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 0x7f70081975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.734+0000 7f70067fc640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70081009e0 0x7f7008197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 -- 172.21.15.45:0/370787674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 msgr2=0x7f70081975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 0x7f70081975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 -- 172.21.15.45:0/370787674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70081009e0 msgr2=0x7f7008197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70081009e0 0x7f7008197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 -- 172.21.15.45:0/370787674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000009900 con 0x7f7008106100 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f70077fe640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f700002c790 tx=0x7f7000034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f6feffff640 1 -- 172.21.15.45:0/370787674 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003d80 con 0x7f7008106100 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700819ca10 con 0x7f7008106100 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f6feffff640 1 -- 172.21.15.45:0/370787674 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7000032b30 con 0x7f7008106100 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f6feffff640 1 -- 172.21.15.45:0/370787674 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000046c0 con 0x7f7008106100 2024-03-12T05:11:48.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.735+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700819cce0 con 0x7f7008106100 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f6feffff640 1 -- 172.21.15.45:0/370787674 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f700003b040 con 0x7f7008106100 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 Processor -- start 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 -- start start 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca32630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca3db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc9ed960 con 0x5648cca32b70 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc9a1310 con 0x5648cca3b750 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f700ffcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc99e840 con 0x5648cca32250 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f7006ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.738+0000 7f6feffff640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 0x7f6ff00a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51244/0 (socket says 172.21.15.45:51244) 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 -- 172.21.15.45:0/2433094888 learned_addr learned my addr 172.21.15.45:0/2433094888 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f70077fe640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 0x7f6ff00a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f70067fc640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca32630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f70067fc640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca3db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 -- 172.21.15.45:0/2433094888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 msgr2=0x5648cca32630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f6feffff640 1 -- 172.21.15.45:0/370787674 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f70000f0290 con 0x7f7008106100 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca32630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 -- 172.21.15.45:0/2433094888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 msgr2=0x5648cca3db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca3db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 -- 172.21.15.45:0/2433094888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648cc95c060 con 0x5648cca32b70 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f70067fc640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca32630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800bd10 tx=0x7f6ff80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f5d2a001d82419 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f70077fe640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 0x7f6ff00a4180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70000ed9d0 tx=0x7f70000a7af0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f6fed7fa640 1 -- 172.21.15.45:0/2433094888 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8004030 con 0x5648cca32b70 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f6fed7fa640 1 -- 172.21.15.45:0/2433094888 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ff8015070 con 0x5648cca32b70 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.739+0000 7f6fed7fa640 1 -- 172.21.15.45:0/2433094888 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80116a0 con 0x5648cca32b70 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 -- 172.21.15.45:0/2433094888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 msgr2=0x5648cca3b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800bd10 tx=0x7f6ff80107a0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 -- 172.21.15.45:0/2433094888 shutdown_connections 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca3b210 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca3db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2433094888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca32630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 -- 172.21.15.45:0/2433094888 >> 172.21.15.45:0/2433094888 conn(0x5648cca2c980 msgr2=0x5648cca2cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 -- 172.21.15.45:0/2433094888 shutdown_connections 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.740+0000 7f700ffcc740 1 -- 172.21.15.45:0/2433094888 wait complete. 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 -- start start 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 Processor -- start 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 0x5648cca38c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca39190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc99a020 con 0x5648cca32250 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc66d430 con 0x5648cca3b750 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc629e00 con 0x5648cca32b70 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f700ffcc740 10 client.0 _ll_register_callbacks cb 0x5648cc941cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51246/0 (socket says 172.21.15.45:51246) 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 -- 172.21.15.45:0/2624815719 learned_addr learned my addr 172.21.15.45:0/2624815719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 0x5648cca38c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca39190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 -- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 msgr2=0x5648cca38c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 0x5648cca38c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 -- 172.21.15.45:0/2624815719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 msgr2=0x5648cca39190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca39190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f70067fc640 1 -- 172.21.15.45:0/2624815719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff800b940 con 0x5648cca32250 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.741+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 0x5648cca38c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f70067fc640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0045f0 tx=0x7f6ffc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89ef4c1b535d73d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc003dc0 con 0x5648cca32250 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6ffc025430 con 0x5648cca32250 2024-03-12T05:11:48.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648cc6997e0 con 0x5648cca32250 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ffc0255f0 con 0x5648cca32250 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5648cc9986a0 con 0x5648cca32250 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5648cc6997e0 con 0x5648cca32250 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6ffc02e640 con 0x5648cca32250 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 10 client.15846 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 20 client.15846 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648cca4df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 4 client.15846 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.742+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648cc6d6720 con 0x5648cca4bb40 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.743+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648cca4df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.743+0000 7f6feaff5640 10 client.15846 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.743+0000 7f7006ffd640 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648cca4df60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0b10c306c6dbe13 server_cookie=668724d14ae948d3 in_seq=0 out_seq=0 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f6feaff5640 1 -- 172.21.15.45:0/2624815719 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5648cc6d6720 con 0x5648cca4bb40 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f6feaff5640 10 client.15846 handle_command_reply: tid=0 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 client.15846 shutdown 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 msgr2=0x5648cca38710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0045f0 tx=0x7f6ffc005850 comp rx=0 tx=0).stop 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 shutdown_connections 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648cca4df60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f6feaff5640 0 client.15846 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca39190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32b70 0x5648cca38c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 --2- 172.21.15.45:0/2624815719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca38710 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 >> 172.21.15.45:0/2624815719 conn(0x5648cca2c980 msgr2=0x5648cca2e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 shutdown_connections 2024-03-12T05:11:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 1 -- 172.21.15.45:0/2624815719 wait complete. 2024-03-12T05:11:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.744+0000 7f700ffcc740 20 client.15846 trim_cache size 0 max 16384 2024-03-12T05:11:48.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:48 smithi116 ceph-mon[35486]: pgmap v30: 97 pgs: 97 active+clean; 643 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 30 op/s 2024-03-12T05:11:48.749 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:48 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3675331522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:11:48.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 Processor -- start 2024-03-12T05:11:48.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 -- start start 2024-03-12T05:11:48.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 0x5648cca528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca3b750 0x5648cca35970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cc91f130 con 0x5648cca32250 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cca34340 con 0x5648cca32b70 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f700ffcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cca285c0 con 0x5648cca3b750 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f70067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f70067fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51248/0 (socket says 172.21.15.45:51248) 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.758+0000 7f70067fc640 1 -- 172.21.15.45:0/1601860140 learned_addr learned my addr 172.21.15.45:0/1601860140 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f7006ffd640 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 0x5648cca528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 -- 172.21.15.45:0/1601860140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca3b750 msgr2=0x5648cca35970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca3b750 0x5648cca35970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 -- 172.21.15.45:0/1601860140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 msgr2=0x5648cca528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 0x5648cca528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 -- 172.21.15.45:0/1601860140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648cc934c20 con 0x5648cca32250 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f7006ffd640 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 0x5648cca528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f70067fc640 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0059f0 tx=0x7f6ffc0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88f0dab6d5c76c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f6fecff9640 1 -- 172.21.15.45:0/1601860140 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc009800 con 0x5648cca32250 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f6fecff9640 1 -- 172.21.15.45:0/1601860140 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ffc003dc0 con 0x5648cca32250 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.759+0000 7f6fecff9640 1 -- 172.21.15.45:0/1601860140 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc005330 con 0x5648cca32250 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- 172.21.15.45:0/1601860140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 msgr2=0x5648cca3ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0059f0 tx=0x7f6ffc0096a0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- 172.21.15.45:0/1601860140 shutdown_connections 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca3b750 0x5648cca35970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca32b70 0x5648cca528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1601860140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32250 0x5648cca3ae40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- 172.21.15.45:0/1601860140 >> 172.21.15.45:0/1601860140 conn(0x5648cca2c980 msgr2=0x5648cca27a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- 172.21.15.45:0/1601860140 shutdown_connections 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- 172.21.15.45:0/1601860140 wait complete. 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 -- start start 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.760+0000 7f700ffcc740 1 Processor -- start 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca42b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca41600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648cca41cd0 con 0x5648cca32b70 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ccaa2050 con 0x5648cca3b750 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ccaa2190 con 0x5648cca32250 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f700ffcc740 10 client.0 _ll_register_callbacks cb 0x7f6ff800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51256/0 (socket says 172.21.15.45:51256) 2024-03-12T05:11:48.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 -- 172.21.15.45:0/1470747969 learned_addr learned my addr 172.21.15.45:0/1470747969 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f70067fc640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca42b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 -- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 msgr2=0x5648cca42b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca42b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 -- 172.21.15.45:0/1470747969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 msgr2=0x5648cca41600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f70067fc640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca41600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca41600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 -- 172.21.15.45:0/1470747969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648cc9986a0 con 0x5648cca32b70 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f70067fc640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca41600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f7006ffd640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6ff801a000 tx=0x7f6ff800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9fdfd10bc06c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f6feaff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.761+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80035f0 con 0x5648cca32b70 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6ff802ae40 con 0x5648cca32b70 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ff8015070 con 0x5648cca32b70 2024-03-12T05:11:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ff8003750 con 0x5648cca32b70 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 10 client.15852 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5648cc95a8c0 con 0x5648cca32b70 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ff8015070 con 0x5648cca32b70 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6ff802a150 con 0x5648cca32b70 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 10 client.15852 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 10 client.15852 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 20 client.15852 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648ccaa59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 4 client.15852 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5648cc725890 con 0x5648cca4bb40 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.762+0000 7f70067fc640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648ccaa59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f70067fc640 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648ccaa59a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18a00de78561d654 server_cookie=e5f54c897f163a43 in_seq=0 out_seq=0 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f6feaff5640 10 client.15852 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f6feaff5640 1 -- 172.21.15.45:0/1470747969 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5648cc725890 con 0x5648cca4bb40 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f6feaff5640 10 client.15852 handle_command_reply: tid=0 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 client.15852 shutdown 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 msgr2=0x5648cca40ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6ff801a000 tx=0x7f6ff800b8b0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 shutdown_connections 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5648cca32b70 0x5648cca40ec0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5648cca4bb40 0x5648ccaa59a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5648cca3b750 0x5648cca41600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f6feaff5640 0 client.15852 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 --2- 172.21.15.45:0/1470747969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5648cca32250 0x5648cca42b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 >> 172.21.15.45:0/1470747969 conn(0x5648cca2c980 msgr2=0x5648cca528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 shutdown_connections 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 1 -- 172.21.15.45:0/1470747969 wait complete. 2024-03-12T05:11:48.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.763+0000 7f700ffcc740 20 client.15852 trim_cache size 0 max 16384 2024-03-12T05:11:48.768 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 msgr2=0x7f6ff00a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 0x7f6ff00a4180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70000ed9d0 tx=0x7f70000a7af0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 msgr2=0x7f700819be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f700002c790 tx=0x7f7000034830 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 shutdown_connections 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ff00a1cc0 0x7f6ff00a4180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7008106100 0x7f700819be60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70081009e0 0x7f7008197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 --2- 172.21.15.45:0/370787674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7008100040 0x7f70081975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 >> 172.21.15.45:0/370787674 conn(0x7f7008077110 msgr2=0x7f70080ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 shutdown_connections 2024-03-12T05:11:48.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.764+0000 7f700d320640 1 -- 172.21.15.45:0/370787674 wait complete. 2024-03-12T05:11:48.774 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:11:48.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:11:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.895+0000 7fcf1953a640 1 -- 172.21.15.45:0/964891221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14068f80 msgr2=0x7fcf14105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.895+0000 7fcf1953a640 1 --2- 172.21.15.45:0/964891221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14068f80 0x7fcf14105d50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009930 tx=0x7fcefc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- 172.21.15.45:0/964891221 shutdown_connections 2024-03-12T05:11:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- 172.21.15.45:0/964891221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14068f80 0x7fcf14105d50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- 172.21.15.45:0/964891221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf14106430 0x7fcf1410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- 172.21.15.45:0/964891221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf140685d0 0x7fcf140689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- 172.21.15.45:0/964891221 >> 172.21.15.45:0/964891221 conn(0x7fcf14077180 msgr2=0x7fcf14077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- 172.21.15.45:0/964891221 shutdown_connections 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- 172.21.15.45:0/964891221 wait complete. 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 Processor -- start 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- start start 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 0x7fcf14197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf1419bfd0 0x7fcf1419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419c9c0 con 0x7fcf14106430 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419cb30 con 0x7fcf140685d0 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf1953a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1419cca0 con 0x7fcf1419bfd0 2024-03-12T05:11:48.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.896+0000 7fcf0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51278/0 (socket says 172.21.15.45:51278) 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 -- 172.21.15.45:0/367775238 learned_addr learned my addr 172.21.15.45:0/367775238 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf137fe640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf1419bfd0 0x7fcf1419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 0x7fcf14197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 -- 172.21.15.45:0/367775238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf1419bfd0 msgr2=0x7fcf1419c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf1419bfd0 0x7fcf1419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 -- 172.21.15.45:0/367775238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 msgr2=0x7fcf14197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 0x7fcf14197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 -- 172.21.15.45:0/367775238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009560 con 0x7fcf14106430 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 0x7fcf14197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005ab0 tx=0x7fcefc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf10ff9640 1 -- 172.21.15.45:0/367775238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0044c0 con 0x7fcf14106430 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf10ff9640 1 -- 172.21.15.45:0/367775238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcefc003aa0 con 0x7fcf14106430 2024-03-12T05:11:48.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf141a4280 con 0x7fcf14106430 2024-03-12T05:11:48.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf10ff9640 1 -- 172.21.15.45:0/367775238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0054d0 con 0x7fcf14106430 2024-03-12T05:11:48.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.897+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf141a4730 con 0x7fcf14106430 2024-03-12T05:11:48.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.899+0000 7fcf10ff9640 1 -- 172.21.15.45:0/367775238 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fcefc0035f0 con 0x7fcf14106430 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 Processor -- start 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 -- start start 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf10ff9640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 0x7fcf000a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf10ff9640 1 -- 172.21.15.45:0/367775238 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcefc02de40 con 0x7fcf14106430 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf137fe640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 0x7fcf000a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80ac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8067960 con 0x561fd80ac8a0 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd801b310 con 0x561fd80b9b60 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.900+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8018840 con 0x561fd80abf80 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51286/0 (socket says 172.21.15.45:51286) 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 -- 172.21.15.45:0/2712476439 learned_addr learned my addr 172.21.15.45:0/2712476439 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf137fe640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 -- 172.21.15.45:0/2712476439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 msgr2=0x561fd80ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80ac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 -- 172.21.15.45:0/2712476439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 msgr2=0x561fd80bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 -- 172.21.15.45:0/2712476439 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd7fd6060 con 0x561fd80ac8a0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf137fe640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf137fe640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 0x7fcf000a4200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0092f0 tx=0x7fcf0c009200 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0bcaa0 tx=0x7fcefc0bd560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3467f0b556392e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf097fa640 1 -- 172.21.15.45:0/2712476439 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0c5540 con 0x561fd80ac8a0 2024-03-12T05:11:48.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf097fa640 1 -- 172.21.15.45:0/2712476439 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcefc004dc0 con 0x561fd80ac8a0 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.901+0000 7fcf097fa640 1 -- 172.21.15.45:0/2712476439 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0bd000 con 0x561fd80ac8a0 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2712476439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 msgr2=0x561fd80b9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0bcaa0 tx=0x7fcefc0bd560 comp rx=0 tx=0).stop 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2712476439 shutdown_connections 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80ac8a0 0x561fd80b9620 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2712476439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2712476439 >> 172.21.15.45:0/2712476439 conn(0x561fd80a6980 msgr2=0x561fd80a6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2712476439 shutdown_connections 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2712476439 wait complete. 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.902+0000 7fcf1c1e2740 1 -- start start 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 Processor -- start 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80b9b60 0x561fd80b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8014020 con 0x561fd80abf80 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd7ce7430 con 0x561fd80ac8a0 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd7ca3e00 con 0x561fd80b9b60 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf1c1e2740 10 client.0 _ll_register_callbacks cb 0x561fd7fbbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51290/0 (socket says 172.21.15.45:51290) 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 -- 172.21.15.45:0/2798067172 learned_addr learned my addr 172.21.15.45:0/2798067172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf137fe640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80b9b60 0x561fd80b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 -- 172.21.15.45:0/2798067172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80b9b60 msgr2=0x561fd80b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80b9b60 0x561fd80b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 -- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 msgr2=0x561fd80b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf12ffd640 1 -- 172.21.15.45:0/2798067172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf141a4730 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.903+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400f750 tx=0x7fcf0400fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ad3f9841a1e00b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400bd10 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcf0400fe60 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561fd7d137e0 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf0400be70 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 10 client.15864 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561fd80126a0 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561fd7d137e0 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcf040106f0 con 0x561fd80abf80 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 10 client.15864 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 20 client.15864 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd80c9db0 0x561fd80cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 4 client.15864 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.904+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fd7d50720 con 0x561fd80c9db0 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.905+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd80c9db0 0x561fd80cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.905+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd80c9db0 0x561fd80cc1f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2be45099053e478c server_cookie=c7abaf289245b882 in_seq=0 out_seq=0 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.905+0000 7fcefa7fc640 10 client.15864 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcefa7fc640 1 -- 172.21.15.45:0/2798067172 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561fd7d50720 con 0x561fd80c9db0 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcefa7fc640 10 client.15864 handle_command_reply: tid=0 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 client.15864 shutdown 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 msgr2=0x561fd80b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400f750 tx=0x7fcf0400fc10 comp rx=0 tx=0).stop 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 shutdown_connections 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd80c9db0 0x561fd80cc1f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80b9b60 0x561fd80b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/2798067172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80b6c00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 >> 172.21.15.45:0/2798067172 conn(0x561fd80a6980 msgr2=0x561fd80a86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcefa7fc640 0 client.15864 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 shutdown_connections 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/2798067172 wait complete. 2024-03-12T05:11:48.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.906+0000 7fcf1c1e2740 20 client.15864 trim_cache size 0 max 16384 2024-03-12T05:11:48.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.921+0000 7fcf1c1e2740 1 Processor -- start 2024-03-12T05:11:48.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.921+0000 7fcf1c1e2740 1 -- start start 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80b9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd7f99130 con 0x561fd80b9b60 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd80acfe0 con 0x561fd80ac8a0 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd80a25c0 con 0x561fd80abf80 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51306/0 (socket says 172.21.15.45:51306) 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 -- 172.21.15.45:0/3721950375 learned_addr learned my addr 172.21.15.45:0/3721950375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 -- 172.21.15.45:0/3721950375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 msgr2=0x561fd80b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 -- 172.21.15.45:0/3721950375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 msgr2=0x561fd80b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 -- 172.21.15.45:0/3721950375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fd7faec20 con 0x561fd80b9b60 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.922+0000 7fcf137fe640 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0115b0 tx=0x7fcf0c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99e581a0edd80b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.923+0000 7fcf08ff9640 1 -- 172.21.15.45:0/3721950375 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c01f770 con 0x561fd80b9b60 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.923+0000 7fcf08ff9640 1 -- 172.21.15.45:0/3721950375 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcf0c01fd80 con 0x561fd80b9b60 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.923+0000 7fcf08ff9640 1 -- 172.21.15.45:0/3721950375 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c027e40 con 0x561fd80b9b60 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.923+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/3721950375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 msgr2=0x561fd80b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.923+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0115b0 tx=0x7fcf0c012000 comp rx=0 tx=0).stop 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/3721950375 shutdown_connections 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80b9b60 0x561fd80b1f40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80ac8a0 0x561fd80b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/3721950375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80abf80 0x561fd80b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/3721950375 >> 172.21.15.45:0/3721950375 conn(0x561fd80a6980 msgr2=0x561fd80d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/3721950375 shutdown_connections 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/3721950375 wait complete. 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- start start 2024-03-12T05:11:48.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 Processor -- start 2024-03-12T05:11:48.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 0x561fd80bf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8120320 con 0x561fd80abf80 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8120490 con 0x561fd80b9b60 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fd8120650 con 0x561fd80ac8a0 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf1c1e2740 10 client.0 _ll_register_callbacks cb 0x7fcefc0b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf12ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51320/0 (socket says 172.21.15.45:51320) 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.924+0000 7fcf12ffd640 1 -- 172.21.15.45:0/1588593563 learned_addr learned my addr 172.21.15.45:0/1588593563 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf137fe640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 0x561fd80bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 -- 172.21.15.45:0/1588593563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 msgr2=0x561fd80bf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 0x561fd80bf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 -- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 msgr2=0x561fd80bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 -- 172.21.15.45:0/1588593563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c009e80 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 0x561fd80bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf137fe640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf12ffd640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcf04010b00 tx=0x7fcf0400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13b415aed95ecd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcefa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04032070 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcefc004dc0 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcf04000e40 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf0400b6f0 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf1c1e2740 10 client.15870 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561fd7fd48c0 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.925+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcefc004dc0 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcf04035020 con 0x561fd80abf80 2024-03-12T05:11:48.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 10 client.15870 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 10 client.15870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 20 client.15870 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd8121ec0 0x561fd81242e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 4 client.15870 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561fd7d9f890 con 0x561fd8121ec0 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd8121ec0 0x561fd81242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcefa7fc640 10 client.15870 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf0bfff640 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd8121ec0 0x561fd81242e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc3a67a34bee749f server_cookie=3a9fd808bdc1cd09 in_seq=0 out_seq=0 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcefa7fc640 1 -- 172.21.15.45:0/1588593563 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561fd7d9f890 con 0x561fd8121ec0 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcefa7fc640 10 client.15870 handle_command_reply: tid=0 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.926+0000 7fcf1c1e2740 1 client.15870 shutdown 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 msgr2=0x561fd80bf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcf04010b00 tx=0x7fcf0400bdc0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 shutdown_connections 2024-03-12T05:11:48.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561fd8121ec0 0x561fd81242e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561fd80b9b60 0x561fd80bfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561fd80ac8a0 0x561fd80bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 --2- 172.21.15.45:0/1588593563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561fd80abf80 0x561fd80bf340 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcefa7fc640 0 client.15870 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 >> 172.21.15.45:0/1588593563 conn(0x561fd80a6980 msgr2=0x561fd80b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 shutdown_connections 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 1 -- 172.21.15.45:0/1588593563 wait complete. 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1c1e2740 20 client.15870 trim_cache size 0 max 16384 2024-03-12T05:11:48.930 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 msgr2=0x7fcf000a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 0x7fcf000a4200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0092f0 tx=0x7fcf0c009200 comp rx=0 tx=0).stop 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 msgr2=0x7fcf14197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcefc005ab0 tx=0x7fcefc0040c0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 shutdown_connections 2024-03-12T05:11:48.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcf000a1d40 0x7fcf000a4200 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf14106430 0x7fcf14197c50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf1419bfd0 0x7fcf1419c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 --2- 172.21.15.45:0/367775238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf140685d0 0x7fcf14197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.927+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 >> 172.21.15.45:0/367775238 conn(0x7fcf14077180 msgr2=0x7fcf140fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.928+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 shutdown_connections 2024-03-12T05:11:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:48.928+0000 7fcf1953a640 1 -- 172.21.15.45:0/367775238 wait complete. 2024-03-12T05:11:49.362 INFO:teuthology.orchestra.run.smithi045.stderr:3100000 of 3200000 tuples (96%) done (elapsed 30.81 s, remaining 0.99 s) 2024-03-12T05:11:49.937 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:11:49.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:11:49.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:11:50.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 -- 172.21.15.45:0/756863893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50068fa0 msgr2=0x7fbb50105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 --2- 172.21.15.45:0/756863893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50068fa0 0x7fbb50105de0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009930 tx=0x7fbb4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 -- 172.21.15.45:0/756863893 shutdown_connections 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 --2- 172.21.15.45:0/756863893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb50068fa0 0x7fbb50105de0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 --2- 172.21.15.45:0/756863893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb501064c0 0x7fbb5010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 --2- 172.21.15.45:0/756863893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb500685f0 0x7fbb500689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 -- 172.21.15.45:0/756863893 >> 172.21.15.45:0/756863893 conn(0x7fbb50077180 msgr2=0x7fbb50077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 -- 172.21.15.45:0/756863893 shutdown_connections 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.061+0000 7fbb565bc640 1 -- 172.21.15.45:0/756863893 wait complete. 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 Processor -- start 2024-03-12T05:11:50.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 -- start start 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb500685f0 0x7fbb501976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb5019bfb0 0x7fbb5019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5019c9a0 con 0x7fbb501064c0 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5019cb10 con 0x7fbb5019bfb0 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb565bc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5019cc80 con 0x7fbb500685f0 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51338/0 (socket says 172.21.15.45:51338) 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb4ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb500685f0 0x7fbb501976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/369364906 learned_addr learned my addr 172.21.15.45:0/369364906 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.062+0000 7fbb54b59640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb5019bfb0 0x7fbb5019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/369364906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb500685f0 msgr2=0x7fbb501976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb500685f0 0x7fbb501976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/369364906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb5019bfb0 msgr2=0x7fbb5019c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb5019bfb0 0x7fbb5019c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/369364906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44009560 con 0x7fbb501064c0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000c00 tx=0x7fbb440040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4d7fa640 1 -- 172.21.15.45:0/369364906 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb440044e0 con 0x7fbb501064c0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb501a4260 con 0x7fbb501064c0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4d7fa640 1 -- 172.21.15.45:0/369364906 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb4402de90 con 0x7fbb501064c0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb4d7fa640 1 -- 172.21.15.45:0/369364906 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb440053c0 con 0x7fbb501064c0 2024-03-12T05:11:50.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.063+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb501a46e0 con 0x7fbb501064c0 2024-03-12T05:11:50.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.065+0000 7fbb4d7fa640 1 -- 172.21.15.45:0/369364906 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fbb44005520 con 0x7fbb501064c0 2024-03-12T05:11:50.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 Processor -- start 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 -- start start 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4d7fa640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 0x7fbb380a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358d30 0x555c8f3612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f363d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f2c6f90 con 0x555c8f358380 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f2c44c0 con 0x555c8f3618b0 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb59190740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f2bfca0 con 0x555c8f358d30 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51350/0 (socket says 172.21.15.45:51350) 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 -- 172.21.15.45:0/2663655437 learned_addr learned my addr 172.21.15.45:0/2663655437 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb54b59640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 0x7fbb380a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4d7fa640 1 -- 172.21.15.45:0/369364906 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbb440efaf0 con 0x7fbb501064c0 2024-03-12T05:11:50.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358d30 0x555c8f3612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f363d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 -- 172.21.15.45:0/2663655437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358d30 msgr2=0x555c8f3612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358d30 0x555c8f3612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.066+0000 7fbb4ffff640 1 -- 172.21.15.45:0/2663655437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 msgr2=0x555c8f363d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f363d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb4ffff640 1 -- 172.21.15.45:0/2663655437 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c8f2820b0 con 0x555c8f358380 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f363d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000ad00 tx=0x7fbb4000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d0947f2849ef87 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb54b59640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 0x7fbb380a4180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb48013590 tx=0x7fbb48013990 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb367fc640 1 -- 172.21.15.45:0/2663655437 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4000a800 con 0x555c8f358380 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb367fc640 1 -- 172.21.15.45:0/2663655437 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb40018af0 con 0x555c8f358380 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.067+0000 7fbb367fc640 1 -- 172.21.15.45:0/2663655437 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40010b20 con 0x555c8f358380 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- 172.21.15.45:0/2663655437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 msgr2=0x555c8f358760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000ad00 tx=0x7fbb4000f790 comp rx=0 tx=0).stop 2024-03-12T05:11:50.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- 172.21.15.45:0/2663655437 shutdown_connections 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f363d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358d30 0x555c8f3612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 --2- 172.21.15.45:0/2663655437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358380 0x555c8f358760 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- 172.21.15.45:0/2663655437 >> 172.21.15.45:0/2663655437 conn(0x555c8f352840 msgr2=0x555c8f352c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- 172.21.15.45:0/2663655437 shutdown_connections 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- 172.21.15.45:0/2663655437 wait complete. 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 -- start start 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.068+0000 7fbb59190740 1 Processor -- start 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f35f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f264f40 con 0x555c8f358d30 2024-03-12T05:11:50.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8ef934b0 con 0x555c8f3618b0 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f245670 con 0x555c8f358380 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 10 client.0 _ll_register_callbacks cb 0x555c8f2b2f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51354/0 (socket says 172.21.15.45:51354) 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/2485293777 learned_addr learned my addr 172.21.15.45:0/2485293777 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/2485293777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 msgr2=0x555c8f35e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f35f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 msgr2=0x555c8f35f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f35f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 -- 172.21.15.45:0/2485293777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4000a960 con 0x555c8f358d30 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:50.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbb440bc570 tx=0x7fbb440bc250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f03b4c5bcc6277 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb440bc5e0 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c8f299f40 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.069+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbb440c0730 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb440c4d80 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 10 client.15882 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555c8f2be320 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555c8f299f40 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fbb4403c070 con 0x555c8f358d30 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 10 client.15882 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 20 client.15882 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f371a90 0x555c8f373ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 4 client.15882 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c8f2b6310 con 0x555c8f371a90 2024-03-12T05:11:50.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.070+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f371a90 0x555c8f373ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.071+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f371a90 0x555c8f373ec0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50cd06eac5cf9a69 server_cookie=f216da2bac431d88 in_seq=0 out_seq=0 2024-03-12T05:11:50.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.071+0000 7fbb2f7fe640 10 client.15882 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.071+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/2485293777 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c8f2b6310 con 0x555c8f371a90 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.071+0000 7fbb2f7fe640 10 client.15882 handle_command_reply: tid=0 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.071+0000 7fbb59190740 1 client.15882 shutdown 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 msgr2=0x555c8f35ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbb440bc570 tx=0x7fbb440bc250 comp rx=0 tx=0).stop 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 shutdown_connections 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f371a90 0x555c8f373ec0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f358d30 0x555c8f35ed80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f3618b0 0x555c8f35f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb2f7fe640 0 client.15882 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 --2- 172.21.15.45:0/2485293777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 >> 172.21.15.45:0/2485293777 conn(0x555c8f352840 msgr2=0x555c8f3547a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 shutdown_connections 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 1 -- 172.21.15.45:0/2485293777 wait complete. 2024-03-12T05:11:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.072+0000 7fbb59190740 20 client.15882 trim_cache size 0 max 16384 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.089+0000 7fbb59190740 1 Processor -- start 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.089+0000 7fbb59190740 1 -- start start 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f351760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f371890 con 0x555c8f3618b0 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f34ccf0 con 0x555c8f358d30 2024-03-12T05:11:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb59190740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f3576a0 con 0x555c8f358380 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51368/0 (socket says 172.21.15.45:51368) 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 -- 172.21.15.45:0/2148718638 learned_addr learned my addr 172.21.15.45:0/2148718638 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f351760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 -- 172.21.15.45:0/2148718638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 msgr2=0x555c8f35f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 -- 172.21.15.45:0/2148718638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 msgr2=0x555c8f351760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f351760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 -- 172.21.15.45:0/2148718638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c8f25ac00 con 0x555c8f3618b0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.090+0000 7fbb54b59640 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbb48013230 tx=0x7fbb4801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7d979a00e305fc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.091+0000 7fbb35ffb640 1 -- 172.21.15.45:0/2148718638 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48013260 con 0x555c8f3618b0 2024-03-12T05:11:50.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.091+0000 7fbb35ffb640 1 -- 172.21.15.45:0/2148718638 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb48003970 con 0x555c8f3618b0 2024-03-12T05:11:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.091+0000 7fbb35ffb640 1 -- 172.21.15.45:0/2148718638 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004610 con 0x555c8f3618b0 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- 172.21.15.45:0/2148718638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 msgr2=0x555c8f3688e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbb48013230 tx=0x7fbb4801a000 comp rx=0 tx=0).stop 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- 172.21.15.45:0/2148718638 shutdown_connections 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f3688e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f351760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 --2- 172.21.15.45:0/2148718638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f35f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- 172.21.15.45:0/2148718638 >> 172.21.15.45:0/2148718638 conn(0x555c8f352840 msgr2=0x555c8f351050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- 172.21.15.45:0/2148718638 shutdown_connections 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- 172.21.15.45:0/2148718638 wait complete. 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 -- start start 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 Processor -- start 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.092+0000 7fbb59190740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f3c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f3c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f3687d0 con 0x555c8f3618b0 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f368940 con 0x555c8f358d30 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c8f368b00 con 0x555c8f358380 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 10 client.0 _ll_register_callbacks cb 0x7fbb4000a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51370/0 (socket says 172.21.15.45:51370) 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 -- 172.21.15.45:0/311379080 learned_addr learned my addr 172.21.15.45:0/311379080 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f3c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb4f7fe640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f3c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 -- 172.21.15.45:0/311379080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 msgr2=0x555c8f3c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f3c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 -- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 msgr2=0x555c8f3c2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f3c2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 -- 172.21.15.45:0/311379080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c8f2820b0 con 0x555c8f3618b0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb54b59640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800b010 tx=0x7fbb480045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a721b6fe9065d9b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004e70 con 0x555c8f3618b0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbb48013260 con 0x555c8f3618b0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.093+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c8f2e5900 con 0x555c8f3618b0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb480047e0 con 0x555c8f3618b0 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 10 client.15888 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fbb48019050 con 0x555c8f3618b0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555c8f2e5900 con 0x555c8f3618b0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fbb480056d0 con 0x555c8f3618b0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 10 client.15888 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 20 client.15888 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f3ca890 0x555c8f3cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 4 client.15888 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555c8efbf860 con 0x555c8f3ca890 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.094+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f3ca890 0x555c8f3cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.095+0000 7fbb4ffff640 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f3ca890 0x555c8f3cccb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1c896b8091f8355 server_cookie=dfeb66bd2fa22174 in_seq=0 out_seq=0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.095+0000 7fbb2f7fe640 10 client.15888 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb2f7fe640 1 -- 172.21.15.45:0/311379080 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555c8efbf860 con 0x555c8f3ca890 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb2f7fe640 10 client.15888 handle_command_reply: tid=0 2024-03-12T05:11:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 client.15888 shutdown 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 msgr2=0x555c8f368210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800b010 tx=0x7fbb480045c0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 shutdown_connections 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555c8f3ca890 0x555c8f3cccb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb2f7fe640 0 client.15888 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555c8f3618b0 0x555c8f368210 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555c8f358d30 0x555c8f3c2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 --2- 172.21.15.45:0/311379080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c8f358380 0x555c8f3c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 >> 172.21.15.45:0/311379080 conn(0x555c8f352840 msgr2=0x555c8f3518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 shutdown_connections 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 1 -- 172.21.15.45:0/311379080 wait complete. 2024-03-12T05:11:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.096+0000 7fbb59190740 20 client.15888 trim_cache size 0 max 16384 2024-03-12T05:11:50.099 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:50.099 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:11:50.099 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "dcd3c477-74b4-45e7-8ab9-e285fc9673be", 2024-03-12T05:11:50.099 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:11:50.099 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 msgr2=0x7fbb380a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 0x7fbb380a4180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb48013590 tx=0x7fbb48013990 comp rx=0 tx=0).stop 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 msgr2=0x7fbb50197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000c00 tx=0x7fbb440040e0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 shutdown_connections 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbb380a1cc0 0x7fbb380a4180 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb501064c0 0x7fbb50197c30 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb5019bfb0 0x7fbb5019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 --2- 172.21.15.45:0/369364906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb500685f0 0x7fbb501976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 >> 172.21.15.45:0/369364906 conn(0x7fbb50077180 msgr2=0x7fbb500fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 shutdown_connections 2024-03-12T05:11:50.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.097+0000 7fbb565bc640 1 -- 172.21.15.45:0/369364906 wait complete. 2024-03-12T05:11:50.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:11:50.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:11:50.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 -- 172.21.15.45:0/3582223607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c0fff20 msgr2=0x7fb59c100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/3582223607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c0fff20 0x7fb59c100300 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb598009900 tx=0x7fb59802bc30 comp rx=0 tx=0).stop 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 -- 172.21.15.45:0/3582223607 shutdown_connections 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/3582223607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c10fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/3582223607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c1008d0 0x7fb59c10d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/3582223607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c0fff20 0x7fb59c100300 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 -- 172.21.15.45:0/3582223607 >> 172.21.15.45:0/3582223607 conn(0x7fb59c077210 msgr2=0x7fb59c077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 -- 172.21.15.45:0/3582223607 shutdown_connections 2024-03-12T05:11:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.242+0000 7fb5a3f35640 1 -- 172.21.15.45:0/3582223607 wait complete. 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 Processor -- start 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 -- start start 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c0fff20 0x7fb59c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c199e90 con 0x7fb59c1008d0 2024-03-12T05:11:50.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c19a000 con 0x7fb59c10dad0 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a3f35640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c19a1c0 con 0x7fb59c0fff20 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51390/0 (socket says 172.21.15.45:51390) 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a14d0640 1 -- 172.21.15.45:0/1496895978 learned_addr learned my addr 172.21.15.45:0/1496895978 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c0fff20 0x7fb59c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a14d0640 1 -- 172.21.15.45:0/1496895978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c0fff20 msgr2=0x7fb59c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.243+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c0fff20 0x7fb59c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a14d0640 1 -- 172.21.15.45:0/1496895978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 msgr2=0x7fb59c199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a14d0640 1 -- 172.21.15.45:0/1496895978 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009560 con 0x7fb59c1008d0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c920 tx=0x7fb58c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb592ffd640 1 -- 172.21.15.45:0/1496895978 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c007c70 con 0x7fb59c1008d0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c19a4a0 con 0x7fb59c1008d0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb592ffd640 1 -- 172.21.15.45:0/1496895978 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb58c007dd0 con 0x7fb59c1008d0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb592ffd640 1 -- 172.21.15.45:0/1496895978 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0187c0 con 0x7fb59c1008d0 2024-03-12T05:11:50.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.244+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c1a65c0 con 0x7fb59c1008d0 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.246+0000 7fb592ffd640 1 -- 172.21.15.45:0/1496895978 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fb58c00f450 con 0x7fb59c1008d0 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.246+0000 7fb5a6be0740 1 Processor -- start 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 -- start start 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7efba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f753280 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f7507b0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f74bf90 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a24d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a24d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54798/0 (socket says 172.21.15.45:54798) 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a24d2640 1 -- 172.21.15.45:0/3019081459 learned_addr learned my addr 172.21.15.45:0/3019081459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb592ffd640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 0x7fb5880a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.247+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 0x7fb5880a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb592ffd640 1 -- 172.21.15.45:0/1496895978 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb58c0cdc80 con 0x7fb59c1008d0 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 -- 172.21.15.45:0/3019081459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e4bd0 msgr2=0x55eb0f7ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 -- 172.21.15.45:0/3019081459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 msgr2=0x55eb0f7e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 -- 172.21.15.45:0/3019081459 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb0f70a570 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7efba0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0cb250 tx=0x7fb58c096300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd7b1e000617833 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb587fff640 1 -- 172.21.15.45:0/3019081459 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00fa90 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 0x7fb5880a4180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb594005cc0 tx=0x7fb594004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb587fff640 1 -- 172.21.15.45:0/3019081459 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb58c09ad60 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.248+0000 7fb587fff640 1 -- 172.21.15.45:0/3019081459 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0a2e00 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- 172.21.15.45:0/3019081459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 msgr2=0x55eb0f7efba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7efba0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0cb250 tx=0x7fb58c096300 comp rx=0 tx=0).stop 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- 172.21.15.45:0/3019081459 shutdown_connections 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7efba0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/3019081459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7e4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- 172.21.15.45:0/3019081459 >> 172.21.15.45:0/3019081459 conn(0x55eb0f7de990 msgr2=0x55eb0f7ded70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- 172.21.15.45:0/3019081459 shutdown_connections 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- 172.21.15.45:0/3019081459 wait complete. 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.249+0000 7fb5a6be0740 1 -- start start 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 Processor -- start 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f41f430 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f3dbe00 con 0x55eb0f7e42b0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f6d0fd0 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a6be0740 10 client.0 _ll_register_callbacks cb 0x55eb0f726fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a1cd1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ea5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51406/0 (socket says 172.21.15.45:51406) 2024-03-12T05:11:50.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a14d0640 1 -- 172.21.15.45:0/1011595463 learned_addr learned my addr 172.21.15.45:0/1011595463 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54812/0 (socket says 172.21.15.45:54812) 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a1cd1640 1 -- 172.21.15.45:0/1011595463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 msgr2=0x55eb0f7eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.250+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a1cd1640 1 -- 172.21.15.45:0/1011595463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 msgr2=0x55eb0f7ea5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ea5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a1cd1640 1 -- 172.21.15.45:0/1011595463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c1a65c0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb598009230 tx=0x7fb59802ca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9194c2253492910 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5857fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598004510 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb5980035f0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb0f44b7e0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5980037b0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a6be0740 10 client.25358 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55eb0f74a610 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.251+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb598003dc0 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb59804f020 con 0x55eb0f7e42b0 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 10 client.25358 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 10 client.25358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 20 client.25358 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f7fd940 0x55eb0f7ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 4 client.25358 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb0f44b7e0 con 0x55eb0f7fd940 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f7fd940 0x55eb0f7ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5857fa640 10 client.25358 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.252+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f7fd940 0x55eb0f7ffd70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ec5aee10ddd4e1 server_cookie=45d026dc320f1dcd in_seq=0 out_seq=0 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5857fa640 1 -- 172.21.15.45:0/1011595463 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eb0f44b7e0 con 0x55eb0f7fd940 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5857fa640 10 client.25358 handle_command_reply: tid=0 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 client.25358 shutdown 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 msgr2=0x55eb0f7ed220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb598009230 tx=0x7fb59802ca70 comp rx=0 tx=0).stop 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 shutdown_connections 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f7fd940 0x55eb0f7ffd70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5857fa640 0 client.25358 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/1011595463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ed220 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 >> 172.21.15.45:0/1011595463 conn(0x55eb0f7de990 msgr2=0x55eb0f7e0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 shutdown_connections 2024-03-12T05:11:50.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 1 -- 172.21.15.45:0/1011595463 wait complete. 2024-03-12T05:11:50.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.253+0000 7fb5a6be0740 20 client.25358 trim_cache size 0 max 16384 2024-03-12T05:11:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 Processor -- start 2024-03-12T05:11:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 -- start start 2024-03-12T05:11:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.273+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f7e3f10 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f7e3630 con 0x55eb0f7e42b0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f804f90 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51418/0 (socket says 172.21.15.45:51418) 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 -- 172.21.15.45:0/467113947 learned_addr learned my addr 172.21.15.45:0/467113947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 -- 172.21.15.45:0/467113947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 msgr2=0x55eb0f7e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 -- 172.21.15.45:0/467113947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 msgr2=0x55eb0f7ecce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ecce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 -- 172.21.15.45:0/467113947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb0f6e6ac0 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ecce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:11:50.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00fe40 tx=0x7fb58c085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d07a4060d2685e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5877fe640 1 -- 172.21.15.45:0/467113947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00dea0 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5877fe640 1 -- 172.21.15.45:0/467113947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb58c09a660 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.274+0000 7fb5877fe640 1 -- 172.21.15.45:0/467113947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c0cdf40 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- 172.21.15.45:0/467113947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 msgr2=0x55eb0f7ed220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00fe40 tx=0x7fb58c085040 comp rx=0 tx=0).stop 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- 172.21.15.45:0/467113947 shutdown_connections 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7ed220 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/467113947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7ecce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- 172.21.15.45:0/467113947 >> 172.21.15.45:0/467113947 conn(0x55eb0f7de990 msgr2=0x55eb0f7ded70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- 172.21.15.45:0/467113947 shutdown_connections 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- 172.21.15.45:0/467113947 wait complete. 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 -- start start 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.275+0000 7fb5a6be0740 1 Processor -- start 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f7e7c20 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f7e7d90 con 0x55eb0f7e4bd0 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0f853e60 con 0x55eb0f7e42b0 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a6be0740 10 client.0 _ll_register_callbacks cb 0x7fb58c096980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:11:50.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51434/0 (socket says 172.21.15.45:51434) 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 -- 172.21.15.45:0/2628337378 learned_addr learned my addr 172.21.15.45:0/2628337378 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a14d0640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 -- 172.21.15.45:0/2628337378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e42b0 msgr2=0x55eb0f7f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 -- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e4bd0 msgr2=0x55eb0f7e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7e6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 -- 172.21.15.45:0/2628337378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb0f70a570 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5a24d2640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb594013740 tx=0x7fb594004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7b53084323b6c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.276+0000 7fb5857fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594010de0 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb594003970 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb58c09ad60 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb594003b30 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 10 client.15906 fetch_fsmap learned FSMap version 29 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55eb0f70c6a0 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb58c09ad60 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb594023400 con 0x55eb0f7ed7b0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 10 client.15906 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 20 client.15906 populate_metadata read hostname 'smithi045' 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f8555c0 0x55eb0f8579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 4 client.15906 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eb0f488720 con 0x55eb0f8555c0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.277+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f8555c0 0x55eb0f8579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5857fa640 10 client.15906 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a1cd1640 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f8555c0 0x55eb0f8579e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5fa0d842f35cfb8 server_cookie=500653225e61d3c in_seq=0 out_seq=0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5857fa640 1 -- 172.21.15.45:0/2628337378 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55eb0f488720 con 0x55eb0f8555c0 2024-03-12T05:11:50.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5857fa640 10 client.15906 handle_command_reply: tid=0 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 client.15906 shutdown 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 msgr2=0x55eb0f7e7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb594013740 tx=0x7fb594004a40 comp rx=0 tx=0).stop 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 shutdown_connections 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55eb0f8555c0 0x55eb0f8579e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55eb0f7ed7b0 0x55eb0f7e7520 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eb0f7e4bd0 0x55eb0f7e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 --2- 172.21.15.45:0/2628337378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb0f7e42b0 0x55eb0f7f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5857fa640 0 client.15906 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 >> 172.21.15.45:0/2628337378 conn(0x55eb0f7de990 msgr2=0x55eb0f7dfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 shutdown_connections 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 1 -- 172.21.15.45:0/2628337378 wait complete. 2024-03-12T05:11:50.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.278+0000 7fb5a6be0740 20 client.15906 trim_cache size 0 max 16384 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "dcd3c477-74b4-45e7-8ab9-e285fc9673be": { 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "dcd3c477-74b4-45e7-8ab9-e285fc9673be", 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:11:50.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:11:50.286 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 msgr2=0x7fb5880a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 0x7fb5880a4180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb594005cc0 tx=0x7fb594004ab0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 msgr2=0x7fb59c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00c920 tx=0x7fb58c00cde0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 shutdown_connections 2024-03-12T05:11:50.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5880a1cc0 0x7fb5880a4180 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb59c1008d0 0x7fb59c19fb80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb59c10dad0 0x7fb59c199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 --2- 172.21.15.45:0/1496895978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb59c0fff20 0x7fb59c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 >> 172.21.15.45:0/1496895978 conn(0x7fb59c077210 msgr2=0x7fb59c10f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 shutdown_connections 2024-03-12T05:11:50.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:11:50.279+0000 7fb5a3f35640 1 -- 172.21.15.45:0/1496895978 wait complete. 2024-03-12T05:11:50.288 INFO:tasks.cephfs.filesystem:scrub status for tag:dcd3c477-74b4-45e7-8ab9-e285fc9673be - {'path': '/', 'tag': 'dcd3c477-74b4-45e7-8ab9-e285fc9673be', 'options': 'recursive,force'} 2024-03-12T05:11:50.561 INFO:teuthology.orchestra.run.smithi045.stderr:3200000 of 3200000 tuples (100%) done (elapsed 32.01 s, remaining 0.00 s) 2024-03-12T05:11:50.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:50 smithi045 ceph-mon[32174]: pgmap v31: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 MiB/s wr, 44 op/s 2024-03-12T05:11:50.633 INFO:teuthology.orchestra.run.smithi045.stderr:vacuuming... 2024-03-12T05:11:50.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:50 smithi130 ceph-mon[35398]: pgmap v31: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 MiB/s wr, 44 op/s 2024-03-12T05:11:50.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:50 smithi116 ceph-mon[35486]: pgmap v31: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 MiB/s wr, 44 op/s 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:51.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:11:51.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:51 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:11:51.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.720 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:51 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:51.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:51 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:11:52.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:52 smithi045 ceph-mon[32174]: pgmap v32: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:52.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:52 smithi130 ceph-mon[35398]: pgmap v32: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:52.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:52 smithi116 ceph-mon[35486]: pgmap v32: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:54.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:54 smithi045 ceph-mon[32174]: pgmap v33: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:54.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:54 smithi130 ceph-mon[35398]: pgmap v33: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:54.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:54 smithi116 ceph-mon[35486]: pgmap v33: 97 pgs: 97 active+clean; 751 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 MiB/s wr, 29 op/s 2024-03-12T05:11:56.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:56 smithi045 ceph-mon[32174]: pgmap v34: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 59 MiB/s wr, 46 op/s 2024-03-12T05:11:56.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:56 smithi130 ceph-mon[35398]: pgmap v34: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 59 MiB/s wr, 46 op/s 2024-03-12T05:11:56.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:56 smithi116 ceph-mon[35486]: pgmap v34: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 59 MiB/s wr, 46 op/s 2024-03-12T05:11:58.279 INFO:teuthology.orchestra.run.smithi045.stderr:creating primary keys... 2024-03-12T05:11:58.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:58 smithi045 ceph-mon[32174]: pgmap v35: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:58.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:58 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:11:58.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:58 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:58.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:58 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:58.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:58 smithi130 ceph-mon[35398]: pgmap v35: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:58 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:11:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:58 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:58 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:58 smithi116 ceph-mon[35486]: pgmap v35: 97 pgs: 97 active+clean; 799 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 39 MiB/s wr, 31 op/s 2024-03-12T05:11:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:58 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:11:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:58 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:58 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:59.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:59 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:11:59.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:59 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:11:59.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:11:59 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:59.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:59 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:11:59.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:59 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:11:59.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:11:59 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:11:59.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:59 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:11:59.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:59 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:11:59.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:11:59 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:12:00.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:00 smithi045 ceph-mon[32174]: pgmap v36: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 61 MiB/s wr, 47 op/s 2024-03-12T05:12:00.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:00 smithi130 ceph-mon[35398]: pgmap v36: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 61 MiB/s wr, 47 op/s 2024-03-12T05:12:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:00 smithi116 ceph-mon[35486]: pgmap v36: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 61 MiB/s wr, 47 op/s 2024-03-12T05:12:02.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:02 smithi045 ceph-mon[32174]: pgmap v37: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:02.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:02 smithi130 ceph-mon[35398]: pgmap v37: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:02.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:02 smithi116 ceph-mon[35486]: pgmap v37: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:03.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:03.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:03.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:03.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:03.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:03.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:03.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:03.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:03.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:04.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:04 smithi045 ceph-mon[32174]: pgmap v38: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:04.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:04 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:04.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:04 smithi130 ceph-mon[35398]: pgmap v38: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:04.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:04 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:04.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:04 smithi116 ceph-mon[35486]: pgmap v38: 97 pgs: 97 active+clean; 816 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 MiB/s wr, 33 op/s 2024-03-12T05:12:04.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:04 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:04.825 INFO:teuthology.orchestra.run.smithi045.stderr:done in 46.35 s (drop tables 0.01 s, create tables 0.05 s, client-side generate 32.10 s, vacuum 7.65 s, primary keys 6.55 s). 2024-03-12T05:12:04.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-03-12T05:12:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:starting vacuum...end. 2024-03-12T05:12:06.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:06 smithi045 ceph-mon[32174]: pgmap v39: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 63 MiB/s wr, 64 op/s 2024-03-12T05:12:06.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:06 smithi130 ceph-mon[35398]: pgmap v39: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 63 MiB/s wr, 64 op/s 2024-03-12T05:12:06.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:06 smithi116 ceph-mon[35486]: pgmap v39: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 63 MiB/s wr, 64 op/s 2024-03-12T05:12:08.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:08 smithi045 ceph-mon[32174]: pgmap v40: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 47 op/s 2024-03-12T05:12:08.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:08 smithi130 ceph-mon[35398]: pgmap v40: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 47 op/s 2024-03-12T05:12:08.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:08 smithi116 ceph-mon[35486]: pgmap v40: 97 pgs: 97 active+clean; 1010 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 47 op/s 2024-03-12T05:12:09.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 5.0 s, 1917.2 tps, lat 48.964 ms stddev 60.712 2024-03-12T05:12:10.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:10 smithi045 ceph-mon[32174]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 50 MiB/s wr, 127 op/s 2024-03-12T05:12:10.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:10 smithi130 ceph-mon[35398]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 50 MiB/s wr, 127 op/s 2024-03-12T05:12:10.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:10 smithi116 ceph-mon[35486]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 50 MiB/s wr, 127 op/s 2024-03-12T05:12:12.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:12 smithi045 ceph-mon[32174]: pgmap v42: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:12.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:12 smithi130 ceph-mon[35398]: pgmap v42: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:12.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:12 smithi116 ceph-mon[35486]: pgmap v42: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:14.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:14 smithi045 ceph-mon[32174]: pgmap v43: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:14.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:14 smithi130 ceph-mon[35398]: pgmap v43: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:14.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:14 smithi116 ceph-mon[35486]: pgmap v43: 97 pgs: 97 active+clean; 1.0 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 MiB/s wr, 110 op/s 2024-03-12T05:12:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 10.0 s, 1648.8 tps, lat 60.667 ms stddev 23.351 2024-03-12T05:12:16.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:16 smithi045 ceph-mon[32174]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 34 MiB/s wr, 710 op/s 2024-03-12T05:12:16.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:16 smithi130 ceph-mon[35398]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 34 MiB/s wr, 710 op/s 2024-03-12T05:12:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:16 smithi116 ceph-mon[35486]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 34 MiB/s wr, 710 op/s 2024-03-12T05:12:18.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:18 smithi045 ceph-mon[32174]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 14 MiB/s wr, 679 op/s 2024-03-12T05:12:18.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:18 smithi130 ceph-mon[35398]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 14 MiB/s wr, 679 op/s 2024-03-12T05:12:18.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:18 smithi116 ceph-mon[35486]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 14 MiB/s wr, 679 op/s 2024-03-12T05:12:19.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 15.0 s, 1508.8 tps, lat 66.024 ms stddev 16.786 2024-03-12T05:12:20.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:12:20.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:12:20.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 -- 172.21.15.45:0/2664132712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 msgr2=0x7f0854108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 --2- 172.21.15.45:0/2664132712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f0854108390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0844009900 tx=0x7f084402bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:20.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 -- 172.21.15.45:0/2664132712 shutdown_connections 2024-03-12T05:12:20.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 --2- 172.21.15.45:0/2664132712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0854116210 0x7f0854118600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 --2- 172.21.15.45:0/2664132712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08541088d0 0x7f0854108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 --2- 172.21.15.45:0/2664132712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f0854108390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 -- 172.21.15.45:0/2664132712 >> 172.21.15.45:0/2664132712 conn(0x7f085406c270 msgr2=0x7f085406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 -- 172.21.15.45:0/2664132712 shutdown_connections 2024-03-12T05:12:20.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.471+0000 7f085b22e640 1 -- 172.21.15.45:0/2664132712 wait complete. 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.472+0000 7f085b22e640 1 Processor -- start 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.472+0000 7f085b22e640 1 -- start start 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 0x7f08541a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f08541a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a6f80 con 0x7f08541088d0 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a70f0 con 0x7f0854116210 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f085b22e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a72b0 con 0x7f085410b860 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38706/0 (socket says 172.21.15.45:38706) 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 -- 172.21.15.45:0/1476299951 learned_addr learned my addr 172.21.15.45:0/1476299951 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f0853fff640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f08541a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f0858fca640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 0x7f08541a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 -- 172.21.15.45:0/1476299951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 msgr2=0x7f08541a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f08541a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 -- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 msgr2=0x7f08541a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 0x7f08541a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f08597cb640 1 -- 172.21.15.45:0/1476299951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844009560 con 0x7f0854116210 2024-03-12T05:12:20.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.473+0000 7f0858fca640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 0x7f08541a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:20.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f08597cb640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f084c00eb40 tx=0x7f084c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f0851ffb640 1 -- 172.21.15.45:0/1476299951 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c004b20 con 0x7f0854116210 2024-03-12T05:12:20.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08541a7590 con 0x7f0854116210 2024-03-12T05:12:20.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f0851ffb640 1 -- 172.21.15.45:0/1476299951 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f084c004c80 con 0x7f0854116210 2024-03-12T05:12:20.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f0851ffb640 1 -- 172.21.15.45:0/1476299951 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c010690 con 0x7f0854116210 2024-03-12T05:12:20.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.474+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541aee10 con 0x7f0854116210 2024-03-12T05:12:20.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.477+0000 7f0851ffb640 1 -- 172.21.15.45:0/1476299951 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f084c005020 con 0x7f0854116210 2024-03-12T05:12:20.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.478+0000 7f0851ffb640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 0x7f083c0a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.478+0000 7f0851ffb640 1 -- 172.21.15.45:0/1476299951 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f084c014070 con 0x7f0854116210 2024-03-12T05:12:20.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0858fca640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 0x7f083c0a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 Processor -- start 2024-03-12T05:12:20.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 -- start start 2024-03-12T05:12:20.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 0x55e772fb4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fbd7b0 0x55e772fbfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772f23280 con 0x55e772fb42b0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772f207b0 con 0x55e772fb4bd0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f085de00740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772f1bf90 con 0x55e772fbd7b0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0858fca640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 0x7f083c0a4200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0844006010 tx=0x7f0844005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0858fca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 0x55e772fb4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0858fca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 0x55e772fb4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34792/0 (socket says 172.21.15.45:34792) 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0858fca640 1 -- 172.21.15.45:0/2288725833 learned_addr learned my addr 172.21.15.45:0/2288725833 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f0853fff640 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.479+0000 7f08597cb640 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fbd7b0 0x55e772fbfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 -- 172.21.15.45:0/2288725833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fbd7b0 msgr2=0x55e772fbfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fbd7b0 0x55e772fbfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 -- 172.21.15.45:0/2288725833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 msgr2=0x55e772fb4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 0x55e772fb4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 -- 172.21.15.45:0/2288725833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e772eda570 con 0x55e772fb4bd0 2024-03-12T05:12:20.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.480+0000 7f0853fff640 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0848009bf0 tx=0x7f0848011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=357f760b3040bc00 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.481+0000 7f083affd640 1 -- 172.21.15.45:0/2288725833 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848004050 con 0x55e772fb4bd0 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.481+0000 7f083affd640 1 -- 172.21.15.45:0/2288725833 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0848005020 con 0x55e772fb4bd0 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.481+0000 7f083affd640 1 -- 172.21.15.45:0/2288725833 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848004c30 con 0x55e772fb4bd0 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 -- 172.21.15.45:0/2288725833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 msgr2=0x55e772fbd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0848009bf0 tx=0x7f0848011810 comp rx=0 tx=0).stop 2024-03-12T05:12:20.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 -- 172.21.15.45:0/2288725833 shutdown_connections 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fbd7b0 0x55e772fbfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 --2- 172.21.15.45:0/2288725833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fb42b0 0x55e772fb4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 -- 172.21.15.45:0/2288725833 >> 172.21.15.45:0/2288725833 conn(0x55e772fae990 msgr2=0x55e772faed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 -- 172.21.15.45:0/2288725833 shutdown_connections 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.482+0000 7f085de00740 1 -- 172.21.15.45:0/2288725833 wait complete. 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 -- start start 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 Processor -- start 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fba5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fbacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772bef430 con 0x55e772fbd7b0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772babe00 con 0x55e772fb4bd0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.483+0000 7f085de00740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772ea0fd0 con 0x55e772fb42b0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f085de00740 10 client.0 _ll_register_callbacks cb 0x55e772ef6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34292/0 (socket says 172.21.15.45:34292) 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f08597cb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fbacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 -- 172.21.15.45:0/414990158 learned_addr learned my addr 172.21.15.45:0/414990158 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0853fff640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fba5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 -- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 msgr2=0x55e772fba5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fba5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 -- 172.21.15.45:0/414990158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 msgr2=0x55e772fbacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fbacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 -- 172.21.15.45:0/414990158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0848009850 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0858fca640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6804ce4186f0d660 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.484+0000 7f0833fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844003b40 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0844046040 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e772c1b7e0 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0844003ca0 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f085de00740 10 client.25372 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e772f1a610 con 0x55e772fb42b0 2024-03-12T05:12:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.485+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e772c1b7e0 con 0x55e772fb42b0 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f084403c040 con 0x55e772fb42b0 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 10 client.25372 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 10 client.25372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 20 client.25372 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e772fcd9e0 0x55e772fcfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 4 client.25372 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.487+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e772c58720 con 0x55e772fcd9e0 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.488+0000 7f0853fff640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e772fcd9e0 0x55e772fcfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.488+0000 7f0853fff640 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e772fcd9e0 0x55e772fcfe00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c01d6e67980435f8 server_cookie=4c34af037ccab647 in_seq=0 out_seq=0 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.488+0000 7f0833fff640 10 client.25372 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f0833fff640 1 -- 172.21.15.45:0/414990158 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e772c58720 con 0x55e772fcd9e0 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f0833fff640 10 client.25372 handle_command_reply: tid=0 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 client.25372 shutdown 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 msgr2=0x55e772fbd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844009260 comp rx=0 tx=0).stop 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 shutdown_connections 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e772fcd9e0 0x55e772fcfe00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fbacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fba5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 --2- 172.21.15.45:0/414990158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbd250 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 >> 172.21.15.45:0/414990158 conn(0x55e772fae990 msgr2=0x55e772fb0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f0833fff640 0 client.25372 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 shutdown_connections 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 1 -- 172.21.15.45:0/414990158 wait complete. 2024-03-12T05:12:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.489+0000 7f085de00740 20 client.25372 trim_cache size 0 max 16384 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.519+0000 7f085de00740 1 Processor -- start 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.519+0000 7f085de00740 1 -- start start 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fb8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c58720 con 0x55e772fbd7b0 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0853fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbcd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34300/0 (socket says 172.21.15.45:34300) 2024-03-12T05:12:20.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772c1b7e0 con 0x55e772fb4bd0 2024-03-12T05:12:20.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0858fca640 1 -- 172.21.15.45:0/1228700578 learned_addr learned my addr 172.21.15.45:0/1228700578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f08597cb640 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fb8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772faa0f0 con 0x55e772fb42b0 2024-03-12T05:12:20.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0853fff640 1 -- 172.21.15.45:0/1228700578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 msgr2=0x55e772fbcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0853fff640 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0853fff640 1 -- 172.21.15.45:0/1228700578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 msgr2=0x55e772fb8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.520+0000 7f0853fff640 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fb8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f0853fff640 1 -- 172.21.15.45:0/1228700578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e772eb6ac0 con 0x55e772fb4bd0 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f08597cb640 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fb8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f0853fff640 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0848000c00 tx=0x7f0848014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e28f69f8e0b285dc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f083a7fc640 1 -- 172.21.15.45:0/1228700578 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848032070 con 0x55e772fb4bd0 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f083a7fc640 1 -- 172.21.15.45:0/1228700578 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0848009ec0 con 0x55e772fb4bd0 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.521+0000 7f083a7fc640 1 -- 172.21.15.45:0/1228700578 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848032070 con 0x55e772fb4bd0 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 msgr2=0x55e772fbd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0848000c00 tx=0x7f0848014000 comp rx=0 tx=0).stop 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 shutdown_connections 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e772fb8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fbd250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 --2- 172.21.15.45:0/1228700578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fbcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 >> 172.21.15.45:0/1228700578 conn(0x55e772fae990 msgr2=0x55e772fb58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.522+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 shutdown_connections 2024-03-12T05:12:20.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.523+0000 7f085de00740 1 -- 172.21.15.45:0/1228700578 wait complete. 2024-03-12T05:12:20.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.523+0000 7f085de00740 1 -- start start 2024-03-12T05:12:20.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.524+0000 7f085de00740 1 Processor -- start 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.525+0000 7f085de00740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fc1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.525+0000 7f085de00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.525+0000 7f085de00740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e773023ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.525+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fc1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.525+0000 7f0858fca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fc1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34304/0 (socket says 172.21.15.45:34304) 2024-03-12T05:12:20.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f0853fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f0853fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38752/0 (socket says 172.21.15.45:38752) 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f0858fca640 1 -- 172.21.15.45:0/1309327036 learned_addr learned my addr 172.21.15.45:0/1309327036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e772fc23e0 con 0x55e772fbd7b0 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f08597cb640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e773023ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e773024420 con 0x55e772fb4bd0 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7730245e0 con 0x55e772fb42b0 2024-03-12T05:12:20.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.526+0000 7f085de00740 10 client.0 _ll_register_callbacks cb 0x7f08480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 -- 172.21.15.45:0/1309327036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 msgr2=0x55e772fc1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fc1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 -- 172.21.15.45:0/1309327036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 msgr2=0x55e773023ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e773023ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541aee10 con 0x55e772fb4bd0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0853fff640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55e772fae2a0 tx=0x7f08480049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15dc9b9635635515 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0833fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848032030 con 0x55e772fb4bd0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0848033070 con 0x55e772fb4bd0 2024-03-12T05:12:20.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.527+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e772ca7890 con 0x55e772fb4bd0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f084802a040 con 0x55e772fb4bd0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 10 client.25379 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e772eda570 con 0x55e772fb4bd0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0848005190 con 0x55e772fb4bd0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f0848021040 con 0x55e772fb4bd0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 10 client.25379 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 10 client.25379 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 20 client.25379 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e773025aa0 0x55e773027e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 4 client.25379 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.528+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e772ca7890 con 0x55e773025aa0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.529+0000 7f0858fca640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e773025aa0 0x55e773027e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.529+0000 7f0833fff640 10 client.25379 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:20.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.529+0000 7f0858fca640 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e773025aa0 0x55e773027e90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed828226705925a server_cookie=a4c36b2320529750 in_seq=0 out_seq=0 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.529+0000 7f0833fff640 1 -- 172.21.15.45:0/1309327036 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e772ca7890 con 0x55e773025aa0 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.529+0000 7f0833fff640 10 client.25379 handle_command_reply: tid=0 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 client.25379 shutdown 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 msgr2=0x55e772fc1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55e772fae2a0 tx=0x7f08480049d0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 shutdown_connections 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e773025aa0 0x55e773027e90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e772fbd7b0 0x55e773023ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f0833fff640 0 client.25379 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:20.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e772fb4bd0 0x55e772fc1c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 --2- 172.21.15.45:0/1309327036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e772fb42b0 0x55e772fc1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 >> 172.21.15.45:0/1309327036 conn(0x55e772fae990 msgr2=0x55e772fb8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 shutdown_connections 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 1 -- 172.21.15.45:0/1309327036 wait complete. 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.530+0000 7f085de00740 20 client.25379 trim_cache size 0 max 16384 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:12:20.530 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:20.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 msgr2=0x7f083c0a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 0x7f083c0a4200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0844006010 tx=0x7f0844005a20 comp rx=0 tx=0).stop 2024-03-12T05:12:20.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 msgr2=0x7f08541a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f084c00eb40 tx=0x7f084c00c2d0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 shutdown_connections 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f083c0a1d40 0x7f083c0a4200 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0854116210 0x7f08541a6980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f085410b860 0x7f08541a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 --2- 172.21.15.45:0/1476299951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08541088d0 0x7f08541a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.531+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 >> 172.21.15.45:0/1476299951 conn(0x7f085406c270 msgr2=0x7f0854071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.532+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 shutdown_connections 2024-03-12T05:12:20.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.532+0000 7f085b22e640 1 -- 172.21.15.45:0/1476299951 wait complete. 2024-03-12T05:12:20.546 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:12:20.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:12:20.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:20 smithi045 ceph-mon[32174]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 21 MiB/s wr, 1.43k op/s 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 -- 172.21.15.45:0/2694476189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101d30 msgr2=0x7f519c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 --2- 172.21.15.45:0/2694476189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101d30 0x7f519c105d70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5190009900 tx=0x7f519002bc40 comp rx=0 tx=0).stop 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 -- 172.21.15.45:0/2694476189 shutdown_connections 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 --2- 172.21.15.45:0/2694476189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 --2- 172.21.15.45:0/2694476189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101d30 0x7f519c105d70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 --2- 172.21.15.45:0/2694476189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c101380 0x7f519c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 -- 172.21.15.45:0/2694476189 >> 172.21.15.45:0/2694476189 conn(0x7f519c078580 msgr2=0x7f519c078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:20.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 -- 172.21.15.45:0/2694476189 shutdown_connections 2024-03-12T05:12:20.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.714+0000 7f51a233a640 1 -- 172.21.15.45:0/2694476189 wait complete. 2024-03-12T05:12:20.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.715+0000 7f51a233a640 1 Processor -- start 2024-03-12T05:12:20.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.715+0000 7f51a233a640 1 -- start start 2024-03-12T05:12:20.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 0x7f519c1a2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 0x7f519c10d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c10dae0 con 0x7f519c106450 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c10dc50 con 0x7f519c10d030 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a233a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c10de10 con 0x7f519c101380 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f519bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 0x7f519c1a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34840/0 (socket says 172.21.15.45:34840) 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f519b7fe640 1 -- 172.21.15.45:0/3536804622 learned_addr learned my addr 172.21.15.45:0/3536804622 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.716+0000 7f51a08d7640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 0x7f519c10d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 -- 172.21.15.45:0/3536804622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 msgr2=0x7f519c1a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 0x7f519c1a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 -- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 msgr2=0x7f519c1a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:20.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 -- 172.21.15.45:0/3536804622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194009840 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f519bfff640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 0x7f519c1a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f519b7fe640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51a08d7640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 0x7f519c10d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f519400ec50 tx=0x7f519400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.717+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194003b70 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.718+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f519400ce70 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.718+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5190009560 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.718+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c1aac50 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.718+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194010670 con 0x7f519c10d030 2024-03-12T05:12:20.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.719+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5188005290 con 0x7f519c10d030 2024-03-12T05:12:20.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.720+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f51940108b0 con 0x7f519c10d030 2024-03-12T05:12:20.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.721+0000 7f51997fa640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 0x7f51840a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:20.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.722+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5194014070 con 0x7f519c10d030 2024-03-12T05:12:20.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:20 smithi130 ceph-mon[35398]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 21 MiB/s wr, 1.43k op/s 2024-03-12T05:12:20.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.722+0000 7f519bfff640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 0x7f51840a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:20.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.723+0000 7f519bfff640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 0x7f51840a4200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f518c005d00 tx=0x7f518c00b410 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:20.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:20.726+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5194096ca0 con 0x7f519c10d030 2024-03-12T05:12:20.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:20 smithi116 ceph-mon[35486]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 21 MiB/s wr, 1.43k op/s 2024-03-12T05:12:21.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.122+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5188005810 con 0x7f519c10d030 2024-03-12T05:12:21.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.123+0000 7f51997fa640 1 -- 172.21.15.45:0/3536804622 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11403 (secure 0 0 0) 0x7f51940ba020 con 0x7f519c10d030 2024-03-12T05:12:21.120 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:12:21.120 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:11:18.408486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:12:21.120 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 29 2024-03-12T05:12:21.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 msgr2=0x7f51840a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 0x7f51840a4200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f518c005d00 tx=0x7f518c00b410 comp rx=0 tx=0).stop 2024-03-12T05:12:21.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 msgr2=0x7f519c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 0x7f519c10d4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f519400ec50 tx=0x7f519400c6a0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 shutdown_connections 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f51840a1d40 0x7f51840a4200 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f518c005d00 tx=0x7f518c00b410 comp rx=0 tx=0).stop 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f519c106450 0x7f519c1a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c10d030 0x7f519c10d4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 --2- 172.21.15.45:0/3536804622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f519c101380 0x7f519c1a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.127+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 >> 172.21.15.45:0/3536804622 conn(0x7f519c078580 msgr2=0x7f519c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.128+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 shutdown_connections 2024-03-12T05:12:21.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.128+0000 7f51a233a640 1 -- 172.21.15.45:0/3536804622 wait complete. 2024-03-12T05:12:21.143 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:21.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:12:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 -- 172.21.15.45:0/1533080255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1508100260 msgr2=0x7f1508100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 --2- 172.21.15.45:0/1533080255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1508100260 0x7f1508100640 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009900 tx=0x7f14f802bc40 comp rx=0 tx=0).stop 2024-03-12T05:12:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 -- 172.21.15.45:0/1533080255 shutdown_connections 2024-03-12T05:12:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 --2- 172.21.15.45:0/1533080255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508105330 0x7f150810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 --2- 172.21.15.45:0/1533080255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508100c10 0x7f1508104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 --2- 172.21.15.45:0/1533080255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1508100260 0x7f1508100640 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 -- 172.21.15.45:0/1533080255 >> 172.21.15.45:0/1533080255 conn(0x7f1508077180 msgr2=0x7f1508077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 -- 172.21.15.45:0/1533080255 shutdown_connections 2024-03-12T05:12:21.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.357+0000 7f150cedf640 1 -- 172.21.15.45:0/1533080255 wait complete. 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.358+0000 7f150cedf640 1 Processor -- start 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.358+0000 7f150cedf640 1 -- start start 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 0x7f1508197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 0x7f150819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38792/0 (socket says 172.21.15.45:38792) 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 0x7f1508197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 0x7f1508197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34338/0 (socket says 172.21.15.45:34338) 2024-03-12T05:12:21.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150659c640 1 -- 172.21.15.45:0/2592548585 learned_addr learned my addr 172.21.15.45:0/2592548585 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819c7a0 con 0x7f150819bdb0 2024-03-12T05:12:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819c910 con 0x7f1508100260 2024-03-12T05:12:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819ca80 con 0x7f1508105330 2024-03-12T05:12:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.359+0000 7f1506d9d640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 0x7f150819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 -- 172.21.15.45:0/2592548585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 msgr2=0x7f1508197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 0x7f1508197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 -- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 msgr2=0x7f150819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 0x7f150819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f8009560 con 0x7f1508100260 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f1506d9d640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 0x7f150819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150659c640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f14f802c180 tx=0x7f14f80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f14f77fe640 1 -- 172.21.15.45:0/2592548585 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80044e0 con 0x7f1508100260 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f14f77fe640 1 -- 172.21.15.45:0/2592548585 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f14f8005d50 con 0x7f1508100260 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15081a4060 con 0x7f1508100260 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f14f77fe640 1 -- 172.21.15.45:0/2592548585 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8005680 con 0x7f1508100260 2024-03-12T05:12:21.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.360+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15081a44e0 con 0x7f1508100260 2024-03-12T05:12:21.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.362+0000 7f14f77fe640 1 -- 172.21.15.45:0/2592548585 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f14f8049460 con 0x7f1508100260 2024-03-12T05:12:21.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.363+0000 7f14f77fe640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 0x7f14f00a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.363+0000 7f14f77fe640 1 -- 172.21.15.45:0/2592548585 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f14f80f0ef0 con 0x7f1508100260 2024-03-12T05:12:21.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.363+0000 7f1505d9b640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 0x7f14f00a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.365+0000 7f1505d9b640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 0x7f14f00a4280 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15080fd620 tx=0x7f14fc00b390 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.365+0000 7f150fb91740 1 Processor -- start 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.365+0000 7f150fb91740 1 -- start start 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.365+0000 7f150fb91740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a25a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.365+0000 7f150fb91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f150fb91740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a30f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f150fb91740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c89dc960 con 0x5565c8a2eb60 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f150fb91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8990310 con 0x5565c8a25f80 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f150fb91740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c898d840 con 0x5565c8a25660 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1506d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a30f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38802/0 (socket says 172.21.15.45:38802) 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 -- 172.21.15.45:0/601604379 learned_addr learned my addr 172.21.15.45:0/601604379 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f150659c640 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a25a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 -- 172.21.15.45:0/601604379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 msgr2=0x5565c8a25a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a25a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 -- 172.21.15.45:0/601604379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 msgr2=0x5565c8a30f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a30f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.366+0000 7f1505d9b640 1 -- 172.21.15.45:0/601604379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565c894b060 con 0x5565c8a25f80 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.367+0000 7f1505d9b640 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b6e0 tx=0x7f14fc013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77bc8dfd992b86d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.367+0000 7f14f4ff9640 1 -- 172.21.15.45:0/601604379 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc005220 con 0x5565c8a25f80 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.367+0000 7f14f4ff9640 1 -- 172.21.15.45:0/601604379 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f14fc015070 con 0x5565c8a25f80 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.367+0000 7f14f4ff9640 1 -- 172.21.15.45:0/601604379 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc01e680 con 0x5565c8a25f80 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- 172.21.15.45:0/601604379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 msgr2=0x5565c8a2e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b6e0 tx=0x7f14fc013000 comp rx=0 tx=0).stop 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- 172.21.15.45:0/601604379 shutdown_connections 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a30f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2e620 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 --2- 172.21.15.45:0/601604379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a25a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- 172.21.15.45:0/601604379 >> 172.21.15.45:0/601604379 conn(0x5565c8a1fd90 msgr2=0x5565c8a20170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- 172.21.15.45:0/601604379 shutdown_connections 2024-03-12T05:12:21.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- 172.21.15.45:0/601604379 wait complete. 2024-03-12T05:12:21.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.368+0000 7f150fb91740 1 -- start start 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 Processor -- start 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a2c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8989020 con 0x5565c8a2eb60 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c865c430 con 0x5565c8a25f80 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150fb91740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8618e00 con 0x5565c8a25660 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34358/0 (socket says 172.21.15.45:34358) 2024-03-12T05:12:21.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f150659c640 1 -- 172.21.15.45:0/155015377 learned_addr learned my addr 172.21.15.45:0/155015377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150fb91740 10 client.0 _ll_register_callbacks cb 0x5565c8930cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.369+0000 7f1505d9b640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f1506d9d640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a2c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 -- 172.21.15.45:0/155015377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 msgr2=0x5565c8a2c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 -- 172.21.15.45:0/155015377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 msgr2=0x5565c8a2c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a2c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 -- 172.21.15.45:0/155015377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc010c90 con 0x5565c8a25660 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f150659c640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009850 tx=0x7f14f8004f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=201176a534c35ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f14edffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.370+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80c6120 con 0x5565c8a25660 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f14f80ee740 con 0x5565c8a25660 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565c86887e0 con 0x5565c8a25660 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14f80ee900 con 0x5565c8a25660 2024-03-12T05:12:21.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f150fb91740 10 client.25396 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5565c89876a0 con 0x5565c8a25660 2024-03-12T05:12:21.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.371+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5565c86887e0 con 0x5565c8a25660 2024-03-12T05:12:21.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f14f803c070 con 0x5565c8a25660 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 10 client.25396 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 10 client.25396 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 20 client.25396 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a1e750 0x5565c8a1eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 4 client.25396 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565c86c5720 con 0x5565c8a1e750 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.372+0000 7f1505d9b640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a1e750 0x5565c8a1eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.373+0000 7f1505d9b640 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a1e750 0x5565c8a1eb30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c55bd4110a135ad8 server_cookie=a818444481f582ea in_seq=0 out_seq=0 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.373+0000 7f14edffb640 10 client.25396 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.373+0000 7f14edffb640 1 -- 172.21.15.45:0/155015377 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5565c86c5720 con 0x5565c8a1e750 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.373+0000 7f14edffb640 10 client.25396 handle_command_reply: tid=0 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.373+0000 7f150fb91740 1 client.25396 shutdown 2024-03-12T05:12:21.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 msgr2=0x5565c8a2bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009850 tx=0x7f14f8004f00 comp rx=0 tx=0).stop 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 shutdown_connections 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a1e750 0x5565c8a1eb30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a2eb60 0x5565c8a2c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f14edffb640 0 client.25396 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a2c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 --2- 172.21.15.45:0/155015377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2bc20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 >> 172.21.15.45:0/155015377 conn(0x5565c8a1fd90 msgr2=0x5565c8a21250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 shutdown_connections 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 1 -- 172.21.15.45:0/155015377 wait complete. 2024-03-12T05:12:21.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.374+0000 7f150fb91740 20 client.25396 trim_cache size 0 max 16384 2024-03-12T05:12:21.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.396+0000 7f150fb91740 1 Processor -- start 2024-03-12T05:12:21.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.396+0000 7f150fb91740 1 -- start start 2024-03-12T05:12:21.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.396+0000 7f150fb91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.396+0000 7f150fb91740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.396+0000 7f150fb91740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f150659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f150659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38820/0 (socket says 172.21.15.45:38820) 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1505d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34372/0 (socket says 172.21.15.45:34372) 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1506d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1506d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34894/0 (socket says 172.21.15.45:34894) 2024-03-12T05:12:21.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f150fb91740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c890e130 con 0x5565c8a25f80 2024-03-12T05:12:21.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8a1f650 con 0x5565c8a1e750 2024-03-12T05:12:21.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1506d9d640 1 -- 172.21.15.45:0/274156830 learned_addr learned my addr 172.21.15.45:0/274156830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8a20d90 con 0x5565c8a25660 2024-03-12T05:12:21.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1505d9b640 1 -- 172.21.15.45:0/274156830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 msgr2=0x5565c8a2e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.397+0000 7f1505d9b640 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f1505d9b640 1 -- 172.21.15.45:0/274156830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 msgr2=0x5565c8a3f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f1505d9b640 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f1505d9b640 1 -- 172.21.15.45:0/274156830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565c8923c20 con 0x5565c8a25660 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f1506d9d640 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f150659c640 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f1505d9b640 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b9c0 tx=0x7f14fc00b5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed3a0bfe46ee36e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f14effff640 1 -- 172.21.15.45:0/274156830 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc005560 con 0x5565c8a25660 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f14effff640 1 -- 172.21.15.45:0/274156830 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f14fc004b30 con 0x5565c8a25660 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.398+0000 7f14effff640 1 -- 172.21.15.45:0/274156830 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc035330 con 0x5565c8a25660 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 msgr2=0x5565c8a2ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00b9c0 tx=0x7f14fc00b5c0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 shutdown_connections 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25f80 0x5565c8a3f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a25660 0x5565c8a2ad80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 --2- 172.21.15.45:0/274156830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a1e750 0x5565c8a2e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.399+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 >> 172.21.15.45:0/274156830 conn(0x5565c8a1fd90 msgr2=0x5565c8a16a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.400+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 shutdown_connections 2024-03-12T05:12:21.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.400+0000 7f150fb91740 1 -- 172.21.15.45:0/274156830 wait complete. 2024-03-12T05:12:21.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.400+0000 7f150fb91740 1 -- start start 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.400+0000 7f150fb91740 1 Processor -- start 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 0x5565c8a3df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a90020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8a90560 con 0x5565c8a25660 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8a906a0 con 0x5565c8a25f80 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565c8a90830 con 0x5565c8a1e750 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34388/0 (socket says 172.21.15.45:34388) 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 -- 172.21.15.45:0/726871686 learned_addr learned my addr 172.21.15.45:0/726871686 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f1506d9d640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a90020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f1505d9b640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 0x5565c8a3df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150fb91740 10 client.0 _ll_register_callbacks cb 0x7f14fc00b8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 -- 172.21.15.45:0/726871686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 msgr2=0x5565c8a90020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a90020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 -- 172.21.15.45:0/726871686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 msgr2=0x5565c8a3df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 0x5565c8a3df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.401+0000 7f150659c640 1 -- 172.21.15.45:0/726871686 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1500009df0 con 0x5565c8a1e750 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f1505d9b640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 0x5565c8a3df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14fc015070 con 0x5565c8a1e750 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f150659c640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14f80d3040 tx=0x7f14f80d6430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756c9bad051047b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f14edffb640 10 client.25402 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f14edffb640 1 -- 172.21.15.45:0/726871686 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80b9c60 con 0x5565c8a1e750 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f14edffb640 1 -- 172.21.15.45:0/726871686 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f14f80c6120 con 0x5565c8a1e750 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f14edffb640 1 -- 172.21.15.45:0/726871686 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14f80498d0 con 0x5565c8a1e750 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f150fb91740 10 client.25402 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.402+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x5565c894b060 con 0x5565c8a1e750 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f14edffb640 1 -- 172.21.15.45:0/726871686 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f14f803c070 con 0x5565c8a1e750 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 10 client.25402 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 10 client.25402 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 20 client.25402 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a2eb60 0x5565c8a939c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 4 client.25402 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5565c8714890 con 0x5565c8a2eb60 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f1505d9b640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a2eb60 0x5565c8a939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f1505d9b640 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a2eb60 0x5565c8a939c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b006fb4b749f0813 server_cookie=c9498054cbedb4e2 in_seq=0 out_seq=0 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.403+0000 7f14edffb640 10 client.25402 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f14edffb640 1 -- 172.21.15.45:0/726871686 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5565c8714890 con 0x5565c8a2eb60 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f14edffb640 10 client.25402 handle_command_reply: tid=0 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 client.25402 shutdown 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 msgr2=0x5565c8a3d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f14f80d3040 tx=0x7f14f80d6430 comp rx=0 tx=0).stop 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 shutdown_connections 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5565c8a2eb60 0x5565c8a939c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f14edffb640 0 client.25402 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5565c8a25660 0x5565c8a3df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565c8a25f80 0x5565c8a90020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 --2- 172.21.15.45:0/726871686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565c8a1e750 0x5565c8a3d9c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 >> 172.21.15.45:0/726871686 conn(0x5565c8a1fd90 msgr2=0x5565c8a2ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 shutdown_connections 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.404+0000 7f150fb91740 1 -- 172.21.15.45:0/726871686 wait complete. 2024-03-12T05:12:21.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.405+0000 7f150fb91740 20 client.25402 trim_cache size 0 max 16384 2024-03-12T05:12:21.406 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.405+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 msgr2=0x7f14f00a4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.405+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 0x7f14f00a4280 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f15080fd620 tx=0x7f14fc00b390 comp rx=0 tx=0).stop 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.405+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 msgr2=0x7f15081974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.405+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f14f802c180 tx=0x7f14f80042f0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 shutdown_connections 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f14f00a1dc0 0x7f14f00a4280 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f150819bdb0 0x7f150819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1508105330 0x7f1508197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 --2- 172.21.15.45:0/2592548585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1508100260 0x7f15081974f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 >> 172.21.15.45:0/2592548585 conn(0x7f1508077180 msgr2=0x7f15080fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 shutdown_connections 2024-03-12T05:12:21.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.406+0000 7f150cedf640 1 -- 172.21.15.45:0/2592548585 wait complete. 2024-03-12T05:12:21.418 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:21 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3536804622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:21.421 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:21.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:12:21.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1888905024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb0100bf0 msgr2=0x7f7eb0101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1888905024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0101070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009900 tx=0x7f7e9c02bc40 comp rx=0 tx=0).stop 2024-03-12T05:12:21.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1888905024 shutdown_connections 2024-03-12T05:12:21.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1888905024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb01015b0 0x7f7eb010cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1888905024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0101070 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1888905024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0105080 0x7f7eb0105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1888905024 >> 172.21.15.45:0/1888905024 conn(0x7f7eb00761d0 msgr2=0x7f7eb00fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1888905024 shutdown_connections 2024-03-12T05:12:21.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.598+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1888905024 wait complete. 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 Processor -- start 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 -- start start 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 0x7f7eb0196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb01a5ac0 con 0x7f7eb0100bf0 2024-03-12T05:12:21.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb01a5c30 con 0x7f7eb0105080 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.599+0000 7f7eb6e0e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb01a5df0 con 0x7f7eb01015b0 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eb53ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 0x7f7eb0196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34414/0 (socket says 172.21.15.45:34414) 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34918/0 (socket says 172.21.15.45:34918) 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eb53ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 0x7f7eb0196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38846/0 (socket says 172.21.15.45:38846) 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 -- 172.21.15.45:0/1549849768 learned_addr learned my addr 172.21.15.45:0/1549849768 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 -- 172.21.15.45:0/1549849768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 msgr2=0x7f7eb0196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 0x7f7eb0196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 -- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 msgr2=0x7f7eb0196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 -- 172.21.15.45:0/1549849768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c009560 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.600+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c000c00 tx=0x7f7e9c02cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.601+0000 7f7eadffb640 1 -- 172.21.15.45:0/1549849768 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c004490 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.601+0000 7f7eadffb640 1 -- 172.21.15.45:0/1549849768 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e9c003aa0 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.601+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb01a6070 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.601+0000 7f7eadffb640 1 -- 172.21.15.45:0/1549849768 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c005480 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.601+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb01a6580 con 0x7f7eb01015b0 2024-03-12T05:12:21.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.604+0000 7f7eadffb640 1 -- 172.21.15.45:0/1549849768 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f7e9c003c10 con 0x7f7eb01015b0 2024-03-12T05:12:21.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 Processor -- start 2024-03-12T05:12:21.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 -- start start 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eadffb640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 0x7f7ea00a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x557609435a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609438350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576093df960 con 0x557609424a10 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557609393310 con 0x557609424060 2024-03-12T05:12:21.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557609390840 con 0x557609435f60 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eadffb640 1 -- 172.21.15.45:0/1549849768 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7e9c02de40 con 0x7f7eb01015b0 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 0x7f7ea00a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38852/0 (socket says 172.21.15.45:38852) 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.605+0000 7f7eb4baa640 1 -- 172.21.15.45:0/451011835 learned_addr learned my addr 172.21.15.45:0/451011835 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609438350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 -- 172.21.15.45:0/451011835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 msgr2=0x557609438350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609438350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 -- 172.21.15.45:0/451011835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 msgr2=0x557609435a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eaffff640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x557609435a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x557609435a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 -- 172.21.15.45:0/451011835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55760934e060 con 0x557609424060 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eaffff640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x557609435a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609438350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.606+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4009ca0 tx=0x7f7ea4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1eaa0ca7e1de451 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.607+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 0x7f7ea00a4280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55760941fda0 tx=0x7f7ea8009920 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.607+0000 7f7e9affd640 1 -- 172.21.15.45:0/451011835 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4004230 con 0x557609424060 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.607+0000 7f7e9affd640 1 -- 172.21.15.45:0/451011835 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ea4004390 con 0x557609424060 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.607+0000 7f7e9affd640 1 -- 172.21.15.45:0/451011835 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4019780 con 0x557609424060 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- 172.21.15.45:0/451011835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 msgr2=0x557609424440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4009ca0 tx=0x7f7ea4011810 comp rx=0 tx=0).stop 2024-03-12T05:12:21.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- 172.21.15.45:0/451011835 shutdown_connections 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x557609435a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609438350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/451011835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424060 0x557609424440 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- 172.21.15.45:0/451011835 >> 172.21.15.45:0/451011835 conn(0x55760941e980 msgr2=0x55760941ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- 172.21.15.45:0/451011835 shutdown_connections 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- 172.21.15.45:0/451011835 wait complete. 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.608+0000 7f7eb99e9740 1 -- start start 2024-03-12T05:12:21.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.609+0000 7f7eb99e9740 1 Processor -- start 2024-03-12T05:12:21.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.609+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760942f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609435f60 0x557609435520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55760938c020 con 0x557609435f60 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55760905f430 con 0x557609424a10 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55760901be00 con 0x557609424060 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb99e9740 10 client.0 _ll_register_callbacks cb 0x557609333cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38866/0 (socket says 172.21.15.45:38866) 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 -- 172.21.15.45:0/960288139 learned_addr learned my addr 172.21.15.45:0/960288139 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609435f60 0x557609435520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760942f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 -- 172.21.15.45:0/960288139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 msgr2=0x55760942f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760942f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 -- 172.21.15.45:0/960288139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609435f60 msgr2=0x557609435520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609435f60 0x557609435520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 -- 172.21.15.45:0/960288139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea40098d0 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760942f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7eaffff640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006010 tx=0x7f7e9c0bccf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ecfc7d35b51b0f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.610+0000 7f7e8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0bce90 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7e9c0bcff0 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55760908b7e0 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e9c0c4f40 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7eb99e9740 10 client.25415 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.611+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55760938a6a0 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55760908b7e0 con 0x557609424a10 2024-03-12T05:12:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f7e9c03c070 con 0x557609424a10 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 10 client.25415 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 10 client.25415 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 20 client.25415 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55760944e830 0x557609450c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 4 client.25415 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576090c8720 con 0x55760944e830 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.612+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55760944e830 0x557609450c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.613+0000 7f7e8ffff640 10 client.25415 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:21.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.613+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55760944e830 0x557609450c50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=daccf5ad20fa3d94 server_cookie=e38b3b1885805a1f in_seq=0 out_seq=0 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.613+0000 7f7e8ffff640 1 -- 172.21.15.45:0/960288139 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5576090c8720 con 0x55760944e830 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.613+0000 7f7e8ffff640 10 client.25415 handle_command_reply: tid=0 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.613+0000 7f7eb99e9740 1 client.25415 shutdown 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 msgr2=0x55760942fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006010 tx=0x7f7e9c0bccf0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 shutdown_connections 2024-03-12T05:12:21.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55760944e830 0x557609450c50 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609435f60 0x557609435520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x55760942fd40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/960288139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760942f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7e8ffff640 0 client.25415 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 >> 172.21.15.45:0/960288139 conn(0x55760941e980 msgr2=0x5576094206d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 shutdown_connections 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 1 -- 172.21.15.45:0/960288139 wait complete. 2024-03-12T05:12:21.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.614+0000 7f7eb99e9740 20 client.25415 trim_cache size 0 max 16384 2024-03-12T05:12:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.639+0000 7f7eb99e9740 1 Processor -- start 2024-03-12T05:12:21.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.639+0000 7f7eb99e9740 1 -- start start 2024-03-12T05:12:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.641+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.641+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.641+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.641+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34934/0 (socket says 172.21.15.45:34934) 2024-03-12T05:12:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609427e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38872/0 (socket says 172.21.15.45:38872) 2024-03-12T05:12:21.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb4baa640 1 -- 172.21.15.45:0/1817403954 learned_addr learned my addr 172.21.15.45:0/1817403954 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557609311130 con 0x557609424060 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609427e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576090c8720 con 0x557609424a10 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.642+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55760908b7e0 con 0x557609435f60 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 -- 172.21.15.45:0/1817403954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 msgr2=0x557609427e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609427e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 -- 172.21.15.45:0/1817403954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 msgr2=0x5576094354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 -- 172.21.15.45:0/1817403954 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557609326c20 con 0x557609424a10 2024-03-12T05:12:21.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c005ab0 tx=0x7f7e9c0c5550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=840c9af89ef44853 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7e9a7fc640 1 -- 172.21.15.45:0/1817403954 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0bc590 con 0x557609424a10 2024-03-12T05:12:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7e9a7fc640 1 -- 172.21.15.45:0/1817403954 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7e9c0b8ed0 con 0x557609424a10 2024-03-12T05:12:21.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.643+0000 7f7e9a7fc640 1 -- 172.21.15.45:0/1817403954 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0bd0f0 con 0x557609424a10 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 msgr2=0x557609429be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c005ab0 tx=0x7f7e9c0c5550 comp rx=0 tx=0).stop 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 shutdown_connections 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609435f60 0x557609427e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609424a10 0x557609429be0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1817403954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424060 0x5576094354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.644+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 >> 172.21.15.45:0/1817403954 conn(0x55760941e980 msgr2=0x5576094294d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.645+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 shutdown_connections 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.645+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1817403954 wait complete. 2024-03-12T05:12:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.645+0000 7f7eb99e9740 1 -- start start 2024-03-12T05:12:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.645+0000 7f7eb99e9740 1 Processor -- start 2024-03-12T05:12:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x55760949e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 0x557609498cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557609499280 con 0x557609424a10 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576094993f0 con 0x557609435f60 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576094995b0 con 0x557609424060 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb99e9740 10 client.0 _ll_register_callbacks cb 0x7f7ea40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eaffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x55760949e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34458/0 (socket says 172.21.15.45:34458) 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 -- 172.21.15.45:0/1651411908 learned_addr learned my addr 172.21.15.45:0/1651411908 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 -- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 msgr2=0x557609498cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 0x557609498cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 0x557609498cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 -- 172.21.15.45:0/1651411908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 msgr2=0x55760949e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x55760949e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb4baa640 1 -- 172.21.15.45:0/1651411908 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea401e040 con 0x557609424060 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x55760949e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.646+0000 7f7eb53ab640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 0x557609498cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.647+0000 7f7eb4baa640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4009460 tx=0x7f7ea4003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d0fa2bbc6f4d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.647+0000 7f7e8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.647+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4004170 con 0x557609424060 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.647+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ea4004390 con 0x557609424060 2024-03-12T05:12:21.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7ea4005040 con 0x557609424060 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ea4005200 con 0x557609424060 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7eb99e9740 10 client.25417 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55760938a6a0 con 0x557609424060 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ea4004390 con 0x557609424060 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f7ea40292e0 con 0x557609424060 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7eb99e9740 10 client.25417 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7eb99e9740 10 client.25417 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.648+0000 7f7eb99e9740 20 client.25417 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5576094a6360 0x5576094a8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7eb99e9740 4 client.25417 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557609117890 con 0x5576094a6360 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5576094a6360 0x5576094a8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7e8ffff640 10 client.25417 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7eaffff640 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5576094a6360 0x5576094a8780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4a5810ae8e3bb43 server_cookie=db1e35124be74019 in_seq=0 out_seq=0 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7e8ffff640 1 -- 172.21.15.45:0/1651411908 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557609117890 con 0x5576094a6360 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.649+0000 7f7e8ffff640 10 client.25417 handle_command_reply: tid=0 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 client.25417 shutdown 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 msgr2=0x55760949e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4009460 tx=0x7f7ea4003f10 comp rx=0 tx=0).stop 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 shutdown_connections 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5576094a6360 0x5576094a8780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557609424a10 0x55760949e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557609435f60 0x557609498cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7e8ffff640 0 client.25417 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 --2- 172.21.15.45:0/1651411908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557609424060 0x55760949e430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 >> 172.21.15.45:0/1651411908 conn(0x55760941e980 msgr2=0x55760941d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 shutdown_connections 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 1 -- 172.21.15.45:0/1651411908 wait complete. 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.650+0000 7f7eb99e9740 20 client.25417 trim_cache size 0 max 16384 2024-03-12T05:12:21.650 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 msgr2=0x7f7ea00a4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 0x7f7ea00a4280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55760941fda0 tx=0x7f7ea8009920 comp rx=0 tx=0).stop 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 msgr2=0x7f7eb01968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c000c00 tx=0x7f7e9c02cd20 comp rx=0 tx=0).stop 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 shutdown_connections 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ea00a1dc0 0x7f7ea00a4280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7eb0105080 0x7f7eb0196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb01015b0 0x7f7eb01968c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 --2- 172.21.15.45:0/1549849768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7eb0100bf0 0x7f7eb0196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 >> 172.21.15.45:0/1549849768 conn(0x7f7eb00761d0 msgr2=0x7f7eb01030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 shutdown_connections 2024-03-12T05:12:21.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.651+0000 7f7eb6e0e640 1 -- 172.21.15.45:0/1549849768 wait complete. 2024-03-12T05:12:21.666 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:21.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:12:21.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:21 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3536804622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:21.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:21 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3536804622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 -- 172.21.15.45:0/2499809961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb0105370 msgr2=0x7f0cb010cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/2499809961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb0105370 0x7f0cb010cf40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009ca0 tx=0x7f0ca802c100 comp rx=0 tx=0).stop 2024-03-12T05:12:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 -- 172.21.15.45:0/2499809961 shutdown_connections 2024-03-12T05:12:21.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/2499809961 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb0100c50 0x7f0cb0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/2499809961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb0105370 0x7f0cb010cf40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/2499809961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb01002a0 0x7f0cb0100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 -- 172.21.15.45:0/2499809961 >> 172.21.15.45:0/2499809961 conn(0x7f0cb0076400 msgr2=0x7f0cb0076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.844+0000 7f0cb50f4640 1 -- 172.21.15.45:0/2499809961 shutdown_connections 2024-03-12T05:12:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.845+0000 7f0cb50f4640 1 -- 172.21.15.45:0/2499809961 wait complete. 2024-03-12T05:12:21.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.846+0000 7f0cb50f4640 1 Processor -- start 2024-03-12T05:12:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.846+0000 7f0cb50f4640 1 -- start start 2024-03-12T05:12:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 0x7f0cb010cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 0x7f0cb0109fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 0x7f0cb01aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb010ad30 con 0x7f0cb01002a0 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb010aea0 con 0x7f0cb010a630 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cb50f4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb01aa560 con 0x7f0cb0105370 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 0x7f0cb0109fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 0x7f0cb010cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 0x7f0cb0109fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34486/0 (socket says 172.21.15.45:34486) 2024-03-12T05:12:21.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 0x7f0cb01aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 0x7f0cb010cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34968/0 (socket says 172.21.15.45:34968) 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0cae59c640 1 -- 172.21.15.45:0/3117513236 learned_addr learned my addr 172.21.15.45:0/3117513236 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 -- 172.21.15.45:0/3117513236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 msgr2=0x7f0cb0109fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 0x7f0cb0109fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 -- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 msgr2=0x7f0cb010cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 0x7f0cb010cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.847+0000 7f0caf59e640 1 -- 172.21.15.45:0/3117513236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca8009900 con 0x7f0cb010a630 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0caf59e640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 0x7f0cb01aa020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009bf0 tx=0x7f0ca8034440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0c97fff640 1 -- 172.21.15.45:0/3117513236 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80325f0 con 0x7f0cb010a630 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0c97fff640 1 -- 172.21.15.45:0/3117513236 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ca8003970 con 0x7f0cb010a630 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb01aa7b0 con 0x7f0cb010a630 2024-03-12T05:12:21.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0c97fff640 1 -- 172.21.15.45:0/3117513236 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80046e0 con 0x7f0cb010a630 2024-03-12T05:12:21.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.848+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01aacb0 con 0x7f0cb010a630 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.852+0000 7f0c97fff640 1 -- 172.21.15.45:0/3117513236 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f0ca803b040 con 0x7f0cb010a630 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.852+0000 7f0cb7da5740 1 Processor -- start 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 -- start start 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e79630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e84b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58e34960 con 0x55ab58e79250 2024-03-12T05:12:21.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58de8310 con 0x55ab58e79b70 2024-03-12T05:12:21.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58de5840 con 0x55ab58e82750 2024-03-12T05:12:21.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0c97fff640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 0x7f0c980a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0caf59e640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 0x7f0c980a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38900/0 (socket says 172.21.15.45:38900) 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0caed9d640 1 -- 172.21.15.45:0/3242494219 learned_addr learned my addr 172.21.15.45:0/3242494219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.853+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e84b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e79630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 -- 172.21.15.45:0/3242494219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 msgr2=0x55ab58e84b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e84b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 -- 172.21.15.45:0/3242494219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 msgr2=0x55ab58e79630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0c97fff640 1 -- 172.21.15.45:0/3117513236 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0ca80f0210 con 0x7f0cb010a630 2024-03-12T05:12:21.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e79630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 -- 172.21.15.45:0/3242494219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab58da3060 con 0x55ab58e79b70 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e79630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caf59e640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 0x7f0c980a4200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ed950 tx=0x7f0ca80e73c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000bd10 tx=0x7f0ca00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=582f1f819abfc0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0c957fa640 1 -- 172.21.15.45:0/3242494219 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0004030 con 0x55ab58e79b70 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.854+0000 7f0c957fa640 1 -- 172.21.15.45:0/3242494219 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ca0015070 con 0x55ab58e79b70 2024-03-12T05:12:21.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0c957fa640 1 -- 172.21.15.45:0/3242494219 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00116a0 con 0x55ab58e79b70 2024-03-12T05:12:21.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3242494219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 msgr2=0x55ab58e82210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000bd10 tx=0x7f0ca00107a0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3242494219 shutdown_connections 2024-03-12T05:12:21.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e84b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79b70 0x55ab58e82210 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3242494219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e79630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.855+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3242494219 >> 172.21.15.45:0/3242494219 conn(0x55ab58e73980 msgr2=0x55ab58e73d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.856+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3242494219 shutdown_connections 2024-03-12T05:12:21.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.856+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3242494219 wait complete. 2024-03-12T05:12:21.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.856+0000 7f0cb7da5740 1 -- start start 2024-03-12T05:12:21.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.856+0000 7f0cb7da5740 1 Processor -- start 2024-03-12T05:12:21.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 0x55ab58e7f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e801e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34980/0 (socket says 172.21.15.45:34980) 2024-03-12T05:12:21.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 0x55ab58e7f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 0x55ab58e7f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38902/0 (socket says 172.21.15.45:38902) 2024-03-12T05:12:21.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58de1020 con 0x55ab58e79b70 2024-03-12T05:12:21.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58ab4430 con 0x55ab58e79250 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58a70e00 con 0x55ab58e82750 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cb7da5740 10 client.0 _ll_register_callbacks cb 0x55ab58d88cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0caed9d640 1 -- 172.21.15.45:0/3809185826 learned_addr learned my addr 172.21.15.45:0/3809185826 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.857+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 -- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 msgr2=0x55ab58e801e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e801e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 -- 172.21.15.45:0/3809185826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 msgr2=0x55ab58e7f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 0x55ab58e7f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 -- 172.21.15.45:0/3809185826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca000b940 con 0x55ab58e79b70 2024-03-12T05:12:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab58ae07e0 con 0x55ab58e79b70 2024-03-12T05:12:21.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0caed9d640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000b620 tx=0x7f0ca0003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80aa18aa3bf9d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0c92ff5640 10 client.15972 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0c92ff5640 1 -- 172.21.15.45:0/3809185826 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0020070 con 0x55ab58e79b70 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.858+0000 7f0c92ff5640 1 -- 172.21.15.45:0/3809185826 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0ca0017070 con 0x55ab58e79b70 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0c92ff5640 1 -- 172.21.15.45:0/3809185826 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca0003900 con 0x55ab58e79b70 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 10 client.15972 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x55ab58ddf6a0 con 0x55ab58e79b70 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0c92ff5640 1 -- 172.21.15.45:0/3809185826 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f0ca0003eb0 con 0x55ab58e79b70 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 10 client.15972 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 10 client.15972 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 20 client.15972 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58e92980 0x55ab58e94da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 4 client.15972 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab58b1d720 con 0x55ab58e92980 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.859+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58e92980 0x55ab58e94da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.860+0000 7f0cae59c640 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58e92980 0x55ab58e94da0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6131f268277cf601 server_cookie=59df32d2b1880cc7 in_seq=0 out_seq=0 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.860+0000 7f0c92ff5640 10 client.15972 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.860+0000 7f0c92ff5640 1 -- 172.21.15.45:0/3809185826 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ab58b1d720 con 0x55ab58e92980 2024-03-12T05:12:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.860+0000 7f0c92ff5640 10 client.15972 handle_command_reply: tid=0 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.860+0000 7f0cb7da5740 1 client.15972 shutdown 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 msgr2=0x55ab58e7fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000b620 tx=0x7f0ca0003da0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 shutdown_connections 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58e92980 0x55ab58e94da0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79b70 0x55ab58e7fca0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e82750 0x55ab58e801e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/3809185826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e79250 0x55ab58e7f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 >> 172.21.15.45:0/3809185826 conn(0x55ab58e73980 msgr2=0x55ab58e756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0c92ff5640 0 client.15972 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 shutdown_connections 2024-03-12T05:12:21.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 1 -- 172.21.15.45:0/3809185826 wait complete. 2024-03-12T05:12:21.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.861+0000 7f0cb7da5740 20 client.15972 trim_cache size 0 max 16384 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.883+0000 7f0cb7da5740 1 Processor -- start 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 -- start start 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e81e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e80000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e81e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e81e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34994/0 (socket says 172.21.15.45:34994) 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e7aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cae59c640 1 -- 172.21.15.45:0/2862729899 learned_addr learned my addr 172.21.15.45:0/2862729899 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58d66130 con 0x55ab58e79250 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58e78eb0 con 0x55ab58e82750 2024-03-12T05:12:21.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58e6f080 con 0x55ab58e79b70 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.884+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e80000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e7aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 -- 172.21.15.45:0/2862729899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 msgr2=0x55ab58e7aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e7aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 -- 172.21.15.45:0/2862729899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 msgr2=0x55ab58e81e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e81e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 -- 172.21.15.45:0/2862729899 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab58d7bc20 con 0x55ab58e79b70 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e7aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e80000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000c010 tx=0x7f0ca0020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b882c1dc3f18f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.885+0000 7f0c94ff9640 1 -- 172.21.15.45:0/2862729899 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0003900 con 0x55ab58e79b70 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.886+0000 7f0c94ff9640 1 -- 172.21.15.45:0/2862729899 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ab58ae07e0 con 0x55ab58e79b70 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.886+0000 7f0c94ff9640 1 -- 172.21.15.45:0/2862729899 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00046c0 con 0x55ab58e79b70 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 msgr2=0x55ab58e80000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e80000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000c010 tx=0x7f0ca0020000 comp rx=0 tx=0).stop 2024-03-12T05:12:21.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 shutdown_connections 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e7aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e80000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2862729899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e81e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 >> 172.21.15.45:0/2862729899 conn(0x55ab58e73980 msgr2=0x55ab58e99720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 shutdown_connections 2024-03-12T05:12:21.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.887+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2862729899 wait complete. 2024-03-12T05:12:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 -- start start 2024-03-12T05:12:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 Processor -- start 2024-03-12T05:12:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e87dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e88350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58e88a50 con 0x55ab58e79250 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58e88bc0 con 0x55ab58e82750 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab58ee8eb0 con 0x55ab58e79b70 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cb7da5740 10 client.0 _ll_register_callbacks cb 0x7f0ca000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.888+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0caed9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e87dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35010/0 (socket says 172.21.15.45:35010) 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 -- 172.21.15.45:0/2485965729 learned_addr learned my addr 172.21.15.45:0/2485965729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e88350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 -- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 msgr2=0x55ab58e87dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e87dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 -- 172.21.15.45:0/2485965729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 msgr2=0x55ab58e88350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:21.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e88350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 -- 172.21.15.45:0/2485965729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab58ddf6a0 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e87dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cae59c640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009600 tx=0x7f0ca4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b5fee8660f62f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0c92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4029070 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0ca4003ac0 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.889+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ca0015070 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.890+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca4003c80 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.890+0000 7f0cb7da5740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.890+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ab58da18c0 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.890+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ca0015070 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.890+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f0ca402c020 con 0x55ab58e79250 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 10 client.15978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:12:21.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 20 client.15978 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58eea670 0x55ab58eeca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 4 client.15978 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ab58b6c890 con 0x55ab58eea670 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.891+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58eea670 0x55ab58eeca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0c92ff5640 10 client.15978 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0caed9d640 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58eea670 0x55ab58eeca90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=984144d06916fd7d server_cookie=1e848f232790ebb8 in_seq=0 out_seq=0 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0c92ff5640 1 -- 172.21.15.45:0/2485965729 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ab58b6c890 con 0x55ab58eea670 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0c92ff5640 10 client.15978 handle_command_reply: tid=0 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 client.15978 shutdown 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 msgr2=0x55ab58e89a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009600 tx=0x7f0ca4005f90 comp rx=0 tx=0).stop 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 shutdown_connections 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ab58eea670 0x55ab58eeca90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab58e82750 0x55ab58e88350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0c92ff5640 0 client.15978 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab58e79b70 0x55ab58e87dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 --2- 172.21.15.45:0/2485965729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ab58e79250 0x55ab58e89a30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 >> 172.21.15.45:0/2485965729 conn(0x55ab58e73980 msgr2=0x55ab58e7aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 shutdown_connections 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.892+0000 7f0cb7da5740 1 -- 172.21.15.45:0/2485965729 wait complete. 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb7da5740 20 client.15978 trim_cache size 0 max 16384 2024-03-12T05:12:21.893 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:21.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 msgr2=0x7f0c980a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 0x7f0c980a4200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80ed950 tx=0x7f0ca80e73c0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 msgr2=0x7f0cb01aa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 0x7f0cb01aa020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009bf0 tx=0x7f0ca8034440 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 shutdown_connections 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c980a1d40 0x7f0c980a4200 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb010a630 0x7f0cb01aa020 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cb0105370 0x7f0cb0109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 --2- 172.21.15.45:0/3117513236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0cb01002a0 0x7f0cb010cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.893+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 >> 172.21.15.45:0/3117513236 conn(0x7f0cb0076400 msgr2=0x7f0cb00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.894+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 shutdown_connections 2024-03-12T05:12:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:21.894+0000 7f0cb50f4640 1 -- 172.21.15.45:0/3117513236 wait complete. 2024-03-12T05:12:21.908 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:21.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:12:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 -- 172.21.15.45:0/1617541839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f201080e0 msgr2=0x7f9f201145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/1617541839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f201080e0 0x7f9f201145f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009930 tx=0x7f9f1002bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 -- 172.21.15.45:0/1617541839 shutdown_connections 2024-03-12T05:12:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/1617541839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f201080e0 0x7f9f201145f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/1617541839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f20107720 0x7f9f20107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/1617541839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 0x7f9f2010db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 -- 172.21.15.45:0/1617541839 >> 172.21.15.45:0/1617541839 conn(0x7f9f200756c0 msgr2=0x7f9f20077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 -- 172.21.15.45:0/1617541839 shutdown_connections 2024-03-12T05:12:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.089+0000 7f9f26cf1640 1 -- 172.21.15.45:0/1617541839 wait complete. 2024-03-12T05:12:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.090+0000 7f9f26cf1640 1 Processor -- start 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.090+0000 7f9f26cf1640 1 -- start start 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.090+0000 7f9f26cf1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.090+0000 7f9f26cf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f201080e0 0x7f9f2019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f26cf1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 0x7f9f201a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f26cf1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201a2b60 con 0x7f9f20107720 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f26cf1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201a2cd0 con 0x7f9f201080e0 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f26cf1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201a2e90 con 0x7f9f2010d730 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35024/0 (socket says 172.21.15.45:35024) 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2335306288 learned_addr learned my addr 172.21.15.45:0/2335306288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f201080e0 0x7f9f2019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.091+0000 7f9f2528e640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 0x7f9f201a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.092+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2335306288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 msgr2=0x7f9f201a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.092+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 0x7f9f201a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.092+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2335306288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f201080e0 msgr2=0x7f9f2019e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.092+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f201080e0 0x7f9f2019e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.092+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2335306288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009560 con 0x7f9f20107720 2024-03-12T05:12:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.093+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00e9f0 tx=0x7f9f0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.093+0000 7f9f1dffb640 1 -- 172.21.15.45:0/2335306288 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c009700 con 0x7f9f20107720 2024-03-12T05:12:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.093+0000 7f9f1dffb640 1 -- 172.21.15.45:0/2335306288 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f0c00cd00 con 0x7f9f20107720 2024-03-12T05:12:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.093+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f201a3170 con 0x7f9f20107720 2024-03-12T05:12:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.093+0000 7f9f1dffb640 1 -- 172.21.15.45:0/2335306288 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c0187a0 con 0x7f9f20107720 2024-03-12T05:12:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.094+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f201aa9f0 con 0x7f9f20107720 2024-03-12T05:12:22.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 Processor -- start 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 -- start start 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f1dffb640 1 -- 172.21.15.45:0/2335306288 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f9f0c010430 con 0x7f9f20107720 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab500210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab502b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab4b2960 con 0x5602ab4f7250 2024-03-12T05:12:22.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab466310 con 0x5602ab4f7b70 2024-03-12T05:12:22.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2999f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab463840 con 0x5602ab500750 2024-03-12T05:12:22.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2528e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2528e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35032/0 (socket says 172.21.15.45:35032) 2024-03-12T05:12:22.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f2528e640 1 -- 172.21.15.45:0/683638375 learned_addr learned my addr 172.21.15.45:0/683638375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.098+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab500210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab502b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 -- 172.21.15.45:0/683638375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 msgr2=0x5602ab500210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab500210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 -- 172.21.15.45:0/683638375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 msgr2=0x5602ab4f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 -- 172.21.15.45:0/683638375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602ab421060 con 0x5602ab500750 2024-03-12T05:12:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab500210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f2528e640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1dffb640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 0x7f9f040a4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab502b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9f2019ebd0 tx=0x7f9f18007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1560e9c2c5a2b2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.099+0000 7f9f1dffb640 1 -- 172.21.15.45:0/2335306288 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9f0c014070 con 0x7f9f20107720 2024-03-12T05:12:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.100+0000 7f9f2528e640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 0x7f9f040a4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.100+0000 7f9f16ffd640 1 -- 172.21.15.45:0/683638375 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18004d20 con 0x5602ab500750 2024-03-12T05:12:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.100+0000 7f9f16ffd640 1 -- 172.21.15.45:0/683638375 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f18004e80 con 0x5602ab500750 2024-03-12T05:12:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.100+0000 7f9f16ffd640 1 -- 172.21.15.45:0/683638375 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18004030 con 0x5602ab500750 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2528e640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 0x7f9f040a4250 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f1002c710 tx=0x7f9f10005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- 172.21.15.45:0/683638375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 msgr2=0x5602ab502b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab502b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9f2019ebd0 tx=0x7f9f18007f90 comp rx=0 tx=0).stop 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- 172.21.15.45:0/683638375 shutdown_connections 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab502b40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab500210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 --2- 172.21.15.45:0/683638375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab4f7250 0x5602ab4f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- 172.21.15.45:0/683638375 >> 172.21.15.45:0/683638375 conn(0x5602ab4f1980 msgr2=0x5602ab4f1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- 172.21.15.45:0/683638375 shutdown_connections 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- 172.21.15.45:0/683638375 wait complete. 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 -- start start 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.101+0000 7f9f2999f740 1 Processor -- start 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 0x5602ab4fd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab500750 0x5602ab4fe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab45f020 con 0x5602ab500750 2024-03-12T05:12:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab132430 con 0x5602ab4f7b70 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab0eee00 con 0x5602ab4f7250 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f2999f740 10 client.0 _ll_register_callbacks cb 0x5602ab406cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.102+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38972/0 (socket says 172.21.15.45:38972) 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f1ffff640 1 -- 172.21.15.45:0/1683401180 learned_addr learned my addr 172.21.15.45:0/1683401180 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 0x5602ab4fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f2528e640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab500750 0x5602ab4fe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1683401180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 msgr2=0x5602ab4fdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1683401180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab500750 msgr2=0x5602ab4fe1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab500750 0x5602ab4fe1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1683401180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18007b30 con 0x5602ab4f7250 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 0x5602ab4fd770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0cb310 tx=0x7f9f0c09a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=154ac38669aaafdf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f03fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c083ea0 con 0x5602ab4f7250 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9f0c09ae20 con 0x5602ab4f7250 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.103+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602ab19b720 con 0x5602ab4f7250 2024-03-12T05:12:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f0c09a720 con 0x5602ab4f7250 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 10 client.25447 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5602ab45d6a0 con 0x5602ab4f7250 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5602ab19b720 con 0x5602ab4f7250 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9f0c0850b0 con 0x5602ab4f7250 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 10 client.25447 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 10 client.25447 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 20 client.25447 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab510a90 0x5602ab512ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.104+0000 7f9f2999f740 4 client.25447 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.105+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602ab1ea890 con 0x5602ab510a90 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.105+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab510a90 0x5602ab512ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f03fff640 10 client.25447 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab510a90 0x5602ab512ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5c4ad8e2fade4ed server_cookie=8458edda436da250 in_seq=0 out_seq=0 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f03fff640 1 -- 172.21.15.45:0/1683401180 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5602ab1ea890 con 0x5602ab510a90 2024-03-12T05:12:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f03fff640 10 client.25447 handle_command_reply: tid=0 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f2999f740 1 client.25447 shutdown 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 msgr2=0x5602ab4fd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.106+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 0x5602ab4fd770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0cb310 tx=0x7f9f0c09a5b0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 shutdown_connections 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab500750 0x5602ab4fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab510a90 0x5602ab512ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1683401180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab4f7250 0x5602ab4fd770 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0cb310 tx=0x7f9f0c09a5b0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 >> 172.21.15.45:0/1683401180 conn(0x5602ab4f1980 msgr2=0x5602ab4f36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f03fff640 0 client.25447 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 shutdown_connections 2024-03-12T05:12:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.107+0000 7f9f2999f740 1 -- 172.21.15.45:0/1683401180 wait complete. 2024-03-12T05:12:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.108+0000 7f9f2999f740 20 client.25447 trim_cache size 0 max 16384 2024-03-12T05:12:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.126+0000 7f9f2999f740 1 Processor -- start 2024-03-12T05:12:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.126+0000 7f9f2999f740 1 -- start start 2024-03-12T05:12:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.126+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.126+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab507a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38988/0 (socket says 172.21.15.45:38988) 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2528e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab507a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34560/0 (socket says 172.21.15.45:34560) 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2528e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab507a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35046/0 (socket says 172.21.15.45:35046) 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2999f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab3e4130 con 0x5602ab507600 2024-03-12T05:12:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2999f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab4ed5c0 con 0x5602ab4f7b70 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f2999f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab510890 con 0x5602ab500750 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1044081814 learned_addr learned my addr 172.21.15.45:0/1044081814 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1044081814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 msgr2=0x5602ab4f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1044081814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 msgr2=0x5602ab507a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.127+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab507a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f24a8d640 1 -- 172.21.15.45:0/1044081814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602ab3f9c20 con 0x5602ab4f7b70 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c09a6b0 tx=0x7f9f0c010cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abd36202d65bafb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f167fc640 1 -- 172.21.15.45:0/1044081814 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c083ea0 con 0x5602ab4f7b70 2024-03-12T05:12:22.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f167fc640 1 -- 172.21.15.45:0/1044081814 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f0c0b8e90 con 0x5602ab4f7b70 2024-03-12T05:12:22.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.128+0000 7f9f167fc640 1 -- 172.21.15.45:0/1044081814 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c09e870 con 0x5602ab4f7b70 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 -- 172.21.15.45:0/1044081814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 msgr2=0x5602ab4fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c09a6b0 tx=0x7f9f0c010cd0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 -- 172.21.15.45:0/1044081814 shutdown_connections 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab507a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab4f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 --2- 172.21.15.45:0/1044081814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab4fe040 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.129+0000 7f9f2999f740 1 -- 172.21.15.45:0/1044081814 >> 172.21.15.45:0/1044081814 conn(0x5602ab4f1980 msgr2=0x5602ab4f2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.130+0000 7f9f2999f740 1 -- 172.21.15.45:0/1044081814 shutdown_connections 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.130+0000 7f9f2999f740 1 -- 172.21.15.45:0/1044081814 wait complete. 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.130+0000 7f9f2999f740 1 -- start start 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.130+0000 7f9f2999f740 1 Processor -- start 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.131+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.131+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab5063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.131+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.131+0000 7f9f2999f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab561010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.131+0000 7f9f24a8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39002/0 (socket says 172.21.15.45:39002) 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2999f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab561550 con 0x5602ab507600 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2999f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab5616c0 con 0x5602ab4f7b70 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602ab561880 con 0x5602ab500750 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab5063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2574035729 learned_addr learned my addr 172.21.15.45:0/2574035729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2999f740 10 client.0 _ll_register_callbacks cb 0x7f9f18007690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2528e640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab561010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 msgr2=0x5602ab5063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab5063f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2574035729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 msgr2=0x5602ab561010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab561010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 -- 172.21.15.45:0/2574035729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602ab421060 con 0x5602ab4f7b70 2024-03-12T05:12:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f2528e640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab561010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f24a8d640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0cb4b0 tx=0x7f9f0c09a320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af58bedf197a74a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f03fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:22.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.132+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c0b2070 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.133+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9f0c09ae20 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.133+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602ab15e7e0 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.133+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f0c083ea0 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.133+0000 7f9f2999f740 10 client.25448 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.133+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5602ab45d6a0 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5602ab15e7e0 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9f0c0ba110 con 0x5602ab4f7b70 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 10 client.25448 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 10 client.25448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:12:22.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 20 client.25448 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab568be0 0x5602ab56afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 4 client.25448 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5602ab10a700 con 0x5602ab568be0 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.134+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab568be0 0x5602ab56afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.135+0000 7f9f1ffff640 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab568be0 0x5602ab56afd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0c6ac7683b1e465 server_cookie=1b86fa35070dd035 in_seq=0 out_seq=0 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.135+0000 7f9f03fff640 10 client.25448 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.135+0000 7f9f03fff640 1 -- 172.21.15.45:0/2574035729 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5602ab10a700 con 0x5602ab568be0 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.135+0000 7f9f03fff640 10 client.25448 handle_command_reply: tid=0 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.135+0000 7f9f2999f740 1 client.25448 shutdown 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 msgr2=0x5602ab505eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0cb4b0 tx=0x7f9f0c09a320 comp rx=0 tx=0).stop 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 shutdown_connections 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5602ab507600 0x5602ab561010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5602ab568be0 0x5602ab56afd0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5602ab500750 0x5602ab5063f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 --2- 172.21.15.45:0/2574035729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602ab4f7b70 0x5602ab505eb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f03fff640 0 client.25448 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.136+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 >> 172.21.15.45:0/2574035729 conn(0x5602ab4f1980 msgr2=0x5602ab4f3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.137+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 shutdown_connections 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.137+0000 7f9f2999f740 1 -- 172.21.15.45:0/2574035729 wait complete. 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.137+0000 7f9f2999f740 20 client.25448 trim_cache size 0 max 16384 2024-03-12T05:12:22.137 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 msgr2=0x7f9f040a4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 0x7f9f040a4250 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f1002c710 tx=0x7f9f10005fd0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 msgr2=0x7f9f2019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00e9f0 tx=0x7f9f0c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 shutdown_connections 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9f040a1d90 0x7f9f040a4250 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f2010d730 0x7f9f201a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f201080e0 0x7f9f2019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 --2- 172.21.15.45:0/2335306288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f20107720 0x7f9f2019dc70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 >> 172.21.15.45:0/2335306288 conn(0x7f9f200756c0 msgr2=0x7f9f20075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 shutdown_connections 2024-03-12T05:12:22.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.138+0000 7f9f26cf1640 1 -- 172.21.15.45:0/2335306288 wait complete. 2024-03-12T05:12:22.154 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:22.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.327+0000 7fab42895640 1 -- 172.21.15.45:0/3663597008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1166c0 msgr2=0x7fab3c118ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.327+0000 7fab42895640 1 --2- 172.21.15.45:0/3663597008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1166c0 0x7fab3c118ab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab34009cd0 tx=0x7fab3402c0e0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 -- 172.21.15.45:0/3663597008 shutdown_connections 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 --2- 172.21.15.45:0/3663597008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c1166c0 0x7fab3c118ab0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 --2- 172.21.15.45:0/3663597008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 0x7fab3c116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 --2- 172.21.15.45:0/3663597008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c108360 0x7fab3c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 -- 172.21.15.45:0/3663597008 >> 172.21.15.45:0/3663597008 conn(0x7fab3c06c260 msgr2=0x7fab3c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 -- 172.21.15.45:0/3663597008 shutdown_connections 2024-03-12T05:12:22.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.328+0000 7fab42895640 1 -- 172.21.15.45:0/3663597008 wait complete. 2024-03-12T05:12:22.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.330+0000 7fab42895640 1 Processor -- start 2024-03-12T05:12:22.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.330+0000 7fab42895640 1 -- start start 2024-03-12T05:12:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 0x7fab3c1a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 0x7fab3c1a6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39016/0 (socket says 172.21.15.45:39016) 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab3bfff640 1 -- 172.21.15.45:0/2290391173 learned_addr learned my addr 172.21.15.45:0/2290391173 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 0x7fab3c1a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1a7020 con 0x7fab3c1166c0 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1a7190 con 0x7fab3c108360 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.331+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1a7350 con 0x7fab3c108c80 2024-03-12T05:12:22.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab40e32640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 0x7fab3c1a6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab3bfff640 1 -- 172.21.15.45:0/2290391173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 msgr2=0x7fab3c1a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab3bfff640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 0x7fab3c1a26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab3bfff640 1 -- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 msgr2=0x7fab3c1a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab3bfff640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 0x7fab3c1a6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab3bfff640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c009490 con 0x7fab3c108360 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.332+0000 7fab40e32640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 0x7fab3c1a6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab3bfff640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00ea60 tx=0x7fab2c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab397fa640 1 -- 172.21.15.45:0/2290391173 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c00cde0 con 0x7fab3c108360 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab397fa640 1 -- 172.21.15.45:0/2290391173 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab2c0044d0 con 0x7fab3c108360 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab397fa640 1 -- 172.21.15.45:0/2290391173 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c010690 con 0x7fab3c108360 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab34009900 con 0x7fab3c108360 2024-03-12T05:12:22.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.333+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c1aecd0 con 0x7fab3c108360 2024-03-12T05:12:22.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.337+0000 7fab397fa640 1 -- 172.21.15.45:0/2290391173 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fab2c003e40 con 0x7fab3c108360 2024-03-12T05:12:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.338+0000 7fab397fa640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 0x7fab280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.338+0000 7fab397fa640 1 -- 172.21.15.45:0/2290391173 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fab2c014070 con 0x7fab3c108360 2024-03-12T05:12:22.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.338+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 0x7fab280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 Processor -- start 2024-03-12T05:12:22.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 0x7fab280a42d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab3c10af20 tx=0x7fab2400a2d0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 -- start start 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aac5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 0x562d2aad0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aa80960 con 0x562d2aac5250 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aa34310 con 0x562d2aace750 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.339+0000 7fab4546a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aa31840 con 0x562d2aac5b70 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aac5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34596/0 (socket says 172.21.15.45:34596) 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3b7fe640 1 -- 172.21.15.45:0/578215024 learned_addr learned my addr 172.21.15.45:0/578215024 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab40e32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 0x562d2aad0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 -- 172.21.15.45:0/578215024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 msgr2=0x562d2aace210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 -- 172.21.15.45:0/578215024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 msgr2=0x562d2aad0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 0x562d2aad0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 -- 172.21.15.45:0/578215024 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2a9ef060 con 0x562d2aac5250 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab40e32640 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 0x562d2aad0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab3bfff640 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aac5630 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cb4d0 tx=0x7fab2c09a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c9eb2ae9b65b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.340+0000 7fab327fc640 1 -- 172.21.15.45:0/578215024 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0b2070 con 0x562d2aac5250 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.341+0000 7fab327fc640 1 -- 172.21.15.45:0/578215024 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab2c09ea30 con 0x562d2aac5250 2024-03-12T05:12:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.341+0000 7fab327fc640 1 -- 172.21.15.45:0/578215024 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c096f50 con 0x562d2aac5250 2024-03-12T05:12:22.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- 172.21.15.45:0/578215024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 msgr2=0x562d2aac5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aac5630 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cb4d0 tx=0x7fab2c09a240 comp rx=0 tx=0).stop 2024-03-12T05:12:22.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- 172.21.15.45:0/578215024 shutdown_connections 2024-03-12T05:12:22.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aace750 0x562d2aad0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aace210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 --2- 172.21.15.45:0/578215024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aac5630 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- 172.21.15.45:0/578215024 >> 172.21.15.45:0/578215024 conn(0x562d2aabf980 msgr2=0x562d2aabfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- 172.21.15.45:0/578215024 shutdown_connections 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- 172.21.15.45:0/578215024 wait complete. 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.342+0000 7fab4546a740 1 -- start start 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.343+0000 7fab4546a740 1 Processor -- start 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aacb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 0x562d2aacbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aa2d020 con 0x562d2aac5b70 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a700430 con 0x562d2aac5250 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a6bce00 con 0x562d2aace750 2024-03-12T05:12:22.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab4546a740 10 client.0 _ll_register_callbacks cb 0x562d2a9d4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 0x562d2aacbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 0x562d2aacbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35092/0 (socket says 172.21.15.45:35092) 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aacb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab3b7fe640 1 -- 172.21.15.45:0/2091427757 learned_addr learned my addr 172.21.15.45:0/2091427757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab40e32640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab40e32640 1 -- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 msgr2=0x562d2aacb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab40e32640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aacb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab40e32640 1 -- 172.21.15.45:0/2091427757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 msgr2=0x562d2aacbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.344+0000 7fab40e32640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 0x562d2aacbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab40e32640 1 -- 172.21.15.45:0/2091427757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c1aecd0 con 0x562d2aace750 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab40e32640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab3402c6f0 tx=0x7fab34034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83c643010b9b3400 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34034c60 con 0x562d2aace750 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fab34034dc0 con 0x562d2aace750 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d2a72c7e0 con 0x562d2aace750 2024-03-12T05:12:22.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab340325d0 con 0x562d2aace750 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab4546a740 10 client.25462 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.345+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562d2aa2b6a0 con 0x562d2aace750 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab34043070 con 0x562d2aace750 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fab34004060 con 0x562d2aace750 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 10 client.25462 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 10 client.25462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 20 client.25462 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2aadebc0 0x562d2aae1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 4 client.25462 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.346+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d2a72c7e0 con 0x562d2aadebc0 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.347+0000 7fab3bfff640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2aadebc0 0x562d2aae1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.347+0000 7fab3bfff640 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2aadebc0 0x562d2aae1000 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e98f6041e5a5e1 server_cookie=53141abe4d809262 in_seq=0 out_seq=0 2024-03-12T05:12:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.347+0000 7fab1b7fe640 10 client.25462 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.348+0000 7fab1b7fe640 1 -- 172.21.15.45:0/2091427757 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562d2a72c7e0 con 0x562d2aadebc0 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.348+0000 7fab1b7fe640 10 client.25462 handle_command_reply: tid=0 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.348+0000 7fab4546a740 1 client.25462 shutdown 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 msgr2=0x562d2aacc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab3402c6f0 tx=0x7fab34034670 comp rx=0 tx=0).stop 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 shutdown_connections 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2aadebc0 0x562d2aae1000 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5b70 0x562d2aacbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc290 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 --2- 172.21.15.45:0/2091427757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aacb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 >> 172.21.15.45:0/2091427757 conn(0x562d2aabf980 msgr2=0x562d2aac16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab1b7fe640 0 client.25462 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 shutdown_connections 2024-03-12T05:12:22.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 1 -- 172.21.15.45:0/2091427757 wait complete. 2024-03-12T05:12:22.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.349+0000 7fab4546a740 20 client.25462 trim_cache size 0 max 16384 2024-03-12T05:12:22.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.373+0000 7fab4546a740 1 Processor -- start 2024-03-12T05:12:22.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.373+0000 7fab4546a740 1 -- start start 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aacdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a9b2130 con 0x562d2aac5250 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aabf4b0 con 0x562d2aac5b70 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.374+0000 7fab4546a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aabb5c0 con 0x562d2aace750 2024-03-12T05:12:22.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aacdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39042/0 (socket says 172.21.15.45:39042) 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 -- 172.21.15.45:0/4002828175 learned_addr learned my addr 172.21.15.45:0/4002828175 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab40e32640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 -- 172.21.15.45:0/4002828175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 msgr2=0x562d2aacc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 -- 172.21.15.45:0/4002828175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 msgr2=0x562d2aacdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aacdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3b7fe640 1 -- 172.21.15.45:0/4002828175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2a9c7c20 con 0x562d2aac5b70 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab40e32640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.375+0000 7fab3bfff640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aacdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.376+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab240104e0 tx=0x7fab24012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae2a96142333e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.376+0000 7fab31ffb640 1 -- 172.21.15.45:0/4002828175 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24021050 con 0x562d2aac5b70 2024-03-12T05:12:22.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.376+0000 7fab31ffb640 1 -- 172.21.15.45:0/4002828175 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab24022040 con 0x562d2aac5b70 2024-03-12T05:12:22.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.376+0000 7fab31ffb640 1 -- 172.21.15.45:0/4002828175 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2401c920 con 0x562d2aac5b70 2024-03-12T05:12:22.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.377+0000 7fab4546a740 1 -- 172.21.15.45:0/4002828175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 msgr2=0x562d2aabac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.377+0000 7fab4546a740 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab240104e0 tx=0x7fab24012000 comp rx=0 tx=0).stop 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.378+0000 7fab4546a740 1 -- 172.21.15.45:0/4002828175 shutdown_connections 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.378+0000 7fab4546a740 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aace750 0x562d2aacc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.378+0000 7fab4546a740 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5b70 0x562d2aabac40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.378+0000 7fab4546a740 1 --2- 172.21.15.45:0/4002828175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aac5250 0x562d2aacdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.378+0000 7fab4546a740 1 -- 172.21.15.45:0/4002828175 >> 172.21.15.45:0/4002828175 conn(0x562d2aabf980 msgr2=0x562d2aac6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.379+0000 7fab4546a740 1 -- 172.21.15.45:0/4002828175 shutdown_connections 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.379+0000 7fab4546a740 1 -- 172.21.15.45:0/4002828175 wait complete. 2024-03-12T05:12:22.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.379+0000 7fab4546a740 1 -- start start 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.380+0000 7fab4546a740 1 Processor -- start 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.380+0000 7fab4546a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aad3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.380+0000 7fab4546a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aad4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab4546a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 0x562d2aad4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aad3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aad4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aad3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39048/0 (socket says 172.21.15.45:39048) 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3bfff640 1 -- 172.21.15.45:0/1134197815 learned_addr learned my addr 172.21.15.45:0/1134197815 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2aad5e60 con 0x562d2aace750 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2ab35130 con 0x562d2aac5250 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2ab352a0 con 0x562d2aac5b70 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab4546a740 10 client.0 _ll_register_callbacks cb 0x7fab2c096460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:22.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab40e32640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 0x562d2aad4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3b7fe640 1 -- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 msgr2=0x562d2aad3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aad3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3b7fe640 1 -- 172.21.15.45:0/1134197815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 msgr2=0x562d2aad4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.381+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 0x562d2aad4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab3b7fe640 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2400fdd0 con 0x562d2aac5b70 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab40e32640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 0x562d2aad4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab3b7fe640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aad4450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x562d2aac46a0 tx=0x7fab24028f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=558cfaa0df0233e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24027760 con 0x562d2aac5b70 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fab240278c0 con 0x562d2aac5b70 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.382+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab2c09ea30 con 0x562d2aac5b70 2024-03-12T05:12:22.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab24027ac0 con 0x562d2aac5b70 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 10 client.25468 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562d2a9ed8c0 con 0x562d2aac5b70 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab2c09ea30 con 0x562d2aac5b70 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fab24018040 con 0x562d2aac5b70 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 10 client.25468 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 10 client.25468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 20 client.25468 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2ab369e0 0x562d2ab38e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 4 client.25468 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562d2a769720 con 0x562d2ab369e0 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.383+0000 7fab3bfff640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2ab369e0 0x562d2ab38e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab1b7fe640 10 client.25468 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab3bfff640 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2ab369e0 0x562d2ab38e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb15d22d7f14902 server_cookie=c78662c37165b29 in_seq=0 out_seq=0 2024-03-12T05:12:22.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab1b7fe640 1 -- 172.21.15.45:0/1134197815 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562d2a769720 con 0x562d2ab369e0 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab1b7fe640 10 client.25468 handle_command_reply: tid=0 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab4546a740 1 client.25468 shutdown 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 msgr2=0x562d2aad4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aad4450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x562d2aac46a0 tx=0x7fab24028f20 comp rx=0 tx=0).stop 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 shutdown_connections 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.385+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562d2ab369e0 0x562d2ab38e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab1b7fe640 0 client.25468 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562d2aace750 0x562d2aad4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562d2aac5b70 0x562d2aad4450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 --2- 172.21.15.45:0/1134197815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d2aac5250 0x562d2aad3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 >> 172.21.15.45:0/1134197815 conn(0x562d2aabf980 msgr2=0x562d2aabe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 shutdown_connections 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 1 -- 172.21.15.45:0/1134197815 wait complete. 2024-03-12T05:12:22.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.386+0000 7fab4546a740 20 client.25468 trim_cache size 0 max 16384 2024-03-12T05:12:22.386 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:22.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 msgr2=0x7fab280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 0x7fab280a42d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fab3c10af20 tx=0x7fab2400a2d0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 msgr2=0x7fab3c1a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00ea60 tx=0x7fab2c00ef20 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 shutdown_connections 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab280a1e10 0x7fab280a42d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab3c1166c0 0x7fab3c1a6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab3c108c80 0x7fab3c1a26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 --2- 172.21.15.45:0/2290391173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab3c108360 0x7fab3c1a2160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.387+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 >> 172.21.15.45:0/2290391173 conn(0x7fab3c06c260 msgr2=0x7fab3c116de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.388+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 shutdown_connections 2024-03-12T05:12:22.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:22.388+0000 7fab42895640 1 -- 172.21.15.45:0/2290391173 wait complete. 2024-03-12T05:12:22.401 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:22 smithi045 ceph-mon[32174]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:22.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:22 smithi130 ceph-mon[35398]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:22.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:22 smithi116 ceph-mon[35486]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:23.404 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:12:23.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:12:23.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:12:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 -- 172.21.15.45:0/2830972239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100c10 msgr2=0x7ff8c0104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 --2- 172.21.15.45:0/2830972239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100c10 0x7ff8c0104c50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009900 tx=0x7ff8a802bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 -- 172.21.15.45:0/2830972239 shutdown_connections 2024-03-12T05:12:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 --2- 172.21.15.45:0/2830972239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100c10 0x7ff8c0104c50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 --2- 172.21.15.45:0/2830972239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c0105330 0x7ff8c010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 --2- 172.21.15.45:0/2830972239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0100260 0x7ff8c0100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 -- 172.21.15.45:0/2830972239 >> 172.21.15.45:0/2830972239 conn(0x7ff8c0077180 msgr2=0x7ff8c0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.581+0000 7ff8c5572640 1 -- 172.21.15.45:0/2830972239 shutdown_connections 2024-03-12T05:12:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.582+0000 7ff8c5572640 1 -- 172.21.15.45:0/2830972239 wait complete. 2024-03-12T05:12:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.582+0000 7ff8c5572640 1 Processor -- start 2024-03-12T05:12:23.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 -- start start 2024-03-12T05:12:23.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100260 0x7ff8c010c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0105330 0x7ff8c010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c01a3000 con 0x7ff8c0100260 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c01a3170 con 0x7ff8c0105330 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8c5572640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c01a32e0 con 0x7ff8c01a2610 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100260 0x7ff8c010c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34628/0 (socket says 172.21.15.45:34628) 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.583+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/3190199391 learned_addr learned my addr 172.21.15.45:0/3190199391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0105330 0x7ff8c010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/3190199391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0105330 msgr2=0x7ff8c010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0105330 0x7ff8c010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100260 msgr2=0x7ff8c010c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100260 0x7ff8c010c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/3190199391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0009490 con 0x7ff8c01a2610 2024-03-12T05:12:23.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.584+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800efd0 tx=0x7ff8b800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.585+0000 7ff8b7fff640 1 -- 172.21.15.45:0/3190199391 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8009b40 con 0x7ff8c01a2610 2024-03-12T05:12:23.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.585+0000 7ff8b7fff640 1 -- 172.21.15.45:0/3190199391 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff8b8003b70 con 0x7ff8c01a2610 2024-03-12T05:12:23.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.585+0000 7ff8b7fff640 1 -- 172.21.15.45:0/3190199391 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80107e0 con 0x7ff8c01a2610 2024-03-12T05:12:23.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.585+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b8009840 con 0x7ff8c01a2610 2024-03-12T05:12:23.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.585+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a8009560 con 0x7ff8c01a2610 2024-03-12T05:12:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.588+0000 7ff8b7fff640 1 -- 172.21.15.45:0/3190199391 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff8b8003ce0 con 0x7ff8c01a2610 2024-03-12T05:12:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8c8217740 1 Processor -- start 2024-03-12T05:12:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8c8217740 1 -- start start 2024-03-12T05:12:23.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8b7fff640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 0x7ff8ac0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8b7fff640 1 -- 172.21.15.45:0/3190199391 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff8b8014070 con 0x7ff8c01a2610 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.589+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d6523f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65241d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8c8217740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d651a4f90 con 0x555d65236380 2024-03-12T05:12:23.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8c8217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d651a24c0 con 0x555d6523f8b0 2024-03-12T05:12:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8c8217740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6519dca0 con 0x555d65236d30 2024-03-12T05:12:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 0x7ff8ac0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35124/0 (socket says 172.21.15.45:35124) 2024-03-12T05:12:23.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 -- 172.21.15.45:0/2257408002 learned_addr learned my addr 172.21.15.45:0/2257408002 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65241d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 -- 172.21.15.45:0/2257408002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 msgr2=0x555d6523f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d6523f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 -- 172.21.15.45:0/2257408002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 msgr2=0x555d65241d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65241d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.590+0000 7ff8beffd640 1 -- 172.21.15.45:0/2257408002 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d651600b0 con 0x555d65236380 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.591+0000 7ff8beffd640 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009e80 tx=0x7ff8b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3406ad6674aa4bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.591+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 0x7ff8ac0a41b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8a802c240 tx=0x7ff8a8005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.591+0000 7ff8b57fa640 1 -- 172.21.15.45:0/2257408002 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0019630 con 0x555d65236380 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.591+0000 7ff8b57fa640 1 -- 172.21.15.45:0/2257408002 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff8b0019c40 con 0x555d65236380 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.591+0000 7ff8b57fa640 1 -- 172.21.15.45:0/2257408002 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b000eca0 con 0x555d65236380 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 -- 172.21.15.45:0/2257408002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 msgr2=0x555d65236760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009e80 tx=0x7ff8b0011810 comp rx=0 tx=0).stop 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 -- 172.21.15.45:0/2257408002 shutdown_connections 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65241d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d6523f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 --2- 172.21.15.45:0/2257408002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65236760 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 -- 172.21.15.45:0/2257408002 >> 172.21.15.45:0/2257408002 conn(0x555d65230840 msgr2=0x555d65230c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 -- 172.21.15.45:0/2257408002 shutdown_connections 2024-03-12T05:12:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.592+0000 7ff8c8217740 1 -- 172.21.15.45:0/2257408002 wait complete. 2024-03-12T05:12:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.593+0000 7ff8c8217740 1 -- start start 2024-03-12T05:12:23.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.593+0000 7ff8c8217740 1 Processor -- start 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.593+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.593+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 0x555d6523ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.593+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d6523d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 0x555d6523ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34640/0 (socket says 172.21.15.45:34640) 2024-03-12T05:12:23.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 0x555d6523ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35136/0 (socket says 172.21.15.45:35136) 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8be7fc640 1 -- 172.21.15.45:0/784256033 learned_addr learned my addr 172.21.15.45:0/784256033 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65142f40 con 0x555d65236d30 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d64e714b0 con 0x555d6523f8b0 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65123670 con 0x555d65236380 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d6523d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8c8217740 10 client.0 _ll_register_callbacks cb 0x555d65190f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 -- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 msgr2=0x555d6523d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d6523d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 -- 172.21.15.45:0/784256033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 msgr2=0x555d6523ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 0x555d6523ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b0009ae0 con 0x555d65236380 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d6523d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8beffd640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e09fc4adafd75b32 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.594+0000 7ff8a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0004230 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff8b0004390 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d65177f40 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8b0005040 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8c8217740 10 client.25477 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555d6519c320 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555d65177f40 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.595+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff8b00035f0 con 0x555d65236380 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 10 client.25477 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 10 client.25477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 20 client.25477 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:23.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d65252260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 4 client.25477 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d65194310 con 0x555d6524fe20 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.596+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d65252260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.597+0000 7ff8a67fc640 10 client.25477 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.597+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d65252260 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5b4ba3c418e36b0 server_cookie=d570057e05370c6 in_seq=0 out_seq=0 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.597+0000 7ff8a67fc640 1 -- 172.21.15.45:0/784256033 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555d65194310 con 0x555d6524fe20 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8a67fc640 10 client.25477 handle_command_reply: tid=0 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 client.25477 shutdown 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 msgr2=0x555d6523c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b000ec50 comp rx=0 tx=0).stop 2024-03-12T05:12:23.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 shutdown_connections 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d65252260 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8a67fc640 0 client.25477 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236d30 0x555d6523ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d6523d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 --2- 172.21.15.45:0/784256033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523c8f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.598+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 >> 172.21.15.45:0/784256033 conn(0x555d65230840 msgr2=0x555d652327a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.599+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 shutdown_connections 2024-03-12T05:12:23.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.599+0000 7ff8c8217740 1 -- 172.21.15.45:0/784256033 wait complete. 2024-03-12T05:12:23.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.599+0000 7ff8c8217740 20 client.25477 trim_cache size 0 max 16384 2024-03-12T05:12:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.626+0000 7ff8c8217740 1 Processor -- start 2024-03-12T05:12:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.626+0000 7ff8c8217740 1 -- start start 2024-03-12T05:12:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 0x555d6523d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34648/0 (socket says 172.21.15.45:34648) 2024-03-12T05:12:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8be7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 0x555d6523d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8be7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 0x555d6523d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39090/0 (socket says 172.21.15.45:39090) 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65235cd0 con 0x555d6523f8b0 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35140/0 (socket says 172.21.15.45:35140) 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/1975446814 learned_addr learned my addr 172.21.15.45:0/1975446814 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65231c80 con 0x555d65236d30 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.627+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d6522c180 con 0x555d65236380 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/1975446814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 msgr2=0x555d6523f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/1975446814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 msgr2=0x555d6523d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 0x555d6523d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 -- 172.21.15.45:0/1975446814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d65138c00 con 0x555d6523f8b0 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff8b809a600 tx=0x7ff8b8096c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253b4a519a2bd98c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8b4ff9640 1 -- 172.21.15.45:0/1975446814 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80a30a0 con 0x555d6523f8b0 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8b4ff9640 1 -- 172.21.15.45:0/1975446814 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff8b8096f70 con 0x555d6523f8b0 2024-03-12T05:12:23.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.628+0000 7ff8b4ff9640 1 -- 172.21.15.45:0/1975446814 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80a2e30 con 0x555d6523f8b0 2024-03-12T05:12:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.629+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 msgr2=0x555d6523b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.629+0000 7ff8c8217740 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff8b809a600 tx=0x7ff8b8096c20 comp rx=0 tx=0).stop 2024-03-12T05:12:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.629+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 shutdown_connections 2024-03-12T05:12:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d6523f8b0 0x555d6523b1c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d65236d30 0x555d6523d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 --2- 172.21.15.45:0/1975446814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236380 0x555d6523f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 >> 172.21.15.45:0/1975446814 conn(0x555d65230840 msgr2=0x555d65237c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 shutdown_connections 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 -- 172.21.15.45:0/1975446814 wait complete. 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 -- start start 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.630+0000 7ff8c8217740 1 Processor -- start 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d65244e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8be7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d65244e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35150/0 (socket says 172.21.15.45:35150) 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8beffd640 1 -- 172.21.15.45:0/624294916 learned_addr learned my addr 172.21.15.45:0/624294916 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65245340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65245880 con 0x555d65236380 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d652459f0 con 0x555d6523f8b0 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d65245bb0 con 0x555d65236d30 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.631+0000 7ff8c8217740 10 client.0 _ll_register_callbacks cb 0x7ff8b0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8bf7fe640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65245340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 -- 172.21.15.45:0/624294916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 msgr2=0x555d65244e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d65244e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 -- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 msgr2=0x555d65245340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65245340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a8009560 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8beffd640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009230 tx=0x7ff8b0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6c723b73d0dcf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0009d00 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff8b00053e0 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.632+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d651c3900 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8b0003c80 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 10 client.16029 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555d6515e910 con 0x555d65236380 2024-03-12T05:12:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff8b0042080 con 0x555d65236380 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff8b003c030 con 0x555d65236380 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 10 client.16029 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 10 client.16029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 20 client.16029 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d652a9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 4 client.16029 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:12:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.633+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555d64e9d860 con 0x555d6524fe20 2024-03-12T05:12:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.634+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d652a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.634+0000 7ff8be7fc640 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d652a9520 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cefb63ff78da0b28 server_cookie=7c52a5a93aab6744 in_seq=0 out_seq=0 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.634+0000 7ff8a67fc640 10 client.16029 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.635+0000 7ff8a67fc640 1 -- 172.21.15.45:0/624294916 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555d64e9d860 con 0x555d6524fe20 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.635+0000 7ff8a67fc640 10 client.16029 handle_command_reply: tid=0 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.635+0000 7ff8c8217740 1 client.16029 shutdown 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 msgr2=0x555d65246a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009230 tx=0x7ff8b0005270 comp rx=0 tx=0).stop 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 shutdown_connections 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555d6524fe20 0x555d652a9520 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d6523f8b0 0x555d65245340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8a67fc640 0 client.16029 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d65236d30 0x555d65244e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 --2- 172.21.15.45:0/624294916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555d65236380 0x555d65246a60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 >> 172.21.15.45:0/624294916 conn(0x555d65230840 msgr2=0x555d6523b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 shutdown_connections 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 1 -- 172.21.15.45:0/624294916 wait complete. 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.636+0000 7ff8c8217740 20 client.16029 trim_cache size 0 max 16384 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "4093d745-4115-4c2a-973c-07179768ceac", 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:12:23.637 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 msgr2=0x7ff8ac0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 0x7ff8ac0a41b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff8a802c240 tx=0x7ff8a8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 msgr2=0x7ff8c01a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800efd0 tx=0x7ff8b800c6a0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 shutdown_connections 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8ac0a1cf0 0x7ff8ac0a41b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8c01a2610 0x7ff8c01a2ac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c0105330 0x7ff8c010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 --2- 172.21.15.45:0/3190199391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8c0100260 0x7ff8c010c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 >> 172.21.15.45:0/3190199391 conn(0x7ff8c0077180 msgr2=0x7ff8c00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.637+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 shutdown_connections 2024-03-12T05:12:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.638+0000 7ff8c5572640 1 -- 172.21.15.45:0/3190199391 wait complete. 2024-03-12T05:12:23.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:12:23.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:12:23.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 -- 172.21.15.45:0/1756402917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54103f50 msgr2=0x7fda54107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 --2- 172.21.15.45:0/1756402917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54103f50 0x7fda54107f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fda44009930 tx=0x7fda4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 -- 172.21.15.45:0/1756402917 shutdown_connections 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 --2- 172.21.15.45:0/1756402917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda54108670 0x7fda54110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 --2- 172.21.15.45:0/1756402917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54103f50 0x7fda54107f90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 --2- 172.21.15.45:0/1756402917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda541035a0 0x7fda54103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 -- 172.21.15.45:0/1756402917 >> 172.21.15.45:0/1756402917 conn(0x7fda540fee40 msgr2=0x7fda54101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 -- 172.21.15.45:0/1756402917 shutdown_connections 2024-03-12T05:12:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.824+0000 7fda5b30c640 1 -- 172.21.15.45:0/1756402917 wait complete. 2024-03-12T05:12:23.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 Processor -- start 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 -- start start 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 0x7fda5410fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 0x7fda5410cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 0x7fda5410d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda541aa200 con 0x7fda541035a0 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda541aa340 con 0x7fda5410d520 2024-03-12T05:12:23.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda5b30c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda541aa480 con 0x7fda54108670 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda588a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 0x7fda5410cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda588a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 0x7fda5410cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34678/0 (socket says 172.21.15.45:34678) 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda588a7640 1 -- 172.21.15.45:0/3682599347 learned_addr learned my addr 172.21.15.45:0/3682599347 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda590a8640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 0x7fda5410fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.826+0000 7fda598a9640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 0x7fda5410d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 -- 172.21.15.45:0/3682599347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 msgr2=0x7fda5410cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 0x7fda5410cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 -- 172.21.15.45:0/3682599347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 msgr2=0x7fda5410d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 0x7fda5410d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 -- 172.21.15.45:0/3682599347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c009840 con 0x7fda541035a0 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda590a8640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 0x7fda5410fef0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fda5000e9f0 tx=0x7fda5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda598a9640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 0x7fda5410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda4a7fc640 1 -- 172.21.15.45:0/3682599347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000cc60 con 0x7fda541035a0 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda4a7fc640 1 -- 172.21.15.45:0/3682599347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda500044d0 con 0x7fda541035a0 2024-03-12T05:12:23.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.827+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda44009560 con 0x7fda541035a0 2024-03-12T05:12:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.828+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda541aaa90 con 0x7fda541035a0 2024-03-12T05:12:23.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.828+0000 7fda4a7fc640 1 -- 172.21.15.45:0/3682599347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda50004a00 con 0x7fda541035a0 2024-03-12T05:12:23.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda4a7fc640 1 -- 172.21.15.45:0/3682599347 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fda50004b60 con 0x7fda541035a0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 Processor -- start 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 -- start start 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65b2a280 con 0x55da65bd57c0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65b277b0 con 0x55da65bbafa0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65b22f90 con 0x55da65bbb8c0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda588a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda598a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.831+0000 7fda598a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39126/0 (socket says 172.21.15.45:39126) 2024-03-12T05:12:23.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda590a8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda588a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35192/0 (socket says 172.21.15.45:35192) 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda588a7640 1 -- 172.21.15.45:0/2373217898 learned_addr learned my addr 172.21.15.45:0/2373217898 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda4a7fc640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 0x7fda400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 -- 172.21.15.45:0/2373217898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 msgr2=0x55da65bd5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 -- 172.21.15.45:0/2373217898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 msgr2=0x55da65bd7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 -- 172.21.15.45:0/2373217898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da65ae1570 con 0x55da65bbafa0 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda588a7640 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda4a7fc640 1 -- 172.21.15.45:0/3682599347 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fda50014070 con 0x7fda541035a0 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.832+0000 7fda598a9640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 0x7fda400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.833+0000 7fda598a9640 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009fb0 tx=0x7fda4c00f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d7858bf5a346835 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.833+0000 7fda3f7fe640 1 -- 172.21.15.45:0/2373217898 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c017030 con 0x55da65bbafa0 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.833+0000 7fda598a9640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 0x7fda400a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda4c020000 tx=0x7fda4c023000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.833+0000 7fda3f7fe640 1 -- 172.21.15.45:0/2373217898 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda4c010d50 con 0x55da65bbafa0 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.833+0000 7fda3f7fe640 1 -- 172.21.15.45:0/2373217898 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c01dc10 con 0x55da65bbafa0 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2373217898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 msgr2=0x55da65bbb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009fb0 tx=0x7fda4c00f7f0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2373217898 shutdown_connections 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bd7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2373217898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bbb380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2373217898 >> 172.21.15.45:0/2373217898 conn(0x55da65bb5990 msgr2=0x55da65bb5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2373217898 shutdown_connections 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.834+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2373217898 wait complete. 2024-03-12T05:12:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.835+0000 7fda5dfc1740 1 -- start start 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.835+0000 7fda5dfc1740 1 Processor -- start 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 0x55da65bd2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da657f6430 con 0x55da65bbafa0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da657b2e00 con 0x55da65bd57c0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65aa7fd0 con 0x55da65bbb8c0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda5dfc1740 10 client.0 _ll_register_callbacks cb 0x55da65afdfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda590a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda590a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35198/0 (socket says 172.21.15.45:35198) 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda590a8640 1 -- 172.21.15.45:0/2460266711 learned_addr learned my addr 172.21.15.45:0/2460266711 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda598a9640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 0x55da65bd2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 -- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 msgr2=0x55da65bd2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 0x55da65bd2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 -- 172.21.15.45:0/2460266711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 msgr2=0x55da65bd50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda588a7640 1 -- 172.21.15.45:0/2460266711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c009c90 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda590a8640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.836+0000 7fda598a9640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 0x55da65bd2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda588a7640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd2450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda44009900 tx=0x7fda44005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ef706c47c1823c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44004310 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da658227e0 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fda44004470 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda440035f0 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda5dfc1740 10 client.25495 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.837+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55da65b21610 con 0x55da65bbb8c0 2024-03-12T05:12:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda4402de90 con 0x55da65bbb8c0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fda4403c040 con 0x55da65bbb8c0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 10 client.25495 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 10 client.25495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 20 client.25495 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65be5ac0 0x55da65be7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 4 client.25495 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da658227e0 con 0x55da65be5ac0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda590a8640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65be5ac0 0x55da65be7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.838+0000 7fda3cff9640 10 client.25495 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.839+0000 7fda590a8640 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65be5ac0 0x55da65be7ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcad1ec1a0caf8ce server_cookie=75426114dcdf8f98 in_seq=0 out_seq=0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.839+0000 7fda3cff9640 1 -- 172.21.15.45:0/2460266711 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55da658227e0 con 0x55da65be5ac0 2024-03-12T05:12:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.839+0000 7fda3cff9640 10 client.25495 handle_command_reply: tid=0 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.839+0000 7fda5dfc1740 1 client.25495 shutdown 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 msgr2=0x55da65bd2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd2450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda44009900 tx=0x7fda44005bb0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 shutdown_connections 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65be5ac0 0x55da65be7ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bd57c0 0x55da65bd2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd2450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/2460266711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbafa0 0x55da65bd50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 >> 172.21.15.45:0/2460266711 conn(0x55da65bb5990 msgr2=0x55da65bb7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda3cff9640 0 client.25495 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 shutdown_connections 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.840+0000 7fda5dfc1740 1 -- 172.21.15.45:0/2460266711 wait complete. 2024-03-12T05:12:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.841+0000 7fda5dfc1740 20 client.25495 trim_cache size 0 max 16384 2024-03-12T05:12:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 Processor -- start 2024-03-12T05:12:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 -- start start 2024-03-12T05:12:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bdc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65bbc070 con 0x55da65bd57c0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65bba4f0 con 0x55da65bbafa0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65be1620 con 0x55da65bbb8c0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda590a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda588a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda598a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bdc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda588a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:34710/0 (socket says 172.21.15.45:34710) 2024-03-12T05:12:23.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.866+0000 7fda588a7640 1 -- 172.21.15.45:0/4134088904 learned_addr learned my addr 172.21.15.45:0/4134088904 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 -- 172.21.15.45:0/4134088904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 msgr2=0x55da65bd4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 -- 172.21.15.45:0/4134088904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 msgr2=0x55da65bdc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bdc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 -- 172.21.15.45:0/4134088904 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da65abdac0 con 0x55da65bbb8c0 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda588a7640 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda4402c330 tx=0x7fda4402c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=500a3e3460cc3160 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.867+0000 7fda3effd640 1 -- 172.21.15.45:0/4134088904 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44005d50 con 0x55da65bbb8c0 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.868+0000 7fda3effd640 1 -- 172.21.15.45:0/4134088904 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda44004310 con 0x55da65bbb8c0 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.868+0000 7fda3effd640 1 -- 172.21.15.45:0/4134088904 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4403e9a0 con 0x55da65bbb8c0 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.868+0000 7fda5dfc1740 1 -- 172.21.15.45:0/4134088904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 msgr2=0x55da65bd50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.868+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda4402c330 tx=0x7fda4402c360 comp rx=0 tx=0).stop 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 -- 172.21.15.45:0/4134088904 shutdown_connections 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bd57c0 0x55da65bdc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bbb8c0 0x55da65bd50c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/4134088904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 -- 172.21.15.45:0/4134088904 >> 172.21.15.45:0/4134088904 conn(0x55da65bb5990 msgr2=0x55da65bcd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 -- 172.21.15.45:0/4134088904 shutdown_connections 2024-03-12T05:12:23.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 -- 172.21.15.45:0/4134088904 wait complete. 2024-03-12T05:12:23.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 -- start start 2024-03-12T05:12:23.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.869+0000 7fda5dfc1740 1 Processor -- start 2024-03-12T05:12:23.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 0x55da65bce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65bdcfe0 con 0x55da65bbb8c0 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65bcec00 con 0x55da65bbafa0 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da65bced70 con 0x55da65bd57c0 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda5dfc1740 10 client.0 _ll_register_callbacks cb 0x7fda4c009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda588a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda588a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:35228/0 (socket says 172.21.15.45:35228) 2024-03-12T05:12:23.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda588a7640 1 -- 172.21.15.45:0/1276752 learned_addr learned my addr 172.21.15.45:0/1276752 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda590a8640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.870+0000 7fda598a9640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 0x55da65bce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 -- 172.21.15.45:0/1276752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 msgr2=0x55da65bce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 0x55da65bce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 -- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 msgr2=0x55da65bd0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 -- 172.21.15.45:0/1276752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da65b21610 con 0x55da65bbb8c0 2024-03-12T05:12:23.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda598a9640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 0x55da65bce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda588a7640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fda44001eb0 tx=0x7fda440051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc88a77fad4315c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44005d50 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fda44004480 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda4c010d50 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda4403e970 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda5dfc1740 10 client.16047 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.871+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55da65ae36a0 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fda4c010d50 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fda4403c040 con 0x55da65bbb8c0 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 10 client.16047 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:23.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 20 client.16047 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65c3ca30 0x55da65c3ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 4 client.16047 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55da6585f720 con 0x55da65c3ca30 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.872+0000 7fda590a8640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65c3ca30 0x55da65c3ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda590a8640 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65c3ca30 0x55da65c3ee50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64122ecfbb37655f server_cookie=8ee6a2f7d41261d0 in_seq=0 out_seq=0 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda3cff9640 10 client.16047 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda3cff9640 1 -- 172.21.15.45:0/1276752 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55da6585f720 con 0x55da65c3ca30 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda3cff9640 10 client.16047 handle_command_reply: tid=0 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 client.16047 shutdown 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 msgr2=0x55da65bdc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fda44001eb0 tx=0x7fda440051e0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 shutdown_connections 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55da65c3ca30 0x55da65c3ee50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da65bbb8c0 0x55da65bdc710 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da65bd57c0 0x55da65bce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 --2- 172.21.15.45:0/1276752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da65bbafa0 0x55da65bd0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 >> 172.21.15.45:0/1276752 conn(0x55da65bb5990 msgr2=0x55da65bb41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.873+0000 7fda3cff9640 0 client.16047 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 shutdown_connections 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5dfc1740 1 -- 172.21.15.45:0/1276752 wait complete. 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5dfc1740 20 client.16047 trim_cache size 0 max 16384 2024-03-12T05:12:23.876 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "4093d745-4115-4c2a-973c-07179768ceac": { 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "4093d745-4115-4c2a-973c-07179768ceac", 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:12:23.877 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 msgr2=0x7fda400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 0x7fda400a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fda4c020000 tx=0x7fda4c023000 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 msgr2=0x7fda5410fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.874+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 0x7fda5410fef0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fda5000e9f0 tx=0x7fda5000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 shutdown_connections 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fda400a1c20 0x7fda400a40e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda5410d520 0x7fda5410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda54108670 0x7fda5410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 --2- 172.21.15.45:0/3682599347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda541035a0 0x7fda5410fef0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 >> 172.21.15.45:0/3682599347 conn(0x7fda540fee40 msgr2=0x7fda540ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 shutdown_connections 2024-03-12T05:12:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:23.875+0000 7fda5b30c640 1 -- 172.21.15.45:0/3682599347 wait complete. 2024-03-12T05:12:23.889 INFO:tasks.cephfs.filesystem:scrub status for tag:4093d745-4115-4c2a-973c-07179768ceac - {'path': '/', 'tag': '4093d745-4115-4c2a-973c-07179768ceac', 'options': 'recursive,force'} 2024-03-12T05:12:24.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: mgrmap e27: x(active, since 90s), standbys: z, y 2024-03-12T05:12:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:24.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:24 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:24.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: mgrmap e27: x(active, since 90s), standbys: z, y 2024-03-12T05:12:24.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:24.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:24.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:24 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:12:24.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: mgrmap e27: x(active, since 90s), standbys: z, y 2024-03-12T05:12:24.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:24.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:24.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:24 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:24.951 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 20.0 s, 1302.3 tps, lat 76.857 ms stddev 30.615 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:25 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:12:25.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:25.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:25.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:25 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:12:25.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:25.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:25.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:25.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:25 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:26.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:26 smithi045 ceph-mon[32174]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 19 MiB/s wr, 2.12k op/s 2024-03-12T05:12:26.644 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:26 smithi116 ceph-mon[35486]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 19 MiB/s wr, 2.12k op/s 2024-03-12T05:12:26.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:26 smithi130 ceph-mon[35398]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 19 MiB/s wr, 2.12k op/s 2024-03-12T05:12:28.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:28 smithi045 ceph-mon[32174]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:12:28.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:28 smithi130 ceph-mon[35398]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:12:28.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:28 smithi116 ceph-mon[35486]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:12:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 25.0 s, 1348.9 tps, lat 74.119 ms stddev 66.928 2024-03-12T05:12:30.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:30 smithi045 ceph-mon[32174]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.14k op/s 2024-03-12T05:12:30.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:30 smithi130 ceph-mon[35398]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.14k op/s 2024-03-12T05:12:30.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:30 smithi116 ceph-mon[35486]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.14k op/s 2024-03-12T05:12:32.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:32 smithi130 ceph-mon[35398]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:32.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:32 smithi116 ceph-mon[35486]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:32.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:32 smithi045 ceph-mon[32174]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:34.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:34 smithi045 ceph-mon[32174]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:34.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:34 smithi130 ceph-mon[35398]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:34 smithi116 ceph-mon[35486]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-03-12T05:12:34.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 30.0 s, 1496.0 tps, lat 66.834 ms stddev 16.540 2024-03-12T05:12:36.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:36 smithi130 ceph-mon[35398]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 19 MiB/s wr, 2.15k op/s 2024-03-12T05:12:36.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:36 smithi116 ceph-mon[35486]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 19 MiB/s wr, 2.15k op/s 2024-03-12T05:12:36.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:36 smithi045 ceph-mon[32174]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 19 MiB/s wr, 2.15k op/s 2024-03-12T05:12:38.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:38 smithi130 ceph-mon[35398]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:12:38.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:38.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:38.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:38.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:38 smithi116 ceph-mon[35486]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:12:38.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:38.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:38.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:38.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:38 smithi045 ceph-mon[32174]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:12:38.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:38.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:38.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:39.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:39 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:12:39.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:39 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:12:39.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:39 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:12:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 35.0 s, 1434.3 tps, lat 39.593 ms stddev 14.565 2024-03-12T05:12:40.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:40 smithi130 ceph-mon[35398]: pgmap v56: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:12:40.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:40 smithi116 ceph-mon[35486]: pgmap v56: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:12:40.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:40 smithi045 ceph-mon[32174]: pgmap v56: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:12:42.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:42 smithi130 ceph-mon[35398]: pgmap v57: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:42.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:42 smithi116 ceph-mon[35486]: pgmap v57: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:42.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:42 smithi045 ceph-mon[32174]: pgmap v57: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:44.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:44 smithi130 ceph-mon[35398]: pgmap v58: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:44.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:44 smithi116 ceph-mon[35486]: pgmap v58: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:44.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:44 smithi045 ceph-mon[32174]: pgmap v58: 97 pgs: 97 active+clean; 1.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:44.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 40.0 s, 1473.9 tps, lat 97.202 ms stddev 498.662 2024-03-12T05:12:46.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:46 smithi130 ceph-mon[35398]: pgmap v59: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 2.29k op/s 2024-03-12T05:12:46.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:46 smithi116 ceph-mon[35486]: pgmap v59: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 2.29k op/s 2024-03-12T05:12:46.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:46 smithi045 ceph-mon[32174]: pgmap v59: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 20 MiB/s wr, 2.29k op/s 2024-03-12T05:12:48.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:48 smithi130 ceph-mon[35398]: pgmap v60: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:48.746 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:48 smithi116 ceph-mon[35486]: pgmap v60: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:48.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:48 smithi045 ceph-mon[32174]: pgmap v60: 97 pgs: 97 active+clean; 1.3 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:12:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 45.0 s, 1453.2 tps, lat 68.714 ms stddev 60.811 2024-03-12T05:12:50.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:50 smithi130 ceph-mon[35398]: pgmap v61: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 20 MiB/s wr, 2.32k op/s 2024-03-12T05:12:50.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:50 smithi116 ceph-mon[35486]: pgmap v61: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 20 MiB/s wr, 2.32k op/s 2024-03-12T05:12:50.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:50 smithi045 ceph-mon[32174]: pgmap v61: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 20 MiB/s wr, 2.32k op/s 2024-03-12T05:12:52.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:52 smithi130 ceph-mon[35398]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:52.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:52 smithi116 ceph-mon[35486]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:52.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:52 smithi045 ceph-mon[32174]: pgmap v62: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:53.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:12:53.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 -- 172.21.15.45:0/4170598581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 msgr2=0x7fcaa8075a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 --2- 172.21.15.45:0/4170598581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa8075a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca98009890 tx=0x7fca9802bb70 comp rx=0 tx=0).stop 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 -- 172.21.15.45:0/4170598581 shutdown_connections 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 --2- 172.21.15.45:0/4170598581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaa8075fd0 0x7fcaa8111960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 --2- 172.21.15.45:0/4170598581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8111ea0 0x7fcaa8114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 --2- 172.21.15.45:0/4170598581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa8075a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 -- 172.21.15.45:0/4170598581 >> 172.21.15.45:0/4170598581 conn(0x7fcaa80fec90 msgr2=0x7fcaa81010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 -- 172.21.15.45:0/4170598581 shutdown_connections 2024-03-12T05:12:54.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.066+0000 7fcaacac5640 1 -- 172.21.15.45:0/4170598581 wait complete. 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.067+0000 7fcaacac5640 1 Processor -- start 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.067+0000 7fcaacac5640 1 -- start start 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa81a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaa8111ea0 0x7fcaa819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819e070 con 0x7fcaa8111ea0 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819e1e0 con 0x7fcaa80756b0 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaacac5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa819e3a0 con 0x7fcaa8075fd0 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:52864/0 (socket says 172.21.15.45:52864) 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/1020891218 learned_addr learned my addr 172.21.15.45:0/1020891218 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa81a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/1020891218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 msgr2=0x7fcaa81a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa81a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaa8111ea0 msgr2=0x7fcaa819da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.068+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaa8111ea0 0x7fcaa819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/1020891218 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca98009490 con 0x7fcaa8075fd0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00e9f0 tx=0x7fca9c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaa4ff9640 1 -- 172.21.15.45:0/1020891218 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c009700 con 0x7fcaa8075fd0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaa4ff9640 1 -- 172.21.15.45:0/1020891218 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca9c00cd80 con 0x7fcaa8075fd0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa819e680 con 0x7fcaa8075fd0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.069+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa81aa8c0 con 0x7fcaa8075fd0 2024-03-12T05:12:54.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.070+0000 7fcaa4ff9640 1 -- 172.21.15.45:0/1020891218 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c018840 con 0x7fcaa8075fd0 2024-03-12T05:12:54.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.073+0000 7fcaa4ff9640 1 -- 172.21.15.45:0/1020891218 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fca9c0189a0 con 0x7fcaa8075fd0 2024-03-12T05:12:54.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.073+0000 7fcab09ce740 1 Processor -- start 2024-03-12T05:12:54.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.074+0000 7fcaa4ff9640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 0x7fca900a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.074+0000 7fcaa4ff9640 1 -- 172.21.15.45:0/1020891218 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fca9c014070 con 0x7fcaa8075fd0 2024-03-12T05:12:54.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 -- start start 2024-03-12T05:12:54.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653b490 0x55980653b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806546d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598064aa280 con 0x559806544990 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598064a77b0 con 0x55980653bdb0 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcab09ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598064a2f90 con 0x55980653b490 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.075+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 0x7fca900a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46510/0 (socket says 172.21.15.45:46510) 2024-03-12T05:12:54.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/3144731390 learned_addr learned my addr 172.21.15.45:0/3144731390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653b490 0x55980653b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806546d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/3144731390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653b490 msgr2=0x55980653b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653b490 0x55980653b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/3144731390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 msgr2=0x559806546d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806546d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.076+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/3144731390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559806461570 con 0x55980653bdb0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.077+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009380 tx=0x7fca9c09a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b32e8cf29db4119 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.077+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 0x7fca900a4160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0005c40 tx=0x7fcaa0009a60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.077+0000 7fca8dffb640 1 -- 172.21.15.45:0/3144731390 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c09e7d0 con 0x55980653bdb0 2024-03-12T05:12:54.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.077+0000 7fca8dffb640 1 -- 172.21.15.45:0/3144731390 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca9c0a2e20 con 0x55980653bdb0 2024-03-12T05:12:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fca8dffb640 1 -- 172.21.15.45:0/3144731390 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c096ed0 con 0x55980653bdb0 2024-03-12T05:12:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 -- 172.21.15.45:0/3144731390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 msgr2=0x559806544450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009380 tx=0x7fca9c09a560 comp rx=0 tx=0).stop 2024-03-12T05:12:54.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 -- 172.21.15.45:0/3144731390 shutdown_connections 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806546d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806544450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 --2- 172.21.15.45:0/3144731390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653b490 0x55980653b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 -- 172.21.15.45:0/3144731390 >> 172.21.15.45:0/3144731390 conn(0x559806535bc0 msgr2=0x559806535fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.078+0000 7fcab09ce740 1 -- 172.21.15.45:0/3144731390 shutdown_connections 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.079+0000 7fcab09ce740 1 -- 172.21.15.45:0/3144731390 wait complete. 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.079+0000 7fcab09ce740 1 -- start start 2024-03-12T05:12:54.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 Processor -- start 2024-03-12T05:12:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x559806544360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x559806541d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559806176430 con 0x55980653bdb0 2024-03-12T05:12:54.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559806132e00 con 0x55980653b490 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559806427fd0 con 0x559806544990 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.080+0000 7fcab09ce740 10 client.0 _ll_register_callbacks cb 0x55980647dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.081+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.081+0000 7fcaa6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44334/0 (socket says 172.21.15.45:44334) 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.081+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/414875177 learned_addr learned my addr 172.21.15.45:0/414875177 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.081+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x559806541d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/414875177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 msgr2=0x559806541d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x559806541d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 msgr2=0x559806544360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x559806544360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 -- 172.21.15.45:0/414875177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa81aa8c0 con 0x55980653bdb0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0cb210 tx=0x7fca9c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ee73f15443835c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.082+0000 7fca86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c0b2070 con 0x55980653bdb0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598061a27e0 con 0x55980653bdb0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fca9c098720 con 0x55980653bdb0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca9c0852f0 con 0x55980653bdb0 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fcab09ce740 10 client.16053 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:54.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.083+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5598064a1610 con 0x55980653bdb0 2024-03-12T05:12:54.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.084+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca9c0a8020 con 0x55980653bdb0 2024-03-12T05:12:54.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.084+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fca9c096570 con 0x55980653bdb0 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.084+0000 7fcab09ce740 10 client.16053 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcab09ce740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcab09ce740 20 client.16053 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065659e0 0x559806567ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcab09ce740 4 client.16053 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598061a27e0 con 0x5598065659e0 2024-03-12T05:12:54.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.085+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065659e0 0x559806567ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.086+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065659e0 0x559806567ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10e25a9b35cbd4c6 server_cookie=54c3aa1fa09ea926 in_seq=0 out_seq=0 2024-03-12T05:12:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.086+0000 7fca86ffd640 10 client.16053 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.086+0000 7fca86ffd640 1 -- 172.21.15.45:0/414875177 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5598061a27e0 con 0x5598065659e0 2024-03-12T05:12:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.086+0000 7fca86ffd640 10 client.16053 handle_command_reply: tid=0 2024-03-12T05:12:54.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 client.16053 shutdown 2024-03-12T05:12:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 msgr2=0x5598065416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca9c0cb210 tx=0x7fca9c009310 comp rx=0 tx=0).stop 2024-03-12T05:12:54.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 shutdown_connections 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065659e0 0x559806567ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653bdb0 0x5598065416f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x559806541d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 --2- 172.21.15.45:0/414875177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x559806544360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 >> 172.21.15.45:0/414875177 conn(0x559806535bc0 msgr2=0x559806537910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.087+0000 7fca86ffd640 0 client.16053 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.088+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 shutdown_connections 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.088+0000 7fcab09ce740 1 -- 172.21.15.45:0/414875177 wait complete. 2024-03-12T05:12:54.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.088+0000 7fcab09ce740 20 client.16053 trim_cache size 0 max 16384 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 Processor -- start 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 -- start start 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x55980653ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653bdb0 0x55980656c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598061a27e0 con 0x559806544990 2024-03-12T05:12:54.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980653b130 con 0x55980653b490 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.115+0000 7fcab09ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598065626e0 con 0x55980653bdb0 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44350/0 (socket says 172.21.15.45:44350) 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 -- 172.21.15.45:0/497686244 learned_addr learned my addr 172.21.15.45:0/497686244 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x55980653ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653bdb0 0x55980656c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 -- 172.21.15.45:0/497686244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653bdb0 msgr2=0x55980656c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653bdb0 0x55980656c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 -- 172.21.15.45:0/497686244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 msgr2=0x55980653ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x55980653ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.116+0000 7fcaa7fff640 1 -- 172.21.15.45:0/497686244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55980643dac0 con 0x559806544990 2024-03-12T05:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.117+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0010320 tx=0x7fcaa0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3291ce183a9cd3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.117+0000 7fca8d7fa640 1 -- 172.21.15.45:0/497686244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0020070 con 0x559806544990 2024-03-12T05:12:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.117+0000 7fca8d7fa640 1 -- 172.21.15.45:0/497686244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcaa000ed80 con 0x559806544990 2024-03-12T05:12:54.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.117+0000 7fca8d7fa640 1 -- 172.21.15.45:0/497686244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0015070 con 0x559806544990 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.118+0000 7fcab09ce740 1 -- 172.21.15.45:0/497686244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 msgr2=0x559806543e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.118+0000 7fcab09ce740 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0010320 tx=0x7fcaa0019000 comp rx=0 tx=0).stop 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 -- 172.21.15.45:0/497686244 shutdown_connections 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559806544990 0x559806543e20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55980653bdb0 0x55980656c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 --2- 172.21.15.45:0/497686244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653b490 0x55980653ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 -- 172.21.15.45:0/497686244 >> 172.21.15.45:0/497686244 conn(0x559806535bc0 msgr2=0x559806535fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 -- 172.21.15.45:0/497686244 shutdown_connections 2024-03-12T05:12:54.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 -- 172.21.15.45:0/497686244 wait complete. 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.119+0000 7fcab09ce740 1 -- start start 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.120+0000 7fcab09ce740 1 Processor -- start 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.120+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.120+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806559eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.120+0000 7fcab09ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x5598065bbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcab09ce740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980655a560 con 0x55980653b490 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcab09ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598065bc4d0 con 0x55980653bdb0 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcab09ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598065bc610 con 0x559806544990 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44356/0 (socket says 172.21.15.45:44356) 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa77fe640 1 -- 172.21.15.45:0/2134707423 learned_addr learned my addr 172.21.15.45:0/2134707423 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcab09ce740 10 client.0 _ll_register_callbacks cb 0x7fca9c0962c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x5598065bbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa6ffd640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806559eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 -- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 msgr2=0x559806559eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806559eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 -- 172.21.15.45:0/2134707423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 msgr2=0x559806559970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 -- 172.21.15.45:0/2134707423 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa0010050 con 0x559806544990 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fcaa7fff640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x5598065bbf90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0006010 tx=0x7fcaa00095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd0d560cb8e6f9fe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.121+0000 7fca86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.122+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0015070 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.122+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcaa001b620 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.122+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca9c0a2e20 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.122+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcaa0020040 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.123+0000 7fcab09ce740 10 client.25519 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.123+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5598064636a0 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.123+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fca9c0a2e20 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcaa0004310 con 0x559806544990 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 10 client.25519 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 10 client.25519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 20 client.25519 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065bdd70 0x5598065c0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 4 client.25519 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:12:54.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.124+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5598061df720 con 0x5598065bdd70 2024-03-12T05:12:54.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.125+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065bdd70 0x5598065c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.125+0000 7fcaa77fe640 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065bdd70 0x5598065c0190 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f2ec0672ffd315 server_cookie=8d8cb64d6bf8b7a4 in_seq=0 out_seq=0 2024-03-12T05:12:54.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fca86ffd640 10 client.25519 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:54.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fca86ffd640 1 -- 172.21.15.45:0/2134707423 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5598061df720 con 0x5598065bdd70 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fca86ffd640 10 client.25519 handle_command_reply: tid=0 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 client.25519 shutdown 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 msgr2=0x5598065bbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x5598065bbf90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0006010 tx=0x7fcaa00095e0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 shutdown_connections 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5598065bdd70 0x5598065c0190 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559806544990 0x5598065bbf90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55980653bdb0 0x559806559eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 --2- 172.21.15.45:0/2134707423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55980653b490 0x559806559970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fca86ffd640 0 client.25519 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 >> 172.21.15.45:0/2134707423 conn(0x559806535bc0 msgr2=0x55980656c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 shutdown_connections 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.126+0000 7fcab09ce740 1 -- 172.21.15.45:0/2134707423 wait complete. 2024-03-12T05:12:54.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.127+0000 7fcab09ce740 20 client.25519 trim_cache size 0 max 16384 2024-03-12T05:12:54.125 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:54.125 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:12:54.125 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:12:54.125 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 msgr2=0x7fca900a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 0x7fca900a4160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0005c40 tx=0x7fcaa0009a60 comp rx=0 tx=0).stop 2024-03-12T05:12:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 msgr2=0x7fcaa81a3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00e9f0 tx=0x7fca9c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 shutdown_connections 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fca900a1ca0 0x7fca900a4160 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcaa8111ea0 0x7fcaa819da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcaa8075fd0 0x7fcaa81a3d60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 --2- 172.21.15.45:0/1020891218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcaa80756b0 0x7fcaa81a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 >> 172.21.15.45:0/1020891218 conn(0x7fcaa80fec90 msgr2=0x7fcaa8101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 shutdown_connections 2024-03-12T05:12:54.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.128+0000 7fcaacac5640 1 -- 172.21.15.45:0/1020891218 wait complete. 2024-03-12T05:12:54.145 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:12:54.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 -- 172.21.15.45:0/49510767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1600ff250 msgr2=0x7ff1600ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 --2- 172.21.15.45:0/49510767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1600ff250 0x7ff1600ff6d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff154009860 tx=0x7ff15402bb70 comp rx=0 tx=0).stop 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 -- 172.21.15.45:0/49510767 shutdown_connections 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 --2- 172.21.15.45:0/49510767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ffc10 0x7ff16010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 --2- 172.21.15.45:0/49510767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1600ff250 0x7ff1600ff6d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 --2- 172.21.15.45:0/49510767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160107260 0x7ff160107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.317+0000 7ff16692c640 1 -- 172.21.15.45:0/49510767 >> 172.21.15.45:0/49510767 conn(0x7ff1600761e0 msgr2=0x7ff1600fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.318+0000 7ff16692c640 1 -- 172.21.15.45:0/49510767 shutdown_connections 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.318+0000 7ff16692c640 1 -- 172.21.15.45:0/49510767 wait complete. 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.318+0000 7ff16692c640 1 Processor -- start 2024-03-12T05:12:54.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.318+0000 7ff16692c640 1 -- start start 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff16692c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 0x7ff160197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 0x7ff160197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff15ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 0x7ff160197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44376/0 (socket says 172.21.15.45:44376) 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff16692c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 0x7ff160197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff15ffff640 1 -- 172.21.15.45:0/2734029323 learned_addr learned my addr 172.21.15.45:0/2734029323 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 0x7ff16019bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.319+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16019c510 con 0x7ff1600ff250 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.320+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16019c680 con 0x7ff1600ffc10 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.320+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16019c840 con 0x7ff160107260 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.320+0000 7ff164ec9640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 0x7ff16019bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.320+0000 7ff15f7fe640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 0x7ff160197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff164ec9640 1 -- 172.21.15.45:0/2734029323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 msgr2=0x7ff160197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff164ec9640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 0x7ff160197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff164ec9640 1 -- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 msgr2=0x7ff160197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff164ec9640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 0x7ff160197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff164ec9640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff158009840 con 0x7ff160107260 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.321+0000 7ff15f7fe640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 0x7ff160197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff164ec9640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 0x7ff16019bf10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff15800ec50 tx=0x7ff15800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158003b70 con 0x7ff160107260 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff15800ce70 con 0x7ff160107260 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158010670 con 0x7ff160107260 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff154009490 con 0x7ff160107260 2024-03-12T05:12:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.322+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff160100bf0 con 0x7ff160107260 2024-03-12T05:12:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.324+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1600683c0 con 0x7ff160107260 2024-03-12T05:12:54.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.325+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff158004020 con 0x7ff160107260 2024-03-12T05:12:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.326+0000 7ff15d7fa640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 0x7ff1480a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.326+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff158014070 con 0x7ff160107260 2024-03-12T05:12:54.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.326+0000 7ff15ffff640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 0x7ff1480a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.327+0000 7ff15ffff640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 0x7ff1480a8640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1500058c0 tx=0x7ff150009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.330+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff158096dd0 con 0x7ff160107260 2024-03-12T05:12:54.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:54 smithi045 ceph-mon[32174]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:54.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:54 smithi130 ceph-mon[35398]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:54.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.746+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff160109660 con 0x7ff160107260 2024-03-12T05:12:54.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:54 smithi116 ceph-mon[35486]: pgmap v63: 97 pgs: 97 active+clean; 1.3 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:12:54.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.747+0000 7ff15d7fa640 1 -- 172.21.15.45:0/2734029323 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11403 (secure 0 0 0) 0x7ff15809a490 con 0x7ff160107260 2024-03-12T05:12:54.744 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:12:54.744 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:11:18.408486+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:12:54.744 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 29 2024-03-12T05:12:54.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.750+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 msgr2=0x7ff1480a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.750+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 0x7ff1480a8640 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1500058c0 tx=0x7ff150009260 comp rx=0 tx=0).stop 2024-03-12T05:12:54.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.750+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 msgr2=0x7ff16019bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.750+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 0x7ff16019bf10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff15800ec50 tx=0x7ff15800c6a0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 shutdown_connections 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1480a6180 0x7ff1480a8640 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff160107260 0x7ff16019bf10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1600ffc10 0x7ff160197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 --2- 172.21.15.45:0/2734029323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1600ff250 0x7ff160197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 >> 172.21.15.45:0/2734029323 conn(0x7ff1600761e0 msgr2=0x7ff1600fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 shutdown_connections 2024-03-12T05:12:54.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.751+0000 7ff16692c640 1 -- 172.21.15.45:0/2734029323 wait complete. 2024-03-12T05:12:54.766 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:54.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:12:54.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 -- 172.21.15.45:0/3741689637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f841410d7e0 msgr2=0x7f841410fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 --2- 172.21.15.45:0/3741689637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f841410d7e0 0x7f841410fbd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f840c009ca0 tx=0x7f840c02c180 comp rx=0 tx=0).stop 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 -- 172.21.15.45:0/3741689637 shutdown_connections 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 --2- 172.21.15.45:0/3741689637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8414073e50 0x7f841410d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 --2- 172.21.15.45:0/3741689637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f841410d7e0 0x7f841410fbd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 --2- 172.21.15.45:0/3741689637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 -- 172.21.15.45:0/3741689637 >> 172.21.15.45:0/3741689637 conn(0x7f84140fca90 msgr2=0x7f84140feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 -- 172.21.15.45:0/3741689637 shutdown_connections 2024-03-12T05:12:54.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.939+0000 7f8413fff640 1 -- 172.21.15.45:0/3741689637 wait complete. 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 Processor -- start 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 -- start start 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 0x7f8414101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841410d7e0 0x7f8414101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a9c80 con 0x7f841410d7e0 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a9dc0 con 0x7f8414073e50 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8413fff640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141a9f30 con 0x7f8414073530 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f84127fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 0x7f8414101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:52950/0 (socket says 172.21.15.45:52950) 2024-03-12T05:12:54.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.940+0000 7f8412ffd640 1 -- 172.21.15.45:0/1496325023 learned_addr learned my addr 172.21.15.45:0/1496325023 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84137fe640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841410d7e0 0x7f8414101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 -- 172.21.15.45:0/1496325023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 msgr2=0x7f8414100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 -- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841410d7e0 msgr2=0x7f8414101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841410d7e0 0x7f8414101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 -- 172.21.15.45:0/1496325023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8404009490 con 0x7f8414073e50 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84127fc640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 0x7f8414101440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f840800c9a0 tx=0x7f840800ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.941+0000 7f84188fd640 1 -- 172.21.15.45:0/1496325023 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408007bb0 con 0x7f8414073e50 2024-03-12T05:12:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.942+0000 7f84188fd640 1 -- 172.21.15.45:0/1496325023 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8408007d10 con 0x7f8414073e50 2024-03-12T05:12:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.942+0000 7f84188fd640 1 -- 172.21.15.45:0/1496325023 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080186f0 con 0x7f8414073e50 2024-03-12T05:12:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.942+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c009900 con 0x7f8414073e50 2024-03-12T05:12:54.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.942+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141aa540 con 0x7f8414073e50 2024-03-12T05:12:54.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.944+0000 7f84188fd640 1 -- 172.21.15.45:0/1496325023 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f840800fa90 con 0x7f8414073e50 2024-03-12T05:12:54.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.945+0000 7f841c5aa740 1 Processor -- start 2024-03-12T05:12:54.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.945+0000 7f841c5aa740 1 -- start start 2024-03-12T05:12:54.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.945+0000 7f84188fd640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 0x7f83fc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f763960 con 0x55e26f7a8460 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f717310 con 0x55e26f7a8d80 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f841c5aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f714840 con 0x55e26f7b1960 2024-03-12T05:12:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84137fe640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 0x7f83fc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84127fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84137fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84127fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46582/0 (socket says 172.21.15.45:46582) 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84127fc640 1 -- 172.21.15.45:0/4106344963 learned_addr learned my addr 172.21.15.45:0/4106344963 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f8412ffd640 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.946+0000 7f84188fd640 1 -- 172.21.15.45:0/1496325023 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f84080cd790 con 0x7f8414073e50 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 -- 172.21.15.45:0/4106344963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 msgr2=0x55e26f7b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 -- 172.21.15.45:0/4106344963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 msgr2=0x55e26f7a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7a8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 -- 172.21.15.45:0/4106344963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e26f6d2060 con 0x55e26f7a8d80 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f8412ffd640 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f84127fc640 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84080967c0 tx=0x7f840809a260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eb2c15d589f1eab server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.947+0000 7f83f9ffb640 1 -- 172.21.15.45:0/4106344963 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840801de50 con 0x55e26f7a8d80 2024-03-12T05:12:54.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.948+0000 7f83f9ffb640 1 -- 172.21.15.45:0/4106344963 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f840809aa60 con 0x55e26f7a8d80 2024-03-12T05:12:54.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.948+0000 7f83f9ffb640 1 -- 172.21.15.45:0/4106344963 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840809e540 con 0x55e26f7a8d80 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 -- 172.21.15.45:0/4106344963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 msgr2=0x55e26f7b1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84080967c0 tx=0x7f840809a260 comp rx=0 tx=0).stop 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f84137fe640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 0x7f83fc0a40e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f840c004af0 tx=0x7f840c005f30 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 -- 172.21.15.45:0/4106344963 shutdown_connections 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8d80 0x55e26f7b1420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 --2- 172.21.15.45:0/4106344963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 -- 172.21.15.45:0/4106344963 >> 172.21.15.45:0/4106344963 conn(0x55e26f7a2b90 msgr2=0x55e26f7a2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 -- 172.21.15.45:0/4106344963 shutdown_connections 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.949+0000 7f841c5aa740 1 -- 172.21.15.45:0/4106344963 wait complete. 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 -- start start 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 Processor -- start 2024-03-12T05:12:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 0x55e26f7aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f7af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f710020 con 0x55e26f7a8460 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f3e3430 con 0x55e26f7b1960 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f39fe00 con 0x55e26f7a8d80 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44394/0 (socket says 172.21.15.45:44394) 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f8412ffd640 1 -- 172.21.15.45:0/1694677077 learned_addr learned my addr 172.21.15.45:0/1694677077 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.950+0000 7f841c5aa740 10 client.0 _ll_register_callbacks cb 0x55e26f6b7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f84127fc640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 0x55e26f7aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f84137fe640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f7af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 -- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 msgr2=0x55e26f7aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 0x55e26f7aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 -- 172.21.15.45:0/1694677077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 msgr2=0x55e26f7af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f7af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 -- 172.21.15.45:0/1694677077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141aa540 con 0x55e26f7a8460 2024-03-12T05:12:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f84137fe640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f7af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f84127fc640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 0x55e26f7aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8404012980 tx=0x7f8404012d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4e313951cb57de server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f83f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.951+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404017070 con 0x55e26f7a8460 2024-03-12T05:12:54.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f840400f9a0 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e26f40f7e0 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84040122e0 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f841c5aa740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e26f70e6a0 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.952+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8404028080 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f8404022030 con 0x55e26f7a8460 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 10 client.16074 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 20 client.16074 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f7c1d90 0x55e26f7c4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 4 client.16074 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e26f44c720 con 0x55e26f7c1d90 2024-03-12T05:12:54.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.953+0000 7f84127fc640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f7c1d90 0x55e26f7c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.953 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 50.0 s, 1436.6 tps, lat 69.652 ms stddev 18.063 2024-03-12T05:12:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.954+0000 7f83f2ffd640 10 client.16074 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.954+0000 7f84127fc640 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f7c1d90 0x55e26f7c4220 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb3ddc8f67c290f2 server_cookie=a99c65020ee7f3bf in_seq=0 out_seq=0 2024-03-12T05:12:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.955+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1694677077 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e26f44c720 con 0x55e26f7c1d90 2024-03-12T05:12:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.955+0000 7f83f2ffd640 10 client.16074 handle_command_reply: tid=0 2024-03-12T05:12:54.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.955+0000 7f841c5aa740 1 client.16074 shutdown 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 msgr2=0x55e26f7aea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8404012980 tx=0x7f8404012d80 comp rx=0 tx=0).stop 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 shutdown_connections 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f7c1d90 0x55e26f7c4220 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f7af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8d80 0x55e26f7aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1694677077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8460 0x55e26f7aea10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f83f2ffd640 0 client.16074 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 >> 172.21.15.45:0/1694677077 conn(0x55e26f7a2b90 msgr2=0x55e26f7a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 shutdown_connections 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 1 -- 172.21.15.45:0/1694677077 wait complete. 2024-03-12T05:12:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.956+0000 7f841c5aa740 20 client.16074 trim_cache size 0 max 16384 2024-03-12T05:12:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.977+0000 7f841c5aa740 1 Processor -- start 2024-03-12T05:12:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 -- start start 2024-03-12T05:12:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7abc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f695130 con 0x55e26f7a8d80 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f7a7ea0 con 0x55e26f7a8460 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46604/0 (socket says 172.21.15.45:46604) 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f8412ffd640 1 -- 172.21.15.45:0/10942564 learned_addr learned my addr 172.21.15.45:0/10942564 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f7a80c0 con 0x55e26f7b1960 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f84137fe640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.978+0000 7f84127fc640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 -- 172.21.15.45:0/10942564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 msgr2=0x55e26f7ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 -- 172.21.15.45:0/10942564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 msgr2=0x55e26f7abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 -- 172.21.15.45:0/10942564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e26f6aac20 con 0x55e26f7a8460 2024-03-12T05:12:54.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f84127fc640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:54.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f8412ffd640 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8404001eb0 tx=0x7f840400f840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca14c0d708b122ec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f83f97fa640 1 -- 172.21.15.45:0/10942564 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404017040 con 0x55e26f7a8460 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f83f97fa640 1 -- 172.21.15.45:0/10942564 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e26f40f7e0 con 0x55e26f7a8460 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.979+0000 7f83f97fa640 1 -- 172.21.15.45:0/10942564 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840401cba0 con 0x55e26f7a8460 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 msgr2=0x55e26f7b1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8404001eb0 tx=0x7f840400f840 comp rx=0 tx=0).stop 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 shutdown_connections 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7b1960 0x55e26f7ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 --2- 172.21.15.45:0/10942564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7a8460 0x55e26f7b1140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.980+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 >> 172.21.15.45:0/10942564 conn(0x55e26f7a2b90 msgr2=0x55e26f7ad2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.981+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 shutdown_connections 2024-03-12T05:12:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.981+0000 7f841c5aa740 1 -- 172.21.15.45:0/10942564 wait complete. 2024-03-12T05:12:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.981+0000 7f841c5aa740 1 -- start start 2024-03-12T05:12:54.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.981+0000 7f841c5aa740 1 Processor -- start 2024-03-12T05:12:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f8412ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:52984/0 (socket says 172.21.15.45:52984) 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f84127fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f84127fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44416/0 (socket says 172.21.15.45:44416) 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f841c5aa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f812470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f84127fc640 1 -- 172.21.15.45:0/1662132126 learned_addr learned my addr 172.21.15.45:0/1662132126 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f84137fe640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f812470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.982+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f8129b0 con 0x55e26f7a8d80 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.983+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f812af0 con 0x55e26f7b1960 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.983+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e26f812cb0 con 0x55e26f7a8460 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.983+0000 7f841c5aa740 10 client.0 _ll_register_callbacks cb 0x7f8408096600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 -- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 msgr2=0x55e26f7b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 -- 172.21.15.45:0/1662132126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 msgr2=0x55e26f812470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f812470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e26f70e6a0 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84137fe640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f812470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f84127fc640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f84080967c0 tx=0x7f840800a3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26338d8f6c9d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f83f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840801cce0 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f840801ce40 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f840809aa60 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84080969d0 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f841c5aa740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.984+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e26f6d08c0 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8408096b30 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f8408016020 con 0x55e26f7a8d80 2024-03-12T05:12:54.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 10 client.16080 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:12:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 20 client.16080 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f81a220 0x55e26f81c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 4 client.16080 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.985+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f840809aa60 con 0x55e26f81a220 2024-03-12T05:12:54.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f81a220 0x55e26f81c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f83f2ffd640 10 client.16080 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f8412ffd640 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f81a220 0x55e26f81c640 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=19e6fd88b086165 server_cookie=10e4587d3b49de1c in_seq=0 out_seq=0 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f83f2ffd640 1 -- 172.21.15.45:0/1662132126 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f840809aa60 con 0x55e26f81a220 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f83f2ffd640 10 client.16080 handle_command_reply: tid=0 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.986+0000 7f841c5aa740 1 client.16080 shutdown 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 msgr2=0x55e26f7b8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f84080967c0 tx=0x7f840800a3a0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 shutdown_connections 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e26f81a220 0x55e26f81c640 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e26f7a8d80 0x55e26f7b8090 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e26f7b1960 0x55e26f812470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 --2- 172.21.15.45:0/1662132126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e26f7a8460 0x55e26f7b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 >> 172.21.15.45:0/1662132126 conn(0x55e26f7a2b90 msgr2=0x55e26f7a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f83f2ffd640 0 client.16080 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:12:54.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.987+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 shutdown_connections 2024-03-12T05:12:54.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f841c5aa740 1 -- 172.21.15.45:0/1662132126 wait complete. 2024-03-12T05:12:54.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f841c5aa740 20 client.16080 trim_cache size 0 max 16384 2024-03-12T05:12:54.987 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 msgr2=0x7f83fc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 0x7f83fc0a40e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f840c004af0 tx=0x7f840c005f30 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 msgr2=0x7f8414101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 0x7f8414101440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f840800c9a0 tx=0x7f840800ce60 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 shutdown_connections 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.988+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f83fc0a1c20 0x7f83fc0a40e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f841410d7e0 0x7f8414101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8414073e50 0x7f8414101440 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 --2- 172.21.15.45:0/1496325023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8414073530 0x7f8414100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 >> 172.21.15.45:0/1496325023 conn(0x7f84140fca90 msgr2=0x7f84140fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:54.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 shutdown_connections 2024-03-12T05:12:54.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:54.989+0000 7f8413fff640 1 -- 172.21.15.45:0/1496325023 wait complete. 2024-03-12T05:12:55.002 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:55.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:12:55.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 -- 172.21.15.45:0/768546511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e010d770 msgr2=0x7f26e010db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 --2- 172.21.15.45:0/768546511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e010d770 0x7f26e010db50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009ca0 tx=0x7f26d002c190 comp rx=0 tx=0).stop 2024-03-12T05:12:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 -- 172.21.15.45:0/768546511 shutdown_connections 2024-03-12T05:12:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 --2- 172.21.15.45:0/768546511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e0108980 0x7f26e01145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 --2- 172.21.15.45:0/768546511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0107fc0 0x7f26e0108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.171+0000 7f26e68ae640 1 --2- 172.21.15.45:0/768546511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e010d770 0x7f26e010db50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.172+0000 7f26e68ae640 1 -- 172.21.15.45:0/768546511 >> 172.21.15.45:0/768546511 conn(0x7f26e0076d30 msgr2=0x7f26e0079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.172+0000 7f26e68ae640 1 -- 172.21.15.45:0/768546511 shutdown_connections 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.172+0000 7f26e68ae640 1 -- 172.21.15.45:0/768546511 wait complete. 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 Processor -- start 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 -- start start 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e0107fc0 0x7f26e019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 0x7f26e01a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01a31b0 con 0x7f26e0107fc0 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01a3320 con 0x7f26e01a27c0 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26e68ae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01a3490 con 0x7f26e0108980 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.173+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53014/0 (socket says 172.21.15.45:53014) 2024-03-12T05:12:55.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 -- 172.21.15.45:0/570319591 learned_addr learned my addr 172.21.15.45:0/570319591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26dffff640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e0107fc0 0x7f26e019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 0x7f26e01a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 -- 172.21.15.45:0/570319591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 msgr2=0x7f26e01a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 0x7f26e01a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 -- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e0107fc0 msgr2=0x7f26e019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e0107fc0 0x7f26e019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 -- 172.21.15.45:0/570319591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0009900 con 0x7f26e0108980 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 0x7f26e01a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.174+0000 7f26df7fe640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26c800d950 tx=0x7f26c800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.175+0000 7f26dd7fa640 1 -- 172.21.15.45:0/570319591 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8009a30 con 0x7f26e0108980 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.175+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e01aaa70 con 0x7f26e0108980 2024-03-12T05:12:55.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.175+0000 7f26dd7fa640 1 -- 172.21.15.45:0/570319591 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26c80044d0 con 0x7f26e0108980 2024-03-12T05:12:55.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.175+0000 7f26dd7fa640 1 -- 172.21.15.45:0/570319591 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c80047a0 con 0x7f26e0108980 2024-03-12T05:12:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.175+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e01aaf70 con 0x7f26e0108980 2024-03-12T05:12:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.177+0000 7f26dd7fa640 1 -- 172.21.15.45:0/570319591 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f26c8004020 con 0x7f26e0108980 2024-03-12T05:12:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.178+0000 7f26dd7fa640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 0x7f26cc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 Processor -- start 2024-03-12T05:12:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26dffff640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 0x7f26cc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 -- start start 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26dd7fa640 1 -- 172.21.15.45:0/570319591 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f26c80cd100 con 0x7f26e0108980 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d373630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d37c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d32e960 con 0x55638d373b70 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d2e2310 con 0x55638d37c750 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e9122740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d2df840 con 0x55638d373250 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26dffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d373630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e4e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e4e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46634/0 (socket says 172.21.15.45:46634) 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26e4e4b640 1 -- 172.21.15.45:0/1146929484 learned_addr learned my addr 172.21.15.45:0/1146929484 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26dffff640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 0x7f26cc0a40e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009bf0 tx=0x7f26d0000f40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.179+0000 7f26df7fe640 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d37c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 -- 172.21.15.45:0/1146929484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 msgr2=0x55638d373630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d373630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 -- 172.21.15.45:0/1146929484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 msgr2=0x55638d37c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d37c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 -- 172.21.15.45:0/1146929484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55638d29d060 con 0x55638d37c750 2024-03-12T05:12:55.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26d800aee0 tx=0x7f26d800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b86c88cedcea6b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26d67fc640 1 -- 172.21.15.45:0/1146929484 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8010840 con 0x55638d37c750 2024-03-12T05:12:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26d67fc640 1 -- 172.21.15.45:0/1146929484 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26d8010e50 con 0x55638d37c750 2024-03-12T05:12:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.180+0000 7f26d67fc640 1 -- 172.21.15.45:0/1146929484 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8016340 con 0x55638d37c750 2024-03-12T05:12:55.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- 172.21.15.45:0/1146929484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 msgr2=0x55638d37eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f26d800aee0 tx=0x7f26d800f790 comp rx=0 tx=0).stop 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- 172.21.15.45:0/1146929484 shutdown_connections 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d37c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d37eb40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 --2- 172.21.15.45:0/1146929484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d373630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- 172.21.15.45:0/1146929484 >> 172.21.15.45:0/1146929484 conn(0x55638d36d980 msgr2=0x55638d36dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- 172.21.15.45:0/1146929484 shutdown_connections 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- 172.21.15.45:0/1146929484 wait complete. 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.181+0000 7f26e9122740 1 -- start start 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.182+0000 7f26e9122740 1 Processor -- start 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.182+0000 7f26e9122740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.182+0000 7f26e9122740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d379d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.182+0000 7f26e9122740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d37a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26e9122740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d2db020 con 0x55638d37c750 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46638/0 (socket says 172.21.15.45:46638) 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 -- 172.21.15.45:0/1678457260 learned_addr learned my addr 172.21.15.45:0/1678457260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638cfae430 con 0x55638d373250 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638cf6ae00 con 0x55638d373b70 2024-03-12T05:12:55.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26e9122740 10 client.0 _ll_register_callbacks cb 0x55638d282cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 -- 172.21.15.45:0/1678457260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 msgr2=0x55638d379d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d379d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 -- 172.21.15.45:0/1678457260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 msgr2=0x55638d37a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d37a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d800ab40 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26dffff640 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009ca0 tx=0x7f26d002e050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e74a9084038f952 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.183+0000 7f26bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0032b80 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55638cfda7e0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f26d0032ce0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26d0032ea0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26e9122740 10 client.25541 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55638d2d96a0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55638cfda7e0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.184+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f26d005d4c0 con 0x55638d373250 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 10 client.25541 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 10 client.25541 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:12:55.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 20 client.25541 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d38cbc0 0x55638d38f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 4 client.25541 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.185+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55638d017720 con 0x55638d38cbc0 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.186+0000 7f26df7fe640 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d38cbc0 0x55638d38f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.187+0000 7f26df7fe640 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d38cbc0 0x55638d38f040 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66fb2b124ba65b14 server_cookie=71a45fcef68c5ba6 in_seq=0 out_seq=0 2024-03-12T05:12:55.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.187+0000 7f26bf7fe640 10 client.25541 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:55.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.187+0000 7f26bf7fe640 1 -- 172.21.15.45:0/1678457260 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55638d017720 con 0x55638d38cbc0 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.187+0000 7f26bf7fe640 10 client.25541 handle_command_reply: tid=0 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.187+0000 7f26e9122740 1 client.25541 shutdown 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 msgr2=0x55638d3797d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009ca0 tx=0x7f26d002e050 comp rx=0 tx=0).stop 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 shutdown_connections 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d38cbc0 0x55638d38f040 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d37a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d379d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26bf7fe640 0 client.25541 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 --2- 172.21.15.45:0/1678457260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d3797d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 >> 172.21.15.45:0/1678457260 conn(0x55638d36d980 msgr2=0x55638d36f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 shutdown_connections 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 1 -- 172.21.15.45:0/1678457260 wait complete. 2024-03-12T05:12:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.188+0000 7f26e9122740 20 client.25541 trim_cache size 0 max 16384 2024-03-12T05:12:55.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.206+0000 7f26e9122740 1 Processor -- start 2024-03-12T05:12:55.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.206+0000 7f26e9122740 1 -- start start 2024-03-12T05:12:55.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.206+0000 7f26e9122740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d37bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.206+0000 7f26e9122740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d376a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e9122740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e9122740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d260130 con 0x55638d373b70 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e9122740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d017720 con 0x55638d37c750 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e9122740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d38f410 con 0x55638d373250 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46654/0 (socket says 172.21.15.45:46654) 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 -- 172.21.15.45:0/141428377 learned_addr learned my addr 172.21.15.45:0/141428377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26dffff640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d37bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26df7fe640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d376a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 -- 172.21.15.45:0/141428377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 msgr2=0x55638d37bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d37bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 -- 172.21.15.45:0/141428377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 msgr2=0x55638d376a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d376a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26e4e4b640 1 -- 172.21.15.45:0/141428377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55638d275c20 con 0x55638d37c750 2024-03-12T05:12:55.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.207+0000 7f26df7fe640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d376a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.208+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f26d800fff0 tx=0x7f26d800a3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6a34f37dcc25d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.208+0000 7f26d5ffb640 1 -- 172.21.15.45:0/141428377 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8010840 con 0x55638d37c750 2024-03-12T05:12:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.208+0000 7f26d5ffb640 1 -- 172.21.15.45:0/141428377 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26d8018440 con 0x55638d37c750 2024-03-12T05:12:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.208+0000 7f26d5ffb640 1 -- 172.21.15.45:0/141428377 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8023680 con 0x55638d37c750 2024-03-12T05:12:55.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 -- 172.21.15.45:0/141428377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 msgr2=0x55638d376f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f26d800fff0 tx=0x7f26d800a3e0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 -- 172.21.15.45:0/141428377 shutdown_connections 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d373b70 0x55638d376a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d37c750 0x55638d376f90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 --2- 172.21.15.45:0/141428377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373250 0x55638d37bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.209+0000 7f26e9122740 1 -- 172.21.15.45:0/141428377 >> 172.21.15.45:0/141428377 conn(0x55638d36d980 msgr2=0x55638d3780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.210+0000 7f26e9122740 1 -- 172.21.15.45:0/141428377 shutdown_connections 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.210+0000 7f26e9122740 1 -- 172.21.15.45:0/141428377 wait complete. 2024-03-12T05:12:55.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.210+0000 7f26e9122740 1 -- start start 2024-03-12T05:12:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 Processor -- start 2024-03-12T05:12:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d382100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d382b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d3e3170 con 0x55638d37c750 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d3e32e0 con 0x55638d373250 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55638d3e34a0 con 0x55638d373b70 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e9122740 10 client.0 _ll_register_callbacks cb 0x7f26d800a930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d382100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d382100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46660/0 (socket says 172.21.15.45:46660) 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26dffff640 1 -- 172.21.15.45:0/2097245668 learned_addr learned my addr 172.21.15.45:0/2097245668 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26df7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53028/0 (socket says 172.21.15.45:53028) 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.211+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d382b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 -- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 msgr2=0x55638d382100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d382100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 -- 172.21.15.45:0/2097245668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 msgr2=0x55638d382b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d382b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 -- 172.21.15.45:0/2097245668 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55638d29d060 con 0x55638d373b70 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26e4e4b640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d382b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26df7fe640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26c80ca850 tx=0x7f26c800d8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73f7fd2aec3deea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c809de80 con 0x55638d373b70 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f26c80a24d0 con 0x55638d373b70 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26d8010e50 con 0x55638d373b70 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.212+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26c80b8250 con 0x55638d373b70 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.213+0000 7f26e9122740 10 client.25558 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.213+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55638d29b8c0 con 0x55638d373b70 2024-03-12T05:12:55.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.213+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f26d8010e50 con 0x55638d373b70 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.213+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f26c80963a0 con 0x55638d373b70 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.213+0000 7f26e9122740 10 client.25558 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26e9122740 10 client.25558 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26e9122740 20 client.25558 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d3e4d20 0x55638d3e7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26e9122740 4 client.25558 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55638d066890 con 0x55638d3e4d20 2024-03-12T05:12:55.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.214+0000 7f26dffff640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d3e4d20 0x55638d3e7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26dffff640 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d3e4d20 0x55638d3e7140 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=93028612c958dcd5 server_cookie=2f9c61a2492b149c in_seq=0 out_seq=0 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26bf7fe640 10 client.25558 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26bf7fe640 1 -- 172.21.15.45:0/2097245668 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55638d066890 con 0x55638d3e4d20 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26bf7fe640 10 client.25558 handle_command_reply: tid=0 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 client.25558 shutdown 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 msgr2=0x55638d382640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f26c80ca850 tx=0x7f26c800d8d0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 shutdown_connections 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55638d3e4d20 0x55638d3e7140 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55638d37c750 0x55638d382b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26bf7fe640 0 client.25558 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55638d373b70 0x55638d382640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.215+0000 7f26e9122740 1 --2- 172.21.15.45:0/2097245668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638d373250 0x55638d382100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.216+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 >> 172.21.15.45:0/2097245668 conn(0x55638d36d980 msgr2=0x55638d36c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.216+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 shutdown_connections 2024-03-12T05:12:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.216+0000 7f26e9122740 1 -- 172.21.15.45:0/2097245668 wait complete. 2024-03-12T05:12:55.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.216+0000 7f26e9122740 20 client.25558 trim_cache size 0 max 16384 2024-03-12T05:12:55.216 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 msgr2=0x7f26cc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 0x7f26cc0a40e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009bf0 tx=0x7f26d0000f40 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 msgr2=0x7f26e019e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26c800d950 tx=0x7f26c800de10 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 shutdown_connections 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f26cc0a1c20 0x7f26cc0a40e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26e01a27c0 0x7f26e01a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26e0108980 0x7f26e019e490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 --2- 172.21.15.45:0/570319591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26e0107fc0 0x7f26e019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.217+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 >> 172.21.15.45:0/570319591 conn(0x7f26e0076d30 msgr2=0x7f26e0079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.218+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 shutdown_connections 2024-03-12T05:12:55.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.218+0000 7f26e68ae640 1 -- 172.21.15.45:0/570319591 wait complete. 2024-03-12T05:12:55.230 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:55.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:12:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 -- 172.21.15.45:0/2056891808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc10d720 msgr2=0x7f37dc10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 --2- 172.21.15.45:0/2056891808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc10d720 0x7f37dc10db00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009930 tx=0x7f37cc02bc20 comp rx=0 tx=0).stop 2024-03-12T05:12:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 -- 172.21.15.45:0/2056891808 shutdown_connections 2024-03-12T05:12:55.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 --2- 172.21.15.45:0/2056891808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc1145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 --2- 172.21.15.45:0/2056891808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc107710 0x7f37dc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 --2- 172.21.15.45:0/2056891808 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc10d720 0x7f37dc10db00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 -- 172.21.15.45:0/2056891808 >> 172.21.15.45:0/2056891808 conn(0x7f37dc075620 msgr2=0x7f37dc077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 -- 172.21.15.45:0/2056891808 shutdown_connections 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.399+0000 7f37e4298640 1 -- 172.21.15.45:0/2056891808 wait complete. 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 Processor -- start 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 -- start start 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc19e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc10d720 0x7f37dc1a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc1a2c30 con 0x7f37dc107710 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc1a2da0 con 0x7f37dc10d720 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.400+0000 7f37e4298640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc1a2f60 con 0x7f37dc1080d0 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44492/0 (socket says 172.21.15.45:44492) 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 -- 172.21.15.45:0/1606556581 learned_addr learned my addr 172.21.15.45:0/1606556581 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e1833640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc19e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2835640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc10d720 0x7f37dc1a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 -- 172.21.15.45:0/1606556581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 msgr2=0x7f37dc19e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc19e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 -- 172.21.15.45:0/1606556581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc10d720 msgr2=0x7f37dc1a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc10d720 0x7f37dc1a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 -- 172.21.15.45:0/1606556581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc009560 con 0x7f37dc107710 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e1833640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc19e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.401+0000 7f37e2034640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005ab0 tx=0x7f37cc005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.402+0000 7f37db7fe640 1 -- 172.21.15.45:0/1606556581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0043d0 con 0x7f37dc107710 2024-03-12T05:12:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.402+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc1a31e0 con 0x7f37dc107710 2024-03-12T05:12:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.402+0000 7f37db7fe640 1 -- 172.21.15.45:0/1606556581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f37cc003aa0 con 0x7f37dc107710 2024-03-12T05:12:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.402+0000 7f37db7fe640 1 -- 172.21.15.45:0/1606556581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0053d0 con 0x7f37dc107710 2024-03-12T05:12:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.402+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc1aaac0 con 0x7f37dc107710 2024-03-12T05:12:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.405+0000 7f37db7fe640 1 -- 172.21.15.45:0/1606556581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f37cc005530 con 0x7f37dc107710 2024-03-12T05:12:55.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.406+0000 7f37e6e5f740 1 Processor -- start 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.406+0000 7f37e6e5f740 1 -- start start 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.406+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.406+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edfab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797edaa960 con 0x55797edef250 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ed5e310 con 0x55797edf8750 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ed5b840 con 0x55797edefb70 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2835640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53044/0 (socket says 172.21.15.45:53044) 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 -- 172.21.15.45:0/3736793081 learned_addr learned my addr 172.21.15.45:0/3736793081 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e1833640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edfab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 -- 172.21.15.45:0/3736793081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 msgr2=0x55797edfab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37db7fe640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 0x7f37c40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edfab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 -- 172.21.15.45:0/3736793081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 msgr2=0x55797edef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edef630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e2034640 1 -- 172.21.15.45:0/3736793081 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55797ed19060 con 0x55797edefb70 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.407+0000 7f37e1833640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edfab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37e2835640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37e2835640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 0x7f37c40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37db7fe640 1 -- 172.21.15.45:0/1606556581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f37cc0efd60 con 0x7f37dc107710 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37e2034640 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0ed530 tx=0x7f37cc0b8e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40bb93cab2daa108 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37d8bfe640 1 -- 172.21.15.45:0/3736793081 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0f3380 con 0x55797edefb70 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.408+0000 7f37d8bfe640 1 -- 172.21.15.45:0/3736793081 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f37cc0bc870 con 0x55797edefb70 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.409+0000 7f37d8bfe640 1 -- 172.21.15.45:0/3736793081 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0bc400 con 0x55797edefb70 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.409+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3736793081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 msgr2=0x55797edf8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.409+0000 7f37e2835640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 0x7f37c40a40e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37d4005fd0 tx=0x7f37d4004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.409+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0ed530 tx=0x7f37cc0b8e00 comp rx=0 tx=0).stop 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3736793081 shutdown_connections 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edfab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf8210 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3736793081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3736793081 >> 172.21.15.45:0/3736793081 conn(0x55797ede9980 msgr2=0x55797ede9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3736793081 shutdown_connections 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3736793081 wait complete. 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.410+0000 7f37e6e5f740 1 -- start start 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.411+0000 7f37e6e5f740 1 Processor -- start 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.411+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edf5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.411+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 0x55797edf6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ed57020 con 0x55797edf8750 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ea2a430 con 0x55797edef250 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797e9e6e00 con 0x55797edefb70 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e2034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edf5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e2034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edf5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46698/0 (socket says 172.21.15.45:46698) 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e2034640 1 -- 172.21.15.45:0/3507951803 learned_addr learned my addr 172.21.15.45:0/3507951803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e2835640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 0x55797edf6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e1833640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.412+0000 7f37e6e5f740 10 client.0 _ll_register_callbacks cb 0x55797ecfecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 -- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 msgr2=0x55797edf5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 -- 172.21.15.45:0/3507951803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 msgr2=0x55797edf5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edf5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 -- 172.21.15.45:0/3507951803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc1aaac0 con 0x55797edf8750 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37e2835640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 0x55797edf6190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f37d400fc30 tx=0x7f37d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcccb829d574f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.413+0000 7f37c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4022070 con 0x55797edf8750 2024-03-12T05:12:55.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f37d401d0d0 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55797ea567e0 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37d401d290 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 10 client.16104 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55797ed556a0 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55797ea567e0 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f37d40264a0 con 0x55797edf8750 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 10 client.16104 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 10 client.16104 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 20 client.16104 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee08b40 0x55797ee0af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 4 client.16104 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.414+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55797ea93720 con 0x55797ee08b40 2024-03-12T05:12:55.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.415+0000 7f37e2034640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee08b40 0x55797ee0af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.416+0000 7f37e2034640 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee08b40 0x55797ee0af60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a72f315379e4b551 server_cookie=d854bb92e47ddfe2 in_seq=0 out_seq=0 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.416+0000 7f37c1ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3507951803 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55797ea93720 con 0x55797ee08b40 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37c1ffb640 10 client.16104 handle_command_reply: tid=0 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 client.16104 shutdown 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 msgr2=0x55797edf6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 0x55797edf6190 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f37d400fc30 tx=0x7f37d4004ab0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 shutdown_connections 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee08b40 0x55797ee0af60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edf8750 0x55797edf6190 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37c1ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:55.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3507951803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edf5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.417+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 >> 172.21.15.45:0/3507951803 conn(0x55797ede9980 msgr2=0x55797edeb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.418+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 shutdown_connections 2024-03-12T05:12:55.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.418+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3507951803 wait complete. 2024-03-12T05:12:55.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.418+0000 7f37e6e5f740 20 client.16104 trim_cache size 0 max 16384 2024-03-12T05:12:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.440+0000 7f37e6e5f740 1 Processor -- start 2024-03-12T05:12:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.440+0000 7f37e6e5f740 1 -- start start 2024-03-12T05:12:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.440+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.440+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edf2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ecdc130 con 0x55797edef250 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797edeeeb0 con 0x55797edf8750 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ee10130 con 0x55797edefb70 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44524/0 (socket says 172.21.15.45:44524) 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 -- 172.21.15.45:0/179630055 learned_addr learned my addr 172.21.15.45:0/179630055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e1833640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2835640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edf2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 -- 172.21.15.45:0/179630055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 msgr2=0x55797edf5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 -- 172.21.15.45:0/179630055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 msgr2=0x55797edf2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edf2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 -- 172.21.15.45:0/179630055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55797ecf1c20 con 0x55797edef250 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2835640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edf2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.441+0000 7f37e2034640 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0edab0 tx=0x7f37cc0b8e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8b5617e358eeaf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.442+0000 7f37c3fff640 1 -- 172.21.15.45:0/179630055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0f3380 con 0x55797edef250 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.442+0000 7f37c3fff640 1 -- 172.21.15.45:0/179630055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f37cc0bc400 con 0x55797edef250 2024-03-12T05:12:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.442+0000 7f37c3fff640 1 -- 172.21.15.45:0/179630055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc0bcaa0 con 0x55797edef250 2024-03-12T05:12:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- 172.21.15.45:0/179630055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 msgr2=0x55797edf7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f37cc0edab0 tx=0x7f37cc0b8e70 comp rx=0 tx=0).stop 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- 172.21.15.45:0/179630055 shutdown_connections 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edf8750 0x55797edf2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edefb70 0x55797edf5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/179630055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edef250 0x55797edf7e40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- 172.21.15.45:0/179630055 >> 172.21.15.45:0/179630055 conn(0x55797ede9980 msgr2=0x55797edf09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- 172.21.15.45:0/179630055 shutdown_connections 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- 172.21.15.45:0/179630055 wait complete. 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- start start 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 Processor -- start 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edff780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edf8750 0x55797edfe060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797edfe760 con 0x55797edefb70 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797edfe8d0 con 0x55797edef250 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.443+0000 7f37e6e5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797ee5f010 con 0x55797edf8750 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e6e5f740 10 client.0 _ll_register_callbacks cb 0x7f37cc0f3740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e2034640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edff780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e2835640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edf8750 0x55797edfe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44526/0 (socket says 172.21.15.45:44526) 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 -- 172.21.15.45:0/3878698926 learned_addr learned my addr 172.21.15.45:0/3878698926 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 -- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edf8750 msgr2=0x55797edfe060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edf8750 0x55797edfe060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 -- 172.21.15.45:0/3878698926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 msgr2=0x55797edff780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edff780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 -- 172.21.15.45:0/3878698926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55797ed19060 con 0x55797edefb70 2024-03-12T05:12:55.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e2034640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.444+0000 7f37e1833640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f37d0005fd0 tx=0x7f37d000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5710cdf2c80c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:55.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d0005220 con 0x55797edefb70 2024-03-12T05:12:55.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f37d0005380 con 0x55797edefb70 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37cc0bc870 con 0x55797edefb70 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37d0003ec0 con 0x55797edefb70 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37e6e5f740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.445+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55797ed178c0 con 0x55797edefb70 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.446+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f37d0018040 con 0x55797edefb70 2024-03-12T05:12:55.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.446+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f37d0011700 con 0x55797edefb70 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 10 client.16110 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 20 client.16110 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee60910 0x55797ee62d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 4 client.16110 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f37cc0bc870 con 0x55797ee60910 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.447+0000 7f37e2034640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee60910 0x55797ee62d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.448+0000 7f37e2034640 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee60910 0x55797ee62d30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e747e036a289867 server_cookie=afb1dcfacd81965 in_seq=0 out_seq=0 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.448+0000 7f37c1ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:55.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.448+0000 7f37c1ffb640 1 -- 172.21.15.45:0/3878698926 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f37cc0bc870 con 0x55797ee60910 2024-03-12T05:12:55.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.448+0000 7f37c1ffb640 10 client.16110 handle_command_reply: tid=0 2024-03-12T05:12:55.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 client.16110 shutdown 2024-03-12T05:12:55.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 msgr2=0x55797edfdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f37d0005fd0 tx=0x7f37d000bf50 comp rx=0 tx=0).stop 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 shutdown_connections 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55797ee60910 0x55797ee62d30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37c1ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55797edefb70 0x55797edfdb20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55797edf8750 0x55797edfe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 --2- 172.21.15.45:0/3878698926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55797edef250 0x55797edff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.449+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 >> 172.21.15.45:0/3878698926 conn(0x55797ede9980 msgr2=0x55797ede82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.450+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 shutdown_connections 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.450+0000 7f37e6e5f740 1 -- 172.21.15.45:0/3878698926 wait complete. 2024-03-12T05:12:55.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.450+0000 7f37e6e5f740 20 client.16110 trim_cache size 0 max 16384 2024-03-12T05:12:55.451 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:55.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 msgr2=0x7f37c40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 0x7f37c40a40e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f37d4005fd0 tx=0x7f37d4004a40 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 msgr2=0x7f37dc19dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005ab0 tx=0x7f37cc005e60 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 shutdown_connections 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37c40a1c20 0x7f37c40a40e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37dc10d720 0x7f37dc1a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37dc1080d0 0x7f37dc19e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 --2- 172.21.15.45:0/1606556581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37dc107710 0x7f37dc19dd70 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 >> 172.21.15.45:0/1606556581 conn(0x7f37dc075620 msgr2=0x7f37dc075f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 shutdown_connections 2024-03-12T05:12:55.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.451+0000 7f37e4298640 1 -- 172.21.15.45:0/1606556581 wait complete. 2024-03-12T05:12:55.464 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:55.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:12:55.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.630+0000 7fd69a9d0640 1 -- 172.21.15.45:0/1365444783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694103f20 msgr2=0x7fd694107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.630+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/1365444783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694103f20 0x7fd694107f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd688009930 tx=0x7fd68802bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 -- 172.21.15.45:0/1365444783 shutdown_connections 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/1365444783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd694108640 0x7fd694110210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/1365444783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694103f20 0x7fd694107f60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/1365444783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd694103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 -- 172.21.15.45:0/1365444783 >> 172.21.15.45:0/1365444783 conn(0x7fd6940fedb0 msgr2=0x7fd6941011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 -- 172.21.15.45:0/1365444783 shutdown_connections 2024-03-12T05:12:55.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.631+0000 7fd69a9d0640 1 -- 172.21.15.45:0/1365444783 wait complete. 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 Processor -- start 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 -- start start 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694108640 0x7fd69419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6941997a0 0x7fd694199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69419a270 con 0x7fd694103570 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69419a3e0 con 0x7fd694108640 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd69a9d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69419a5a0 con 0x7fd6941997a0 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44544/0 (socket says 172.21.15.45:44544) 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.632+0000 7fd693fff640 1 -- 172.21.15.45:0/4099556974 learned_addr learned my addr 172.21.15.45:0/4099556974 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd6937fe640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694108640 0x7fd69419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd698f6d640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6941997a0 0x7fd694199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 -- 172.21.15.45:0/4099556974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6941997a0 msgr2=0x7fd694199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6941997a0 0x7fd694199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 -- 172.21.15.45:0/4099556974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694108640 msgr2=0x7fd69419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694108640 0x7fd69419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 -- 172.21.15.45:0/4099556974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009560 con 0x7fd694103570 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.633+0000 7fd693fff640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd68400c890 tx=0x7fd68400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.634+0000 7fd6917fa640 1 -- 172.21.15.45:0/4099556974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684004220 con 0x7fd694103570 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.634+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6941a64f0 con 0x7fd694103570 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.634+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6941a6a20 con 0x7fd694103570 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.635+0000 7fd6917fa640 1 -- 172.21.15.45:0/4099556974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd684007e60 con 0x7fd694103570 2024-03-12T05:12:55.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.635+0000 7fd6917fa640 1 -- 172.21.15.45:0/4099556974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684004a00 con 0x7fd694103570 2024-03-12T05:12:55.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.637+0000 7fd6917fa640 1 -- 172.21.15.45:0/4099556974 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd684004b60 con 0x7fd694103570 2024-03-12T05:12:55.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.637+0000 7fd69d5a2740 1 Processor -- start 2024-03-12T05:12:55.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.637+0000 7fd69d5a2740 1 -- start start 2024-03-12T05:12:55.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e954630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e95fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e90f960 con 0x559c2e954b70 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e8c3310 con 0x559c2e954250 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e8c0840 con 0x559c2e95d750 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e95fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44552/0 (socket says 172.21.15.45:44552) 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd693fff640 1 -- 172.21.15.45:0/3561374195 learned_addr learned my addr 172.21.15.45:0/3561374195 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e954630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 -- 172.21.15.45:0/3561374195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 msgr2=0x559c2e95fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e95fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 -- 172.21.15.45:0/3561374195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 msgr2=0x559c2e95d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd698f6d640 1 -- 172.21.15.45:0/3561374195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c2e87e060 con 0x559c2e954250 2024-03-12T05:12:55.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.638+0000 7fd6937fe640 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e95fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.639+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e954630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009f00 tx=0x7fd68c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3bf61bce459c3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.639+0000 7fd6917fa640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 0x7fd67c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.639+0000 7fd67a7fc640 1 -- 172.21.15.45:0/3561374195 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c004d20 con 0x559c2e954250 2024-03-12T05:12:55.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.639+0000 7fd67a7fc640 1 -- 172.21.15.45:0/3561374195 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd68c004e80 con 0x559c2e954250 2024-03-12T05:12:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd693fff640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 0x7fd67c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd67a7fc640 1 -- 172.21.15.45:0/3561374195 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c005280 con 0x559c2e954250 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd6917fa640 1 -- 172.21.15.45:0/4099556974 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fd6840cd250 con 0x7fd694103570 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3561374195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 msgr2=0x559c2e954630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd693fff640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 0x7fd67c0a40e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd684007fd0 tx=0x7fd6840d0d60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.640+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e954630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009f00 tx=0x7fd68c011810 comp rx=0 tx=0).stop 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3561374195 shutdown_connections 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e95fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3561374195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e954630 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3561374195 >> 172.21.15.45:0/3561374195 conn(0x559c2e94e980 msgr2=0x559c2e94ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3561374195 shutdown_connections 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3561374195 wait complete. 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.641+0000 7fd69d5a2740 1 -- start start 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 Processor -- start 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e95d750 0x559c2e95b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e8bc020 con 0x559c2e954b70 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd698f6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e58f430 con 0x559c2e95d750 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e54be00 con 0x559c2e954250 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd698f6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53100/0 (socket says 172.21.15.45:53100) 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd698f6d640 1 -- 172.21.15.45:0/4252805629 learned_addr learned my addr 172.21.15.45:0/4252805629 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.642+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd69d5a2740 10 client.0 _ll_register_callbacks cb 0x559c2e863cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd698f6d640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e95d750 0x559c2e95b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 -- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 msgr2=0x559c2e95a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 -- 172.21.15.45:0/4252805629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e95d750 msgr2=0x559c2e95b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e95d750 0x559c2e95b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 -- 172.21.15.45:0/4252805629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c009be0 con 0x559c2e954b70 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd6937fe640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95abe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd688009230 tx=0x7fd68802cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227b3db4083398b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.643+0000 7fd698f6d640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd6737fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688004250 con 0x559c2e954b70 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c2e5f8720 con 0x559c2e954b70 2024-03-12T05:12:55.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fd688046070 con 0x559c2e954b70 2024-03-12T05:12:55.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6880043b0 con 0x559c2e954b70 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.644+0000 7fd69d5a2740 10 client.16119 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559c2e8ba6a0 con 0x559c2e954b70 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd688044070 con 0x559c2e954b70 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd68803c040 con 0x559c2e954b70 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 10 client.16119 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 10 client.16119 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 20 client.16119 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e96dac0 0x559c2e96ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.645+0000 7fd69d5a2740 4 client.16119 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.646+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c2e5f8720 con 0x559c2e96dac0 2024-03-12T05:12:55.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.646+0000 7fd698f6d640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e96dac0 0x559c2e96ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.646+0000 7fd6737fe640 10 client.16119 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.646+0000 7fd698f6d640 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e96dac0 0x559c2e96ff40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d1f59f23618e030 server_cookie=5395b16ac3106644 in_seq=0 out_seq=0 2024-03-12T05:12:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.647+0000 7fd6737fe640 1 -- 172.21.15.45:0/4252805629 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559c2e5f8720 con 0x559c2e96dac0 2024-03-12T05:12:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.647+0000 7fd6737fe640 10 client.16119 handle_command_reply: tid=0 2024-03-12T05:12:55.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.647+0000 7fd69d5a2740 1 client.16119 shutdown 2024-03-12T05:12:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 msgr2=0x559c2e95abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95abe0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd688009230 tx=0x7fd68802cae0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 shutdown_connections 2024-03-12T05:12:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e95abe0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e96dac0 0x559c2e96ff40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e95d750 0x559c2e95b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/4252805629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e95a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 >> 172.21.15.45:0/4252805629 conn(0x559c2e94e980 msgr2=0x559c2e9506d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd6737fe640 0 client.16119 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 shutdown_connections 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 1 -- 172.21.15.45:0/4252805629 wait complete. 2024-03-12T05:12:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.648+0000 7fd69d5a2740 20 client.16119 trim_cache size 0 max 16384 2024-03-12T05:12:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 Processor -- start 2024-03-12T05:12:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 -- start start 2024-03-12T05:12:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e95cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e957e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e841130 con 0x559c2e954b70 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e95d5b0 con 0x559c2e954250 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e950cc0 con 0x559c2e95d750 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44562/0 (socket says 172.21.15.45:44562) 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd6937fe640 1 -- 172.21.15.45:0/3100345318 learned_addr learned my addr 172.21.15.45:0/3100345318 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e95cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.667+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e957e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 -- 172.21.15.45:0/3100345318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 msgr2=0x559c2e957e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e957e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 -- 172.21.15.45:0/3100345318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 msgr2=0x559c2e957900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 -- 172.21.15.45:0/3100345318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c2e856c20 con 0x559c2e954250 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd6937fe640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd698f6d640 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e95cdd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x559c2e970830 tx=0x7fd68c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9725d5f00d95576 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd679ffb640 1 -- 172.21.15.45:0/3100345318 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02dea0 con 0x559c2e954250 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd679ffb640 1 -- 172.21.15.45:0/3100345318 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd68c033070 con 0x559c2e954250 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.668+0000 7fd679ffb640 1 -- 172.21.15.45:0/3100345318 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02d390 con 0x559c2e954250 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3100345318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 msgr2=0x559c2e95cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e95cdd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x559c2e970830 tx=0x7fd68c011810 comp rx=0 tx=0).stop 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3100345318 shutdown_connections 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e954b70 0x559c2e957900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e95d750 0x559c2e957e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/3100345318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954250 0x559c2e95cdd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.669+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3100345318 >> 172.21.15.45:0/3100345318 conn(0x559c2e94e980 msgr2=0x559c2e9748e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.670+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3100345318 shutdown_connections 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.670+0000 7fd69d5a2740 1 -- 172.21.15.45:0/3100345318 wait complete. 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.670+0000 7fd69d5a2740 1 -- start start 2024-03-12T05:12:55.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.670+0000 7fd69d5a2740 1 Processor -- start 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.670+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 0x559c2e962f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e95d750 0x559c2e9635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e963c90 con 0x559c2e95d750 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e963e00 con 0x559c2e954b70 2024-03-12T05:12:55.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c2e9cc8d0 con 0x559c2e954250 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd69d5a2740 10 client.0 _ll_register_callbacks cb 0x7fd68c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53114/0 (socket says 172.21.15.45:53114) 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 -- 172.21.15.45:0/1361537917 learned_addr learned my addr 172.21.15.45:0/1361537917 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd6937fe640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 0x559c2e962f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e95d750 0x559c2e9635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 -- 172.21.15.45:0/1361537917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 msgr2=0x559c2e962f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.671+0000 7fd698f6d640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 0x559c2e962f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd698f6d640 1 -- 172.21.15.45:0/1361537917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e95d750 msgr2=0x559c2e9635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd698f6d640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e95d750 0x559c2e9635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd698f6d640 1 -- 172.21.15.45:0/1361537917 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c2e8ba6a0 con 0x559c2e954250 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd6937fe640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 0x559c2e962f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd698f6d640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd68c004a20 tx=0x7fd68c0046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b227625126b4a258 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd6737fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:55.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02dea0 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fd68c003c30 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c2e5bb7e0 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.672+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd68c003df0 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 10 client.25594 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559c2e87e060 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd68c02d5e0 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fd68c037070 con 0x559c2e954250 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 10 client.25594 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 10 client.25594 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 20 client.25594 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e9cdf00 0x559c2e9d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 4 client.25594 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559c2e5bb7e0 con 0x559c2e9cdf00 2024-03-12T05:12:55.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.673+0000 7fd6937fe640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e9cdf00 0x559c2e9d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd6937fe640 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e9cdf00 0x559c2e9d02f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=689f259b2e40e35d server_cookie=212bd75885df2e68 in_seq=0 out_seq=0 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd6737fe640 10 client.25594 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd6737fe640 1 -- 172.21.15.45:0/1361537917 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559c2e5bb7e0 con 0x559c2e9cdf00 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd6737fe640 10 client.25594 handle_command_reply: tid=0 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd69d5a2740 1 client.25594 shutdown 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 msgr2=0x559c2e964b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.674+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd68c004a20 tx=0x7fd68c0046e0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 shutdown_connections 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559c2e95d750 0x559c2e9635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559c2e9cdf00 0x559c2e9d02f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c2e954b70 0x559c2e962f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 --2- 172.21.15.45:0/1361537917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559c2e954250 0x559c2e964b50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 >> 172.21.15.45:0/1361537917 conn(0x559c2e94e980 msgr2=0x559c2e958f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd6737fe640 0 client.25594 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 shutdown_connections 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 1 -- 172.21.15.45:0/1361537917 wait complete. 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69d5a2740 20 client.25594 trim_cache size 0 max 16384 2024-03-12T05:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 msgr2=0x7fd67c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.675+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 0x7fd67c0a40e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd684007fd0 tx=0x7fd6840d0d60 comp rx=0 tx=0).stop 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 msgr2=0x7fd69419f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd68400c890 tx=0x7fd68400cd50 comp rx=0 tx=0).stop 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 shutdown_connections 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd67c0a1c20 0x7fd67c0a40e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6941997a0 0x7fd694199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd694108640 0x7fd69419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 --2- 172.21.15.45:0/4099556974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd694103570 0x7fd69419f690 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 >> 172.21.15.45:0/4099556974 conn(0x7fd6940fedb0 msgr2=0x7fd694100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 shutdown_connections 2024-03-12T05:12:55.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.676+0000 7fd69a9d0640 1 -- 172.21.15.45:0/4099556974 wait complete. 2024-03-12T05:12:55.687 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:55 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2734029323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:55.689 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:12:55.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:12:55.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:55 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2734029323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:55.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:55 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2734029323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:12:55.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 -- 172.21.15.45:0/1331250756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4140ffae0 msgr2=0x7fc4140fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 --2- 172.21.15.45:0/1331250756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4140ffae0 0x7fc4140fff60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc408009900 tx=0x7fc40802bcd0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 -- 172.21.15.45:0/1331250756 shutdown_connections 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 --2- 172.21.15.45:0/1331250756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4141004a0 0x7fc41410c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 --2- 172.21.15.45:0/1331250756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4140ffae0 0x7fc4140fff60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 --2- 172.21.15.45:0/1331250756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc414105870 0x7fc414105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.846+0000 7fc419c1c640 1 -- 172.21.15.45:0/1331250756 >> 172.21.15.45:0/1331250756 conn(0x7fc4140761d0 msgr2=0x7fc4140fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.847+0000 7fc419c1c640 1 -- 172.21.15.45:0/1331250756 shutdown_connections 2024-03-12T05:12:55.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.847+0000 7fc419c1c640 1 -- 172.21.15.45:0/1331250756 wait complete. 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.847+0000 7fc419c1c640 1 Processor -- start 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 -- start start 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4141004a0 0x7fc414197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 0x7fc41419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41419c590 con 0x7fc4140ffae0 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41419c700 con 0x7fc414105870 2024-03-12T05:12:55.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc419c1c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41419c8c0 con 0x7fc4141004a0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44586/0 (socket says 172.21.15.45:44586) 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc413fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 0x7fc41419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc4137fe640 1 -- 172.21.15.45:0/4077107067 learned_addr learned my addr 172.21.15.45:0/4077107067 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.848+0000 7fc412ffd640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4141004a0 0x7fc414197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 -- 172.21.15.45:0/4077107067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4141004a0 msgr2=0x7fc414197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4141004a0 0x7fc414197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 -- 172.21.15.45:0/4077107067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 msgr2=0x7fc41419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 0x7fc41419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 -- 172.21.15.45:0/4077107067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c00a950 con 0x7fc4140ffae0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc413fff640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 0x7fc41419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc4137fe640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc40400e9f0 tx=0x7fc40400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc410ff9640 1 -- 172.21.15.45:0/4077107067 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404009700 con 0x7fc4140ffae0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc408009560 con 0x7fc4140ffae0 2024-03-12T05:12:55.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc410ff9640 1 -- 172.21.15.45:0/4077107067 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc40400cd80 con 0x7fc4140ffae0 2024-03-12T05:12:55.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc410ff9640 1 -- 172.21.15.45:0/4077107067 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404018840 con 0x7fc4140ffae0 2024-03-12T05:12:55.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.849+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141a4240 con 0x7fc4140ffae0 2024-03-12T05:12:55.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.852+0000 7fc410ff9640 1 -- 172.21.15.45:0/4077107067 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc4040189a0 con 0x7fc4140ffae0 2024-03-12T05:12:55.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 Processor -- start 2024-03-12T05:12:55.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 -- start start 2024-03-12T05:12:55.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408625cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408612eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604085d3960 con 0x5604086262a0 2024-03-12T05:12:55.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408587310 con 0x5604086258f0 2024-03-12T05:12:55.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408584840 con 0x560408612a50 2024-03-12T05:12:55.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc410ff9640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 0x7fc3fc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc410ff9640 1 -- 172.21.15.45:0/4077107067 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc404014070 con 0x7fc4140ffae0 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44594/0 (socket says 172.21.15.45:44594) 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc4137fe640 1 -- 172.21.15.45:0/1173282369 learned_addr learned my addr 172.21.15.45:0/1173282369 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc412ffd640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408612eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.853+0000 7fc413fff640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408625cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc413fff640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 0x7fc3fc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc4137fe640 1 -- 172.21.15.45:0/1173282369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 msgr2=0x560408612eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc4137fe640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408612eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc4137fe640 1 -- 172.21.15.45:0/1173282369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 msgr2=0x560408625cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc4137fe640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408625cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc4137fe640 1 -- 172.21.15.45:0/1173282369 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560408542060 con 0x5604086262a0 2024-03-12T05:12:55.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.854+0000 7fc412ffd640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408612eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:55.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.855+0000 7fc4137fe640 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc40400c6a0 tx=0x7fc404096420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98edce02652b3ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.855+0000 7fc3f9ffb640 1 -- 172.21.15.45:0/1173282369 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40409e8c0 con 0x5604086262a0 2024-03-12T05:12:55.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.855+0000 7fc3f9ffb640 1 -- 172.21.15.45:0/1173282369 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4040a2f10 con 0x5604086262a0 2024-03-12T05:12:55.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.855+0000 7fc3f9ffb640 1 -- 172.21.15.45:0/1173282369 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040a2a80 con 0x5604086262a0 2024-03-12T05:12:55.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.855+0000 7fc413fff640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 0x7fc3fc0a4060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x560408613570 tx=0x7fc40c00a610 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- 172.21.15.45:0/1173282369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 msgr2=0x560408612430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc40400c6a0 tx=0x7fc404096420 comp rx=0 tx=0).stop 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- 172.21.15.45:0/1173282369 shutdown_connections 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408612430 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408612eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/1173282369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408625cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- 172.21.15.45:0/1173282369 >> 172.21.15.45:0/1173282369 conn(0x560408614ef0 msgr2=0x5604086152d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- 172.21.15.45:0/1173282369 shutdown_connections 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- 172.21.15.45:0/1173282369 wait complete. 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.856+0000 7fc41c7f5740 1 -- start start 2024-03-12T05:12:55.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 Processor -- start 2024-03-12T05:12:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 0x560408629870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 0x560408629db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 0x560408629db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604086262a0 0x56040862a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 0x560408629db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44598/0 (socket says 172.21.15.45:44598) 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc412ffd640 1 -- 172.21.15.45:0/401967932 learned_addr learned my addr 172.21.15.45:0/401967932 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408580020 con 0x5604086258f0 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408253430 con 0x560408612a50 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040820fe00 con 0x5604086262a0 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.857+0000 7fc4137fe640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 0x560408629870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc41c7f5740 10 client.0 _ll_register_callbacks cb 0x560408527cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc413fff640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604086262a0 0x56040862a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 -- 172.21.15.45:0/401967932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604086262a0 msgr2=0x56040862a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604086262a0 0x56040862a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 -- 172.21.15.45:0/401967932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 msgr2=0x560408629db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 0x560408629db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141a4240 con 0x560408612a50 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc4137fe640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 0x560408629870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x56040862c900 tx=0x7fc40409a9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f600bbf17c880d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.858+0000 7fc3f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.859+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404096e90 con 0x560408612a50 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.859+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56040827f7e0 con 0x560408612a50 2024-03-12T05:12:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.859+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc40409e8c0 con 0x560408612a50 2024-03-12T05:12:55.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4040a2c90 con 0x560408612a50 2024-03-12T05:12:55.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 10 client.25595 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56040857e6a0 con 0x560408612a50 2024-03-12T05:12:55.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4040b3200 con 0x560408612a50 2024-03-12T05:12:55.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc4040967b0 con 0x560408612a50 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 10 client.25595 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 10 client.25595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 20 client.25595 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.860+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56040864b000 0x56040864d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.861+0000 7fc41c7f5740 4 client.25595 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.861+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56040827f7e0 con 0x56040864b000 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.861+0000 7fc412ffd640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56040864b000 0x56040864d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.862+0000 7fc3f2ffd640 10 client.25595 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.862+0000 7fc412ffd640 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56040864b000 0x56040864d480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83fefcc2230ea904 server_cookie=e5a09db11122e5d7 in_seq=0 out_seq=0 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/401967932 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56040827f7e0 con 0x56040864b000 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc3f2ffd640 10 client.25595 handle_command_reply: tid=0 2024-03-12T05:12:55.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc41c7f5740 1 client.25595 shutdown 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 msgr2=0x560408629870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 0x560408629870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x56040862c900 tx=0x7fc40409a9a0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 shutdown_connections 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.863+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56040864b000 0x56040864d480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086258f0 0x560408629db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604086262a0 0x56040862a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/401967932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560408612a50 0x560408629870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 >> 172.21.15.45:0/401967932 conn(0x560408614ef0 msgr2=0x560408613dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc3f2ffd640 0 client.25595 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 shutdown_connections 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 1 -- 172.21.15.45:0/401967932 wait complete. 2024-03-12T05:12:55.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.864+0000 7fc41c7f5740 20 client.25595 trim_cache size 0 max 16384 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 Processor -- start 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 -- start start 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x56040862bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x5604086248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408505130 con 0x5604086262a0 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560408647e40 con 0x5604086258f0 2024-03-12T05:12:55.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604086136d0 con 0x560408612a50 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46822/0 (socket says 172.21.15.45:46822) 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 -- 172.21.15.45:0/413297610 learned_addr learned my addr 172.21.15.45:0/413297610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc413fff640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x5604086248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc4137fe640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x56040862bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 -- 172.21.15.45:0/413297610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 msgr2=0x56040862bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x56040862bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 -- 172.21.15.45:0/413297610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 msgr2=0x5604086248e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x5604086248e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc412ffd640 1 -- 172.21.15.45:0/413297610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56040851ac20 con 0x5604086258f0 2024-03-12T05:12:55.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc4137fe640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x56040862bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.886+0000 7fc413fff640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x5604086248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.887+0000 7fc412ffd640 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc408009230 tx=0x7fc40802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd28bfc107eb069a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.887+0000 7fc3f97fa640 1 -- 172.21.15.45:0/413297610 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408046070 con 0x5604086258f0 2024-03-12T05:12:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.887+0000 7fc3f97fa640 1 -- 172.21.15.45:0/413297610 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4080047e0 con 0x5604086258f0 2024-03-12T05:12:55.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.887+0000 7fc3f97fa640 1 -- 172.21.15.45:0/413297610 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40802c9c0 con 0x5604086258f0 2024-03-12T05:12:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 -- 172.21.15.45:0/413297610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 msgr2=0x560408651de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc408009230 tx=0x7fc40802e040 comp rx=0 tx=0).stop 2024-03-12T05:12:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 -- 172.21.15.45:0/413297610 shutdown_connections 2024-03-12T05:12:55.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x5604086248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408651de0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/413297610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x56040862bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 -- 172.21.15.45:0/413297610 >> 172.21.15.45:0/413297610 conn(0x560408614ef0 msgr2=0x56040864d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.888+0000 7fc41c7f5740 1 -- 172.21.15.45:0/413297610 shutdown_connections 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.889+0000 7fc41c7f5740 1 -- 172.21.15.45:0/413297610 wait complete. 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.889+0000 7fc41c7f5740 1 -- start start 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.889+0000 7fc41c7f5740 1 Processor -- start 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408640a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408641490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604086a16e0 con 0x5604086262a0 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604086a1850 con 0x5604086258f0 2024-03-12T05:12:55.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604086a1a10 con 0x560408612a50 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc41c7f5740 10 client.0 _ll_register_callbacks cb 0x7fc40400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408640a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408640a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53192/0 (socket says 172.21.15.45:53192) 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc4137fe640 1 -- 172.21.15.45:0/2695685857 learned_addr learned my addr 172.21.15.45:0/2695685857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc412ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46824/0 (socket says 172.21.15.45:46824) 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc413fff640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408641490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc412ffd640 1 -- 172.21.15.45:0/2695685857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 msgr2=0x560408640a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc412ffd640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408640a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.890+0000 7fc412ffd640 1 -- 172.21.15.45:0/2695685857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 msgr2=0x560408641490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc412ffd640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408641490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc412ffd640 1 -- 172.21.15.45:0/2695685857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc408030070 con 0x5604086258f0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc412ffd640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56040862c7c0 tx=0x7fc408004200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73a54a0b02d509e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:55.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc3f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408046070 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc40802ca40 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.891+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4040a2f10 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc408005d50 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc41c7f5740 10 client.25601 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5604085408c0 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4040a2f10 con 0x5604086258f0 2024-03-12T05:12:55.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc40803c040 con 0x5604086258f0 2024-03-12T05:12:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc41c7f5740 10 client.25601 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:55.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc41c7f5740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.892+0000 7fc41c7f5740 20 client.25601 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604086a2f90 0x5604086a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc41c7f5740 4 client.25601 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5604082bc720 con 0x5604086a2f90 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc4137fe640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604086a2f90 0x5604086a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc4137fe640 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604086a2f90 0x5604086a53b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f900810f3c340e88 server_cookie=ea13c5ac47e1e9f5 in_seq=0 out_seq=0 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.893+0000 7fc3f2ffd640 10 client.25601 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc3f2ffd640 1 -- 172.21.15.45:0/2695685857 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5604082bc720 con 0x5604086a2f90 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc3f2ffd640 10 client.25601 handle_command_reply: tid=0 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 client.25601 shutdown 2024-03-12T05:12:55.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 msgr2=0x560408640f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56040862c7c0 tx=0x7fc408004200 comp rx=0 tx=0).stop 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 shutdown_connections 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5604086a2f90 0x5604086a53b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5604086262a0 0x560408641490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5604086258f0 0x560408640f50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 --2- 172.21.15.45:0/2695685857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560408612a50 0x560408640a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 >> 172.21.15.45:0/2695685857 conn(0x560408614ef0 msgr2=0x560408651de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc3f2ffd640 0 client.25601 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 shutdown_connections 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 1 -- 172.21.15.45:0/2695685857 wait complete. 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.894+0000 7fc41c7f5740 20 client.25601 trim_cache size 0 max 16384 2024-03-12T05:12:55.900 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:12:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 msgr2=0x7fc3fc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 0x7fc3fc0a4060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x560408613570 tx=0x7fc40c00a610 comp rx=0 tx=0).stop 2024-03-12T05:12:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 msgr2=0x7fc4141976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc40400e9f0 tx=0x7fc40400eeb0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 shutdown_connections 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3fc0a1ba0 0x7fc3fc0a4060 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc414105870 0x7fc41419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4141004a0 0x7fc414197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 --2- 172.21.15.45:0/4077107067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4140ffae0 0x7fc4141976d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 >> 172.21.15.45:0/4077107067 conn(0x7fc4140761d0 msgr2=0x7fc4140fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 shutdown_connections 2024-03-12T05:12:55.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:55.895+0000 7fc419c1c640 1 -- 172.21.15.45:0/4077107067 wait complete. 2024-03-12T05:12:56.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:56 smithi130 ceph-mon[35398]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 2.19k op/s 2024-03-12T05:12:56.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:56 smithi116 ceph-mon[35486]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 2.19k op/s 2024-03-12T05:12:56.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:56 smithi045 ceph-mon[32174]: pgmap v64: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 2.19k op/s 2024-03-12T05:12:56.909 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:12:56.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:12:56.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 -- 172.21.15.45:0/344858758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4101d10 msgr2=0x7f9ee4105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/344858758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4101d10 0x7f9ee4105d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009930 tx=0x7f9ed802bc30 comp rx=0 tx=0).stop 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 -- 172.21.15.45:0/344858758 shutdown_connections 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/344858758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4106430 0x7f9ee410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/344858758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4101d10 0x7f9ee4105d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/344858758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee4101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.085+0000 7f9eea7eb640 1 -- 172.21.15.45:0/344858758 >> 172.21.15.45:0/344858758 conn(0x7f9ee40fcbe0 msgr2=0x7f9ee40ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.086+0000 7f9eea7eb640 1 -- 172.21.15.45:0/344858758 shutdown_connections 2024-03-12T05:12:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.086+0000 7f9eea7eb640 1 -- 172.21.15.45:0/344858758 wait complete. 2024-03-12T05:12:57.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.086+0000 7f9eea7eb640 1 Processor -- start 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.086+0000 7f9eea7eb640 1 -- start start 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 0x7f9ee419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a300 con 0x7f9ee4199850 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a470 con 0x7f9ee4106430 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9eea7eb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a5e0 con 0x7f9ee4101360 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee8d88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 0x7f9ee419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee8d88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44630/0 (socket says 172.21.15.45:44630) 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53200/0 (socket says 172.21.15.45:53200) 2024-03-12T05:12:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee8d88640 1 -- 172.21.15.45:0/3567253249 learned_addr learned my addr 172.21.15.45:0/3567253249 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee37fe640 1 -- 172.21.15.45:0/3567253249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 msgr2=0x7f9ee419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.087+0000 7f9ee37fe640 1 -- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 msgr2=0x7f9ee4199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee37fe640 1 -- 172.21.15.45:0/3567253249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8009560 con 0x7f9ee4106430 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 0x7f9ee419fbd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009230 tx=0x7f9ed8004bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.088+0000 7f9ee17fa640 1 -- 172.21.15.45:0/3567253249 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80043d0 con 0x7f9ee4106430 2024-03-12T05:12:57.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.089+0000 7f9ee17fa640 1 -- 172.21.15.45:0/3567253249 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ed8003aa0 con 0x7f9ee4106430 2024-03-12T05:12:57.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.089+0000 7f9ee17fa640 1 -- 172.21.15.45:0/3567253249 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8005470 con 0x7f9ee4106430 2024-03-12T05:12:57.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.089+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee41a64e0 con 0x7f9ee4106430 2024-03-12T05:12:57.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.089+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee41a6970 con 0x7f9ee4106430 2024-03-12T05:12:57.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 Processor -- start 2024-03-12T05:12:57.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 -- start start 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc660760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9ee17fa640 1 -- 172.21.15.45:0/3567253249 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ed8003c10 con 0x7f9ee4106430 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc6698b0 0x5615dc66bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc5cef90 con 0x5615dc660d30 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc5cc4c0 con 0x5615dc660380 2024-03-12T05:12:57.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc5c7ca0 con 0x5615dc6698b0 2024-03-12T05:12:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.092+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee8d88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc660760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44646/0 (socket says 172.21.15.45:44646) 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 -- 172.21.15.45:0/4069416191 learned_addr learned my addr 172.21.15.45:0/4069416191 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc6698b0 0x5615dc66bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 -- 172.21.15.45:0/4069416191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc6698b0 msgr2=0x5615dc66bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc6698b0 0x5615dc66bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 -- 172.21.15.45:0/4069416191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 msgr2=0x5615dc660760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc660760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 -- 172.21.15.45:0/4069416191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615dc58a0b0 con 0x5615dc660d30 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee17fa640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 0x7f9ecc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.093+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40115f0 tx=0x7f9ed4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8ca929473ca393 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.094+0000 7f9ee17fa640 1 -- 172.21.15.45:0/3567253249 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9ed802de40 con 0x7f9ee4106430 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.094+0000 7f9eca7fc640 1 -- 172.21.15.45:0/4069416191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400f640 con 0x5615dc660d30 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.094+0000 7f9eca7fc640 1 -- 172.21.15.45:0/4069416191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ed400fc50 con 0x5615dc660d30 2024-03-12T05:12:57.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.094+0000 7f9eca7fc640 1 -- 172.21.15.45:0/4069416191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4019cc0 con 0x5615dc660d30 2024-03-12T05:12:57.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.094+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 0x7f9ecc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- 172.21.15.45:0/4069416191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 msgr2=0x5615dc6692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40115f0 tx=0x7f9ed4011ab0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 0x7f9ecc0a40e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009270 tx=0x7f9edc009680 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- 172.21.15.45:0/4069416191 shutdown_connections 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6692e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc6698b0 0x5615dc66bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/4069416191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc660760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- 172.21.15.45:0/4069416191 >> 172.21.15.45:0/4069416191 conn(0x5615dc65a840 msgr2=0x5615dc65ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- 172.21.15.45:0/4069416191 shutdown_connections 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- 172.21.15.45:0/4069416191 wait complete. 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.095+0000 7f9eed3b6740 1 -- start start 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9eed3b6740 1 Processor -- start 2024-03-12T05:12:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660d30 0x5615dc666db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc6672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc56cf40 con 0x5615dc6698b0 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.096+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc29b4b0 con 0x5615dc660d30 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53216/0 (socket says 172.21.15.45:53216) 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 -- 172.21.15.45:0/1215938568 learned_addr learned my addr 172.21.15.45:0/1215938568 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc54d670 con 0x5615dc660380 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc6672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9eed3b6740 10 client.0 _ll_register_callbacks cb 0x5615dc5baf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660d30 0x5615dc666db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 -- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660d30 msgr2=0x5615dc666db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660d30 0x5615dc666db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 -- 172.21.15.45:0/1215938568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 msgr2=0x5615dc6672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc6672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 -- 172.21.15.45:0/1215938568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed4009d40 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed4019640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=866819d01f4f3b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.097+0000 7f9ec37fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4004240 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615dc5a1f40 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9ed40043a0 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed4005070 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9eed3b6740 10 client.25624 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5615dc5c6320 con 0x5615dc660380 2024-03-12T05:12:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.098+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ed402f020 con 0x5615dc660380 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9ed4004bf0 con 0x5615dc660380 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 10 client.25624 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 10 client.25624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 20 client.25624 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc679cd0 0x5615dc67c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 4 client.25624 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615dc5be310 con 0x5615dc679cd0 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.099+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc679cd0 0x5615dc67c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.100+0000 7f9ec37fe640 10 client.25624 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.100+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc679cd0 0x5615dc67c0e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4996f11c830499d2 server_cookie=46570e15e1606176 in_seq=0 out_seq=0 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9ec37fe640 1 -- 172.21.15.45:0/1215938568 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5615dc5be310 con 0x5615dc679cd0 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9ec37fe640 10 client.25624 handle_command_reply: tid=0 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 client.25624 shutdown 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 msgr2=0x5615dc666870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4000c00 tx=0x7f9ed4019640 comp rx=0 tx=0).stop 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 shutdown_connections 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc679cd0 0x5615dc67c0e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc6672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9ec37fe640 0 client.25624 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660d30 0x5615dc666db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/1215938568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc666870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 >> 172.21.15.45:0/1215938568 conn(0x5615dc65a840 msgr2=0x5615dc65c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 shutdown_connections 2024-03-12T05:12:57.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.101+0000 7f9eed3b6740 1 -- 172.21.15.45:0/1215938568 wait complete. 2024-03-12T05:12:57.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.102+0000 7f9eed3b6740 20 client.25624 trim_cache size 0 max 16384 2024-03-12T05:12:57.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.128+0000 7f9eed3b6740 1 Processor -- start 2024-03-12T05:12:57.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 -- start start 2024-03-12T05:12:57.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 0x5615dc6708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc670e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 0x5615dc6708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 0x5615dc6708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53220/0 (socket says 172.21.15.45:53220) 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46866/0 (socket says 172.21.15.45:46866) 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee37fe640 1 -- 172.21.15.45:0/378368057 learned_addr learned my addr 172.21.15.45:0/378368057 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc670e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc66c3c0 con 0x5615dc6698b0 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc65a370 con 0x5615dc660380 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.129+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc664dc0 con 0x5615dc660d30 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 -- 172.21.15.45:0/378368057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 msgr2=0x5615dc6708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 0x5615dc6708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 -- 172.21.15.45:0/378368057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 msgr2=0x5615dc659760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 -- 172.21.15.45:0/378368057 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615dc562c00 con 0x5615dc6698b0 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.130+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc670e20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9edc012680 tx=0x7f9edc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c13b8875536e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.131+0000 7f9ec9ffb640 1 -- 172.21.15.45:0/378368057 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc025040 con 0x5615dc6698b0 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.131+0000 7f9ec9ffb640 1 -- 172.21.15.45:0/378368057 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9edc011d60 con 0x5615dc6698b0 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.131+0000 7f9ec9ffb640 1 -- 172.21.15.45:0/378368057 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc017070 con 0x5615dc6698b0 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 msgr2=0x5615dc670e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc670e20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9edc012680 tx=0x7f9edc01a000 comp rx=0 tx=0).stop 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 shutdown_connections 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc6698b0 0x5615dc670e20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660d30 0x5615dc6708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/378368057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc660380 0x5615dc659760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 >> 172.21.15.45:0/378368057 conn(0x5615dc65a840 msgr2=0x5615dc659050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 shutdown_connections 2024-03-12T05:12:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- 172.21.15.45:0/378368057 wait complete. 2024-03-12T05:12:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.132+0000 7f9eed3b6740 1 -- start start 2024-03-12T05:12:57.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 Processor -- start 2024-03-12T05:12:57.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc6d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6d3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc6704a0 con 0x5615dc660d30 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc670610 con 0x5615dc6698b0 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615dc6707d0 con 0x5615dc660380 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9eed3b6740 10 client.0 _ll_register_callbacks cb 0x7f9ed4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc6d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee8d88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc6d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:53226/0 (socket says 172.21.15.45:53226) 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee3fff640 1 -- 172.21.15.45:0/3902720392 learned_addr learned my addr 172.21.15.45:0/3902720392 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee8d88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46882/0 (socket says 172.21.15.45:46882) 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.133+0000 7f9ee37fe640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 -- 172.21.15.45:0/3902720392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 msgr2=0x5615dc6d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc6d3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 -- 172.21.15.45:0/3902720392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 msgr2=0x5615dc6d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6d3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 -- 172.21.15.45:0/3902720392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615dc5c6320 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ee8d88640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9edc019040 tx=0x7f9edc005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71e85f8e561347c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.134+0000 7f9ec37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc017040 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615dc5ed900 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9edc02a480 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9edc004e50 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 10 client.25619 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f9edc011a40 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5615dc5ed900 con 0x5615dc6698b0 2024-03-12T05:12:57.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9edc030020 con 0x5615dc6698b0 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 10 client.25619 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 20 client.25619 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc6dae90 0x5615dc6dd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 4 client.25619 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.135+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5615dc2c7860 con 0x5615dc6dae90 2024-03-12T05:12:57.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.136+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc6dae90 0x5615dc6dd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.136+0000 7f9ec37fe640 10 client.25619 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.137+0000 7f9ee3fff640 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc6dae90 0x5615dc6dd2b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c45d7d909a4a972e server_cookie=941061672e99126 in_seq=0 out_seq=0 2024-03-12T05:12:57.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.138+0000 7f9ec37fe640 1 -- 172.21.15.45:0/3902720392 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5615dc2c7860 con 0x5615dc6dae90 2024-03-12T05:12:57.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.138+0000 7f9ec37fe640 10 client.25619 handle_command_reply: tid=0 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.138+0000 7f9eed3b6740 1 client.25619 shutdown 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 msgr2=0x5615dc66fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9edc019040 tx=0x7f9edc005c80 comp rx=0 tx=0).stop 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 shutdown_connections 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5615dc6dae90 0x5615dc6dd2b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5615dc660d30 0x5615dc6d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5615dc6698b0 0x5615dc66fdd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 --2- 172.21.15.45:0/3902720392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615dc660380 0x5615dc6d3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 >> 172.21.15.45:0/3902720392 conn(0x5615dc65a840 msgr2=0x5615dc6598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9ec37fe640 0 client.25619 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 shutdown_connections 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 1 -- 172.21.15.45:0/3902720392 wait complete. 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.139+0000 7f9eed3b6740 20 client.25619 trim_cache size 0 max 16384 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "c63497eb-b4ce-4867-9e4e-446100115ed2", 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:12:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.140+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 msgr2=0x7f9ecc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.140+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 0x7f9ecc0a40e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009270 tx=0x7f9edc009680 comp rx=0 tx=0).stop 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.140+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 msgr2=0x7f9ee419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.140+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 0x7f9ee419fbd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009230 tx=0x7f9ed8004bb0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 shutdown_connections 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ecc0a1c20 0x7f9ecc0a40e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ee4199850 0x7f9ee4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4106430 0x7f9ee419fbd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 --2- 172.21.15.45:0/3567253249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee4101360 0x7f9ee419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 >> 172.21.15.45:0/3567253249 conn(0x7f9ee40fcbe0 msgr2=0x7f9ee40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 shutdown_connections 2024-03-12T05:12:57.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.141+0000 7f9eea7eb640 1 -- 172.21.15.45:0/3567253249 wait complete. 2024-03-12T05:12:57.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:12:57.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 -- 172.21.15.45:0/614689988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 msgr2=0x7f578c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 --2- 172.21.15.45:0/614689988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c10e000 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5784009cd0 tx=0x7f578402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 -- 172.21.15.45:0/614689988 shutdown_connections 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 --2- 172.21.15.45:0/614689988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c10e000 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 --2- 172.21.15.45:0/614689988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 0x7f578c069c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 --2- 172.21.15.45:0/614689988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 0x7f578c106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 -- 172.21.15.45:0/614689988 >> 172.21.15.45:0/614689988 conn(0x7f578c076e20 msgr2=0x7f578c077230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 -- 172.21.15.45:0/614689988 shutdown_connections 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.309+0000 7f57945a7640 1 -- 172.21.15.45:0/614689988 wait complete. 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 Processor -- start 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 -- start start 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 0x7f578c19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 0x7f578c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c199c30 con 0x7f578c06a170 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c199da0 con 0x7f578c106130 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f57945a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c199f60 con 0x7f578c0697b0 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44702/0 (socket says 172.21.15.45:44702) 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 -- 172.21.15.45:0/1979100464 learned_addr learned my addr 172.21.15.45:0/1979100464 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5792b44640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 0x7f578c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5792343640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 0x7f578c19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 -- 172.21.15.45:0/1979100464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 msgr2=0x7f578c19f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 0x7f578c19f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 -- 172.21.15.45:0/1979100464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 msgr2=0x7f578c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 0x7f578c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.310+0000 7f5791b42640 1 -- 172.21.15.45:0/1979100464 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5784009900 con 0x7f578c06a170 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f5791b42640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d770 tx=0x7f577c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f57837fe640 1 -- 172.21.15.45:0/1979100464 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c009c30 con 0x7f578c06a170 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f57837fe640 1 -- 172.21.15.45:0/1979100464 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f577c00de80 con 0x7f578c06a170 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c19a240 con 0x7f578c06a170 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f57837fe640 1 -- 172.21.15.45:0/1979100464 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c004e00 con 0x7f578c06a170 2024-03-12T05:12:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.311+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c1a64d0 con 0x7f578c06a170 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.313+0000 7f57837fe640 1 -- 172.21.15.45:0/1979100464 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f577c004f60 con 0x7f578c06a170 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.313+0000 7f5797177740 1 Processor -- start 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.313+0000 7f5797177740 1 -- start start 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f57837fe640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 0x7f57780a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0bfb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c06ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0b6a280 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0b677b0 con 0x55fbd0c047b0 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5797177740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0b62f90 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5792b44640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 0x7f57780a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f57837fe640 1 -- 172.21.15.45:0/1979100464 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f577c0cd280 con 0x7f578c06a170 2024-03-12T05:12:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44706/0 (socket says 172.21.15.45:44706) 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 -- 172.21.15.45:0/3327993218 learned_addr learned my addr 172.21.15.45:0/3327993218 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5792343640 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0bfb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5792b44640 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c06ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 -- 172.21.15.45:0/3327993218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 msgr2=0x55fbd0bfb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0bfb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 -- 172.21.15.45:0/3327993218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 msgr2=0x55fbd0c06ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c06ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 -- 172.21.15.45:0/3327993218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbd0b21570 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5792b44640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 0x7f57780a4060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5784005cc0 tx=0x7f5784032f80 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5791b42640 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f577c099e40 tx=0x7f577c099b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc00ac99b087808d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.314+0000 7f5780bfe640 1 -- 172.21.15.45:0/3327993218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c099eb0 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5780bfe640 1 -- 172.21.15.45:0/3327993218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f577c09e000 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5780bfe640 1 -- 172.21.15.45:0/3327993218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c096520 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 -- 172.21.15.45:0/3327993218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 msgr2=0x55fbd0c04270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f577c099e40 tx=0x7f577c099b00 comp rx=0 tx=0).stop 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 -- 172.21.15.45:0/3327993218 shutdown_connections 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c04270 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c06ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 --2- 172.21.15.45:0/3327993218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0bfb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 -- 172.21.15.45:0/3327993218 >> 172.21.15.45:0/3327993218 conn(0x55fbd0bf5990 msgr2=0x55fbd0bf5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 -- 172.21.15.45:0/3327993218 shutdown_connections 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.315+0000 7f5797177740 1 -- 172.21.15.45:0/3327993218 wait complete. 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 -- start start 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 Processor -- start 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c01550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c01b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0836430 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd07f2e00 con 0x55fbd0c047b0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0ae7fd0 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5797177740 10 client.0 _ll_register_callbacks cb 0x55fbd0b3dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44708/0 (socket says 172.21.15.45:44708) 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 -- 172.21.15.45:0/324107641 learned_addr learned my addr 172.21.15.45:0/324107641 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5791b42640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c01550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792b44640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c01b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 -- 172.21.15.45:0/324107641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 msgr2=0x55fbd0c01550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c01550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 -- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 msgr2=0x55fbd0c01b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c01b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 -- 172.21.15.45:0/324107641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c1a64d0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5791b42640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c01550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792b44640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c01b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.316+0000 7f5792343640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f578800bf40 tx=0x7f57880128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf89b9cb57e5a8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57880108a0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5788010a00 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fbd08627e0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5788010bc0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 10 client.16173 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fbd0b61610 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5788027020 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f578800f3c0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 10 client.16173 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 20 client.16173 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c16dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 4 client.16173 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fbd089f720 con 0x55fbd0c149b0 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.317+0000 7f5791b42640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c16dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5775ffb640 10 client.16173 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5791b42640 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c16dd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c38cc2358e14dd0a server_cookie=79b4ac0f22477011 in_seq=0 out_seq=0 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5775ffb640 1 -- 172.21.15.45:0/324107641 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fbd089f720 con 0x55fbd0c149b0 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5775ffb640 10 client.16173 handle_command_reply: tid=0 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5797177740 1 client.16173 shutdown 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 msgr2=0x55fbd0c041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.318+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f578800bf40 tx=0x7f57880128e0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 shutdown_connections 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c16dd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5775ffb640 0 client.16173 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c01b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c01550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 --2- 172.21.15.45:0/324107641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c041c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 >> 172.21.15.45:0/324107641 conn(0x55fbd0bf5990 msgr2=0x55fbd0bf7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 shutdown_connections 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 1 -- 172.21.15.45:0/324107641 wait complete. 2024-03-12T05:12:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.319+0000 7f5797177740 20 client.16173 trim_cache size 0 max 16384 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.334+0000 7f5797177740 1 Processor -- start 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.334+0000 7f5797177740 1 -- start start 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c03c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0c06fa0 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0bf57f0 con 0x55fbd0c047b0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5797177740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0c028a0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44724/0 (socket says 172.21.15.45:44724) 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 -- 172.21.15.45:0/2278488285 learned_addr learned my addr 172.21.15.45:0/2278488285 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5792b44640 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5792343640 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c03c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 -- 172.21.15.45:0/2278488285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 msgr2=0x55fbd0c03c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c03c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 -- 172.21.15.45:0/2278488285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 msgr2=0x55fbd0c0b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 -- 172.21.15.45:0/2278488285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fbd0afdac0 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5791b42640 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f577c098400 tx=0x7f577c04b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dd20705788e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5777fff640 1 -- 172.21.15.45:0/2278488285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0b1070 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5777fff640 1 -- 172.21.15.45:0/2278488285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f577c098230 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.335+0000 7f5777fff640 1 -- 172.21.15.45:0/2278488285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c099300 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- 172.21.15.45:0/2278488285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 msgr2=0x55fbd0c041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f577c098400 tx=0x7f577c04b040 comp rx=0 tx=0).stop 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- 172.21.15.45:0/2278488285 shutdown_connections 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c041c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 --2- 172.21.15.45:0/2278488285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c03c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- 172.21.15.45:0/2278488285 >> 172.21.15.45:0/2278488285 conn(0x55fbd0bf5990 msgr2=0x55fbd0c1b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- 172.21.15.45:0/2278488285 shutdown_connections 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- 172.21.15.45:0/2278488285 wait complete. 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.336+0000 7f5797177740 1 -- start start 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 Processor -- start 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c0ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0c0aa70 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0c0abe0 con 0x55fbd0bfb2b0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fbd0c6aec0 con 0x55fbd0c047b0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5797177740 10 client.0 _ll_register_callbacks cb 0x7f577c004760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44738/0 (socket says 172.21.15.45:44738) 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 -- 172.21.15.45:0/941065403 learned_addr learned my addr 172.21.15.45:0/941065403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5792b44640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5792343640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c0ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 -- 172.21.15.45:0/941065403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0c047b0 msgr2=0x55fbd0c0a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 -- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0bfb2b0 msgr2=0x55fbd0c0ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:12:57.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.337+0000 7f5791b42640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c0ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5791b42640 1 -- 172.21.15.45:0/941065403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fbd0b21570 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5791b42640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f577c0962c0 tx=0x7f577c009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2a94159a18dc83 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5775ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0b1050 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f577c0b8270 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f577c09e000 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f577c0b8430 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5797177740 10 client.16179 fetch_fsmap learned FSMap version 29 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.338+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fbd0b236a0 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.339+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f577c096520 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.339+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f577c005210 con 0x55fbd0bfbbd0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 10 client.16179 fetch_fsmap finished waiting for FSMap version 29 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 20 client.16179 populate_metadata read hostname 'smithi045' 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c6e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 4 client.16179 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f577c09e000 con 0x55fbd0c149b0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.340+0000 7f5792343640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c6e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5792343640 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c6e220 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a13f9706274d78 server_cookie=2a9618f61e29c817 in_seq=0 out_seq=0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5775ffb640 10 client.16179 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5775ffb640 1 -- 172.21.15.45:0/941065403 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f577c09e000 con 0x55fbd0c149b0 2024-03-12T05:12:57.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5775ffb640 10 client.16179 handle_command_reply: tid=0 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5797177740 1 client.16179 shutdown 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 msgr2=0x55fbd0c09dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.342+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f577c0962c0 tx=0x7f577c009d90 comp rx=0 tx=0).stop 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 shutdown_connections 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fbd0c149b0 0x55fbd0c6e220 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fbd0bfbbd0 0x55fbd0c09dd0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fbd0c047b0 0x55fbd0c0a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 --2- 172.21.15.45:0/941065403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbd0bfb2b0 0x55fbd0c0ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 >> 172.21.15.45:0/941065403 conn(0x55fbd0bf5990 msgr2=0x55fbd0bffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5775ffb640 0 client.16179 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 shutdown_connections 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 1 -- 172.21.15.45:0/941065403 wait complete. 2024-03-12T05:12:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.343+0000 7f5797177740 20 client.16179 trim_cache size 0 max 16384 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "c63497eb-b4ce-4867-9e4e-446100115ed2": { 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "c63497eb-b4ce-4867-9e4e-446100115ed2", 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:12:57.343 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.344+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 msgr2=0x7f57780a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.344+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 0x7f57780a4060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5784005cc0 tx=0x7f5784032f80 comp rx=0 tx=0).stop 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.344+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 msgr2=0x7f578c19f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.344+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d770 tx=0x7f577c00dc30 comp rx=0 tx=0).stop 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.344+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 shutdown_connections 2024-03-12T05:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57780a1ba0 0x7f57780a4060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f578c06a170 0x7f578c19f970 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f578c106130 0x7f578c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 --2- 172.21.15.45:0/1979100464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f578c0697b0 0x7f578c19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 >> 172.21.15.45:0/1979100464 conn(0x7f578c076e20 msgr2=0x7f578c10c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 shutdown_connections 2024-03-12T05:12:57.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:12:57.345+0000 7f57945a7640 1 -- 172.21.15.45:0/1979100464 wait complete. 2024-03-12T05:12:57.358 INFO:tasks.cephfs.filesystem:scrub status for tag:c63497eb-b4ce-4867-9e4e-446100115ed2 - {'path': '/', 'tag': 'c63497eb-b4ce-4867-9e4e-446100115ed2', 'options': 'recursive,force'} 2024-03-12T05:12:58.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:12:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:12:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:58.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: pgmap v65: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:12:58.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:12:58.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:58.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:12:59.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:12:59 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:12:59.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:12:59 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:12:59.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:12:59 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:12:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 55.0 s, 1474.4 tps, lat 67.955 ms stddev 35.227 2024-03-12T05:13:00.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:13:00.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:13:00.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:13:00.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 20 MiB/s wr, 2.15k op/s 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:13:00.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:13:02.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:02 smithi130 ceph-mon[35398]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:02.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:02 smithi116 ceph-mon[35486]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:02.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:02 smithi045 ceph-mon[32174]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:04.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:04 smithi130 ceph-mon[35398]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:04.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:04 smithi116 ceph-mon[35486]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:04.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:04 smithi045 ceph-mon[32174]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-03-12T05:13:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 60.0 s, 1493.4 tps, lat 66.832 ms stddev 23.999 2024-03-12T05:13:06.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:06 smithi130 ceph-mon[35398]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:06.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:06 smithi116 ceph-mon[35486]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:06.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:06 smithi045 ceph-mon[32174]: pgmap v69: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:08.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:08 smithi130 ceph-mon[35398]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:08.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:08.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:08.719 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:08.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:08 smithi116 ceph-mon[35486]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:08.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:08.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:08.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:08 smithi045 ceph-mon[32174]: pgmap v70: 97 pgs: 97 active+clean; 1.4 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:08.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:09.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:09 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:09.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:09 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:09.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:09 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:09.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 65.0 s, 1627.8 tps, lat 61.382 ms stddev 16.006 2024-03-12T05:13:10.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:10 smithi130 ceph-mon[35398]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 19 MiB/s wr, 2.24k op/s 2024-03-12T05:13:10.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:10 smithi116 ceph-mon[35486]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 19 MiB/s wr, 2.24k op/s 2024-03-12T05:13:10.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:10 smithi045 ceph-mon[32174]: pgmap v71: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 19 MiB/s wr, 2.24k op/s 2024-03-12T05:13:12.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:12 smithi130 ceph-mon[35398]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:12.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:12 smithi116 ceph-mon[35486]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:12.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:12 smithi045 ceph-mon[32174]: pgmap v72: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:14.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:14 smithi130 ceph-mon[35398]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:14.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:14 smithi116 ceph-mon[35486]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:14.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:14 smithi045 ceph-mon[32174]: pgmap v73: 97 pgs: 97 active+clean; 1.5 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:13:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 70.0 s, 1560.6 tps, lat 64.057 ms stddev 60.060 2024-03-12T05:13:16.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:16 smithi130 ceph-mon[35398]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 2.31k op/s 2024-03-12T05:13:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:16 smithi116 ceph-mon[35486]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 2.31k op/s 2024-03-12T05:13:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:16 smithi045 ceph-mon[32174]: pgmap v74: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 MiB/s wr, 2.31k op/s 2024-03-12T05:13:18.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:18 smithi130 ceph-mon[35398]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-03-12T05:13:18.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:18 smithi116 ceph-mon[35486]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-03-12T05:13:18.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:18 smithi045 ceph-mon[32174]: pgmap v75: 97 pgs: 97 active+clean; 1.5 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-03-12T05:13:19.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 75.0 s, 1599.4 tps, lat 62.392 ms stddev 16.123 2024-03-12T05:13:20.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:20 smithi130 ceph-mon[35398]: pgmap v76: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 2.34k op/s 2024-03-12T05:13:20.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:20 smithi116 ceph-mon[35486]: pgmap v76: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 2.34k op/s 2024-03-12T05:13:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:20 smithi045 ceph-mon[32174]: pgmap v76: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 21 MiB/s wr, 2.34k op/s 2024-03-12T05:13:22.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:22 smithi130 ceph-mon[35398]: pgmap v77: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:22.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:22 smithi116 ceph-mon[35486]: pgmap v77: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:22.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:22 smithi045 ceph-mon[32174]: pgmap v77: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:23.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:23 smithi045 ceph-mon[32174]: pgmap v78: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:23.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:23 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:23.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:23 smithi130 ceph-mon[35398]: pgmap v78: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:23.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:23 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:23.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:23 smithi116 ceph-mon[35486]: pgmap v78: 97 pgs: 97 active+clean; 1.6 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.56k op/s 2024-03-12T05:13:23.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:23 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:24.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 80.0 s, 1501.8 tps, lat 66.726 ms stddev 16.467 2024-03-12T05:13:25.769 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:25 smithi130 ceph-mon[35398]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 20 MiB/s wr, 2.26k op/s 2024-03-12T05:13:25.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:25 smithi045 ceph-mon[32174]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 20 MiB/s wr, 2.26k op/s 2024-03-12T05:13:25.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:25 smithi116 ceph-mon[35486]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 20 MiB/s wr, 2.26k op/s 2024-03-12T05:13:27.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:13:27.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:13:27.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 -- 172.21.15.45:0/2400875587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34101390 msgr2=0x7f1d34101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 --2- 172.21.15.45:0/2400875587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34101390 0x7f1d34101770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009930 tx=0x7f1d2802bc30 comp rx=0 tx=0).stop 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 -- 172.21.15.45:0/2400875587 shutdown_connections 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 --2- 172.21.15.45:0/2400875587 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34106460 0x7f1d3410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 --2- 172.21.15.45:0/2400875587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34101d40 0x7f1d34105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 --2- 172.21.15.45:0/2400875587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34101390 0x7f1d34101770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 -- 172.21.15.45:0/2400875587 >> 172.21.15.45:0/2400875587 conn(0x7f1d340fcc30 msgr2=0x7f1d340ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 -- 172.21.15.45:0/2400875587 shutdown_connections 2024-03-12T05:13:27.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.495+0000 7f1d3b076640 1 -- 172.21.15.45:0/2400875587 wait complete. 2024-03-12T05:13:27.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 Processor -- start 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 -- start start 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 0x7f1d340739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34073f00 0x7f1d341a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341a5f40 con 0x7f1d34101390 2024-03-12T05:13:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341a60b0 con 0x7f1d34106460 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d3b076640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341a6270 con 0x7f1d34073f00 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37646/0 (socket says 172.21.15.45:37646) 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d38e12640 1 -- 172.21.15.45:0/281845845 learned_addr learned my addr 172.21.15.45:0/281845845 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d39613640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34073f00 0x7f1d341a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.497+0000 7f1d33fff640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 0x7f1d340739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 -- 172.21.15.45:0/281845845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34073f00 msgr2=0x7f1d341a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34073f00 0x7f1d341a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 -- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 msgr2=0x7f1d34073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 -- 172.21.15.45:0/281845845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28009560 con 0x7f1d34106460 2024-03-12T05:13:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d38e12640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d33fff640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 0x7f1d340739c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000e9c0 tx=0x7f1d2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d31ffb640 1 -- 172.21.15.45:0/281845845 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2000ccc0 con 0x7f1d34106460 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d31ffb640 1 -- 172.21.15.45:0/281845845 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1d2000ce20 con 0x7f1d34106460 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d31ffb640 1 -- 172.21.15.45:0/281845845 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20018710 con 0x7f1d34106460 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.498+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d341a6550 con 0x7f1d34106460 2024-03-12T05:13:27.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.499+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d341a6a10 con 0x7f1d34106460 2024-03-12T05:13:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.502+0000 7f1d31ffb640 1 -- 172.21.15.45:0/281845845 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1d20010430 con 0x7f1d34106460 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.502+0000 7f1d3dc4a740 1 Processor -- start 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 -- start start 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98d6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 0x5635e98e23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e983d280 con 0x5635e98d73e0 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e983a7b0 con 0x5635e98d6ac0 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9835f90 con 0x5635e98dffc0 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d31ffb640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 0x7f1d240a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d31ffb640 1 -- 172.21.15.45:0/281845845 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f1d20014070 con 0x7f1d34106460 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37650/0 (socket says 172.21.15.45:37650) 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d38e12640 1 -- 172.21.15.45:0/2375898743 learned_addr learned my addr 172.21.15.45:0/2375898743 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.503+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 0x5635e98e23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d39613640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 0x7f1d240a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d33fff640 1 -- 172.21.15.45:0/2375898743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 msgr2=0x5635e98d6ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98d6ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d33fff640 1 -- 172.21.15.45:0/2375898743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 msgr2=0x5635e98dfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.504+0000 7f1d33fff640 1 -- 172.21.15.45:0/2375898743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635e97f4570 con 0x5635e98dffc0 2024-03-12T05:13:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d39613640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 0x7f1d240a40a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c005fd0 tx=0x7f1d2c0096a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d38e12640 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:13:27.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 0x5635e98e23b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d200cb290 tx=0x7f1d2009a240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a520bd157c45e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d1effd640 1 -- 172.21.15.45:0/2375898743 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200a1c10 con 0x5635e98dffc0 2024-03-12T05:13:27.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d1effd640 1 -- 172.21.15.45:0/2375898743 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1d2009a820 con 0x5635e98dffc0 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.505+0000 7f1d1effd640 1 -- 172.21.15.45:0/2375898743 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200a19d0 con 0x5635e98dffc0 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2375898743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 msgr2=0x5635e98e23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 0x5635e98e23b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d200cb290 tx=0x7f1d2009a240 comp rx=0 tx=0).stop 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2375898743 shutdown_connections 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d73e0 0x5635e98dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98dffc0 0x5635e98e23b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2375898743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98d6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.506+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2375898743 >> 172.21.15.45:0/2375898743 conn(0x5635e98d11a0 msgr2=0x5635e98d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.507+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2375898743 shutdown_connections 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.507+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2375898743 wait complete. 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.507+0000 7f1d3dc4a740 1 -- start start 2024-03-12T05:13:27.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.507+0000 7f1d3dc4a740 1 Processor -- start 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98dce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9509430 con 0x5635e98dffc0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e94c5e00 con 0x5635e98d6ac0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e97bafd0 con 0x5635e98d73e0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d3dc4a740 10 client.0 _ll_register_callbacks cb 0x5635e9810fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.508+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37652/0 (socket says 172.21.15.45:37652) 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 -- 172.21.15.45:0/1971502079 learned_addr learned my addr 172.21.15.45:0/1971502079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46886/0 (socket says 172.21.15.45:46886) 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 -- 172.21.15.45:0/1971502079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 msgr2=0x5635e98dd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 -- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 msgr2=0x5635e98dce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98dce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.509+0000 7f1d39613640 1 -- 172.21.15.45:0/1971502079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d341a6a10 con 0x5635e98dffc0 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d39613640 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c011970 tx=0x7f1d2c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd33f7b4e8feeca server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d13fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00ec60 con 0x5635e98dffc0 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635e95357e0 con 0x5635e98dffc0 2024-03-12T05:13:27.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f1d2c017070 con 0x5635e98dffc0 2024-03-12T05:13:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d2c00edc0 con 0x5635e98dffc0 2024-03-12T05:13:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d3dc4a740 10 client.16191 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:27.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.510+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5635e9834610 con 0x5635e98dffc0 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.513+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5635e95357e0 con 0x5635e98dffc0 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f1d2c0046a0 con 0x5635e98dffc0 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 10 client.16191 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 20 client.16191 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e98f0260 0x5635e98f2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 4 client.16191 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635e9572720 con 0x5635e98f0260 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d38e12640 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e98f0260 0x5635e98f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.514+0000 7f1d13fff640 10 client.16191 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.515+0000 7f1d38e12640 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e98f0260 0x5635e98f2680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31d2e61d52b9187 server_cookie=cabfcc2bc0eea788 in_seq=0 out_seq=0 2024-03-12T05:13:27.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d13fff640 1 -- 172.21.15.45:0/1971502079 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5635e9572720 con 0x5635e98f0260 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d13fff640 10 client.16191 handle_command_reply: tid=0 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 client.16191 shutdown 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 msgr2=0x5635e98dd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c011970 tx=0x7f1d2c004ab0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 shutdown_connections 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e98f0260 0x5635e98f2680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98dd8c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/1971502079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98dce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 >> 172.21.15.45:0/1971502079 conn(0x5635e98d11a0 msgr2=0x5635e98d2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.516+0000 7f1d13fff640 0 client.16191 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.517+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 shutdown_connections 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.517+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/1971502079 wait complete. 2024-03-12T05:13:27.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.517+0000 7f1d3dc4a740 20 client.16191 trim_cache size 0 max 16384 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 Processor -- start 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 -- start start 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98dffc0 0x5635e98dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9572720 con 0x5635e98d6ac0 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e95357e0 con 0x5635e98dffc0 2024-03-12T05:13:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.542+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e98d2420 con 0x5635e98d73e0 2024-03-12T05:13:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37668/0 (socket says 172.21.15.45:37668) 2024-03-12T05:13:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 -- 172.21.15.45:0/69789667 learned_addr learned my addr 172.21.15.45:0/69789667 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d39613640 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98dffc0 0x5635e98dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d33fff640 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 -- 172.21.15.45:0/69789667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 msgr2=0x5635e98dd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 -- 172.21.15.45:0/69789667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98dffc0 msgr2=0x5635e98dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98dffc0 0x5635e98dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 -- 172.21.15.45:0/69789667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635e97d0ac0 con 0x5635e98d6ac0 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.543+0000 7f1d38e12640 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005ab0 tx=0x7f1d28005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97463d612bb90966 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.544+0000 7f1d1e7fc640 1 -- 172.21.15.45:0/69789667 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28044040 con 0x5635e98d6ac0 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.544+0000 7f1d1e7fc640 1 -- 172.21.15.45:0/69789667 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1d28046040 con 0x5635e98d6ac0 2024-03-12T05:13:27.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.544+0000 7f1d1e7fc640 1 -- 172.21.15.45:0/69789667 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d280035f0 con 0x5635e98d6ac0 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/69789667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 msgr2=0x5635e98df570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1d28005ab0 tx=0x7f1d28005fd0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/69789667 shutdown_connections 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98dffc0 0x5635e98dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98dd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/69789667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98d6ac0 0x5635e98df570 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/69789667 >> 172.21.15.45:0/69789667 conn(0x5635e98d11a0 msgr2=0x5635e98db730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/69789667 shutdown_connections 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/69789667 wait complete. 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.545+0000 7f1d3dc4a740 1 -- start start 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.546+0000 7f1d3dc4a740 1 Processor -- start 2024-03-12T05:13:27.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.546+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98e5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9946770 con 0x5635e98dffc0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e99468b0 con 0x5635e98d6ac0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635e9946a70 con 0x5635e98d73e0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d3dc4a740 10 client.0 _ll_register_callbacks cb 0x7f1d200963c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d38e12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46898/0 (socket says 172.21.15.45:46898) 2024-03-12T05:13:27.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.547+0000 7f1d33fff640 1 -- 172.21.15.45:0/2140074605 learned_addr learned my addr 172.21.15.45:0/2140074605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 -- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 msgr2=0x5635e98e5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98e5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d39613640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 -- 172.21.15.45:0/2140074605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 msgr2=0x5635e98e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98e63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 -- 172.21.15.45:0/2140074605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3410e840 con 0x5635e98d73e0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d39613640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.548+0000 7f1d33fff640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d2009a520 tx=0x7f1d20010d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755dad195ca9d681 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.549+0000 7f1d13fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.549+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200a2f10 con 0x5635e98d73e0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.549+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f1d200a19d0 con 0x5635e98d73e0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.549+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d2009a820 con 0x5635e98d73e0 2024-03-12T05:13:27.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.549+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d200b32c0 con 0x5635e98d73e0 2024-03-12T05:13:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.550+0000 7f1d3dc4a740 10 client.25663 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.550+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5635e97f66a0 con 0x5635e98d73e0 2024-03-12T05:13:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1d200b7080 con 0x5635e98d73e0 2024-03-12T05:13:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f1d200a0e40 con 0x5635e98d73e0 2024-03-12T05:13:27.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 10 client.25663 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 10 client.25663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 20 client.25663 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e9947fd0 0x5635e994a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 4 client.25663 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5635e95c1890 con 0x5635e9947fd0 2024-03-12T05:13:27.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.551+0000 7f1d38e12640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e9947fd0 0x5635e994a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d13fff640 10 client.25663 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d38e12640 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e9947fd0 0x5635e994a3f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=214aa75b533896c0 server_cookie=b9a2f43c0c1818f3 in_seq=0 out_seq=0 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d13fff640 1 -- 172.21.15.45:0/2140074605 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5635e95c1890 con 0x5635e9947fd0 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d13fff640 10 client.25663 handle_command_reply: tid=0 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 client.25663 shutdown 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 msgr2=0x5635e98e5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d2009a520 tx=0x7f1d20010d40 comp rx=0 tx=0).stop 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 shutdown_connections 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5635e9947fd0 0x5635e994a3f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5635e98dffc0 0x5635e98e63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635e98d73e0 0x5635e98e5e70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 --2- 172.21.15.45:0/2140074605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5635e98d6ac0 0x5635e98e5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 >> 172.21.15.45:0/2140074605 conn(0x5635e98d11a0 msgr2=0x5635e98d2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.552+0000 7f1d13fff640 0 client.25663 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.553+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 shutdown_connections 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.553+0000 7f1d3dc4a740 1 -- 172.21.15.45:0/2140074605 wait complete. 2024-03-12T05:13:27.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.553+0000 7f1d3dc4a740 20 client.25663 trim_cache size 0 max 16384 2024-03-12T05:13:27.551 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:13:27.551 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:13:27.551 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:13:27.551 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 msgr2=0x7f1d240a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 0x7f1d240a40a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c005fd0 tx=0x7f1d2c0096a0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 msgr2=0x7f1d340739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 0x7f1d340739c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000e9c0 tx=0x7f1d2000ee80 comp rx=0 tx=0).stop 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 shutdown_connections 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1d240a1be0 0x7f1d240a40a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d34073f00 0x7f1d341a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d34106460 0x7f1d340739c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 --2- 172.21.15.45:0/281845845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d34101390 0x7f1d34073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 >> 172.21.15.45:0/281845845 conn(0x7f1d340fcc30 msgr2=0x7f1d340fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 shutdown_connections 2024-03-12T05:13:27.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.555+0000 7f1d3b076640 1 -- 172.21.15.45:0/281845845 wait complete. 2024-03-12T05:13:27.568 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:13:27.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 -- 172.21.15.45:0/2498611528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7020075fd0 msgr2=0x7f7020111a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 --2- 172.21.15.45:0/2498611528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7020075fd0 0x7f7020111a90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7010009930 tx=0x7f701002bc30 comp rx=0 tx=0).stop 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 -- 172.21.15.45:0/2498611528 shutdown_connections 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 --2- 172.21.15.45:0/2498611528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f70201143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 --2- 172.21.15.45:0/2498611528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7020075fd0 0x7f7020111a90 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7010009930 tx=0x7f701002bc30 comp rx=0 tx=0).stop 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 --2- 172.21.15.45:0/2498611528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f7020075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 -- 172.21.15.45:0/2498611528 >> 172.21.15.45:0/2498611528 conn(0x7f70200fede0 msgr2=0x7f7020101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 -- 172.21.15.45:0/2498611528 shutdown_connections 2024-03-12T05:13:27.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.739+0000 7f7027288640 1 -- 172.21.15.45:0/2498611528 wait complete. 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.740+0000 7f7027288640 1 Processor -- start 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.740+0000 7f7027288640 1 -- start start 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f702019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70201a2810 0x7f70201a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201a3200 con 0x7f7020111fd0 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201a3370 con 0x7f70200756b0 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7027288640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201a34e0 con 0x7f70201a2810 2024-03-12T05:13:27.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7024823640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7024823640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37702/0 (socket says 172.21.15.45:37702) 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7024823640 1 -- 172.21.15.45:0/2835933531 learned_addr learned my addr 172.21.15.45:0/2835933531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7025024640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f702019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.741+0000 7f7025825640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70201a2810 0x7f70201a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 -- 172.21.15.45:0/2835933531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70201a2810 msgr2=0x7f70201a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70201a2810 0x7f70201a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 -- 172.21.15.45:0/2835933531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 msgr2=0x7f702019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:27.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f702019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 -- 172.21.15.45:0/2835933531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010009560 con 0x7f7020111fd0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7025024640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f702019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.742+0000 7f7024823640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7010009230 tx=0x7f7010005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.743+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010004350 con 0x7f7020111fd0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.743+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f701002de90 con 0x7f7020111fd0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.743+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70100051e0 con 0x7f7020111fd0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.743+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70201aaac0 con 0x7f7020111fd0 2024-03-12T05:13:27.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.743+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70201aaef0 con 0x7f7020111fd0 2024-03-12T05:13:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.744+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020076df0 con 0x7f7020111fd0 2024-03-12T05:13:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.746+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7010003bc0 con 0x7f7020111fd0 2024-03-12T05:13:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.747+0000 7f70167fc640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 0x7f700c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:27.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.747+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f70100efdb0 con 0x7f7020111fd0 2024-03-12T05:13:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.747+0000 7f7025024640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 0x7f700c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:27.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.748+0000 7f7025024640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 0x7f700c0a40e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f701c009fc0 tx=0x7f701c005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:27.754+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f70100b8d90 con 0x7f7020111fd0 2024-03-12T05:13:27.795 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:27 smithi116 ceph-mon[35486]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:27 smithi045 ceph-mon[32174]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:27.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:27 smithi130 ceph-mon[35398]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-03-12T05:13:28.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.151+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f702019f2a0 con 0x7f7020111fd0 2024-03-12T05:13:28.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.152+0000 7f70167fc640 1 -- 172.21.15.45:0/2835933531 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f7010043090 con 0x7f7020111fd0 2024-03-12T05:13:28.149 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:13:28.149 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:23.410859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:13:28.149 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 30 2024-03-12T05:13:28.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.155+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 msgr2=0x7f700c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.155+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 0x7f700c0a40e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f701c009fc0 tx=0x7f701c005f90 comp rx=0 tx=0).stop 2024-03-12T05:13:28.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.156+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 msgr2=0x7f702019e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.156+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7010009230 tx=0x7f7010005dc0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.156+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 shutdown_connections 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.156+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f700c0a1c20 0x7f700c0a40e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.156+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7020111fd0 0x7f702019e490 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.157+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70201a2810 0x7f70201a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.157+0000 7f7027288640 1 --2- 172.21.15.45:0/2835933531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70200756b0 0x7f702019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.157+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 >> 172.21.15.45:0/2835933531 conn(0x7f70200fede0 msgr2=0x7f7020112750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.157+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 shutdown_connections 2024-03-12T05:13:28.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.157+0000 7f7027288640 1 -- 172.21.15.45:0/2835933531 wait complete. 2024-03-12T05:13:28.170 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:13:28.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:13:28.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.348+0000 7f86084ae640 1 -- 172.21.15.45:0/465011881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8600103f40 msgr2=0x7f8600107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.348+0000 7f86084ae640 1 --2- 172.21.15.45:0/465011881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8600103f40 0x7f8600107f80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009900 tx=0x7f85fc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:13:28.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 -- 172.21.15.45:0/465011881 shutdown_connections 2024-03-12T05:13:28.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 --2- 172.21.15.45:0/465011881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600108660 0x7f8600110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 --2- 172.21.15.45:0/465011881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8600103f40 0x7f8600107f80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 --2- 172.21.15.45:0/465011881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600103590 0x7f8600103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 -- 172.21.15.45:0/465011881 >> 172.21.15.45:0/465011881 conn(0x7f86000fedf0 msgr2=0x7f8600101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 -- 172.21.15.45:0/465011881 shutdown_connections 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.349+0000 7f86084ae640 1 -- 172.21.15.45:0/465011881 wait complete. 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.350+0000 7f86084ae640 1 Processor -- start 2024-03-12T05:13:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.350+0000 7f86084ae640 1 -- start start 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600108660 0x7f860010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 0x7f860010dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86001aa250 con 0x7f8600103590 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86001aa3c0 con 0x7f8600108660 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f86084ae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86001aa580 con 0x7f860010d600 2024-03-12T05:13:28.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f860624a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 0x7f860010dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f860624a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37710/0 (socket says 172.21.15.45:37710) 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f860624a640 1 -- 172.21.15.45:0/3460605970 learned_addr learned my addr 172.21.15.45:0/3460605970 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8605a49640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600108660 0x7f860010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 -- 172.21.15.45:0/3460605970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600108660 msgr2=0x7f860010d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600108660 0x7f860010d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 -- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 msgr2=0x7f8600110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f8606a4b640 1 -- 172.21.15.45:0/3460605970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc009560 con 0x7f860010d600 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.351+0000 7f860624a640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.352+0000 7f8606a4b640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 0x7f860010dab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009fd0 tx=0x7f85f800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.352+0000 7f85f77fe640 1 -- 172.21.15.45:0/3460605970 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800ce20 con 0x7f860010d600 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.352+0000 7f85f77fe640 1 -- 172.21.15.45:0/3460605970 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85f8005020 con 0x7f860010d600 2024-03-12T05:13:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.352+0000 7f85f77fe640 1 -- 172.21.15.45:0/3460605970 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8010690 con 0x7f860010d600 2024-03-12T05:13:28.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.352+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86001aa860 con 0x7f860010d600 2024-03-12T05:13:28.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.353+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86001aad40 con 0x7f860010d600 2024-03-12T05:13:28.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.356+0000 7f85f77fe640 1 -- 172.21.15.45:0/3460605970 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f85f8004b20 con 0x7f860010d600 2024-03-12T05:13:28.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.358+0000 7f860ad22740 1 Processor -- start 2024-03-12T05:13:28.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.358+0000 7f860ad22740 1 -- start start 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f85f77fe640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 0x7f85ec0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f860ad22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f860ad22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0b8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f8605a49640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 0x7f85ec0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f85f77fe640 1 -- 172.21.15.45:0/3460605970 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f85f8014070 con 0x7f860010d600 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f860624a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.359+0000 7f860624a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46944/0 (socket says 172.21.15.45:46944) 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860ad22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0b8750 0x55f61c0bab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860ad22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c06a960 con 0x55f61c0afb70 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860ad22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c01e310 con 0x55f61c0b8750 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860ad22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c01b840 con 0x55f61c0af250 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860624a640 1 -- 172.21.15.45:0/2907886636 learned_addr learned my addr 172.21.15.45:0/2907886636 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f8606a4b640 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0b8750 0x55f61c0bab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860624a640 1 -- 172.21.15.45:0/2907886636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0b8750 msgr2=0x55f61c0bab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860624a640 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0b8750 0x55f61c0bab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860624a640 1 -- 172.21.15.45:0/2907886636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 msgr2=0x55f61c0b8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.360+0000 7f860624a640 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f860624a640 1 -- 172.21.15.45:0/2907886636 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f61bfd9060 con 0x55f61c0af250 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f8605a49640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 0x7f85ec0a4230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f85fc02c240 tx=0x7f85fc02c700 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f860624a640 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85f0011650 tx=0x7f85f0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dba5cf39571f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f85f4bfe640 1 -- 172.21.15.45:0/2907886636 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f000f6c0 con 0x55f61c0af250 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f85f4bfe640 1 -- 172.21.15.45:0/2907886636 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85f000fcd0 con 0x55f61c0af250 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.361+0000 7f85f4bfe640 1 -- 172.21.15.45:0/2907886636 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0019cb0 con 0x55f61c0af250 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.362+0000 7f860ad22740 1 -- 172.21.15.45:0/2907886636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 msgr2=0x55f61c0af630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.362+0000 7f860ad22740 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85f0011650 tx=0x7f85f0009f90 comp rx=0 tx=0).stop 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.362+0000 7f860ad22740 1 -- 172.21.15.45:0/2907886636 shutdown_connections 2024-03-12T05:13:28.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0b8750 0x55f61c0bab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 --2- 172.21.15.45:0/2907886636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0af630 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 -- 172.21.15.45:0/2907886636 >> 172.21.15.45:0/2907886636 conn(0x55f61c0a9980 msgr2=0x55f61c0a9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 -- 172.21.15.45:0/2907886636 shutdown_connections 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 -- 172.21.15.45:0/2907886636 wait complete. 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.363+0000 7f860ad22740 1 -- start start 2024-03-12T05:13:28.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 Processor -- start 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 0x55f61c0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c017020 con 0x55f61c0af250 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61bcea430 con 0x55f61c0afb70 2024-03-12T05:13:28.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61bca6e00 con 0x55f61c0b8750 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.364+0000 7f860ad22740 10 client.0 _ll_register_callbacks cb 0x55f61bfbecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8605a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46950/0 (socket says 172.21.15.45:46950) 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 -- 172.21.15.45:0/2126439732 learned_addr learned my addr 172.21.15.45:0/2126439732 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f860624a640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 0x55f61c0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 -- 172.21.15.45:0/2126439732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 msgr2=0x55f61c0b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 -- 172.21.15.45:0/2126439732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 msgr2=0x55f61c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 0x55f61c0b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 -- 172.21.15.45:0/2126439732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f0009e00 con 0x55f61c0b8750 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f860624a640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 0x55f61c0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:13:28.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f8606a4b640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85f80cb620 tx=0x7f85f809b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70a0f04ae97d9274 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.365+0000 7f85e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f80a3240 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f85f8097110 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f61bd167e0 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85f80972d0 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f860ad22740 10 client.25681 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f61c0156a0 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85f80a9080 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.366+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f85f801f070 con 0x55f61c0b8750 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 10 client.25681 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 10 client.25681 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 20 client.25681 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c0c8b90 0x55f61c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 4 client.25681 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:28.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f61bd167e0 con 0x55f61c0c8b90 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.367+0000 7f860624a640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c0c8b90 0x55f61c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.368+0000 7f860624a640 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c0c8b90 0x55f61c0caff0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=19144d212356cb90 server_cookie=1e3c04de73d7a47b in_seq=0 out_seq=0 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.368+0000 7f85e9ffb640 10 client.25681 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.368+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2126439732 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f61bd167e0 con 0x55f61c0c8b90 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.368+0000 7f85e9ffb640 10 client.25681 handle_command_reply: tid=0 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.368+0000 7f860ad22740 1 client.25681 shutdown 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 msgr2=0x55f61c0b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85f80cb620 tx=0x7f85f809b260 comp rx=0 tx=0).stop 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 shutdown_connections 2024-03-12T05:13:28.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c0c8b90 0x55f61c0caff0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c0b6210 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 --2- 172.21.15.45:0/2126439732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0af250 0x55f61c0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 >> 172.21.15.45:0/2126439732 conn(0x55f61c0a9980 msgr2=0x55f61c0ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f85e9ffb640 0 client.25681 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 shutdown_connections 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 1 -- 172.21.15.45:0/2126439732 wait complete. 2024-03-12T05:13:28.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.369+0000 7f860ad22740 20 client.25681 trim_cache size 0 max 16384 2024-03-12T05:13:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.390+0000 7f860ad22740 1 Processor -- start 2024-03-12T05:13:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.390+0000 7f860ad22740 1 -- start start 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 0x55f61c0bf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61bf9c130 con 0x55f61c0b8750 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c0d0240 con 0x55f61c0afb70 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860ad22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c0c8940 con 0x55f61c0af250 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f8606a4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 0x55f61c0bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f8606a4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 0x55f61c0bf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37730/0 (socket says 172.21.15.45:37730) 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f8605a49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f8606a4b640 1 -- 172.21.15.45:0/1021298171 learned_addr learned my addr 172.21.15.45:0/1021298171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 -- 172.21.15.45:0/1021298171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 msgr2=0x55f61c0a8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0a8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 -- 172.21.15.45:0/1021298171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 msgr2=0x55f61c0bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 0x55f61c0bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.391+0000 7f860624a640 1 -- 172.21.15.45:0/1021298171 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f61bfb1c20 con 0x55f61c0af250 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.392+0000 7f860624a640 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0b6030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f85f0000c00 tx=0x7f85f0019930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dd4ff3dbe2ff92e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.392+0000 7f85ebfff640 1 -- 172.21.15.45:0/1021298171 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0029ea0 con 0x55f61c0af250 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.392+0000 7f85ebfff640 1 -- 172.21.15.45:0/1021298171 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85f0019410 con 0x55f61c0af250 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.392+0000 7f85ebfff640 1 -- 172.21.15.45:0/1021298171 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00050b0 con 0x55f61c0af250 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- 172.21.15.45:0/1021298171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 msgr2=0x55f61c0b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0b6030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f85f0000c00 tx=0x7f85f0019930 comp rx=0 tx=0).stop 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- 172.21.15.45:0/1021298171 shutdown_connections 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0b8750 0x55f61c0bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0afb70 0x55f61c0a8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 --2- 172.21.15.45:0/1021298171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0af250 0x55f61c0b6030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- 172.21.15.45:0/1021298171 >> 172.21.15.45:0/1021298171 conn(0x55f61c0a9980 msgr2=0x55f61c0a82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- 172.21.15.45:0/1021298171 shutdown_connections 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- 172.21.15.45:0/1021298171 wait complete. 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.393+0000 7f860ad22740 1 -- start start 2024-03-12T05:13:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 Processor -- start 2024-03-12T05:13:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0bef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c119240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c119780 con 0x55f61c0afb70 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c1198f0 con 0x55f61c0af250 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f61c119a60 con 0x55f61c0b8750 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860ad22740 10 client.0 _ll_register_callbacks cb 0x7f85f0009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39568/0 (socket says 172.21.15.45:39568) 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 -- 172.21.15.45:0/2671458059 learned_addr learned my addr 172.21.15.45:0/2671458059 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f8606a4b640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c119240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f8605a49640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0bef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 -- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 msgr2=0x55f61c119240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c119240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.394+0000 7f860624a640 1 -- 172.21.15.45:0/2671458059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 msgr2=0x55f61c0bef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f860624a640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0bef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f860624a640 1 -- 172.21.15.45:0/2671458059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f61bfd9060 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f8605a49640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0bef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f860624a640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85f0010010 tx=0x7f85f00198c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83506388182daf8d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f85e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0029ea0 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f85f000f6c0 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.395+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85f000fcd0 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85f0019580 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 10 client.25670 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f61bfd78c0 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85f000fcd0 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f85f0039070 con 0x55f61c0af250 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 10 client.25670 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 10 client.25670 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:13:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 20 client.25670 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c120de0 0x55f61c123200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 4 client.25670 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.396+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f61bd53720 con 0x55f61c120de0 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f8605a49640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c120de0 0x55f61c123200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f85e9ffb640 10 client.25670 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f8605a49640 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c120de0 0x55f61c123200 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d6540099709a2535 server_cookie=66675b4dfd9affd5 in_seq=0 out_seq=0 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f85e9ffb640 1 -- 172.21.15.45:0/2671458059 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f61bd53720 con 0x55f61c120de0 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f85e9ffb640 10 client.25670 handle_command_reply: tid=0 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.397+0000 7f860ad22740 1 client.25670 shutdown 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 msgr2=0x55f61c0be9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85f0010010 tx=0x7f85f00198c0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 shutdown_connections 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f61c120de0 0x55f61c123200 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f61c0afb70 0x55f61c0bef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f61c0b8750 0x55f61c119240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 --2- 172.21.15.45:0/2671458059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f61c0af250 0x55f61c0be9d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 >> 172.21.15.45:0/2671458059 conn(0x55f61c0a9980 msgr2=0x55f61c0a8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f85e9ffb640 0 client.25670 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 shutdown_connections 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 1 -- 172.21.15.45:0/2671458059 wait complete. 2024-03-12T05:13:28.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.398+0000 7f860ad22740 20 client.25670 trim_cache size 0 max 16384 2024-03-12T05:13:28.399 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:13:28.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 msgr2=0x7f85ec0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 0x7f85ec0a4230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f85fc02c240 tx=0x7f85fc02c700 comp rx=0 tx=0).stop 2024-03-12T05:13:28.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 msgr2=0x7f860010dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 0x7f860010dab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009fd0 tx=0x7f85f800ef10 comp rx=0 tx=0).stop 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 shutdown_connections 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85ec0a1d70 0x7f85ec0a4230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f860010d600 0x7f860010dab0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600108660 0x7f860010d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 --2- 172.21.15.45:0/3460605970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8600103590 0x7f8600110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 >> 172.21.15.45:0/3460605970 conn(0x7f86000fedf0 msgr2=0x7f8600100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 shutdown_connections 2024-03-12T05:13:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.399+0000 7f86084ae640 1 -- 172.21.15.45:0/3460605970 wait complete. 2024-03-12T05:13:28.413 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:13:28.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 -- 172.21.15.45:0/511326304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 msgr2=0x7fa86c10df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 --2- 172.21.15.45:0/511326304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c10df30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa864009cd0 tx=0x7fa86402c0e0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 -- 172.21.15.45:0/511326304 shutdown_connections 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 --2- 172.21.15.45:0/511326304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c10df30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 --2- 172.21.15.45:0/511326304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c101c40 0x7fa86c105c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 --2- 172.21.15.45:0/511326304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 -- 172.21.15.45:0/511326304 >> 172.21.15.45:0/511326304 conn(0x7fa86c0fcb50 msgr2=0x7fa86c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 -- 172.21.15.45:0/511326304 shutdown_connections 2024-03-12T05:13:28.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.591+0000 7fa8714df640 1 -- 172.21.15.45:0/511326304 wait complete. 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.592+0000 7fa8714df640 1 Processor -- start 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.592+0000 7fa8714df640 1 -- start start 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c1a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c1a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 0x7fa86c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c19e400 con 0x7fa86c106360 2024-03-12T05:13:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c19e570 con 0x7fa86c19da10 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa8714df640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c19e730 con 0x7fa86c101290 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c1a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c1a3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46976/0 (socket says 172.21.15.45:46976) 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 0x7fa86c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86affd640 1 -- 172.21.15.45:0/3443167642 learned_addr learned my addr 172.21.15.45:0/3443167642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c1a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.593+0000 7fa86b7fe640 1 -- 172.21.15.45:0/3443167642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 msgr2=0x7fa86c1a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c1a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa86b7fe640 1 -- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 msgr2=0x7fa86c1a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c1a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa86b7fe640 1 -- 172.21.15.45:0/3443167642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86000f790 con 0x7fa86c19da10 2024-03-12T05:13:28.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 0x7fa86c19dec0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa864006010 tx=0x7fa864003ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa853fff640 1 -- 172.21.15.45:0/3443167642 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864032d50 con 0x7fa86c19da10 2024-03-12T05:13:28.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa853fff640 1 -- 172.21.15.45:0/3443167642 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa864032eb0 con 0x7fa86c19da10 2024-03-12T05:13:28.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa853fff640 1 -- 172.21.15.45:0/3443167642 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640349d0 con 0x7fa86c19da10 2024-03-12T05:13:28.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.594+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa864009900 con 0x7fa86c19da10 2024-03-12T05:13:28.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.595+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1aa910 con 0x7fa86c19da10 2024-03-12T05:13:28.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.598+0000 7fa853fff640 1 -- 172.21.15.45:0/3443167642 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa86403b040 con 0x7fa86c19da10 2024-03-12T05:13:28.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.598+0000 7fa87418d740 1 Processor -- start 2024-03-12T05:13:28.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 -- start start 2024-03-12T05:13:28.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556300399960 con 0x5563003de110 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630034d310 con 0x5563003f8970 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa87418d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630034a840 con 0x5563003dea30 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa86a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa86a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37768/0 (socket says 172.21.15.45:37768) 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa86a7fc640 1 -- 172.21.15.45:0/1522157809 learned_addr learned my addr 172.21.15.45:0/1522157809 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa86affd640 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.599+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 -- 172.21.15.45:0/1522157809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 msgr2=0x5563003f8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 -- 172.21.15.45:0/1522157809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 msgr2=0x5563003fad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 -- 172.21.15.45:0/1522157809 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556300308060 con 0x5563003de110 2024-03-12T05:13:28.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa853fff640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 0x7fa8540a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa860009960 tx=0x7fa860005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9734ad43ad85d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa86affd640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 0x7fa8540a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa8517fa640 1 -- 172.21.15.45:0/1522157809 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860004230 con 0x5563003de110 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa8517fa640 1 -- 172.21.15.45:0/1522157809 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa860009e20 con 0x5563003de110 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.600+0000 7fa8517fa640 1 -- 172.21.15.45:0/1522157809 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860003920 con 0x5563003de110 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa853fff640 1 -- 172.21.15.45:0/3443167642 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa8640ef590 con 0x7fa86c19da10 2024-03-12T05:13:28.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 -- 172.21.15.45:0/1522157809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 msgr2=0x5563003de4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa86affd640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 0x7fa8540a4230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa86c100290 tx=0x7fa85c00b4a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa860009960 tx=0x7fa860005f90 comp rx=0 tx=0).stop 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 -- 172.21.15.45:0/1522157809 shutdown_connections 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 --2- 172.21.15.45:0/1522157809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003de4f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.601+0000 7fa87418d740 1 -- 172.21.15.45:0/1522157809 >> 172.21.15.45:0/1522157809 conn(0x5563003d8b90 msgr2=0x5563003d8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.602+0000 7fa87418d740 1 -- 172.21.15.45:0/1522157809 shutdown_connections 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.602+0000 7fa87418d740 1 -- 172.21.15.45:0/1522157809 wait complete. 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.602+0000 7fa87418d740 1 -- start start 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.602+0000 7fa87418d740 1 Processor -- start 2024-03-12T05:13:28.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003dea30 0x5563003f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003f8970 0x5563003f6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556300346020 con 0x5563003f8970 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556300019430 con 0x5563003dea30 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fffd5e00 con 0x5563003de110 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa87418d740 10 client.0 _ll_register_callbacks cb 0x5563002edcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47000/0 (socket says 172.21.15.45:47000) 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 -- 172.21.15.45:0/932684106 learned_addr learned my addr 172.21.15.45:0/932684106 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003f8970 0x5563003f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003dea30 0x5563003f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 -- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003dea30 msgr2=0x5563003f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003dea30 0x5563003f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 -- 172.21.15.45:0/932684106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003f8970 msgr2=0x5563003f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003f8970 0x5563003f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.603+0000 7fa86affd640 1 -- 172.21.15.45:0/932684106 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa860009640 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.604+0000 7fa86affd640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa85c003e60 tx=0x7fa85c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b1c1041326e94e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.604+0000 7fa84eff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.604+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c016070 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563000457e0 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa85c0043a0 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa85c030410 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa87418d740 10 client.25699 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5563003446a0 con 0x5563003de110 2024-03-12T05:13:28.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.605+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa85c034020 con 0x5563003de110 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fa85c0116c0 con 0x5563003de110 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 10 client.25699 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 10 client.25699 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 20 client.25699 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300408e60 0x55630040b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 4 client.25699 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556300082720 con 0x556300408e60 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.606+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300408e60 0x55630040b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.607+0000 7fa84eff5640 10 client.25699 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.607+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300408e60 0x55630040b280 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a29ffffee244c14 server_cookie=ad02b8ab674be2bb in_seq=0 out_seq=0 2024-03-12T05:13:28.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa84eff5640 1 -- 172.21.15.45:0/932684106 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556300082720 con 0x556300408e60 2024-03-12T05:13:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa84eff5640 10 client.25699 handle_command_reply: tid=0 2024-03-12T05:13:28.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 client.25699 shutdown 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 msgr2=0x5563003f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa85c003e60 tx=0x7fa85c004730 comp rx=0 tx=0).stop 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 shutdown_connections 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300408e60 0x55630040b280 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003f8970 0x5563003f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003dea30 0x5563003f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 --2- 172.21.15.45:0/932684106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003de110 0x5563003f58a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 >> 172.21.15.45:0/932684106 conn(0x5563003d8b90 msgr2=0x5563003da8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa84eff5640 0 client.25699 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:13:28.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 shutdown_connections 2024-03-12T05:13:28.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 1 -- 172.21.15.45:0/932684106 wait complete. 2024-03-12T05:13:28.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.608+0000 7fa87418d740 20 client.25699 trim_cache size 0 max 16384 2024-03-12T05:13:28.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.635+0000 7fa87418d740 1 Processor -- start 2024-03-12T05:13:28.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.635+0000 7fa87418d740 1 -- start start 2024-03-12T05:13:28.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003d7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563002cb130 con 0x5563003de110 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563003daee0 con 0x5563003f8970 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa87418d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563003d4540 con 0x5563003dea30 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003d7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47008/0 (socket says 172.21.15.45:47008) 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003d7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39616/0 (socket says 172.21.15.45:39616) 2024-03-12T05:13:28.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 -- 172.21.15.45:0/3574940084 learned_addr learned my addr 172.21.15.45:0/3574940084 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 -- 172.21.15.45:0/3574940084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 msgr2=0x5563003d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003d7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 -- 172.21.15.45:0/3574940084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 msgr2=0x5563003f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86a7fc640 1 -- 172.21.15.45:0/3574940084 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563002e0c20 con 0x5563003dea30 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.636+0000 7fa86affd640 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.637+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa860005860 tx=0x7fa860021040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=450d28b931a5bde7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.637+0000 7fa850ff9640 1 -- 172.21.15.45:0/3574940084 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860024070 con 0x5563003dea30 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.637+0000 7fa850ff9640 1 -- 172.21.15.45:0/3574940084 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa860025040 con 0x5563003dea30 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.637+0000 7fa850ff9640 1 -- 172.21.15.45:0/3574940084 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600035f0 con 0x5563003dea30 2024-03-12T05:13:28.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 -- 172.21.15.45:0/3574940084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 msgr2=0x5563003f6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa860005860 tx=0x7fa860021040 comp rx=0 tx=0).stop 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 -- 172.21.15.45:0/3574940084 shutdown_connections 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003d7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003f6140 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 --2- 172.21.15.45:0/3574940084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 -- 172.21.15.45:0/3574940084 >> 172.21.15.45:0/3574940084 conn(0x5563003d8b90 msgr2=0x5563003d7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 -- 172.21.15.45:0/3574940084 shutdown_connections 2024-03-12T05:13:28.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.638+0000 7fa87418d740 1 -- 172.21.15.45:0/3574940084 wait complete. 2024-03-12T05:13:28.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.639+0000 7fa87418d740 1 -- start start 2024-03-12T05:13:28.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.639+0000 7fa87418d740 1 Processor -- start 2024-03-12T05:13:28.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003ff920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563004001f0 con 0x5563003de110 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563003fe790 con 0x5563003f8970 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563003fe950 con 0x5563003dea30 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003ff920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37788/0 (socket says 172.21.15.45:37788) 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86affd640 1 -- 172.21.15.45:0/1571836025 learned_addr learned my addr 172.21.15.45:0/1571836025 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa87418d740 10 client.0 _ll_register_callbacks cb 0x7fa86000faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86a7fc640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 -- 172.21.15.45:0/1571836025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 msgr2=0x5563003fdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003fdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 -- 172.21.15.45:0/1571836025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 msgr2=0x5563003ff920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003ff920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.640+0000 7fa86b7fe640 1 -- 172.21.15.45:0/1571836025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563003446a0 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa86b7fe640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fe250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa8640ecd60 tx=0x7fa8640bbf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0c779074c9ae741 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa84eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640c4780 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa8640c03c0 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa860009e20 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.641+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8640ed110 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 10 client.25688 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556300308060 con 0x5563003f8970 2024-03-12T05:13:28.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8640c8020 con 0x5563003f8970 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fa8640bb770 con 0x5563003f8970 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 10 client.25688 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 10 client.25688 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 20 client.25688 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300460430 0x556300462850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 4 client.25688 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5563000d1890 con 0x556300460430 2024-03-12T05:13:28.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.642+0000 7fa86affd640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300460430 0x556300462850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.643+0000 7fa84eff5640 10 client.25688 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.643+0000 7fa86affd640 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300460430 0x556300462850 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ec64bb74610e91cb server_cookie=d39344c684c6de64 in_seq=0 out_seq=0 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.643+0000 7fa84eff5640 1 -- 172.21.15.45:0/1571836025 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5563000d1890 con 0x556300460430 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.643+0000 7fa84eff5640 10 client.25688 handle_command_reply: tid=0 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 client.25688 shutdown 2024-03-12T05:13:28.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 msgr2=0x5563003fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fe250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa8640ecd60 tx=0x7fa8640bbf20 comp rx=0 tx=0).stop 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 shutdown_connections 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x556300460430 0x556300462850 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563003f8970 0x5563003fe250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa84eff5640 0 client.25688 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563003dea30 0x5563003fdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 --2- 172.21.15.45:0/1571836025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563003de110 0x5563003ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 >> 172.21.15.45:0/1571836025 conn(0x5563003d8b90 msgr2=0x5563003f6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 shutdown_connections 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 1 -- 172.21.15.45:0/1571836025 wait complete. 2024-03-12T05:13:28.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.644+0000 7fa87418d740 20 client.25688 trim_cache size 0 max 16384 2024-03-12T05:13:28.646 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.645+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 msgr2=0x7fa8540a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.645+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 0x7fa8540a4230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa86c100290 tx=0x7fa85c00b4a0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.645+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 msgr2=0x7fa86c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.645+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 0x7fa86c19dec0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa864006010 tx=0x7fa864003ee0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.645+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 shutdown_connections 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8540a1d70 0x7fa8540a4230 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa86c106360 0x7fa86c1a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c19da10 0x7fa86c19dec0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 --2- 172.21.15.45:0/3443167642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c101290 0x7fa86c1a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 >> 172.21.15.45:0/3443167642 conn(0x7fa86c0fcb50 msgr2=0x7fa86c0feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 shutdown_connections 2024-03-12T05:13:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.646+0000 7fa8714df640 1 -- 172.21.15.45:0/3443167642 wait complete. 2024-03-12T05:13:28.659 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:13:28.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 -- 172.21.15.45:0/2090486346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280107fc0 msgr2=0x7f6280108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 --2- 172.21.15.45:0/2090486346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280107fc0 0x7f6280108440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6270009930 tx=0x7f627002bcd0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 -- 172.21.15.45:0/2090486346 shutdown_connections 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 --2- 172.21.15.45:0/2090486346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6280108980 0x7f62801145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 --2- 172.21.15.45:0/2090486346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280107fc0 0x7f6280108440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 --2- 172.21.15.45:0/2090486346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d770 0x7f628010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 -- 172.21.15.45:0/2090486346 >> 172.21.15.45:0/2090486346 conn(0x7f6280076d30 msgr2=0x7f6280079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 -- 172.21.15.45:0/2090486346 shutdown_connections 2024-03-12T05:13:28.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.833+0000 7f6287c40640 1 -- 172.21.15.45:0/2090486346 wait complete. 2024-03-12T05:13:28.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 Processor -- start 2024-03-12T05:13:28.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 -- start start 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280108980 0x7f628019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d770 0x7f62801a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801a2be0 con 0x7f6280107fc0 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801a2d20 con 0x7f628010d770 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.834+0000 7f6287c40640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801a2ee0 con 0x7f6280108980 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37806/0 (socket says 172.21.15.45:37806) 2024-03-12T05:13:28.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 -- 172.21.15.45:0/4182307750 learned_addr learned my addr 172.21.15.45:0/4182307750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62851db640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280108980 0x7f628019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62861dd640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d770 0x7f62801a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 -- 172.21.15.45:0/4182307750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280108980 msgr2=0x7f628019e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280108980 0x7f628019e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 -- 172.21.15.45:0/4182307750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d770 msgr2=0x7f62801a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d770 0x7f62801a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.835+0000 7f62859dc640 1 -- 172.21.15.45:0/4182307750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6270009560 con 0x7f6280107fc0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.836+0000 7f62859dc640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d950 tx=0x7f627c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.837+0000 7f6276ffd640 1 -- 172.21.15.45:0/4182307750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c009c30 con 0x7f6280107fc0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.837+0000 7f6276ffd640 1 -- 172.21.15.45:0/4182307750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f627c009d90 con 0x7f6280107fc0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.837+0000 7f6276ffd640 1 -- 172.21.15.45:0/4182307750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00f740 con 0x7f6280107fc0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.837+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62801a31c0 con 0x7f6280107fc0 2024-03-12T05:13:28.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.837+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628007bbd0 con 0x7f6280107fc0 2024-03-12T05:13:28.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.840+0000 7f6276ffd640 1 -- 172.21.15.45:0/4182307750 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f627c00f8a0 con 0x7f6280107fc0 2024-03-12T05:13:28.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.840+0000 7f628a80d740 1 Processor -- start 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 -- start start 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 0x5590eb2c4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb276960 con 0x5590eb2c4750 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb22a310 con 0x5590eb2bbb70 2024-03-12T05:13:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f628a80d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb227840 con 0x5590eb2bb250 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f62861dd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f62851db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f62861dd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47050/0 (socket says 172.21.15.45:47050) 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f62861dd640 1 -- 172.21.15.45:0/2350644652 learned_addr learned my addr 172.21.15.45:0/2350644652 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.841+0000 7f62851db640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37808/0 (socket says 172.21.15.45:37808) 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f6276ffd640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 0x7f626c0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62851db640 1 -- 172.21.15.45:0/2350644652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 msgr2=0x5590eb2bb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62851db640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62851db640 1 -- 172.21.15.45:0/2350644652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 msgr2=0x5590eb2c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62859dc640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 0x5590eb2c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62851db640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 0x5590eb2c4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62851db640 1 -- 172.21.15.45:0/2350644652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590eb1e5060 con 0x5590eb2c4750 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62861dd640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f62859dc640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 0x5590eb2c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.842+0000 7f6276ffd640 1 -- 172.21.15.45:0/4182307750 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f627c0cd280 con 0x7f6280107fc0 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.843+0000 7f62851db640 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6270036040 tx=0x7f62700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3568c5da100a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.843+0000 7f62861dd640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 0x7f626c0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.843+0000 7f626bfff640 1 -- 172.21.15.45:0/2350644652 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270004970 con 0x5590eb2c4750 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.843+0000 7f626bfff640 1 -- 172.21.15.45:0/2350644652 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6270004ad0 con 0x5590eb2c4750 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.843+0000 7f626bfff640 1 -- 172.21.15.45:0/2350644652 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270003850 con 0x5590eb2c4750 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.844+0000 7f628a80d740 1 -- 172.21.15.45:0/2350644652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 msgr2=0x5590eb2c6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.844+0000 7f628a80d740 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6270036040 tx=0x7f62700046c0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.844+0000 7f62861dd640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 0x7f626c0a41b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6278003e10 tx=0x7f6278004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 -- 172.21.15.45:0/2350644652 shutdown_connections 2024-03-12T05:13:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2c6b40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bbb70 0x5590eb2c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 --2- 172.21.15.45:0/2350644652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2bb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 -- 172.21.15.45:0/2350644652 >> 172.21.15.45:0/2350644652 conn(0x5590eb2b5980 msgr2=0x5590eb2b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 -- 172.21.15.45:0/2350644652 shutdown_connections 2024-03-12T05:13:28.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.845+0000 7f628a80d740 1 -- 172.21.15.45:0/2350644652 wait complete. 2024-03-12T05:13:28.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.846+0000 7f628a80d740 1 -- start start 2024-03-12T05:13:28.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.846+0000 7f628a80d740 1 Processor -- start 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2c2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb223020 con 0x5590eb2bbb70 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eaef6430 con 0x5590eb2c4750 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eaeb2e00 con 0x5590eb2bb250 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f628a80d740 10 client.0 _ll_register_callbacks cb 0x5590eb1cacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62859dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62861dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37810/0 (socket says 172.21.15.45:37810) 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 -- 172.21.15.45:0/2901796955 learned_addr learned my addr 172.21.15.45:0/2901796955 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 -- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 msgr2=0x5590eb2c17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2c17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 -- 172.21.15.45:0/2901796955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 msgr2=0x5590eb2c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.847+0000 7f62851db640 1 -- 172.21.15.45:0/2901796955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6280114ec0 con 0x5590eb2bbb70 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f62851db640 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f627002c2e0 tx=0x7f627002ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8aa9f1090f8b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f62697fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270004970 con 0x5590eb2bbb70 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590eaf5f720 con 0x5590eb2bbb70 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6270004fc0 con 0x5590eb2bbb70 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6270004520 con 0x5590eb2bbb70 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f628a80d740 10 client.16245 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.848+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5590eb2216a0 con 0x5590eb2bbb70 2024-03-12T05:13:28.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6270005630 con 0x5590eb2bbb70 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f627004e030 con 0x5590eb2bbb70 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f628a80d740 10 client.16245 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f628a80d740 10 client.16245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f628a80d740 20 client.16245 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb2d4850 0x5590eb2d6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.849+0000 7f628a80d740 4 client.16245 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.850+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590eaf5f720 con 0x5590eb2d4850 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.850+0000 7f62859dc640 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb2d4850 0x5590eb2d6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.850+0000 7f62859dc640 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb2d4850 0x5590eb2d6c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a0462c463cfc8b3 server_cookie=dee01d7d9be37587 in_seq=0 out_seq=0 2024-03-12T05:13:28.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.850+0000 7f62697fa640 10 client.16245 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:13:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.851+0000 7f62697fa640 1 -- 172.21.15.45:0/2901796955 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5590eaf5f720 con 0x5590eb2d4850 2024-03-12T05:13:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.851+0000 7f62697fa640 10 client.16245 handle_command_reply: tid=0 2024-03-12T05:13:28.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.851+0000 7f628a80d740 1 client.16245 shutdown 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 msgr2=0x5590eb2c1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f627002c2e0 tx=0x7f627002ca40 comp rx=0 tx=0).stop 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 shutdown_connections 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb2d4850 0x5590eb2d6c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bbb70 0x5590eb2c1ce0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 --2- 172.21.15.45:0/2901796955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bb250 0x5590eb2c17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 >> 172.21.15.45:0/2901796955 conn(0x5590eb2b5980 msgr2=0x5590eb2b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f62697fa640 0 client.16245 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 shutdown_connections 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.852+0000 7f628a80d740 1 -- 172.21.15.45:0/2901796955 wait complete. 2024-03-12T05:13:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.853+0000 7f628a80d740 20 client.16245 trim_cache size 0 max 16384 2024-03-12T05:13:28.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:28 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2835933531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:13:28.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 Processor -- start 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 -- start start 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2c2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb1a8130 con 0x5590eb2c4750 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb2b57f0 con 0x5590eb2bb250 2024-03-12T05:13:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f628a80d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb2c2c30 con 0x5590eb2bbb70 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.874+0000 7f62861dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62851db640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39664/0 (socket says 172.21.15.45:39664) 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62851db640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2c2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47068/0 (socket says 172.21.15.45:47068) 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 -- 172.21.15.45:0/2976315943 learned_addr learned my addr 172.21.15.45:0/2976315943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62861dd640 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37816/0 (socket says 172.21.15.45:37816) 2024-03-12T05:13:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 -- 172.21.15.45:0/2976315943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 msgr2=0x5590eb2c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 -- 172.21.15.45:0/2976315943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 msgr2=0x5590eb2bea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 -- 172.21.15.45:0/2976315943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590eb1bdc20 con 0x5590eb2bb250 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62861dd640 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f62859dc640 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f627c009610 tx=0x7f627c0cf690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3128b6420b092acd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.875+0000 7f626b7fe640 1 -- 172.21.15.45:0/2976315943 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0adea0 con 0x5590eb2bb250 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f626b7fe640 1 -- 172.21.15.45:0/2976315943 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f627c0a25f0 con 0x5590eb2bb250 2024-03-12T05:13:28.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f626b7fe640 1 -- 172.21.15.45:0/2976315943 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0cd040 con 0x5590eb2bb250 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 -- 172.21.15.45:0/2976315943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 msgr2=0x5590eb2c1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f627c009610 tx=0x7f627c0cf690 comp rx=0 tx=0).stop 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 -- 172.21.15.45:0/2976315943 shutdown_connections 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2c4750 0x5590eb2bea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 --2- 172.21.15.45:0/2976315943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2bb250 0x5590eb2c1cf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.876+0000 7f628a80d740 1 -- 172.21.15.45:0/2976315943 >> 172.21.15.45:0/2976315943 conn(0x5590eb2b5980 msgr2=0x5590eb2db5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.877+0000 7f628a80d740 1 -- 172.21.15.45:0/2976315943 shutdown_connections 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.877+0000 7f628a80d740 1 -- 172.21.15.45:0/2976315943 wait complete. 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.877+0000 7f628a80d740 1 -- start start 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.877+0000 7f628a80d740 1 Processor -- start 2024-03-12T05:13:28.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2cab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62859dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62859dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37824/0 (socket says 172.21.15.45:37824) 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62859dc640 1 -- 172.21.15.45:0/1713505519 learned_addr learned my addr 172.21.15.45:0/1713505519 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb32ad80 con 0x5590eb2bb250 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb32aec0 con 0x5590eb2c4750 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590eb32b000 con 0x5590eb2bbb70 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f628a80d740 10 client.0 _ll_register_callbacks cb 0x7f6270009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62851db640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62861dd640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.878+0000 7f62859dc640 1 -- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 msgr2=0x5590eb2ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62859dc640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62859dc640 1 -- 172.21.15.45:0/1713505519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 msgr2=0x5590eb2cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62859dc640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62859dc640 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6278009a50 con 0x5590eb2bb250 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62851db640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62859dc640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f627c0a20c0 tx=0x7f627c09e120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b758fc2008c00a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62697fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0adea0 con 0x5590eb2bb250 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590eaf227e0 con 0x5590eb2bb250 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f627c010aa0 con 0x5590eb2bb250 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627c0cd040 con 0x5590eb2bb250 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f628a80d740 10 client.16251 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:28.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.879+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f628007bbd0 con 0x5590eb2bb250 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5590eaf227e0 con 0x5590eb2bb250 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f627c0b2b80 con 0x5590eb2bb250 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f628a80d740 10 client.16251 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f628a80d740 10 client.16251 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f628a80d740 20 client.16251 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb32c3b0 0x5590eb32e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.880+0000 7f628a80d740 4 client.16251 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.881+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5590eafae890 con 0x5590eb32c3b0 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.881+0000 7f62851db640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb32c3b0 0x5590eb32e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.881+0000 7f62697fa640 10 client.16251 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:13:28.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.881+0000 7f62851db640 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb32c3b0 0x5590eb32e7d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8fdc980e7795becd server_cookie=60d9ef8c8495cfb4 in_seq=0 out_seq=0 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.882+0000 7f62697fa640 1 -- 172.21.15.45:0/1713505519 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5590eafae890 con 0x5590eb32c3b0 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.882+0000 7f62697fa640 10 client.16251 handle_command_reply: tid=0 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.882+0000 7f628a80d740 1 client.16251 shutdown 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.882+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 msgr2=0x5590eb2ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.882+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f627c0a20c0 tx=0x7f627c09e120 comp rx=0 tx=0).stop 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 shutdown_connections 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5590eb32c3b0 0x5590eb32e7d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f62697fa640 0 client.16251 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590eb2c4750 0x5590eb2cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5590eb2bbb70 0x5590eb2ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 --2- 172.21.15.45:0/1713505519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5590eb2bb250 0x5590eb2ca0b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 >> 172.21.15.45:0/1713505519 conn(0x5590eb2b5980 msgr2=0x5590eb2b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.883+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 shutdown_connections 2024-03-12T05:13:28.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.884+0000 7f628a80d740 1 -- 172.21.15.45:0/1713505519 wait complete. 2024-03-12T05:13:28.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.884+0000 7f628a80d740 20 client.16251 trim_cache size 0 max 16384 2024-03-12T05:13:28.883 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 msgr2=0x7f626c0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 0x7f626c0a41b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6278003e10 tx=0x7f6278004a40 comp rx=0 tx=0).stop 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 msgr2=0x7f628019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d950 tx=0x7f627c00de10 comp rx=0 tx=0).stop 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 shutdown_connections 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1cf0 0x7f626c0a41b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d770 0x7f62801a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6280108980 0x7f628019e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 --2- 172.21.15.45:0/4182307750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6280107fc0 0x7f628019dd20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:28.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 >> 172.21.15.45:0/4182307750 conn(0x7f6280076d30 msgr2=0x7f62800797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:28.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 shutdown_connections 2024-03-12T05:13:28.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:28.885+0000 7f6287c40640 1 -- 172.21.15.45:0/4182307750 wait complete. 2024-03-12T05:13:28.899 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:13:28.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:13:28.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:28 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2835933531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:13:28.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:28 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2835933531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.073+0000 7fc1cc793640 1 -- 172.21.15.45:0/2330331795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 msgr2=0x7fc1c4068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.073+0000 7fc1cc793640 1 --2- 172.21.15.45:0/2330331795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c4068d20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009930 tx=0x7fc1b402bc30 comp rx=0 tx=0).stop 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 -- 172.21.15.45:0/2330331795 shutdown_connections 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 --2- 172.21.15.45:0/2330331795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c4103c40 0x7fc1c41040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 --2- 172.21.15.45:0/2330331795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c4068d20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 --2- 172.21.15.45:0/2330331795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c41060d0 0x7fc1c4068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 -- 172.21.15.45:0/2330331795 >> 172.21.15.45:0/2330331795 conn(0x7fc1c4077180 msgr2=0x7fc1c4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 -- 172.21.15.45:0/2330331795 shutdown_connections 2024-03-12T05:13:29.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.074+0000 7fc1cc793640 1 -- 172.21.15.45:0/2330331795 wait complete. 2024-03-12T05:13:29.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.075+0000 7fc1cc793640 1 Processor -- start 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.075+0000 7fc1cc793640 1 -- start start 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c41060d0 0x7fc1c419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c419e090 con 0x7fc1c4103c40 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c419e200 con 0x7fc1c40688c0 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cc793640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c419e3c0 con 0x7fc1c41060d0 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39684/0 (socket says 172.21.15.45:39684) 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37836/0 (socket says 172.21.15.45:37836) 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1ca52f640 1 -- 172.21.15.45:0/466326357 learned_addr learned my addr 172.21.15.45:0/466326357 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1cad30640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c41060d0 0x7fc1c419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 -- 172.21.15.45:0/466326357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c41060d0 msgr2=0x7fc1c419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c41060d0 0x7fc1c419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 -- 172.21.15.45:0/466326357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 msgr2=0x7fc1c41a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1c9d2e640 1 -- 172.21.15.45:0/466326357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4009560 con 0x7fc1c4103c40 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.076+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4036040 tx=0x7fc1b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1bb7fe640 1 -- 172.21.15.45:0/466326357 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4004500 con 0x7fc1c4103c40 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1bb7fe640 1 -- 172.21.15.45:0/466326357 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1b4003aa0 con 0x7fc1c4103c40 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1bb7fe640 1 -- 172.21.15.45:0/466326357 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4005470 con 0x7fc1c4103c40 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c419e640 con 0x7fc1c4103c40 2024-03-12T05:13:29.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.077+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c41aa930 con 0x7fc1c4103c40 2024-03-12T05:13:29.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.080+0000 7fc1cf36f740 1 Processor -- start 2024-03-12T05:13:29.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 -- start start 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1bb7fe640 1 -- 172.21.15.45:0/466326357 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc1b4003c10 con 0x7fc1c4103c40 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41751630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c4175cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c4170c960 con 0x563c41751b70 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c416c0310 con 0x563c4175a750 2024-03-12T05:13:29.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c416bd840 con 0x563c41751250 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37848/0 (socket says 172.21.15.45:37848) 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1ca52f640 1 -- 172.21.15.45:0/1420647036 learned_addr learned my addr 172.21.15.45:0/1420647036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1c9d2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c4175cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.081+0000 7fc1cad30640 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41751630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 -- 172.21.15.45:0/1420647036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 msgr2=0x563c41751630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41751630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 -- 172.21.15.45:0/1420647036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 msgr2=0x563c4175cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c4175cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 -- 172.21.15.45:0/1420647036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c4167b060 con 0x563c41751b70 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c4175cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1bb7fe640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 0x7fc1b00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1bb7fe640 1 -- 172.21.15.45:0/466326357 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc1b402de40 con 0x7fc1c4103c40 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00115f0 tx=0x7fc1c0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ac9d812681f0d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1cad30640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 0x7fc1b00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.082+0000 7fc1b8ff9640 1 -- 172.21.15.45:0/1420647036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0004030 con 0x563c41751b70 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.083+0000 7fc1b8ff9640 1 -- 172.21.15.45:0/1420647036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1c000f660 con 0x563c41751b70 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.083+0000 7fc1b8ff9640 1 -- 172.21.15.45:0/1420647036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0003970 con 0x563c41751b70 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.083+0000 7fc1cf36f740 1 -- 172.21.15.45:0/1420647036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 msgr2=0x563c4175a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.083+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00115f0 tx=0x7fc1c0011ab0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cad30640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 0x7fc1b00a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005fd0 tx=0x7fc1bc009600 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 -- 172.21.15.45:0/1420647036 shutdown_connections 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c4175a210 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c4175cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/1420647036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41751630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 -- 172.21.15.45:0/1420647036 >> 172.21.15.45:0/1420647036 conn(0x563c4174b980 msgr2=0x563c4174bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 -- 172.21.15.45:0/1420647036 shutdown_connections 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 -- 172.21.15.45:0/1420647036 wait complete. 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.084+0000 7fc1cf36f740 1 -- start start 2024-03-12T05:13:29.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 Processor -- start 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c417581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c416b9020 con 0x563c41751b70 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c4138c430 con 0x563c4175a750 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c41348e00 con 0x563c41751250 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cf36f740 10 client.0 _ll_register_callbacks cb 0x563c41660cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47112/0 (socket says 172.21.15.45:47112) 2024-03-12T05:13:29.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1ca52f640 1 -- 172.21.15.45:0/3353834891 learned_addr learned my addr 172.21.15.45:0/3353834891 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cad30640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c417581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cad30640 1 -- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 msgr2=0x563c41757730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cad30640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cad30640 1 -- 172.21.15.45:0/3353834891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 msgr2=0x563c41757c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.085+0000 7fc1cad30640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1cad30640 1 -- 172.21.15.45:0/3353834891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c0009d80 con 0x563c4175a750 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1cad30640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c417581b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc011cf0 tx=0x7fc1bc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92124cb9968f7b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1adffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc020620 con 0x563c4175a750 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc1bc020780 con 0x563c4175a750 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563c413b87e0 con 0x563c4175a750 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1bc01fe80 con 0x563c4175a750 2024-03-12T05:13:29.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1cf36f740 10 client.25715 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.086+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563c416b76a0 con 0x563c4175a750 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563c413b87e0 con 0x563c4175a750 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc1bc005140 con 0x563c4175a750 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 10 client.25715 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 10 client.25715 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 20 client.25715 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c4176acc0 0x563c4176d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 4 client.25715 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c413f5720 con 0x563c4176acc0 2024-03-12T05:13:29.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.087+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c4176acc0 0x563c4176d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.088+0000 7fc1adffb640 10 client.25715 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:13:29.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.088+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c4176acc0 0x563c4176d100 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57555fe46ec105bb server_cookie=2ce4a7d87bfcff6e in_seq=0 out_seq=0 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.089+0000 7fc1adffb640 1 -- 172.21.15.45:0/3353834891 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563c413f5720 con 0x563c4176acc0 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.089+0000 7fc1adffb640 10 client.25715 handle_command_reply: tid=0 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.089+0000 7fc1cf36f740 1 client.25715 shutdown 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.089+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 msgr2=0x563c417581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.089+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c417581b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc011cf0 tx=0x7fc1bc004a40 comp rx=0 tx=0).stop 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 shutdown_connections 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c4176acc0 0x563c4176d100 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c4175a750 0x563c417581b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/3353834891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c41751250 0x563c41757730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 >> 172.21.15.45:0/3353834891 conn(0x563c4174b980 msgr2=0x563c4174d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1adffb640 0 client.25715 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 shutdown_connections 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 1 -- 172.21.15.45:0/3353834891 wait complete. 2024-03-12T05:13:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.090+0000 7fc1cf36f740 20 client.25715 trim_cache size 0 max 16384 2024-03-12T05:13:29.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.110+0000 7fc1cf36f740 1 Processor -- start 2024-03-12T05:13:29.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 -- start start 2024-03-12T05:13:29.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c4163e130 con 0x563c41751b70 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c4174b7f0 con 0x563c41751250 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c417676b0 con 0x563c4175a750 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1ca52f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39706/0 (socket says 172.21.15.45:39706) 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1ca52f640 1 -- 172.21.15.45:0/2343636095 learned_addr learned my addr 172.21.15.45:0/2343636095 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.111+0000 7fc1cad30640 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 -- 172.21.15.45:0/2343636095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 msgr2=0x563c417581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 -- 172.21.15.45:0/2343636095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 msgr2=0x563c41757c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 -- 172.21.15.45:0/2343636095 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c41653c20 con 0x563c41751250 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1cad30640 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0014040 tx=0x7fc1c0009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6e38f4f501fe445 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1affff640 1 -- 172.21.15.45:0/2343636095 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c002d070 con 0x563c41751250 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1affff640 1 -- 172.21.15.45:0/2343636095 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1c0005630 con 0x563c41751250 2024-03-12T05:13:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.112+0000 7fc1affff640 1 -- 172.21.15.45:0/2343636095 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c00039e0 con 0x563c41751250 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 -- 172.21.15.45:0/2343636095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 msgr2=0x563c41759e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0014040 tx=0x7fc1c0009530 comp rx=0 tx=0).stop 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 -- 172.21.15.45:0/2343636095 shutdown_connections 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751b70 0x563c41757c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/2343636095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751250 0x563c41759e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.113+0000 7fc1cf36f740 1 -- 172.21.15.45:0/2343636095 >> 172.21.15.45:0/2343636095 conn(0x563c4174b980 msgr2=0x563c41752a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 -- 172.21.15.45:0/2343636095 shutdown_connections 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 -- 172.21.15.45:0/2343636095 wait complete. 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 -- start start 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 Processor -- start 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 0x563c41761820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 0x563c4175fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.114+0000 7fc1cf36f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c41760700 con 0x563c41751250 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c41760870 con 0x563c41751b70 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cf36f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c41760a30 con 0x563c4175a750 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cf36f740 10 client.0 _ll_register_callbacks cb 0x7fc1c0009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47128/0 (socket says 172.21.15.45:47128) 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 -- 172.21.15.45:0/866656831 learned_addr learned my addr 172.21.15.45:0/866656831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 0x563c4175fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 0x563c41761820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 -- 172.21.15.45:0/866656831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 msgr2=0x563c4175fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 0x563c4175fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 -- 172.21.15.45:0/866656831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 msgr2=0x563c41761820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 0x563c41761820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1cad30640 1 -- 172.21.15.45:0/866656831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c416b76a0 con 0x563c4175a750 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.115+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 0x563c41761820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1cad30640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005570 tx=0x7fc1bc0119e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15b092e0a033ae4b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1c9d2e640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 0x563c4175fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1adffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:29.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc004ec0 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc1bc01fe80 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.116+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1c000f660 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1bc005020 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 10 client.25741 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563c4167b060 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1c000f660 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc1bc020420 con 0x563c4175a750 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 10 client.25741 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 10 client.25741 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 20 client.25741 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c417c2790 0x563c417c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 4 client.25741 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.117+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563c41444890 con 0x563c417c2790 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.118+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c417c2790 0x563c417c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.118+0000 7fc1ca52f640 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c417c2790 0x563c417c4bb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba12339019a8a553 server_cookie=bea61ffd9e670a2c in_seq=0 out_seq=0 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.118+0000 7fc1adffb640 10 client.25741 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1adffb640 1 -- 172.21.15.45:0/866656831 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563c41444890 con 0x563c417c2790 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1adffb640 10 client.25741 handle_command_reply: tid=0 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 client.25741 shutdown 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 msgr2=0x563c417601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005570 tx=0x7fc1bc0119e0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 shutdown_connections 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563c417c2790 0x563c417c4bb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4175a750 0x563c417601c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c41751b70 0x563c4175fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 --2- 172.21.15.45:0/866656831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563c41751250 0x563c41761820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 >> 172.21.15.45:0/866656831 conn(0x563c4174b980 msgr2=0x563c4174a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1adffb640 0 client.25741 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 shutdown_connections 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.119+0000 7fc1cf36f740 1 -- 172.21.15.45:0/866656831 wait complete. 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.120+0000 7fc1cf36f740 20 client.25741 trim_cache size 0 max 16384 2024-03-12T05:13:29.118 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:13:29.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 msgr2=0x7fc1b00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 0x7fc1b00a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005fd0 tx=0x7fc1bc009600 comp rx=0 tx=0).stop 2024-03-12T05:13:29.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 msgr2=0x7fc1c41a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4036040 tx=0x7fc1b4004040 comp rx=0 tx=0).stop 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 shutdown_connections 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc1b00a1ba0 0x7fc1b00a4060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c4103c40 0x7fc1c41a3dd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1c41060d0 0x7fc1c419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 --2- 172.21.15.45:0/466326357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c40688c0 0x7fc1c41a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 >> 172.21.15.45:0/466326357 conn(0x7fc1c4077180 msgr2=0x7fc1c4101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 shutdown_connections 2024-03-12T05:13:29.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.121+0000 7fc1cc793640 1 -- 172.21.15.45:0/466326357 wait complete. 2024-03-12T05:13:29.136 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:13:29.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:13:29.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.313+0000 7f84e3296640 1 -- 172.21.15.45:0/2476987120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc102380 msgr2=0x7f84dc102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.313+0000 7f84e3296640 1 --2- 172.21.15.45:0/2476987120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc102380 0x7f84dc102760 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009900 tx=0x7f84d002bcd0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 -- 172.21.15.45:0/2476987120 shutdown_connections 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 --2- 172.21.15.45:0/2476987120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc10b750 0x7f84dc10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 --2- 172.21.15.45:0/2476987120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102ca0 0x7f84dc10b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 --2- 172.21.15.45:0/2476987120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc102380 0x7f84dc102760 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 -- 172.21.15.45:0/2476987120 >> 172.21.15.45:0/2476987120 conn(0x7f84dc076e30 msgr2=0x7f84dc0fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 -- 172.21.15.45:0/2476987120 shutdown_connections 2024-03-12T05:13:29.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.314+0000 7f84e3296640 1 -- 172.21.15.45:0/2476987120 wait complete. 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 Processor -- start 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 -- start start 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102380 0x7f84dc115460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc102ca0 0x7f84dc1159a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc110d00 con 0x7f84dc10b750 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc110e70 con 0x7f84dc102ca0 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e3296640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc111030 con 0x7f84dc102380 2024-03-12T05:13:29.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e1833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e1032640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102380 0x7f84dc115460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e1833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37878/0 (socket says 172.21.15.45:37878) 2024-03-12T05:13:29.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.315+0000 7f84e1833640 1 -- 172.21.15.45:0/3369163244 learned_addr learned my addr 172.21.15.45:0/3369163244 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e0831640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc102ca0 0x7f84dc1159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 -- 172.21.15.45:0/3369163244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102380 msgr2=0x7f84dc115460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102380 0x7f84dc115460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 -- 172.21.15.45:0/3369163244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc102ca0 msgr2=0x7f84dc1159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc102ca0 0x7f84dc1159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 -- 172.21.15.45:0/3369163244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009560 con 0x7f84dc10b750 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84e1833640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c970 tx=0x7f84d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.316+0000 7f84da7fc640 1 -- 172.21.15.45:0/3369163244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4007bb0 con 0x7f84dc10b750 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.317+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc111310 con 0x7f84dc10b750 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.317+0000 7f84da7fc640 1 -- 172.21.15.45:0/3369163244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84d4007d10 con 0x7f84dc10b750 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.317+0000 7f84da7fc640 1 -- 172.21.15.45:0/3369163244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4018770 con 0x7f84dc10b750 2024-03-12T05:13:29.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.317+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc11b410 con 0x7f84dc10b750 2024-03-12T05:13:29.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.320+0000 7f84da7fc640 1 -- 172.21.15.45:0/3369163244 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f84d40188d0 con 0x7f84dc10b750 2024-03-12T05:13:29.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.321+0000 7f84da7fc640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 0x7f84c40a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.321+0000 7f84e5e61740 1 Processor -- start 2024-03-12T05:13:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.321+0000 7f84e1032640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 0x7f84c40a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.321+0000 7f84e5e61740 1 -- start start 2024-03-12T05:13:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 0x558e84205630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84da7fc640 1 -- 172.21.15.45:0/3369163244 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f84d40cd880 con 0x7f84dc10b750 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e0831640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e0831640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37886/0 (socket says 172.21.15.45:37886) 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420e750 0x558e84210b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e841c0960 con 0x558e84205b70 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84174310 con 0x558e8420e750 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e5e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84171840 con 0x558e84205250 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e0831640 1 -- 172.21.15.45:0/1945645578 learned_addr learned my addr 172.21.15.45:0/1945645578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e1032640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 0x7f84c40a41a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84d0000c00 tx=0x7f84d0005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e1833640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420e750 0x558e84210b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.322+0000 7f84e1032640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 0x558e84205630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 -- 172.21.15.45:0/1945645578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420e750 msgr2=0x558e84210b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420e750 0x558e84210b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 -- 172.21.15.45:0/1945645578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 msgr2=0x558e8420e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 -- 172.21.15.45:0/1945645578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e8412f060 con 0x558e84205250 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e0831640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84e1032640 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 0x558e84205630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009850 tx=0x7f84d003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d9112e4e8b3309 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84c37fe640 1 -- 172.21.15.45:0/1945645578 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0004400 con 0x558e84205250 2024-03-12T05:13:29.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.323+0000 7f84c37fe640 1 -- 172.21.15.45:0/1945645578 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84d002ccc0 con 0x558e84205250 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84c37fe640 1 -- 172.21.15.45:0/1945645578 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0003d10 con 0x558e84205250 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 -- 172.21.15.45:0/1945645578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 msgr2=0x558e84205630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 0x558e84205630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009850 tx=0x7f84d003f000 comp rx=0 tx=0).stop 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 -- 172.21.15.45:0/1945645578 shutdown_connections 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420e750 0x558e84210b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1945645578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e84205250 0x558e84205630 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009850 tx=0x7f84d003f000 comp rx=0 tx=0).stop 2024-03-12T05:13:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.324+0000 7f84e5e61740 1 -- 172.21.15.45:0/1945645578 >> 172.21.15.45:0/1945645578 conn(0x558e841ff980 msgr2=0x558e841ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.325+0000 7f84e5e61740 1 -- 172.21.15.45:0/1945645578 shutdown_connections 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.325+0000 7f84e5e61740 1 -- 172.21.15.45:0/1945645578 wait complete. 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.325+0000 7f84e5e61740 1 -- start start 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.325+0000 7f84e5e61740 1 Processor -- start 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420e750 0x558e8420bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e8416d020 con 0x558e84205b70 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e83e40430 con 0x558e8420c2a0 2024-03-12T05:13:29.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e83dfce00 con 0x558e8420e750 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e5e61740 10 client.0 _ll_register_callbacks cb 0x558e84114cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39746/0 (socket says 172.21.15.45:39746) 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 -- 172.21.15.45:0/1187116933 learned_addr learned my addr 172.21.15.45:0/1187116933 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e0831640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420e750 0x558e8420bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1032640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 -- 172.21.15.45:0/1187116933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420e750 msgr2=0x558e8420bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420e750 0x558e8420bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 -- 172.21.15.45:0/1187116933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 msgr2=0x558e8420b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.326+0000 7f84e1833640 1 -- 172.21.15.45:0/1187116933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc10e870 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84e1032640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84e1833640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84d400afd0 tx=0x7f84d409ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b6eac7850414b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40cb3f0 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f84d40b0210 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e83e6c7e0 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84d40b03d0 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84e5e61740 10 client.25727 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.327+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558e8416b6a0 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558e83e6c7e0 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f84d4016020 con 0x558e8420c2a0 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 10 client.25727 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:29.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 20 client.25727 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:29.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e8421f080 0x558e842214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 4 client.25727 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:29.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e83ea9720 con 0x558e8421f080 2024-03-12T05:13:29.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.328+0000 7f84e1032640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e8421f080 0x558e842214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.329+0000 7f84e1032640 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e8421f080 0x558e842214a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de2e22855503bf7b server_cookie=a48074c2c8e3fda6 in_seq=0 out_seq=0 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.329+0000 7f84c0ff9640 10 client.25727 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84c0ff9640 1 -- 172.21.15.45:0/1187116933 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558e83ea9720 con 0x558e8421f080 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84c0ff9640 10 client.25727 handle_command_reply: tid=0 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 client.25727 shutdown 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 msgr2=0x558e8421a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f84d400afd0 tx=0x7f84d409ad80 comp rx=0 tx=0).stop 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 shutdown_connections 2024-03-12T05:13:29.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e8421f080 0x558e842214a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e8420c2a0 0x558e8421a6a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420e750 0x558e8420bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 --2- 172.21.15.45:0/1187116933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e84205b70 0x558e8420b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 >> 172.21.15.45:0/1187116933 conn(0x558e841ff980 msgr2=0x558e842016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84c0ff9640 0 client.25727 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 shutdown_connections 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 1 -- 172.21.15.45:0/1187116933 wait complete. 2024-03-12T05:13:29.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.330+0000 7f84e5e61740 20 client.25727 trim_cache size 0 max 16384 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.353+0000 7f84e5e61740 1 Processor -- start 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.353+0000 7f84e5e61740 1 -- start start 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.353+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.353+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.353+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e8421a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e5e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e840f2130 con 0x558e8420e750 2024-03-12T05:13:29.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e5e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e841fff90 con 0x558e84205b70 2024-03-12T05:13:29.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e5e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84207240 con 0x558e8420c2a0 2024-03-12T05:13:29.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e1032640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47168/0 (socket says 172.21.15.45:47168) 2024-03-12T05:13:29.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 -- 172.21.15.45:0/234103088 learned_addr learned my addr 172.21.15.45:0/234103088 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 -- 172.21.15.45:0/234103088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 msgr2=0x558e842155f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 -- 172.21.15.45:0/234103088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 msgr2=0x558e8421a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e8421a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 -- 172.21.15.45:0/234103088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e84107c20 con 0x558e8420c2a0 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.354+0000 7f84e0831640 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84cc003a50 tx=0x7f84cc003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc95b6a59fc08c9e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.355+0000 7f84c2ffd640 1 -- 172.21.15.45:0/234103088 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc016070 con 0x558e8420c2a0 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.355+0000 7f84c2ffd640 1 -- 172.21.15.45:0/234103088 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84cc017040 con 0x558e8420c2a0 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.355+0000 7f84c2ffd640 1 -- 172.21.15.45:0/234103088 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc01b6d0 con 0x558e8420c2a0 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.355+0000 7f84e5e61740 1 -- 172.21.15.45:0/234103088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 msgr2=0x558e84215b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.355+0000 7f84e5e61740 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f84cc003a50 tx=0x7f84cc003f10 comp rx=0 tx=0).stop 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 -- 172.21.15.45:0/234103088 shutdown_connections 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e8421a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 --2- 172.21.15.45:0/234103088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 -- 172.21.15.45:0/234103088 >> 172.21.15.45:0/234103088 conn(0x558e841ff980 msgr2=0x558e8420b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 -- 172.21.15.45:0/234103088 shutdown_connections 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.356+0000 7f84e5e61740 1 -- 172.21.15.45:0/234103088 wait complete. 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.357+0000 7f84e5e61740 1 -- start start 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.357+0000 7f84e5e61740 1 Processor -- start 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e5e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e84213be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e5e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e842142b0 con 0x558e8420e750 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e0831640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e0831640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47178/0 (socket says 172.21.15.45:47178) 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e0831640 1 -- 172.21.15.45:0/885242225 learned_addr learned my addr 172.21.15.45:0/885242225 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e1833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e84213be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e84214420 con 0x558e84205b70 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e842145e0 con 0x558e8420c2a0 2024-03-12T05:13:29.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e5e61740 10 client.0 _ll_register_callbacks cb 0x7f84d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.358+0000 7f84e1032640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 -- 172.21.15.45:0/885242225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 msgr2=0x558e84215b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 -- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 msgr2=0x558e842155f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc009d90 con 0x558e8420e750 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e0831640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84e1833640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e84213be0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f84d409b060 tx=0x7f84d40cb480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddcf728545ae425 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.359+0000 7f84c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4018d80 con 0x558e8420e750 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f84d409a7b0 con 0x558e8420e750 2024-03-12T05:13:29.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84d002ccc0 con 0x558e8420e750 2024-03-12T05:13:29.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84d409aea0 con 0x558e8420e750 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84e5e61740 10 client.16275 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.360+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558e8412d8c0 con 0x558e8420e750 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f84d002ccc0 con 0x558e8420e750 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f84d4016070 con 0x558e8420e750 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 10 client.16275 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 20 client.16275 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e84276560 0x558e84278980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 4 client.16275 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558e83ef8890 con 0x558e84276560 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.361+0000 7f84e1032640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e84276560 0x558e84278980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e1032640 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e84276560 0x558e84278980 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c359eaa77b14e3ca server_cookie=658a3f9f9886b301 in_seq=0 out_seq=0 2024-03-12T05:13:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84c0ff9640 10 client.16275 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84c0ff9640 1 -- 172.21.15.45:0/885242225 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558e83ef8890 con 0x558e84276560 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84c0ff9640 10 client.16275 handle_command_reply: tid=0 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 client.16275 shutdown 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 msgr2=0x558e84213be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e84213be0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f84d409b060 tx=0x7f84d40cb480 comp rx=0 tx=0).stop 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 shutdown_connections 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558e84276560 0x558e84278980 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558e8420e750 0x558e84213be0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e8420c2a0 0x558e84215b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 --2- 172.21.15.45:0/885242225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558e84205b70 0x558e842155f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84c0ff9640 0 client.16275 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.362+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 >> 172.21.15.45:0/885242225 conn(0x558e841ff980 msgr2=0x558e8421a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.363+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 shutdown_connections 2024-03-12T05:13:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.363+0000 7f84e5e61740 1 -- 172.21.15.45:0/885242225 wait complete. 2024-03-12T05:13:29.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.363+0000 7f84e5e61740 20 client.16275 trim_cache size 0 max 16384 2024-03-12T05:13:29.362 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 msgr2=0x7f84c40a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 0x7f84c40a41a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84d0000c00 tx=0x7f84d0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 msgr2=0x7f84dc110570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c970 tx=0x7f84d400ce30 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 shutdown_connections 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f84c40a1ce0 0x7f84c40a41a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc10b750 0x7f84dc110570 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84dc102ca0 0x7f84dc1159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 --2- 172.21.15.45:0/3369163244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84dc102380 0x7f84dc115460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 >> 172.21.15.45:0/3369163244 conn(0x7f84dc076e30 msgr2=0x7f84dc0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:29.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 shutdown_connections 2024-03-12T05:13:29.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:29.364+0000 7f84e3296640 1 -- 172.21.15.45:0/3369163244 wait complete. 2024-03-12T05:13:29.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:29 smithi045 ceph-mon[32174]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 18 MiB/s wr, 2.11k op/s 2024-03-12T05:13:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 85.0 s, 1335.4 tps, lat 74.901 ms stddev 23.433 2024-03-12T05:13:29.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:29 smithi130 ceph-mon[35398]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 18 MiB/s wr, 2.11k op/s 2024-03-12T05:13:29.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:29 smithi116 ceph-mon[35486]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 18 MiB/s wr, 2.11k op/s 2024-03-12T05:13:30.380 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:13:30.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:13:30.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:13:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.558+0000 7f4d1253a640 1 -- 172.21.15.45:0/3640906179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 msgr2=0x7f4d0c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.558+0000 7f4d1253a640 1 --2- 172.21.15.45:0/3640906179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c0689b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009890 tx=0x7f4d0402bb70 comp rx=0 tx=0).stop 2024-03-12T05:13:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 -- 172.21.15.45:0/3640906179 shutdown_connections 2024-03-12T05:13:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 --2- 172.21.15.45:0/3640906179 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c106430 0x7f4d0c10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 --2- 172.21.15.45:0/3640906179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c068f80 0x7f4d0c105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 --2- 172.21.15.45:0/3640906179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c0689b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 -- 172.21.15.45:0/3640906179 >> 172.21.15.45:0/3640906179 conn(0x7f4d0c077180 msgr2=0x7f4d0c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 -- 172.21.15.45:0/3640906179 shutdown_connections 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.559+0000 7f4d1253a640 1 -- 172.21.15.45:0/3640906179 wait complete. 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.560+0000 7f4d1253a640 1 Processor -- start 2024-03-12T05:13:30.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.560+0000 7f4d1253a640 1 -- start start 2024-03-12T05:13:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.560+0000 7f4d1253a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.560+0000 7f4d1253a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c106430 0x7f4d0c195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d1253a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 0x7f4d0c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d1253a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c19a780 con 0x7f4d0c199d90 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d1253a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c19a8f0 con 0x7f4d0c0685d0 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d1253a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0c19aa60 con 0x7f4d0c106430 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 0x7f4d0c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 0x7f4d0c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37946/0 (socket says 172.21.15.45:37946) 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39786/0 (socket says 172.21.15.45:39786) 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d10ad7640 1 -- 172.21.15.45:0/1000992511 learned_addr learned my addr 172.21.15.45:0/1000992511 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c106430 0x7f4d0c195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 -- 172.21.15.45:0/1000992511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c106430 msgr2=0x7f4d0c195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c106430 0x7f4d0c195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 -- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 msgr2=0x7f4d0c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 0x7f4d0c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.561+0000 7f4d0bfff640 1 -- 172.21.15.45:0/1000992511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04009490 con 0x7f4d0c0685d0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d04037040 tx=0x7f4d04004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d097fa640 1 -- 172.21.15.45:0/1000992511 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04034910 con 0x7f4d0c0685d0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d097fa640 1 -- 172.21.15.45:0/1000992511 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4d0402ce90 con 0x7f4d0c0685d0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d097fa640 1 -- 172.21.15.45:0/1000992511 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d04032ce0 con 0x7f4d0c0685d0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0c1a2040 con 0x7f4d0c0685d0 2024-03-12T05:13:30.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.562+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c1a2410 con 0x7f4d0c0685d0 2024-03-12T05:13:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.565+0000 7f4d097fa640 1 -- 172.21.15.45:0/1000992511 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4d0403b040 con 0x7f4d0c0685d0 2024-03-12T05:13:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.566+0000 7f4d151e8740 1 Processor -- start 2024-03-12T05:13:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.566+0000 7f4d097fa640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 0x7f4cf80a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.566+0000 7f4d097fa640 1 -- 172.21.15.45:0/1000992511 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4d040ef150 con 0x7f4d0c0685d0 2024-03-12T05:13:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.566+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 0x7f4cf80a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 -- start start 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f56816760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 0x7f4cf80a4160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc009ef0 tx=0x7f4cfc009680 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816d30 0x562f5681f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56784f90 con 0x562f56816380 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f567824c0 con 0x562f5681f8b0 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d151e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5677dca0 con 0x562f56816d30 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f56816760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816d30 0x562f5681f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39798/0 (socket says 172.21.15.45:39798) 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.567+0000 7f4d10ad7640 1 -- 172.21.15.45:0/553205240 learned_addr learned my addr 172.21.15.45:0/553205240 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 -- 172.21.15.45:0/553205240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816d30 msgr2=0x562f5681f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816d30 0x562f5681f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 -- 172.21.15.45:0/553205240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 msgr2=0x562f56816760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f56816760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 -- 172.21.15.45:0/553205240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f567400b0 con 0x562f5681f8b0 2024-03-12T05:13:30.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc01d780 tx=0x7f4cfc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34e98958ccf60d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d027fc640 1 -- 172.21.15.45:0/553205240 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc014070 con 0x562f5681f8b0 2024-03-12T05:13:30.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d027fc640 1 -- 172.21.15.45:0/553205240 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cfc01ba20 con 0x562f5681f8b0 2024-03-12T05:13:30.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.568+0000 7f4d027fc640 1 -- 172.21.15.45:0/553205240 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc025950 con 0x562f5681f8b0 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 -- 172.21.15.45:0/553205240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 msgr2=0x562f56821d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc01d780 tx=0x7f4cfc014000 comp rx=0 tx=0).stop 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 -- 172.21.15.45:0/553205240 shutdown_connections 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56821d00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816d30 0x562f5681f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 --2- 172.21.15.45:0/553205240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f56816760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.569+0000 7f4d151e8740 1 -- 172.21.15.45:0/553205240 >> 172.21.15.45:0/553205240 conn(0x562f56810840 msgr2=0x562f56810c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.570+0000 7f4d151e8740 1 -- 172.21.15.45:0/553205240 shutdown_connections 2024-03-12T05:13:30.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.570+0000 7f4d151e8740 1 -- 172.21.15.45:0/553205240 wait complete. 2024-03-12T05:13:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.570+0000 7f4d151e8740 1 -- start start 2024-03-12T05:13:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.571+0000 7f4d151e8740 1 Processor -- start 2024-03-12T05:13:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f5681cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37960/0 (socket says 172.21.15.45:37960) 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d0bfff640 1 -- 172.21.15.45:0/3982670268 learned_addr learned my addr 172.21.15.45:0/3982670268 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56722f40 con 0x562f56816380 2024-03-12T05:13:30.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.572+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f564514b0 con 0x562f56816d30 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56703670 con 0x562f5681f8b0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d151e8740 10 client.0 _ll_register_callbacks cb 0x562f56770f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f5681cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d10ad7640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/3982670268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 msgr2=0x562f5681d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/3982670268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 msgr2=0x562f5681c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc01d430 con 0x562f56816d30 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.573+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f5681cd90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562f56817130 tx=0x7f4cf4003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85d77ed4f633fd67 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4005220 con 0x562f56816d30 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4cf4005380 con 0x562f56816d30 2024-03-12T05:13:30.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f56757f40 con 0x562f56816d30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cf4005540 con 0x562f56816d30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4d151e8740 10 client.25745 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562f5677c320 con 0x562f56816d30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562f56757f40 con 0x562f56816d30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.574+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4cf401a040 con 0x562f56816d30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 10 client.25745 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:30.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 20 client.25745 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f56831ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 4 client.25745 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f56774310 con 0x562f5682fbd0 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.575+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f56831ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.576+0000 7f4ceb7fe640 10 client.25745 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.576+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f56831ff0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abb92b5f64bb2571 server_cookie=3e782ccf0dabe5dc in_seq=0 out_seq=0 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.576+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/3982670268 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562f56774310 con 0x562f5682fbd0 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.576+0000 7f4ceb7fe640 10 client.25745 handle_command_reply: tid=0 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 client.25745 shutdown 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 msgr2=0x562f5681cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f5681cd90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562f56817130 tx=0x7f4cf4003a80 comp rx=0 tx=0).stop 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 shutdown_connections 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f56831ff0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f5681cd90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4ceb7fe640 0 client.25745 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 --2- 172.21.15.45:0/3982670268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.577+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 >> 172.21.15.45:0/3982670268 conn(0x562f56810840 msgr2=0x562f568127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.578+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 shutdown_connections 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.578+0000 7f4d151e8740 1 -- 172.21.15.45:0/3982670268 wait complete. 2024-03-12T05:13:30.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.578+0000 7f4d151e8740 20 client.25745 trim_cache size 0 max 16384 2024-03-12T05:13:30.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.607+0000 7f4d151e8740 1 Processor -- start 2024-03-12T05:13:30.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.607+0000 7f4d151e8740 1 -- start start 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f56819b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56811440 con 0x562f56816380 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5682e470 con 0x562f56816d30 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d151e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f5680ad20 con 0x562f5681f8b0 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37964/0 (socket says 172.21.15.45:37964) 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d0bfff640 1 -- 172.21.15.45:0/2155808923 learned_addr learned my addr 172.21.15.45:0/2155808923 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.608+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f56819b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/2155808923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 msgr2=0x562f5681a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/2155808923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 msgr2=0x562f5681ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 -- 172.21.15.45:0/2155808923 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f56718c00 con 0x562f56816d30 2024-03-12T05:13:30.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f56819b50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x562f5681c0f0 tx=0x7f4cf4009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ccc3e8377080dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d01ffb640 1 -- 172.21.15.45:0/2155808923 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf400fb50 con 0x562f56816d30 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d01ffb640 1 -- 172.21.15.45:0/2155808923 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cf4015040 con 0x562f56816d30 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.609+0000 7f4d01ffb640 1 -- 172.21.15.45:0/2155808923 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4005360 con 0x562f56816d30 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 -- 172.21.15.45:0/2155808923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 msgr2=0x562f56819b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f56819b50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x562f5681c0f0 tx=0x7f4cf4009f40 comp rx=0 tx=0).stop 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 -- 172.21.15.45:0/2155808923 shutdown_connections 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f5681f8b0 0x562f5681a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f56816d30 0x562f56819b50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 --2- 172.21.15.45:0/2155808923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816380 0x562f5681ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.610+0000 7f4d151e8740 1 -- 172.21.15.45:0/2155808923 >> 172.21.15.45:0/2155808923 conn(0x562f56810840 msgr2=0x562f5681b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.611+0000 7f4d151e8740 1 -- 172.21.15.45:0/2155808923 shutdown_connections 2024-03-12T05:13:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.611+0000 7f4d151e8740 1 -- 172.21.15.45:0/2155808923 wait complete. 2024-03-12T05:13:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.611+0000 7f4d151e8740 1 -- start start 2024-03-12T05:13:30.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d151e8740 1 Processor -- start 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816d30 0x562f56880d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d151e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56825f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816d30 0x562f56880d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.612+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47230/0 (socket says 172.21.15.45:47230) 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56825f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d10ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56825f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39820/0 (socket says 172.21.15.45:39820) 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 -- 172.21.15.45:0/933670540 learned_addr learned my addr 172.21.15.45:0/933670540 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56826550 con 0x562f56816d30 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f568266c0 con 0x562f5681f8b0 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f56826880 con 0x562f56816380 2024-03-12T05:13:30.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d151e8740 10 client.0 _ll_register_callbacks cb 0x7f4cfc00ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 -- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 msgr2=0x562f56825f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56825f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 -- 172.21.15.45:0/933670540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816d30 msgr2=0x562f56880d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816d30 0x562f56880d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0c1a2410 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.613+0000 7f4d0bfff640 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d040bbbe0 tx=0x7f4d04005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4186c9a06a24720f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4ceb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d040bbee0 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4d040b8550 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f567a3900 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d040b8710 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4d151e8740 10 client.25777 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.614+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562f5677c320 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562f567a3900 con 0x562f56816380 2024-03-12T05:13:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4d040a1050 con 0x562f56816380 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 10 client.25777 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 10 client.25777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 20 client.25777 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f5688a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 4 client.25777 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562f5647d860 con 0x562f5682fbd0 2024-03-12T05:13:30.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.615+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f5688a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.616+0000 7f4d0b7fe640 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f5688a2e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb7c054aa898611 server_cookie=5a395881ecdf7eb9 in_seq=0 out_seq=0 2024-03-12T05:13:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.616+0000 7f4ceb7fe640 10 client.25777 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.617+0000 7f4ceb7fe640 1 -- 172.21.15.45:0/933670540 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562f5647d860 con 0x562f5682fbd0 2024-03-12T05:13:30.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.617+0000 7f4ceb7fe640 10 client.25777 handle_command_reply: tid=0 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.617+0000 7f4d151e8740 1 client.25777 shutdown 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.617+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 msgr2=0x562f56880820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.617+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d040bbbe0 tx=0x7f4d04005740 comp rx=0 tx=0).stop 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 shutdown_connections 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562f5682fbd0 0x562f5688a2e0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562f56816d30 0x562f56880d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f5681f8b0 0x562f56825f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 --2- 172.21.15.45:0/933670540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562f56816380 0x562f56880820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 >> 172.21.15.45:0/933670540 conn(0x562f56810840 msgr2=0x562f5681c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4ceb7fe640 0 client.25777 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 shutdown_connections 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 1 -- 172.21.15.45:0/933670540 wait complete. 2024-03-12T05:13:30.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.618+0000 7f4d151e8740 20 client.25777 trim_cache size 0 max 16384 2024-03-12T05:13:30.617 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:13:30.617 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:13:30.617 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "59154b7c-39cb-44a4-ad14-65cf51400e38", 2024-03-12T05:13:30.617 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:13:30.617 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.620+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 msgr2=0x7f4cf80a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.620+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 0x7f4cf80a4160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc009ef0 tx=0x7f4cfc009680 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.620+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 msgr2=0x7f4d0c1954d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.620+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4d04037040 tx=0x7f4d04004430 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 shutdown_connections 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4cf80a1ca0 0x7f4cf80a4160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c199d90 0x7f4d0c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d0c106430 0x7f4d0c195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 --2- 172.21.15.45:0/1000992511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d0c0685d0 0x7f4d0c1954d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 >> 172.21.15.45:0/1000992511 conn(0x7f4d0c077180 msgr2=0x7f4d0c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 shutdown_connections 2024-03-12T05:13:30.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.621+0000 7f4d1253a640 1 -- 172.21.15.45:0/1000992511 wait complete. 2024-03-12T05:13:30.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:13:30.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 -- 172.21.15.45:0/272639218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c103f60 msgr2=0x7fc93c107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 --2- 172.21.15.45:0/272639218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c103f60 0x7fc93c107fa0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc930009930 tx=0x7fc93002bc50 comp rx=0 tx=0).stop 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 -- 172.21.15.45:0/272639218 shutdown_connections 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 --2- 172.21.15.45:0/272639218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c103f60 0x7fc93c107fa0 secure :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc930009930 tx=0x7fc93002bc50 comp rx=0 tx=0).stop 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 --2- 172.21.15.45:0/272639218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c108680 0x7fc93c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 --2- 172.21.15.45:0/272639218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c1035b0 0x7fc93c103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 -- 172.21.15.45:0/272639218 >> 172.21.15.45:0/272639218 conn(0x7fc93c0fedf0 msgr2=0x7fc93c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.817+0000 7fc94302a640 1 -- 172.21.15.45:0/272639218 shutdown_connections 2024-03-12T05:13:30.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.818+0000 7fc94302a640 1 -- 172.21.15.45:0/272639218 wait complete. 2024-03-12T05:13:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.819+0000 7fc94302a640 1 Processor -- start 2024-03-12T05:13:30.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.819+0000 7fc94302a640 1 -- start start 2024-03-12T05:13:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.819+0000 7fc94302a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.819+0000 7fc94302a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 0x7fc93c19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc94302a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 0x7fc93c199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47254/0 (socket says 172.21.15.45:47254) 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc93bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 0x7fc93c19fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc93bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 0x7fc93c19fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39826/0 (socket says 172.21.15.45:39826) 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc940dc6640 1 -- 172.21.15.45:0/1393032400 learned_addr learned my addr 172.21.15.45:0/1393032400 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c19a440 con 0x7fc93c199a50 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c19a5b0 con 0x7fc93c108680 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc9415c7640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 0x7fc93c199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.820+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c19a770 con 0x7fc93c1035b0 2024-03-12T05:13:30.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 -- 172.21.15.45:0/1393032400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 msgr2=0x7fc93c19fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 0x7fc93c19fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 -- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 msgr2=0x7fc93c199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 0x7fc93c199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930009560 con 0x7fc93c1035b0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc9415c7640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 0x7fc93c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.821+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00b450 tx=0x7fc92c00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.822+0000 7fc939ffb640 1 -- 172.21.15.45:0/1393032400 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c011890 con 0x7fc93c1035b0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.822+0000 7fc939ffb640 1 -- 172.21.15.45:0/1393032400 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc92c0103e0 con 0x7fc93c1035b0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.822+0000 7fc939ffb640 1 -- 172.21.15.45:0/1393032400 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c011ea0 con 0x7fc93c1035b0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.822+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c1a6710 con 0x7fc93c1035b0 2024-03-12T05:13:30.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.822+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c1a6b40 con 0x7fc93c1035b0 2024-03-12T05:13:30.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.825+0000 7fc939ffb640 1 -- 172.21.15.45:0/1393032400 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc92c010550 con 0x7fc93c1035b0 2024-03-12T05:13:30.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc939ffb640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 0x7fc9240a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc939ffb640 1 -- 172.21.15.45:0/1393032400 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc92c0cd370 con 0x7fc93c1035b0 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 0x7fc9240a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 Processor -- start 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 -- start start 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b3bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b44a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7aa2280 con 0x5630b7b3bac0 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7a9f7b0 con 0x5630b7b44fc0 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.826+0000 7fc945bff740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7a9af90 con 0x5630b7b3c3e0 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b3bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39836/0 (socket says 172.21.15.45:39836) 2024-03-12T05:13:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 -- 172.21.15.45:0/3454150194 learned_addr learned my addr 172.21.15.45:0/3454150194 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 0x7fc9240a41b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc93002c260 tx=0x7fc930005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc93bfff640 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 -- 172.21.15.45:0/3454150194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 msgr2=0x5630b7b44a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b44a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 -- 172.21.15.45:0/3454150194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 msgr2=0x5630b7b3bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b3bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc9415c7640 1 -- 172.21.15.45:0/3454150194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630b7a59570 con 0x5630b7b44fc0 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.827+0000 7fc940dc6640 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b3bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.828+0000 7fc9415c7640 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9340117e0 tx=0x7fc934011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8042d957488ca741 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.828+0000 7fc922ffd640 1 -- 172.21.15.45:0/3454150194 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934011310 con 0x5630b7b44fc0 2024-03-12T05:13:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.828+0000 7fc922ffd640 1 -- 172.21.15.45:0/3454150194 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc934003e20 con 0x5630b7b44fc0 2024-03-12T05:13:30.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.828+0000 7fc922ffd640 1 -- 172.21.15.45:0/3454150194 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9340044f0 con 0x5630b7b44fc0 2024-03-12T05:13:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 -- 172.21.15.45:0/3454150194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 msgr2=0x5630b7b473b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9340117e0 tx=0x7fc934011ca0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 -- 172.21.15.45:0/3454150194 shutdown_connections 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b44fc0 0x5630b7b473b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b44a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 --2- 172.21.15.45:0/3454150194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b3bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 -- 172.21.15.45:0/3454150194 >> 172.21.15.45:0/3454150194 conn(0x5630b7b361a0 msgr2=0x5630b7b36580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 -- 172.21.15.45:0/3454150194 shutdown_connections 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.829+0000 7fc945bff740 1 -- 172.21.15.45:0/3454150194 wait complete. 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.830+0000 7fc945bff740 1 -- start start 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.830+0000 7fc945bff740 1 Processor -- start 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 0x5630b7b41d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b776e430 con 0x5630b7b3bac0 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc9415c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc9415c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47264/0 (socket says 172.21.15.45:47264) 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc9415c7640 1 -- 172.21.15.45:0/1223347205 learned_addr learned my addr 172.21.15.45:0/1223347205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 0x5630b7b41d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b772ae00 con 0x5630b7b3c3e0 2024-03-12T05:13:30.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.831+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7a1ffd0 con 0x5630b7b44fc0 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc945bff740 10 client.0 _ll_register_callbacks cb 0x5630b7a75fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 -- 172.21.15.45:0/1223347205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 msgr2=0x5630b7b424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 -- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 msgr2=0x5630b7b41d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 0x5630b7b41d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9340114c0 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 0x5630b7b41d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc940dc6640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b449f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0cabd0 tx=0x7fc92c0109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75ccea67e95b8e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.832+0000 7fc91bfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c011bf0 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc92c0a2850 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630b779a7e0 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc92c0b8320 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc945bff740 10 client.16305 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5630b7a99610 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.833+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5630b779a7e0 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc92c098f70 con 0x5630b7b3bac0 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 10 client.16305 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:30.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 20 client.16305 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7b55120 0x5630b7b57550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 4 client.16305 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:13:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630b77d7720 con 0x5630b7b55120 2024-03-12T05:13:30.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.834+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7b55120 0x5630b7b57550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.835+0000 7fc93bfff640 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7b55120 0x5630b7b57550 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b09e632e1cca65 server_cookie=80bad157cf229961 in_seq=0 out_seq=0 2024-03-12T05:13:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.835+0000 7fc91bfff640 10 client.16305 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.836+0000 7fc91bfff640 1 -- 172.21.15.45:0/1223347205 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5630b77d7720 con 0x5630b7b55120 2024-03-12T05:13:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.836+0000 7fc91bfff640 10 client.16305 handle_command_reply: tid=0 2024-03-12T05:13:30.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.837+0000 7fc945bff740 1 client.16305 shutdown 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 msgr2=0x5630b7b449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b449f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0cabd0 tx=0x7fc92c0109e0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 shutdown_connections 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7b55120 0x5630b7b57550 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3c3e0 0x5630b7b41d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc91bfff640 0 client.16305 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 --2- 172.21.15.45:0/1223347205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3bac0 0x5630b7b449f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 >> 172.21.15.45:0/1223347205 conn(0x5630b7b361a0 msgr2=0x5630b7b37f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 shutdown_connections 2024-03-12T05:13:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 1 -- 172.21.15.45:0/1223347205 wait complete. 2024-03-12T05:13:30.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.838+0000 7fc945bff740 20 client.16305 trim_cache size 0 max 16384 2024-03-12T05:13:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 Processor -- start 2024-03-12T05:13:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 -- start start 2024-03-12T05:13:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3c3e0 0x5630b7b444b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7b3b500 con 0x5630b7b3c3e0 2024-03-12T05:13:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7b37420 con 0x5630b7b3bac0 2024-03-12T05:13:30.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc945bff740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7b371f0 con 0x5630b7b44fc0 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39866/0 (socket says 172.21.15.45:39866) 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc940dc6640 1 -- 172.21.15.45:0/407958473 learned_addr learned my addr 172.21.15.45:0/407958473 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc93bfff640 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3c3e0 0x5630b7b444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.866+0000 7fc9415c7640 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 -- 172.21.15.45:0/407958473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 msgr2=0x5630b7b449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 -- 172.21.15.45:0/407958473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3c3e0 msgr2=0x5630b7b444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3c3e0 0x5630b7b444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 -- 172.21.15.45:0/407958473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630b7a35ac0 con 0x5630b7b3bac0 2024-03-12T05:13:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc940dc6640 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc92c099db0 tx=0x7fc92c011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea3c524493c70f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc9227fc640 1 -- 172.21.15.45:0/407958473 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0961c0 con 0x5630b7b3bac0 2024-03-12T05:13:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc9227fc640 1 -- 172.21.15.45:0/407958473 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc92c09e020 con 0x5630b7b3bac0 2024-03-12T05:13:30.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.867+0000 7fc9227fc640 1 -- 172.21.15.45:0/407958473 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0119f0 con 0x5630b7b3bac0 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 -- 172.21.15.45:0/407958473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 msgr2=0x5630b7b41d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc92c099db0 tx=0x7fc92c011000 comp rx=0 tx=0).stop 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 -- 172.21.15.45:0/407958473 shutdown_connections 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b3c3e0 0x5630b7b444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b44fc0 0x5630b7b449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 --2- 172.21.15.45:0/407958473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b41d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 -- 172.21.15.45:0/407958473 >> 172.21.15.45:0/407958473 conn(0x5630b7b361a0 msgr2=0x5630b7b3d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.868+0000 7fc945bff740 1 -- 172.21.15.45:0/407958473 shutdown_connections 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.869+0000 7fc945bff740 1 -- 172.21.15.45:0/407958473 wait complete. 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.869+0000 7fc945bff740 1 -- start start 2024-03-12T05:13:30.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.869+0000 7fc945bff740 1 Processor -- start 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b3ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b3f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b44fc0 0x5630b7bab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7b3fa30 con 0x5630b7b44fc0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7b3fba0 con 0x5630b7b3bac0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630b7babbb0 con 0x5630b7b3c3e0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc945bff740 10 client.0 _ll_register_callbacks cb 0x7fc934009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b3ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc940dc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b3ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:39868/0 (socket says 172.21.15.45:39868) 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc940dc6640 1 -- 172.21.15.45:0/3506649389 learned_addr learned my addr 172.21.15.45:0/3506649389 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc9415c7640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b44fc0 0x5630b7bab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.870+0000 7fc93bfff640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b3f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 -- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 msgr2=0x5630b7b3ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b3ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 -- 172.21.15.45:0/3506649389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b44fc0 msgr2=0x5630b7bab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:13:30.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b44fc0 0x5630b7bab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 -- 172.21.15.45:0/3506649389 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630b7a99610 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc93bfff640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b3f2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc93c1044b0 tx=0x7fc930009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e2208e1a0639e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc91bfff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930053070 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630b7826890 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc930047040 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.871+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc930004510 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.872+0000 7fc945bff740 10 client.25795 fetch_fsmap learned FSMap version 30 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.872+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5630b7a5b6a0 con 0x5630b7b3c3e0 2024-03-12T05:13:30.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.874+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5630b7826890 con 0x5630b7b3c3e0 2024-03-12T05:13:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc930005090 con 0x5630b7b3c3e0 2024-03-12T05:13:30.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 10 client.25795 fetch_fsmap finished waiting for FSMap version 30 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 10 client.25795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 20 client.25795 populate_metadata read hostname 'smithi045' 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7bad470 0x5630b7baf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 4 client.25795 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5630b77894e0 con 0x5630b7bad470 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.875+0000 7fc940dc6640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7bad470 0x5630b7baf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:13:30.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.876+0000 7fc91bfff640 10 client.25795 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc940dc6640 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7bad470 0x5630b7baf890 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9114677de0ddc4c8 server_cookie=c5670144c56f016 in_seq=0 out_seq=0 2024-03-12T05:13:30.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc91bfff640 1 -- 172.21.15.45:0/3506649389 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5630b77894e0 con 0x5630b7bad470 2024-03-12T05:13:30.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc91bfff640 10 client.25795 handle_command_reply: tid=0 2024-03-12T05:13:30.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 client.25795 shutdown 2024-03-12T05:13:30.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 msgr2=0x5630b7b3f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b3f2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc93c1044b0 tx=0x7fc930009260 comp rx=0 tx=0).stop 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 shutdown_connections 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5630b7bad470 0x5630b7baf890 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5630b7b44fc0 0x5630b7bab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc91bfff640 0 client.25795 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630b7b3c3e0 0x5630b7b3f2d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 --2- 172.21.15.45:0/3506649389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630b7b3bac0 0x5630b7b3ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.877+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 >> 172.21.15.45:0/3506649389 conn(0x5630b7b361a0 msgr2=0x5630b7b405d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.878+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 shutdown_connections 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.878+0000 7fc945bff740 1 -- 172.21.15.45:0/3506649389 wait complete. 2024-03-12T05:13:30.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.878+0000 7fc945bff740 20 client.25795 trim_cache size 0 max 16384 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "59154b7c-39cb-44a4-ad14-65cf51400e38": { 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "59154b7c-39cb-44a4-ad14-65cf51400e38", 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:13:30.876 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 msgr2=0x7fc9240a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 0x7fc9240a41b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc93002c260 tx=0x7fc930005a20 comp rx=0 tx=0).stop 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 msgr2=0x7fc93c19f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00b450 tx=0x7fc92c00b910 comp rx=0 tx=0).stop 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 shutdown_connections 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9240a1cf0 0x7fc9240a41b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc93c199a50 0x7fc93c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc93c108680 0x7fc93c19fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 --2- 172.21.15.45:0/1393032400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc93c1035b0 0x7fc93c19f850 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:13:30.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 >> 172.21.15.45:0/1393032400 conn(0x7fc93c0fedf0 msgr2=0x7fc93c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:13:30.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 shutdown_connections 2024-03-12T05:13:30.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:13:30.879+0000 7fc94302a640 1 -- 172.21.15.45:0/1393032400 wait complete. 2024-03-12T05:13:30.894 INFO:tasks.cephfs.filesystem:scrub status for tag:59154b7c-39cb-44a4-ad14-65cf51400e38 - {'path': '/', 'tag': '59154b7c-39cb-44a4-ad14-65cf51400e38', 'options': 'recursive,force'} 2024-03-12T05:13:31.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:31 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:31.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:13:31.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:13:31.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:13:31.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:31 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:31.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:31 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:32 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:13:32.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:32 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:13:32.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:32 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:13:32.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:32 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:13:32.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:32 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:13:32.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:32.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:32 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:13:32.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:13:33.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:33 smithi045 ceph-mon[32174]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:33.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:33 smithi130 ceph-mon[35398]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:33.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:33 smithi116 ceph-mon[35486]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:13:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:34 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:34.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 90.0 s, 1478.8 tps, lat 67.637 ms stddev 31.244 2024-03-12T05:13:34.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:34 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:34 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:35.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:35 smithi045 ceph-mon[32174]: pgmap v84: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:35.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:35 smithi130 ceph-mon[35398]: pgmap v84: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:35.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:35 smithi116 ceph-mon[35486]: pgmap v84: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:37.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:37 smithi045 ceph-mon[32174]: pgmap v85: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:37.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:37 smithi130 ceph-mon[35398]: pgmap v85: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:37.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:37 smithi116 ceph-mon[35486]: pgmap v85: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:39.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:39 smithi045 ceph-mon[32174]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 95.0 s, 1470.4 tps, lat 67.928 ms stddev 64.549 2024-03-12T05:13:39.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:39 smithi130 ceph-mon[35398]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:39.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:39 smithi116 ceph-mon[35486]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:41.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:41 smithi045 ceph-mon[32174]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:41.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:41 smithi130 ceph-mon[35398]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:41.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:41 smithi116 ceph-mon[35486]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:43.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:43 smithi045 ceph-mon[32174]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:43.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:43.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:43.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:43.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:13:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:43 smithi130 ceph-mon[35398]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:13:43.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:43 smithi116 ceph-mon[35486]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.44k op/s 2024-03-12T05:13:43.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:43.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:43.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:43.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:13:44.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 100.0 s, 1587.5 tps, lat 62.980 ms stddev 15.784 2024-03-12T05:13:45.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:45 smithi045 ceph-mon[32174]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:45.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:45 smithi130 ceph-mon[35398]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:45.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:45 smithi116 ceph-mon[35486]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:13:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:47 smithi045 ceph-mon[32174]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:13:47.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:47 smithi130 ceph-mon[35398]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:13:47.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:47 smithi116 ceph-mon[35486]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:13:49.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:49 smithi045 ceph-mon[32174]: pgmap v91: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:49.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:49 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 105.0 s, 1558.9 tps, lat 64.119 ms stddev 17.072 2024-03-12T05:13:49.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:49 smithi130 ceph-mon[35398]: pgmap v91: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:49.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:49 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:49.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:49 smithi116 ceph-mon[35486]: pgmap v91: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 19 MiB/s wr, 2.09k op/s 2024-03-12T05:13:49.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:49 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:51.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:51 smithi045 ceph-mon[32174]: pgmap v92: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:51.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:51 smithi130 ceph-mon[35398]: pgmap v92: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:51.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:51 smithi116 ceph-mon[35486]: pgmap v92: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:53 smithi045 ceph-mon[32174]: pgmap v93: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:53.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:53 smithi130 ceph-mon[35398]: pgmap v93: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:53.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:53 smithi116 ceph-mon[35486]: pgmap v93: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:13:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 110.0 s, 1518.6 tps, lat 65.933 ms stddev 16.468 2024-03-12T05:13:55.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:55 smithi045 ceph-mon[32174]: pgmap v94: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:55.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:55 smithi130 ceph-mon[35398]: pgmap v94: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:55.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:55 smithi116 ceph-mon[35486]: pgmap v94: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 18 MiB/s wr, 2.06k op/s 2024-03-12T05:13:57.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:57 smithi045 ceph-mon[32174]: pgmap v95: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:13:57.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:57 smithi130 ceph-mon[35398]: pgmap v95: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:13:57.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:57 smithi116 ceph-mon[35486]: pgmap v95: 97 pgs: 97 active+clean; 1.8 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:13:59.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:59 smithi045 ceph-mon[32174]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 2.09k op/s 2024-03-12T05:13:59.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:13:59 smithi045 ceph-mon[32174]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 115.0 s, 1549.2 tps, lat 64.491 ms stddev 18.403 2024-03-12T05:13:59.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:59 smithi130 ceph-mon[35398]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 2.09k op/s 2024-03-12T05:13:59.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:13:59 smithi130 ceph-mon[35398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:13:59.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:59 smithi116 ceph-mon[35486]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 2.09k op/s 2024-03-12T05:13:59.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:13:59 smithi116 ceph-mon[35486]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-12T05:14:00.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:00 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:14:00.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:14:00.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:14:00.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:00 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:14:00.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:00 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.098+0000 7f9c81b87640 1 -- 172.21.15.45:0/4194272313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 msgr2=0x7f9c74098740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.098+0000 7f9c81b87640 1 --2- 172.21.15.45:0/4194272313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74098740 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c066e50 tx=0x7f9c7c096e00 comp rx=0 tx=0).stop 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.098+0000 7f9c81b87640 1 -- 172.21.15.45:0/4194272313 shutdown_connections 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.098+0000 7f9c81b87640 1 --2- 172.21.15.45:0/4194272313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74098740 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.098+0000 7f9c81b87640 1 --2- 172.21.15.45:0/4194272313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c740065e0 0x7f9c74006a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.099+0000 7f9c81b87640 1 --2- 172.21.15.45:0/4194272313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c7409ac40 0x7f9c7409b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.099+0000 7f9c81b87640 1 -- 172.21.15.45:0/4194272313 >> 172.21.15.45:0/4194272313 conn(0x7f9c7400a6f0 msgr2=0x7f9c7400ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.099+0000 7f9c81b87640 1 -- 172.21.15.45:0/4194272313 shutdown_connections 2024-03-12T05:14:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.099+0000 7f9c81b87640 1 -- 172.21.15.45:0/4194272313 wait complete. 2024-03-12T05:14:01.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.100+0000 7f9c81b87640 1 Processor -- start 2024-03-12T05:14:01.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.101+0000 7f9c81b87640 1 -- start start 2024-03-12T05:14:01.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.101+0000 7f9c81b87640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 0x7f9c74132600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.101+0000 7f9c81b87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.101+0000 7f9c81b87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 0x7f9c74136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.101+0000 7f9c81b87640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c741374c0 con 0x7f9c74007060 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c81b87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74137630 con 0x7f9c7409ac40 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c81386640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 0x7f9c74136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c81386640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 0x7f9c74136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41128/0 (socket says 172.21.15.45:41128) 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c81386640 1 -- 172.21.15.45:0/3040087294 learned_addr learned my addr 172.21.15.45:0/3040087294 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c741377f0 con 0x7f9c740065e0 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 0x7f9c74132600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.102+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 -- 172.21.15.45:0/3040087294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 msgr2=0x7f9c74136ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 0x7f9c74136ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 -- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 msgr2=0x7f9c74132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 -- 172.21.15.45:0/3040087294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c04f750 con 0x7f9c740065e0 2024-03-12T05:14:01.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.104+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 0x7f9c74132600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000b3f0 tx=0x7f9c7000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.105+0000 7f9c79ffb640 1 -- 172.21.15.45:0/3040087294 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70011830 con 0x7f9c740065e0 2024-03-12T05:14:01.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.105+0000 7f9c79ffb640 1 -- 172.21.15.45:0/3040087294 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c70011e40 con 0x7f9c740065e0 2024-03-12T05:14:01.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.105+0000 7f9c79ffb640 1 -- 172.21.15.45:0/3040087294 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7000f2e0 con 0x7f9c740065e0 2024-03-12T05:14:01.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.105+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74137ad0 con 0x7f9c740065e0 2024-03-12T05:14:01.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.106+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7413f350 con 0x7f9c740065e0 2024-03-12T05:14:01.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.109+0000 7f9c79ffb640 1 -- 172.21.15.45:0/3040087294 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9c7000f440 con 0x7f9c740065e0 2024-03-12T05:14:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.109+0000 7f9c859c2740 1 Processor -- start 2024-03-12T05:14:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 -- start start 2024-03-12T05:14:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650ca5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 0x562650cae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650cae990 0x562650cb0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650c14280 con 0x562650ca5490 2024-03-12T05:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650c117b0 con 0x562650ca5db0 2024-03-12T05:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.110+0000 7f9c859c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650c0cf90 con 0x562650cae990 2024-03-12T05:14:01.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c79ffb640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 0x7f9c6c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c79ffb640 1 -- 172.21.15.45:0/3040087294 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9c700cd190 con 0x7f9c740065e0 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 0x562650cae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 0x562650cae450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41136/0 (socket says 172.21.15.45:41136) 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c80b85640 1 -- 172.21.15.45:0/2616894626 learned_addr learned my addr 172.21.15.45:0/2616894626 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c81386640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 0x7f9c6c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.111+0000 7f9c81386640 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650ca5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.112+0000 7f9c81386640 1 -- 172.21.15.45:0/2616894626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650cae990 msgr2=0x562650cb0d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:01.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.112+0000 7f9c81386640 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650cae990 0x562650cb0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.112+0000 7f9c81386640 1 -- 172.21.15.45:0/2616894626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 msgr2=0x562650cae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:01.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.112+0000 7f9c81386640 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 0x562650cae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.112+0000 7f9c81386640 1 -- 172.21.15.45:0/2616894626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562650bcb570 con 0x562650ca5490 2024-03-12T05:14:01.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.113+0000 7f9c81386640 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650ca5870 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c066a40 tx=0x7f9c7c067df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1b31ff4a760717 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.113+0000 7f9c81386640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 0x7f9c6c0a4270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c06d000 tx=0x7f9c7c0a8000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.114+0000 7f9c66ffd640 1 -- 172.21.15.45:0/2616894626 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0a4040 con 0x562650ca5490 2024-03-12T05:14:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.114+0000 7f9c66ffd640 1 -- 172.21.15.45:0/2616894626 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c7c067ab0 con 0x562650ca5490 2024-03-12T05:14:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.114+0000 7f9c66ffd640 1 -- 172.21.15.45:0/2616894626 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c074cf0 con 0x562650ca5490 2024-03-12T05:14:01.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 -- 172.21.15.45:0/2616894626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 msgr2=0x562650ca5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650ca5870 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c066a40 tx=0x7f9c7c067df0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 -- 172.21.15.45:0/2616894626 shutdown_connections 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650cae990 0x562650cb0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5db0 0x562650cae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2616894626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650ca5870 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 -- 172.21.15.45:0/2616894626 >> 172.21.15.45:0/2616894626 conn(0x562650c9fbc0 msgr2=0x562650c9ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 -- 172.21.15.45:0/2616894626 shutdown_connections 2024-03-12T05:14:01.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.115+0000 7f9c859c2740 1 -- 172.21.15.45:0/2616894626 wait complete. 2024-03-12T05:14:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.116+0000 7f9c859c2740 1 -- start start 2024-03-12T05:14:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.116+0000 7f9c859c2740 1 Processor -- start 2024-03-12T05:14:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.117+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.117+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cab750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.117+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650cabe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41148/0 (socket says 172.21.15.45:41148) 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 -- 172.21.15.45:0/3115368418 learned_addr learned my addr 172.21.15.45:0/3115368418 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626508e0430 con 0x562650cae990 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265089ce00 con 0x562650ca5490 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650b91fd0 con 0x562650ca5db0 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c859c2740 10 client.0 _ll_register_callbacks cb 0x562650be7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 -- 172.21.15.45:0/3115368418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 msgr2=0x562650cab750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cab750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 -- 172.21.15.45:0/3115368418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 msgr2=0x562650cabe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650cabe90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.118+0000 7f9c80b85640 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c740992d0 con 0x562650ca5490 2024-03-12T05:14:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.119+0000 7f9c80b85640 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000b3c0 tx=0x7f9c70097150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eefc5d2c38ba062 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.119+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56265090c7e0 con 0x562650ca5490 2024-03-12T05:14:01.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c5bfff640 10 client.25778 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:01.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c5bfff640 1 -- 172.21.15.45:0/3115368418 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7009adf0 con 0x562650ca5490 2024-03-12T05:14:01.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c5bfff640 1 -- 172.21.15.45:0/3115368418 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9c700aee40 con 0x562650ca5490 2024-03-12T05:14:01.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c5bfff640 1 -- 172.21.15.45:0/3115368418 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7009ef40 con 0x562650ca5490 2024-03-12T05:14:01.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c859c2740 10 client.25778 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:01.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.120+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x562650c0b610 con 0x562650ca5490 2024-03-12T05:14:01.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c5bfff640 1 -- 172.21.15.45:0/3115368418 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c70097460 con 0x562650ca5490 2024-03-12T05:14:01.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 10 client.25778 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:01.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 10 client.25778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 20 client.25778 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650ccf9a0 0x562650cd1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 4 client.25778 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.122+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562650949720 con 0x562650ccf9a0 2024-03-12T05:14:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.124+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650ccf9a0 0x562650cd1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.124+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650ccf9a0 0x562650cd1e00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d87264ca1aa7eee6 server_cookie=e08ccfa676f9b073 in_seq=0 out_seq=0 2024-03-12T05:14:01.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.124+0000 7f9c5bfff640 10 client.25778 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:01.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.125+0000 7f9c5bfff640 1 -- 172.21.15.45:0/3115368418 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562650949720 con 0x562650ccf9a0 2024-03-12T05:14:01.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.125+0000 7f9c5bfff640 10 client.25778 handle_command_reply: tid=0 2024-03-12T05:14:01.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.125+0000 7f9c859c2740 1 client.25778 shutdown 2024-03-12T05:14:01.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 msgr2=0x562650cae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000b3c0 tx=0x7f9c70097150 comp rx=0 tx=0).stop 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 shutdown_connections 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650ccf9a0 0x562650cd1e00 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650cabe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cab750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 --2- 172.21.15.45:0/3115368418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650cae3c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 >> 172.21.15.45:0/3115368418 conn(0x562650c9fbc0 msgr2=0x562650ca1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c5bfff640 0 client.25778 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 shutdown_connections 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.127+0000 7f9c859c2740 1 -- 172.21.15.45:0/3115368418 wait complete. 2024-03-12T05:14:01.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.128+0000 7f9c859c2740 20 client.25778 trim_cache size 0 max 16384 2024-03-12T05:14:01.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.169+0000 7f9c859c2740 1 Processor -- start 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.169+0000 7f9c859c2740 1 -- start start 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.169+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650cade80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 0x562650caa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c859c2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650ca9ce0 con 0x562650ca5490 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c859c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650ca5ba0 con 0x562650cae990 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c859c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650c9b230 con 0x562650ca5db0 2024-03-12T05:14:01.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c81386640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 0x562650caa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c81386640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 0x562650caa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41160/0 (socket says 172.21.15.45:41160) 2024-03-12T05:14:01.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c81386640 1 -- 172.21.15.45:0/2039859713 learned_addr learned my addr 172.21.15.45:0/2039859713 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.170+0000 7f9c80b85640 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650cade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.172+0000 7f9c80b85640 1 -- 172.21.15.45:0/2039859713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 msgr2=0x562650cae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.172+0000 7f9c80b85640 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.172+0000 7f9c80b85640 1 -- 172.21.15.45:0/2039859713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 msgr2=0x562650caa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.172+0000 7f9c80b85640 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 0x562650caa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.172+0000 7f9c80b85640 1 -- 172.21.15.45:0/2039859713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562650ba7ac0 con 0x562650ca5490 2024-03-12T05:14:01.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.174+0000 7f9c80b85640 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650cade80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9c70006e10 tx=0x7f9c7001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1125482f08eee39 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.174+0000 7f9c667fc640 1 -- 172.21.15.45:0/2039859713 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7000d040 con 0x562650ca5490 2024-03-12T05:14:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.174+0000 7f9c667fc640 1 -- 172.21.15.45:0/2039859713 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56265090c7e0 con 0x562650ca5490 2024-03-12T05:14:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.174+0000 7f9c667fc640 1 -- 172.21.15.45:0/2039859713 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c700cab10 con 0x562650ca5490 2024-03-12T05:14:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 -- 172.21.15.45:0/2039859713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 msgr2=0x562650cade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650cade80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9c70006e10 tx=0x7f9c7001b000 comp rx=0 tx=0).stop 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 -- 172.21.15.45:0/2039859713 shutdown_connections 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650cae990 0x562650caa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650cae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 --2- 172.21.15.45:0/2039859713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650ca5490 0x562650cade80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.176+0000 7f9c859c2740 1 -- 172.21.15.45:0/2039859713 >> 172.21.15.45:0/2039859713 conn(0x562650c9fbc0 msgr2=0x562650ca6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.177+0000 7f9c859c2740 1 -- 172.21.15.45:0/2039859713 shutdown_connections 2024-03-12T05:14:01.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.177+0000 7f9c859c2740 1 -- 172.21.15.45:0/2039859713 wait complete. 2024-03-12T05:14:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.177+0000 7f9c859c2740 1 -- start start 2024-03-12T05:14:01.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.177+0000 7f9c859c2740 1 Processor -- start 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.178+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.178+0000 7f9c859c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650ca9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.178+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.178+0000 7f9c80b85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41174/0 (socket says 172.21.15.45:41174) 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.178+0000 7f9c80b85640 1 -- 172.21.15.45:0/826342594 learned_addr learned my addr 172.21.15.45:0/826342594 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650ca9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650d25f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650d26460 con 0x562650cae990 2024-03-12T05:14:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650d265a0 con 0x562650ca5490 2024-03-12T05:14:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562650d266e0 con 0x562650ca5db0 2024-03-12T05:14:01.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.179+0000 7f9c859c2740 10 client.0 _ll_register_callbacks cb 0x7f9c7c069620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c81386640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650d25f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 -- 172.21.15.45:0/826342594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 msgr2=0x562650ca9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650ca9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 -- 172.21.15.45:0/826342594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 msgr2=0x562650d25f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650d25f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562650c0b610 con 0x562650ca5490 2024-03-12T05:14:01.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.180+0000 7f9c80b85640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c700a3060 tx=0x7f9c700047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fec5ea456c1b200f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.183+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650ca9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.184+0000 7f9c5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.184+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7009a750 con 0x562650ca5490 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.184+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9c700aee40 con 0x562650ca5490 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.184+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c7c067ab0 con 0x562650ca5490 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.184+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7009adf0 con 0x562650ca5490 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c859c2740 10 client.25784 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562650bcd6a0 con 0x562650ca5490 2024-03-12T05:14:01.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9c70089080 con 0x562650ca5490 2024-03-12T05:14:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c70096b40 con 0x562650ca5490 2024-03-12T05:14:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c859c2740 10 client.25784 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.185+0000 7f9c859c2740 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.186+0000 7f9c859c2740 20 client.25784 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:01.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.186+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650d27990 0x562650d29d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.186+0000 7f9c859c2740 4 client.25784 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:14:01.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.186+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562650998890 con 0x562650d27990 2024-03-12T05:14:01.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.186+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650d27990 0x562650d29d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.187+0000 7f9c7bfff640 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650d27990 0x562650d29d80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2837eae73d5ec853 server_cookie=356a93be7a69eaa8 in_seq=0 out_seq=0 2024-03-12T05:14:01.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.187+0000 7f9c5bfff640 10 client.25784 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:01.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.189+0000 7f9c5bfff640 1 -- 172.21.15.45:0/826342594 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562650998890 con 0x562650d27990 2024-03-12T05:14:01.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.189+0000 7f9c5bfff640 10 client.25784 handle_command_reply: tid=0 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.189+0000 7f9c859c2740 1 client.25784 shutdown 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.190+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 msgr2=0x562650ca8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.190+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c700a3060 tx=0x7f9c700047d0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 shutdown_connections 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562650d27990 0x562650d29d80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562650cae990 0x562650d25f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562650ca5db0 0x562650ca9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 --2- 172.21.15.45:0/826342594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562650ca5490 0x562650ca8b50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 >> 172.21.15.45:0/826342594 conn(0x562650c9fbc0 msgr2=0x562650caafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.191+0000 7f9c5bfff640 0 client.25784 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:01.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.192+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 shutdown_connections 2024-03-12T05:14:01.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.195+0000 7f9c859c2740 1 -- 172.21.15.45:0/826342594 wait complete. 2024-03-12T05:14:01.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.195+0000 7f9c859c2740 20 client.25784 trim_cache size 0 max 16384 2024-03-12T05:14:01.195 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:14:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:14:01.195 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.205+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 msgr2=0x7f9c6c0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.205+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 0x7f9c6c0a4270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c06d000 tx=0x7f9c7c0a8000 comp rx=0 tx=0).stop 2024-03-12T05:14:01.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.205+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 msgr2=0x7f9c74132600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.205+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 0x7f9c74132600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000b3f0 tx=0x7f9c7000b8b0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 shutdown_connections 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9c6c0a1db0 0x7f9c6c0a4270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c74007060 0x7f9c74132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c7409ac40 0x7f9c74136ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 --2- 172.21.15.45:0/3040087294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c740065e0 0x7f9c74132600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.206+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 >> 172.21.15.45:0/3040087294 conn(0x7f9c7400a6f0 msgr2=0x7f9c740928d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.209+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 shutdown_connections 2024-03-12T05:14:01.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.210+0000 7f9c81b87640 1 -- 172.21.15.45:0/3040087294 wait complete. 2024-03-12T05:14:01.231 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:14:01.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:14:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.434+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3745577716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc0105390 msgr2=0x7f3cc010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.434+0000 7f3cc48eb640 1 --2- 172.21.15.45:0/3745577716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc0105390 0x7f3cc010cf60 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009cd0 tx=0x7f3cb802c0e0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3745577716 shutdown_connections 2024-03-12T05:14:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 --2- 172.21.15.45:0/3745577716 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc0105390 0x7f3cc010cf60 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 --2- 172.21.15.45:0/3745577716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0100c70 0x7f3cc0104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 --2- 172.21.15.45:0/3745577716 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc01002c0 0x7f3cc01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3745577716 >> 172.21.15.45:0/3745577716 conn(0x7f3cc0076400 msgr2=0x7f3cc0076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.436+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3745577716 shutdown_connections 2024-03-12T05:14:01.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.438+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3745577716 wait complete. 2024-03-12T05:14:01.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.439+0000 7f3cc48eb640 1 Processor -- start 2024-03-12T05:14:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 -- start start 2024-03-12T05:14:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 0x7f3cc0072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 0x7f3cc006b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 0x7f3cc006c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc006c790 con 0x7f3cc01002c0 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0070490 con 0x7f3cc0105390 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cc48eb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc0070650 con 0x7f3cc0072fc0 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cbdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 0x7f3cc006b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cbe59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 0x7f3cc0072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cbe59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 0x7f3cc0072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39216/0 (socket says 172.21.15.45:39216) 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.440+0000 7f3cbe59c640 1 -- 172.21.15.45:0/3945899841 learned_addr learned my addr 172.21.15.45:0/3945899841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbed9d640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 0x7f3cc006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 -- 172.21.15.45:0/3945899841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 msgr2=0x7f3cc006c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 0x7f3cc006c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 -- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 msgr2=0x7f3cc0072970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 0x7f3cc0072970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 -- 172.21.15.45:0/3945899841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8009900 con 0x7f3cc0105390 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbdd9b640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 0x7f3cc006b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e9c0 tx=0x7f3cb000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000ccc0 con 0x7f3cc0105390 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3945899841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc0070930 con 0x7f3cc0105390 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3cb000ce20 con 0x7f3cc0105390 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.442+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0018780 con 0x7f3cc0105390 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.441+0000 7f3cbed9d640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 0x7f3cc006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:01.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.442+0000 7f3cc48eb640 1 -- 172.21.15.45:0/3945899841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc0070dd0 con 0x7f3cc0105390 2024-03-12T05:14:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.444+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc0068fe0 con 0x7f3cc0105390 2024-03-12T05:14:01.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.445+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3cb00188e0 con 0x7f3cc0105390 2024-03-12T05:14:01.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.446+0000 7f3caf7fe640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 0x7f3ca80a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:01.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.446+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3cb0014070 con 0x7f3cc0105390 2024-03-12T05:14:01.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.446+0000 7f3cbe59c640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 0x7f3ca80a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:01.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.447+0000 7f3cbe59c640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 0x7f3ca80a41a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40058c0 tx=0x7f3cb400a380 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:01.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.451+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3cb0096c40 con 0x7f3cc0105390 2024-03-12T05:14:01.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:01 smithi045 ceph-mon[32174]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.891+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cc0101790 con 0x7f3cc0105390 2024-03-12T05:14:01.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.892+0000 7f3caf7fe640 1 -- 172.21.15.45:0/3945899841 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f3cb009a300 con 0x7f3cc0105390 2024-03-12T05:14:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:14:01.889 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:14:01.889 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 msgr2=0x7f3ca80a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 0x7f3ca80a41a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40058c0 tx=0x7f3cb400a380 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 msgr2=0x7f3cc006b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 0x7f3cc006b9f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e9c0 tx=0x7f3cb000ee80 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 shutdown_connections 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3ca80a1ce0 0x7f3ca80a41a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cc0072fc0 0x7f3cc006c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3cc0105390 0x7f3cc006b9f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 --2- 172.21.15.45:0/3945899841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc01002c0 0x7f3cc0072970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:01.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.896+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 >> 172.21.15.45:0/3945899841 conn(0x7f3cc0076400 msgr2=0x7f3cc00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:01.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.897+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 shutdown_connections 2024-03-12T05:14:01.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:01.897+0000 7f3cad7fa640 1 -- 172.21.15.45:0/3945899841 wait complete. 2024-03-12T05:14:01.910 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:01.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:14:01.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:01 smithi130 ceph-mon[35398]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:01.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:01 smithi116 ceph-mon[35486]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 -- 172.21.15.45:0/2896084274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 msgr2=0x7f2888100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 --2- 172.21.15.45:0/2896084274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888100640 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2878009ca0 tx=0x7f287802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 -- 172.21.15.45:0/2896084274 shutdown_connections 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 --2- 172.21.15.45:0/2896084274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f288810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 --2- 172.21.15.45:0/2896084274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2888100c10 0x7f2888104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 --2- 172.21.15.45:0/2896084274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888100640 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 -- 172.21.15.45:0/2896084274 >> 172.21.15.45:0/2896084274 conn(0x7f2888077180 msgr2=0x7f2888077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.085+0000 7f288d0ae640 1 -- 172.21.15.45:0/2896084274 shutdown_connections 2024-03-12T05:14:02.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.086+0000 7f288d0ae640 1 -- 172.21.15.45:0/2896084274 wait complete. 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.086+0000 7f288d0ae640 1 Processor -- start 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 -- start start 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f2888199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f2886d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f2886d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41196/0 (socket says 172.21.15.45:41196) 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f2886d9d640 1 -- 172.21.15.45:0/1946207011 learned_addr learned my addr 172.21.15.45:0/1946207011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 0x7f288819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288819e9f0 con 0x7f2888105330 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288819eb60 con 0x7f2888100260 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288819ecd0 con 0x7f288819e000 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f287ffff640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f2888199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.087+0000 7f288759e640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 0x7f288819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 -- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 msgr2=0x7f288819e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 0x7f288819e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 -- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 msgr2=0x7f2888199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f2888199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2878009900 con 0x7f2888100260 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f288759e640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 0x7f288819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f287ffff640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f2888199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2886d9d640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2878004af0 tx=0x7f2878035ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2884d99640 1 -- 172.21.15.45:0/1946207011 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878041e30 con 0x7f2888100260 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2884d99640 1 -- 172.21.15.45:0/1946207011 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2878035e90 con 0x7f2888100260 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.088+0000 7f2884d99640 1 -- 172.21.15.45:0/1946207011 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780042a0 con 0x7f2888100260 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.089+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288810f790 con 0x7f2888100260 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.089+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288810fb90 con 0x7f2888100260 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.092+0000 7f288fc82740 1 Processor -- start 2024-03-12T05:14:02.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.092+0000 7f288fc82740 1 -- start start 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.092+0000 7f288fc82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.092+0000 7f288fc82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba48210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288fc82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654aba4ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288fc82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab9fa960 con 0x5654aba3fb70 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288fc82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab9ae310 con 0x5654aba48750 2024-03-12T05:14:02.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288fc82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab9ab840 con 0x5654aba3f250 2024-03-12T05:14:02.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54054/0 (socket says 172.21.15.45:54054) 2024-03-12T05:14:02.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 -- 172.21.15.45:0/248970904 learned_addr learned my addr 172.21.15.45:0/248970904 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f287ffff640 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654aba4ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 -- 172.21.15.45:0/248970904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 msgr2=0x5654aba4ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654aba4ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 -- 172.21.15.45:0/248970904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 msgr2=0x5654aba48210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba48210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 -- 172.21.15.45:0/248970904 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654ab969060 con 0x5654aba3f250 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f2884d99640 1 -- 172.21.15.45:0/1946207011 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f287800a040 con 0x7f2888100260 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.093+0000 7f288759e640 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2880011440 tx=0x7f2880011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f921bd61c7adb603 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.094+0000 7f287d7fa640 1 -- 172.21.15.45:0/248970904 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880004020 con 0x5654aba3f250 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.094+0000 7f287d7fa640 1 -- 172.21.15.45:0/248970904 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28800035f0 con 0x5654aba3f250 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.094+0000 7f287d7fa640 1 -- 172.21.15.45:0/248970904 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288000f7a0 con 0x5654aba3f250 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.094+0000 7f2884d99640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 0x7f28740a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.094+0000 7f2884d99640 1 -- 172.21.15.45:0/1946207011 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f28780f7f10 con 0x7f2888100260 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f2886d9d640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 0x7f28740a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- 172.21.15.45:0/248970904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 msgr2=0x5654aba3f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2880011440 tx=0x7f2880011900 comp rx=0 tx=0).stop 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- 172.21.15.45:0/248970904 shutdown_connections 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba48210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654aba4ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 --2- 172.21.15.45:0/248970904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba3f630 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- 172.21.15.45:0/248970904 >> 172.21.15.45:0/248970904 conn(0x5654aba39980 msgr2=0x5654aba39d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- 172.21.15.45:0/248970904 shutdown_connections 2024-03-12T05:14:02.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- 172.21.15.45:0/248970904 wait complete. 2024-03-12T05:14:02.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.095+0000 7f288fc82740 1 -- start start 2024-03-12T05:14:02.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.096+0000 7f288fc82740 1 Processor -- start 2024-03-12T05:14:02.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.096+0000 7f2886d9d640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 0x7f28740a4120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5654aba3ed20 tx=0x7f287802c0f0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.096+0000 7f288fc82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.096+0000 7f288fc82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba45d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288fc82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba48750 0x5654aba46250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f287ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba45d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41220/0 (socket says 172.21.15.45:41220) 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f287ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba45d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39230/0 (socket says 172.21.15.45:39230) 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 -- 172.21.15.45:0/3961831245 learned_addr learned my addr 172.21.15.45:0/3961831245 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab9a7020 con 0x5654aba3fb70 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab67a430 con 0x5654aba3f250 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab636e00 con 0x5654aba48750 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288fc82740 10 client.0 _ll_register_callbacks cb 0x5654ab94ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba48750 0x5654aba46250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 -- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba48750 msgr2=0x5654aba46250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba48750 0x5654aba46250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 -- 172.21.15.45:0/3961831245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 msgr2=0x5654aba45d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba45d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.097+0000 7f288759e640 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2880009cc0 con 0x5654aba3f250 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f288759e640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2880009380 tx=0x7f2880009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6dcebf748b7745 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f286e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880005480 con 0x5654aba3f250 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f28800055e0 con 0x5654aba3f250 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654ab6a67e0 con 0x5654aba3f250 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28800046c0 con 0x5654aba3f250 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.098+0000 7f288fc82740 10 client.25802 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.099+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5654ab9a56a0 con 0x5654aba3f250 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2880034020 con 0x5654aba3f250 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f288000f8a0 con 0x5654aba3f250 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 10 client.25802 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 10 client.25802 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 20 client.25802 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654aba58af0 0x5654aba5af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 4 client.25802 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:02.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654ab6e3720 con 0x5654aba58af0 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.100+0000 7f287ffff640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654aba58af0 0x5654aba5af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.101+0000 7f287ffff640 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654aba58af0 0x5654aba5af10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db647c7bd6c2e160 server_cookie=b5ba193f42b99274 in_seq=0 out_seq=0 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.101+0000 7f286e7fc640 10 client.25802 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.101+0000 7f286e7fc640 1 -- 172.21.15.45:0/3961831245 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5654ab6e3720 con 0x5654aba58af0 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.101+0000 7f286e7fc640 10 client.25802 handle_command_reply: tid=0 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 client.25802 shutdown 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 msgr2=0x5654aba457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2880009380 tx=0x7f2880009310 comp rx=0 tx=0).stop 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 shutdown_connections 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654aba58af0 0x5654aba5af10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba45d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba48750 0x5654aba46250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 --2- 172.21.15.45:0/3961831245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba457d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f286e7fc640 0 client.25802 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:02.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 >> 172.21.15.45:0/3961831245 conn(0x5654aba39980 msgr2=0x5654aba3b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 shutdown_connections 2024-03-12T05:14:02.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 1 -- 172.21.15.45:0/3961831245 wait complete. 2024-03-12T05:14:02.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.102+0000 7f288fc82740 20 client.25802 trim_cache size 0 max 16384 2024-03-12T05:14:02.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.127+0000 7f288fc82740 1 Processor -- start 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.127+0000 7f288fc82740 1 -- start start 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3fb70 0x5654aba42a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba48750 0x5654aba42f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ab92c130 con 0x5654aba48750 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654aba3e9f0 con 0x5654aba3f250 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288fc82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654aba588f0 con 0x5654aba3fb70 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41228/0 (socket says 172.21.15.45:41228) 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 -- 172.21.15.45:0/56897130 learned_addr learned my addr 172.21.15.45:0/56897130 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f287ffff640 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3fb70 0x5654aba42a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 -- 172.21.15.45:0/56897130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3fb70 msgr2=0x5654aba42a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3fb70 0x5654aba42a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 -- 172.21.15.45:0/56897130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba48750 msgr2=0x5654aba42f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba48750 0x5654aba42f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.128+0000 7f288759e640 1 -- 172.21.15.45:0/56897130 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654ab941c20 con 0x5654aba3f250 2024-03-12T05:14:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.129+0000 7f288759e640 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2880001eb0 tx=0x7f2880009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fb473d4e171d985 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.129+0000 7f287cff9640 1 -- 172.21.15.45:0/56897130 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880003bb0 con 0x5654aba3f250 2024-03-12T05:14:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.129+0000 7f287cff9640 1 -- 172.21.15.45:0/56897130 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2880034040 con 0x5654aba3f250 2024-03-12T05:14:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.129+0000 7f287cff9640 1 -- 172.21.15.45:0/56897130 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880030570 con 0x5654aba3f250 2024-03-12T05:14:02.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 -- 172.21.15.45:0/56897130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 msgr2=0x5654aba47f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2880001eb0 tx=0x7f2880009c20 comp rx=0 tx=0).stop 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 -- 172.21.15.45:0/56897130 shutdown_connections 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba48750 0x5654aba42f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3fb70 0x5654aba42a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 --2- 172.21.15.45:0/56897130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba3f250 0x5654aba47f00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 -- 172.21.15.45:0/56897130 >> 172.21.15.45:0/56897130 conn(0x5654aba39980 msgr2=0x5654aba40b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 -- 172.21.15.45:0/56897130 shutdown_connections 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.130+0000 7f288fc82740 1 -- 172.21.15.45:0/56897130 wait complete. 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 -- start start 2024-03-12T05:14:02.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 Processor -- start 2024-03-12T05:14:02.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba4e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba4ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654abac0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654aba4f390 con 0x5654aba3fb70 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654abac0550 con 0x5654aba48750 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.131+0000 7f288fc82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654abac0710 con 0x5654aba3f250 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288fc82740 10 client.0 _ll_register_callbacks cb 0x7f2880009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba4e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba4e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54086/0 (socket says 172.21.15.45:54086) 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 -- 172.21.15.45:0/3199787939 learned_addr learned my addr 172.21.15.45:0/3199787939 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654abac0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f287ffff640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba4ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 -- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 msgr2=0x5654aba4e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba4e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 -- 172.21.15.45:0/3199787939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 msgr2=0x5654aba4ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba4ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 -- 172.21.15.45:0/3199787939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654ab9a56a0 con 0x5654aba48750 2024-03-12T05:14:02.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f288759e640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654abac0010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28800043b0 tx=0x7f288000fe50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=109e8fb5715fbdaa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f286e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:02.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.132+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288000f540 con 0x5654aba48750 2024-03-12T05:14:02.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.133+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28800035f0 con 0x5654aba48750 2024-03-12T05:14:02.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.133+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f2880000e40 con 0x5654aba48750 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f288000f6a0 con 0x5654aba48750 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 10 client.25808 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5654ab9678c0 con 0x5654aba48750 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2880042070 con 0x5654aba48750 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2880030b80 con 0x5654aba48750 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 10 client.25808 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 10 client.25808 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 20 client.25808 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654abac1a00 0x5654abac3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 4 client.25808 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.134+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f28800035f0 con 0x5654abac1a00 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.135+0000 7f287ffff640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654abac1a00 0x5654abac3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f287ffff640 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654abac1a00 0x5654abac3e20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37a39a2f4d319c9a server_cookie=fc499c99986de770 in_seq=0 out_seq=0 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f286e7fc640 10 client.25808 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f286e7fc640 1 -- 172.21.15.45:0/3199787939 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f28800035f0 con 0x5654abac1a00 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f286e7fc640 10 client.25808 handle_command_reply: tid=0 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f288fc82740 1 client.25808 shutdown 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 msgr2=0x5654abac0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654abac0010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28800043b0 tx=0x7f288000fe50 comp rx=0 tx=0).stop 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 shutdown_connections 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.136+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5654abac1a00 0x5654abac3e20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654aba3fb70 0x5654aba4ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654aba48750 0x5654abac0010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 --2- 172.21.15.45:0/3199787939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654aba3f250 0x5654aba4e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 >> 172.21.15.45:0/3199787939 conn(0x5654aba39980 msgr2=0x5654aba45790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f286e7fc640 0 client.25808 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 shutdown_connections 2024-03-12T05:14:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 1 -- 172.21.15.45:0/3199787939 wait complete. 2024-03-12T05:14:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.137+0000 7f288fc82740 20 client.25808 trim_cache size 0 max 16384 2024-03-12T05:14:02.136 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:02.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.138+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 msgr2=0x7f28740a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.138+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 0x7f28740a4120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5654aba3ed20 tx=0x7f287802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.138+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 msgr2=0x7f2888199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.138+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2878004af0 tx=0x7f2878035ca0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 shutdown_connections 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f28740a1c60 0x7f28740a4120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2888105330 0x7f2888199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f288819e000 0x7f288819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 --2- 172.21.15.45:0/1946207011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2888100260 0x7f2888199790 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.139+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 >> 172.21.15.45:0/1946207011 conn(0x7f2888077180 msgr2=0x7f28880fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.140+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 shutdown_connections 2024-03-12T05:14:02.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.140+0000 7f288d0ae640 1 -- 172.21.15.45:0/1946207011 wait complete. 2024-03-12T05:14:02.166 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:02.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:14:02.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.348+0000 7f7f8559a640 1 -- 172.21.15.45:0/1258814262 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800329d0 con 0x7f7f88106430 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.348+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/1258814262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f88106430 msgr2=0x7f7f8810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.348+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/1258814262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f88106430 0x7f7f8810e000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009cd0 tx=0x7f7f8002c0e0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/1258814262 shutdown_connections 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/1258814262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f88106430 0x7f7f8810e000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/1258814262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88068f80 0x7f7f88105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/1258814262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f880685d0 0x7f7f880689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/1258814262 >> 172.21.15.45:0/1258814262 conn(0x7f7f88077180 msgr2=0x7f7f88077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/1258814262 shutdown_connections 2024-03-12T05:14:02.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.349+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/1258814262 wait complete. 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 Processor -- start 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 -- start start 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 0x7f7f881a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 0x7f7f8810c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8810c8e0 con 0x7f7f8810bef0 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8810ca50 con 0x7f7f880685d0 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8ccc0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8810cc10 con 0x7f7f88106430 2024-03-12T05:14:02.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41256/0 (socket says 172.21.15.45:41256) 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f8659c640 1 -- 172.21.15.45:0/476236880 learned_addr learned my addr 172.21.15.45:0/476236880 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 0x7f7f8810c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.350+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 0x7f7f881a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 -- 172.21.15.45:0/476236880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 msgr2=0x7f7f881a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 -- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 msgr2=0x7f7f8810c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 0x7f7f8810c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 -- 172.21.15.45:0/476236880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80009900 con 0x7f7f88106430 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 0x7f7f8810c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f8659c640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 0x7f7f881a3e70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00d6a0 tx=0x7f7f7c00db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f777fe640 1 -- 172.21.15.45:0/476236880 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c005220 con 0x7f7f88106430 2024-03-12T05:14:02.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f777fe640 1 -- 172.21.15.45:0/476236880 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f7c004c50 con 0x7f7f88106430 2024-03-12T05:14:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f881aa7f0 con 0x7f7f88106430 2024-03-12T05:14:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.351+0000 7f7f777fe640 1 -- 172.21.15.45:0/476236880 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c009cf0 con 0x7f7f88106430 2024-03-12T05:14:02.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.352+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f881aad50 con 0x7f7f88106430 2024-03-12T05:14:02.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.355+0000 7f7f777fe640 1 -- 172.21.15.45:0/476236880 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7f7c0047a0 con 0x7f7f88106430 2024-03-12T05:14:02.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.355+0000 7f7f8f96e740 1 Processor -- start 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.356+0000 7f7f8f96e740 1 -- start start 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.356+0000 7f7f777fe640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 0x7f7f700a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.356+0000 7f7f777fe640 1 -- 172.21.15.45:0/476236880 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7f7c0cc660 con 0x7f7f88106430 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 0x7f7f700a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fd8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fe3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720f93960 con 0x559720fd8250 2024-03-12T05:14:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720f47310 con 0x559720fd8b70 2024-03-12T05:14:02.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720f44840 con 0x559720fe1750 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fd8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f85d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f85d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41258/0 (socket says 172.21.15.45:41258) 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.357+0000 7f7f85d9b640 1 -- 172.21.15.45:0/384252480 learned_addr learned my addr 172.21.15.45:0/384252480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 0x7f7f700a4120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f80006010 tx=0x7f7f80032500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fe3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 -- 172.21.15.45:0/384252480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 msgr2=0x559720fe3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fe3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 -- 172.21.15.45:0/384252480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 msgr2=0x559720fd8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fd8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 -- 172.21.15.45:0/384252480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559720f02060 con 0x559720fd8b70 2024-03-12T05:14:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c095d10 tx=0x7f7f7c099340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc0472671fe5b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f74ff9640 1 -- 172.21.15.45:0/384252480 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0a1a30 con 0x559720fd8b70 2024-03-12T05:14:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f74ff9640 1 -- 172.21.15.45:0/384252480 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f7c004dc0 con 0x559720fd8b70 2024-03-12T05:14:02.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.358+0000 7f7f74ff9640 1 -- 172.21.15.45:0/384252480 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0959b0 con 0x559720fd8b70 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.359+0000 7f7f8f96e740 1 -- 172.21.15.45:0/384252480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 msgr2=0x559720fe1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.359+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c095d10 tx=0x7f7f7c099340 comp rx=0 tx=0).stop 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.359+0000 7f7f8f96e740 1 -- 172.21.15.45:0/384252480 shutdown_connections 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fe3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fe1210 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/384252480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fd8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 -- 172.21.15.45:0/384252480 >> 172.21.15.45:0/384252480 conn(0x559720fd2980 msgr2=0x559720fd2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 -- 172.21.15.45:0/384252480 shutdown_connections 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 -- 172.21.15.45:0/384252480 wait complete. 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.360+0000 7f7f8f96e740 1 -- start start 2024-03-12T05:14:02.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.361+0000 7f7f8f96e740 1 Processor -- start 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.361+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 0x559720fde7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.361+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.361+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fdf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f85d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f85d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39268/0 (socket says 172.21.15.45:39268) 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f85d9b640 1 -- 172.21.15.45:0/952258496 learned_addr learned my addr 172.21.15.45:0/952258496 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f8659c640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 0x559720fde7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fdf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720f40020 con 0x559720fd8b70 2024-03-12T05:14:02.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720c13430 con 0x559720fe1750 2024-03-12T05:14:02.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.362+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720bcfe00 con 0x559720fd8250 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.363+0000 7f7f8f96e740 10 client.0 _ll_register_callbacks cb 0x559720ee7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 -- 172.21.15.45:0/952258496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 msgr2=0x559720fde7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 0x559720fde7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 -- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 msgr2=0x559720fdf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fdf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f881aad50 con 0x559720fd8b70 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f8659c640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 0x559720fde7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c099040 tx=0x7f7f7c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3032e5affa43c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0959b0 con 0x559720fd8b70 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559720c3f7e0 con 0x559720fd8b70 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.364+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7f7c095b10 con 0x559720fd8b70 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f7c098790 con 0x559720fd8b70 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f8f96e740 10 client.16356 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559720f3e6a0 con 0x559720fd8b70 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559720c3f7e0 con 0x559720fd8b70 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7f7c04f080 con 0x559720fd8b70 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f8f96e740 10 client.16356 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.365+0000 7f7f8f96e740 10 client.16356 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.366+0000 7f7f8f96e740 20 client.16356 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.366+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559720ff15a0 0x559720ff39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.366+0000 7f7f8f96e740 4 client.16356 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.366+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559720c7c720 con 0x559720ff15a0 2024-03-12T05:14:02.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.366+0000 7f7f8659c640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559720ff15a0 0x559720ff39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.367+0000 7f7f8659c640 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559720ff15a0 0x559720ff39c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6822bd7761009e1b server_cookie=a76498555d0a4076 in_seq=0 out_seq=0 2024-03-12T05:14:02.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.367+0000 7f7f6dffb640 10 client.16356 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:02.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.368+0000 7f7f6dffb640 1 -- 172.21.15.45:0/952258496 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559720c7c720 con 0x559720ff15a0 2024-03-12T05:14:02.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.368+0000 7f7f6dffb640 10 client.16356 handle_command_reply: tid=0 2024-03-12T05:14:02.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.368+0000 7f7f8f96e740 1 client.16356 shutdown 2024-03-12T05:14:02.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.368+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 msgr2=0x559720fded20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.368+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c099040 tx=0x7f7f7c0056d0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 shutdown_connections 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559720ff15a0 0x559720ff39c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8b70 0x559720fded20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fdf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/952258496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8250 0x559720fde7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 >> 172.21.15.45:0/952258496 conn(0x559720fd2980 msgr2=0x559720fd46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f6dffb640 0 client.16356 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:02.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 shutdown_connections 2024-03-12T05:14:02.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 1 -- 172.21.15.45:0/952258496 wait complete. 2024-03-12T05:14:02.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.371+0000 7f7f8f96e740 20 client.16356 trim_cache size 0 max 16384 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.394+0000 7f7f8f96e740 1 Processor -- start 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.394+0000 7f7f8f96e740 1 -- start start 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fdf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fdba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720ec5130 con 0x559720fd8250 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720fed610 con 0x559720fd8b70 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8f96e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720ff8c10 con 0x559720fe1750 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39284/0 (socket says 172.21.15.45:39284) 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f8659c640 1 -- 172.21.15.45:0/1991646007 learned_addr learned my addr 172.21.15.45:0/1991646007 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fdba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fdf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 -- 172.21.15.45:0/1991646007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 msgr2=0x559720fdf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fdf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 -- 172.21.15.45:0/1991646007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 msgr2=0x559720fe0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.395+0000 7f7f86d9d640 1 -- 172.21.15.45:0/1991646007 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559720edac20 con 0x559720fe1750 2024-03-12T05:14:02.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.396+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fdba40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7f80037040 tx=0x7f7f8003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=963fefca2632140d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.396+0000 7f7f6ffff640 1 -- 172.21.15.45:0/1991646007 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80032890 con 0x559720fe1750 2024-03-12T05:14:02.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.396+0000 7f7f6ffff640 1 -- 172.21.15.45:0/1991646007 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f80003e20 con 0x559720fe1750 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.396+0000 7f7f6ffff640 1 -- 172.21.15.45:0/1991646007 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80004880 con 0x559720fe1750 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1991646007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 msgr2=0x559720fdba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fdba40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7f80037040 tx=0x7f7f8003f000 comp rx=0 tx=0).stop 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1991646007 shutdown_connections 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fe1750 0x559720fdba40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fd8b70 0x559720fdf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1991646007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1991646007 >> 172.21.15.45:0/1991646007 conn(0x559720fd2980 msgr2=0x559720fd9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1991646007 shutdown_connections 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.397+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1991646007 wait complete. 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8f96e740 1 -- start start 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8f96e740 1 Processor -- start 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8f96e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 0x559720fe6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39296/0 (socket says 172.21.15.45:39296) 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.398+0000 7f7f8659c640 1 -- 172.21.15.45:0/1274803253 learned_addr learned my addr 172.21.15.45:0/1274803253 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fe7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559720fe7d00 con 0x559720fd8250 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559721047af0 con 0x559720fe1750 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 0x559720fe6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f86d9d640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fe7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559721047c30 con 0x559720fd8b70 2024-03-12T05:14:02.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f8f96e740 10 client.0 _ll_register_callbacks cb 0x7f7f7c005380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 -- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 msgr2=0x559720fe7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fe7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 -- 172.21.15.45:0/1274803253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 msgr2=0x559720fe8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.399+0000 7f7f85d9b640 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8810e810 con 0x559720fd8b70 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.400+0000 7f7f85d9b640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 0x559720fe6ee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009610 tx=0x7f7f7c095660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edee9a2edfe264c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.400+0000 7f7f8659c640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.400+0000 7f7f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.400+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c09d1a0 con 0x559720fd8b70 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.400+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f7c004dc0 con 0x559720fd8b70 2024-03-12T05:14:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.401+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7f7c098570 con 0x559720fd8b70 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.401+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f7c04f260 con 0x559720fd8b70 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.401+0000 7f7f8f96e740 10 client.25831 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.401+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559720f008c0 con 0x559720fd8b70 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7f7c004dc0 con 0x559720fd8b70 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7f7c098790 con 0x559720fd8b70 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 10 client.25831 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 10 client.25831 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 20 client.25831 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559721048f40 0x55972104b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 4 client.25831 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559720ccb890 con 0x559721048f40 2024-03-12T05:14:02.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.402+0000 7f7f8659c640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559721048f40 0x55972104b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.403+0000 7f7f6dffb640 10 client.25831 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.403+0000 7f7f8659c640 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559721048f40 0x55972104b330 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1cefcf4094608b20 server_cookie=80bb958d33c2b159 in_seq=0 out_seq=0 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.403+0000 7f7f6dffb640 1 -- 172.21.15.45:0/1274803253 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559720ccb890 con 0x559721048f40 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.403+0000 7f7f6dffb640 10 client.25831 handle_command_reply: tid=0 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.403+0000 7f7f8f96e740 1 client.25831 shutdown 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 msgr2=0x559720fe6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 0x559720fe6ee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009610 tx=0x7f7f7c095660 comp rx=0 tx=0).stop 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 shutdown_connections 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559721048f40 0x55972104b330 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559720fe1750 0x559720fe7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559720fd8b70 0x559720fe6ee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 --2- 172.21.15.45:0/1274803253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559720fd8250 0x559720fe8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 >> 172.21.15.45:0/1274803253 conn(0x559720fd2980 msgr2=0x559720fd12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f6dffb640 0 client.25831 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 shutdown_connections 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 1 -- 172.21.15.45:0/1274803253 wait complete. 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.404+0000 7f7f8f96e740 20 client.25831 trim_cache size 0 max 16384 2024-03-12T05:14:02.403 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 msgr2=0x7f7f700a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 0x7f7f700a4120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7f80006010 tx=0x7f7f80032500 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 msgr2=0x7f7f881a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 0x7f7f881a3e70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00d6a0 tx=0x7f7f7c00db60 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 shutdown_connections 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7f700a1c60 0x7f7f700a4120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f8810bef0 0x7f7f8810c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f88106430 0x7f7f881a3e70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 --2- 172.21.15.45:0/476236880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f880685d0 0x7f7f881a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.405+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 >> 172.21.15.45:0/476236880 conn(0x7f7f88077180 msgr2=0x7f7f88104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.406+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 shutdown_connections 2024-03-12T05:14:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.406+0000 7f7f8ccc0640 1 -- 172.21.15.45:0/476236880 wait complete. 2024-03-12T05:14:02.420 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:02.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 -- 172.21.15.45:0/1761076618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180100260 msgr2=0x7fc180100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 --2- 172.21.15.45:0/1761076618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180100260 0x7fc180100640 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc170009930 tx=0x7fc17002bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 -- 172.21.15.45:0/1761076618 shutdown_connections 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 --2- 172.21.15.45:0/1761076618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180105330 0x7fc18010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 --2- 172.21.15.45:0/1761076618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc180100c10 0x7fc180104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 --2- 172.21.15.45:0/1761076618 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180100260 0x7fc180100640 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 -- 172.21.15.45:0/1761076618 >> 172.21.15.45:0/1761076618 conn(0x7fc180077180 msgr2=0x7fc180077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 -- 172.21.15.45:0/1761076618 shutdown_connections 2024-03-12T05:14:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.566+0000 7fc1869d0640 1 -- 172.21.15.45:0/1761076618 wait complete. 2024-03-12T05:14:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 Processor -- start 2024-03-12T05:14:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 -- start start 2024-03-12T05:14:02.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180100260 0x7fc180197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc18019c010 0x7fc18019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019ca00 con 0x7fc180105330 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019cb70 con 0x7fc180100260 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc1869d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019cce0 con 0x7fc18019c010 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.567+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39320/0 (socket says 172.21.15.45:39320) 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 -- 172.21.15.45:0/139140184 learned_addr learned my addr 172.21.15.45:0/139140184 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc184f6d640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc18019c010 0x7fc18019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17ffff640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180100260 0x7fc180197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 -- 172.21.15.45:0/139140184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc18019c010 msgr2=0x7fc18019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc18019c010 0x7fc18019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 -- 172.21.15.45:0/139140184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180100260 msgr2=0x7fc180197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180100260 0x7fc180197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 -- 172.21.15.45:0/139140184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170009560 con 0x7fc180105330 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17f7fe640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e9f0 tx=0x7fc16800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17d7fa640 1 -- 172.21.15.45:0/139140184 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1680096c0 con 0x7fc180105330 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17d7fa640 1 -- 172.21.15.45:0/139140184 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1680054d0 con 0x7fc180105330 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1801a42c0 con 0x7fc180105330 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc17d7fa640 1 -- 172.21.15.45:0/139140184 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168010430 con 0x7fc180105330 2024-03-12T05:14:02.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.568+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1801a4770 con 0x7fc180105330 2024-03-12T05:14:02.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.570+0000 7fc17d7fa640 1 -- 172.21.15.45:0/139140184 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc168005020 con 0x7fc180105330 2024-03-12T05:14:02.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17d7fa640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 0x7fc16c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 Processor -- start 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17d7fa640 1 -- 172.21.15.45:0/139140184 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc168014070 con 0x7fc180105330 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17ffff640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 0x7fc16c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 -- start start 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912f9fc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 0x55912f9feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f9ae960 con 0x55912f9f3250 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f962310 con 0x55912f9fc750 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc1895a3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f95f840 con 0x55912f9f3b70 2024-03-12T05:14:02.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912f9fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc184f6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 0x55912f9feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39334/0 (socket says 172.21.15.45:39334) 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17ffff640 1 -- 172.21.15.45:0/1971022098 learned_addr learned my addr 172.21.15.45:0/1971022098 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.571+0000 7fc17ffff640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 0x7fc16c0a40e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc170005ab0 tx=0x7fc170005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 -- 172.21.15.45:0/1971022098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 msgr2=0x55912f9fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912f9fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 -- 172.21.15.45:0/1971022098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 msgr2=0x55912f9feb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 0x55912f9feb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 -- 172.21.15.45:0/1971022098 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55912f91d060 con 0x55912f9f3250 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc184f6d640 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 0x55912f9feb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc17ffff640 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc170009930 tx=0x7fc17003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bb56d9e59b709e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc1767fc640 1 -- 172.21.15.45:0/1971022098 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17002c8b0 con 0x55912f9f3250 2024-03-12T05:14:02.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc1767fc640 1 -- 172.21.15.45:0/1971022098 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc17002de90 con 0x55912f9f3250 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.572+0000 7fc1767fc640 1 -- 172.21.15.45:0/1971022098 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170003cd0 con 0x55912f9f3250 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- 172.21.15.45:0/1971022098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 msgr2=0x55912f9f3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc170009930 tx=0x7fc17003f000 comp rx=0 tx=0).stop 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- 172.21.15.45:0/1971022098 shutdown_connections 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9fc750 0x55912f9feb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912f9fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 --2- 172.21.15.45:0/1971022098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9f3630 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- 172.21.15.45:0/1971022098 >> 172.21.15.45:0/1971022098 conn(0x55912f9ed980 msgr2=0x55912f9edd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- 172.21.15.45:0/1971022098 shutdown_connections 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- 172.21.15.45:0/1971022098 wait complete. 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 -- start start 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.573+0000 7fc1895a3740 1 Processor -- start 2024-03-12T05:14:02.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3250 0x55912f9f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9f9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f95b020 con 0x55912f9fc750 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f62e430 con 0x55912f9f3b70 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f5eae00 con 0x55912f9f3250 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 10 client.0 _ll_register_callbacks cb 0x55912f902cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39336/0 (socket says 172.21.15.45:39336) 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 -- 172.21.15.45:0/2933380379 learned_addr learned my addr 172.21.15.45:0/2933380379 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc17f7fe640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9f9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc17ffff640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3250 0x55912f9f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 -- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3250 msgr2=0x55912f9f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3250 0x55912f9f97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 -- 172.21.15.45:0/2933380379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 msgr2=0x55912f9f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9f9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 -- 172.21.15.45:0/2933380379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18010d710 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc17f7fe640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9f9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc15f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc184f6d640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc178013a20 tx=0x7fc178013ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba4798b27bee96d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178003970 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc178003ad0 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.574+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55912f65a7e0 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc178003c90 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 10 client.16374 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55912f9596a0 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc178027080 con 0x55912f9fc750 2024-03-12T05:14:02.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc178021030 con 0x55912f9fc750 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 10 client.16374 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 10 client.16374 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 20 client.16374 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa0caa0 0x55912fa0eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 4 client.16374 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55912f697720 con 0x55912fa0caa0 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.575+0000 7fc17ffff640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa0caa0 0x55912fa0eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc15f7fe640 10 client.16374 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc17ffff640 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa0caa0 0x55912fa0eec0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb539fd070f7b7b4 server_cookie=2e17923ee2a735ca in_seq=0 out_seq=0 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc15f7fe640 1 -- 172.21.15.45:0/2933380379 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55912f697720 con 0x55912fa0caa0 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc15f7fe640 10 client.16374 handle_command_reply: tid=0 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc1895a3740 1 client.16374 shutdown 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.576+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 msgr2=0x55912f9fa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc178013a20 tx=0x7fc178013ee0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 shutdown_connections 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa0caa0 0x55912fa0eec0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912f9fa220 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9f9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc15f7fe640 0 client.16374 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 --2- 172.21.15.45:0/2933380379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3250 0x55912f9f97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 >> 172.21.15.45:0/2933380379 conn(0x55912f9ed980 msgr2=0x55912f9ef6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 shutdown_connections 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 1 -- 172.21.15.45:0/2933380379 wait complete. 2024-03-12T05:14:02.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.577+0000 7fc1895a3740 20 client.16374 trim_cache size 0 max 16384 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 Processor -- start 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 -- start start 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 0x55912f9f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f8e0130 con 0x55912f9f3250 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912f9f2680 con 0x55912f9f3b70 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.605+0000 7fc1895a3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912fa14150 con 0x55912f9fc750 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39352/0 (socket says 172.21.15.45:39352) 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 -- 172.21.15.45:0/226873979 learned_addr learned my addr 172.21.15.45:0/226873979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc184f6d640 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 0x55912f9f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 -- 172.21.15.45:0/226873979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 msgr2=0x55912f9f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 0x55912f9f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 -- 172.21.15.45:0/226873979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 msgr2=0x55912f9fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 -- 172.21.15.45:0/226873979 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55912f8f5c20 con 0x55912f9f3250 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc184f6d640 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 0x55912f9f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:02.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.606+0000 7fc17ffff640 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc170009900 tx=0x7fc170000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7cdba08a7d5d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.607+0000 7fc175ffb640 1 -- 172.21.15.45:0/226873979 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17005a040 con 0x55912f9f3250 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.607+0000 7fc175ffb640 1 -- 172.21.15.45:0/226873979 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc1700044f0 con 0x55912f9f3250 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.607+0000 7fc175ffb640 1 -- 172.21.15.45:0/226873979 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17005ea90 con 0x55912f9f3250 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- 172.21.15.45:0/226873979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 msgr2=0x55912f9fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc170009900 tx=0x7fc170000bc0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- 172.21.15.45:0/226873979 shutdown_connections 2024-03-12T05:14:02.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9fc750 0x55912f9f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3b70 0x55912f9fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 --2- 172.21.15.45:0/226873979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9f3250 0x55912f9fbed0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- 172.21.15.45:0/226873979 >> 172.21.15.45:0/226873979 conn(0x55912f9ed980 msgr2=0x55912f9f4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- 172.21.15.45:0/226873979 shutdown_connections 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- 172.21.15.45:0/226873979 wait complete. 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.608+0000 7fc1895a3740 1 -- start start 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 Processor -- start 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3250 0x55912fa027b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912fa02cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912fa5d530 con 0x55912f9fc750 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912fa5d670 con 0x55912f9f3250 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912fa5d830 con 0x55912f9f3b70 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.609+0000 7fc1895a3740 10 client.0 _ll_register_callbacks cb 0x7fc170009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc17f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912fa02cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39368/0 (socket says 172.21.15.45:39368) 2024-03-12T05:14:02.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 -- 172.21.15.45:0/3100775225 learned_addr learned my addr 172.21.15.45:0/3100775225 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc17ffff640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3250 0x55912fa027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 -- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 msgr2=0x55912fa02cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912fa02cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 -- 172.21.15.45:0/3100775225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3250 msgr2=0x55912fa027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3250 0x55912fa027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.610+0000 7fc184f6d640 1 -- 172.21.15.45:0/3100775225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55912f91d060 con 0x55912f9fc750 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.611+0000 7fc184f6d640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc178000f80 tx=0x7fc1780096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf63587548ff17af server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.611+0000 7fc15f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.614+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc178011ed0 con 0x55912f9fc750 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.614+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc178003970 con 0x55912f9fc750 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.614+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc17002de90 con 0x55912f9fc750 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.614+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc178003b30 con 0x55912f9fc750 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.614+0000 7fc1895a3740 10 client.16380 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55912f91b8c0 con 0x55912f9fc750 2024-03-12T05:14:02.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc17002de90 con 0x55912f9fc750 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc178005290 con 0x55912f9fc750 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 10 client.16380 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 10 client.16380 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 20 client.16380 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa760a0 0x55912fa784c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 4 client.16380 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55912f6e6890 con 0x55912fa760a0 2024-03-12T05:14:02.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.615+0000 7fc17ffff640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa760a0 0x55912fa784c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc17ffff640 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa760a0 0x55912fa784c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6816cbf1c0f53b52 server_cookie=16d1c9e072f29574 in_seq=0 out_seq=0 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc15f7fe640 10 client.16380 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc15f7fe640 1 -- 172.21.15.45:0/3100775225 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55912f6e6890 con 0x55912fa760a0 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc15f7fe640 10 client.16380 handle_command_reply: tid=0 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc1895a3740 1 client.16380 shutdown 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 msgr2=0x55912fa5cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.616+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc178000f80 tx=0x7fc1780096f0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 shutdown_connections 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55912fa760a0 0x55912fa784c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55912f9fc750 0x55912fa5cff0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55912f9f3b70 0x55912fa02cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 --2- 172.21.15.45:0/3100775225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55912f9f3250 0x55912fa027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 >> 172.21.15.45:0/3100775225 conn(0x55912f9ed980 msgr2=0x55912f9ec2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc15f7fe640 0 client.16380 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 shutdown_connections 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 1 -- 172.21.15.45:0/3100775225 wait complete. 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.617+0000 7fc1895a3740 20 client.16380 trim_cache size 0 max 16384 2024-03-12T05:14:02.615 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 msgr2=0x7fc16c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 0x7fc16c0a40e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc170005ab0 tx=0x7fc170005fd0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 msgr2=0x7fc180197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e9f0 tx=0x7fc16800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 shutdown_connections 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc16c0a1c20 0x7fc16c0a40e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc180105330 0x7fc180197c90 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc18019c010 0x7fc18019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 --2- 172.21.15.45:0/139140184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc180100260 0x7fc180197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 >> 172.21.15.45:0/139140184 conn(0x7fc180077180 msgr2=0x7fc1800687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 shutdown_connections 2024-03-12T05:14:02.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.618+0000 7fc1869d0640 1 -- 172.21.15.45:0/139140184 wait complete. 2024-03-12T05:14:02.634 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:02.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:14:02.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:02 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:14:02.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:02 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:14:02.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:02 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:14:02.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:02 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3945899841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 -- 172.21.15.45:0/3101911989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 msgr2=0x7f8e90101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/3101911989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90101760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e88009ca0 tx=0x7f8e8802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 -- 172.21.15.45:0/3101911989 shutdown_connections 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/3101911989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90101d30 0x7f8e90105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/3101911989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90106450 0x7f8e9010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/3101911989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90101760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 -- 172.21.15.45:0/3101911989 >> 172.21.15.45:0/3101911989 conn(0x7f8e900fcbe0 msgr2=0x7f8e900ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 -- 172.21.15.45:0/3101911989 shutdown_connections 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.815+0000 7f8e95a8a640 1 -- 172.21.15.45:0/3101911989 wait complete. 2024-03-12T05:14:02.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.816+0000 7f8e95a8a640 1 Processor -- start 2024-03-12T05:14:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.816+0000 7f8e95a8a640 1 -- start start 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.816+0000 7f8e95a8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.816+0000 7f8e95a8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 0x7f8e90115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e95a8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90110670 0x7f8e90110b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e95a8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e901112b0 con 0x7f8e90106450 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e95a8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e90111420 con 0x7f8e90101380 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e95a8a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e90119260 con 0x7f8e90110670 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41362/0 (socket says 172.21.15.45:41362) 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1293140346 learned_addr learned my addr 172.21.15.45:0/1293140346 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90110670 0x7f8e90110b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 0x7f8e90115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1293140346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90110670 msgr2=0x7f8e90110b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90110670 0x7f8e90110b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 msgr2=0x7f8e90115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 0x7f8e90115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1293140346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e88009900 con 0x7f8e90101380 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.817+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 0x7f8e90115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e8802c680 tx=0x7f8e88035ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e8cff9640 1 -- 172.21.15.45:0/1293140346 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880329e0 con 0x7f8e90101380 2024-03-12T05:14:02.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e901194e0 con 0x7f8e90101380 2024-03-12T05:14:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e8cff9640 1 -- 172.21.15.45:0/1293140346 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e88035e90 con 0x7f8e90101380 2024-03-12T05:14:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e8cff9640 1 -- 172.21.15.45:0/1293140346 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88004750 con 0x7f8e90101380 2024-03-12T05:14:02.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.818+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e901199e0 con 0x7f8e90101380 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e8cff9640 1 -- 172.21.15.45:0/1293140346 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8e8803b040 con 0x7f8e90101380 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e98657740 1 Processor -- start 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e98657740 1 -- start start 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e98657740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e98657740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f17210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.821+0000 7f8e98657740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 0x55f337f19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e98657740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337ec9960 con 0x55f337f0e250 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e98657740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337e7d310 con 0x55f337f0eb70 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e98657740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337e7a840 con 0x55f337f17750 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f17210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39386/0 (socket says 172.21.15.45:39386) 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 -- 172.21.15.45:0/2163412954 learned_addr learned my addr 172.21.15.45:0/2163412954 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8cff9640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 0x7f8e780a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8effd640 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 0x55f337f19b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 0x7f8e780a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 -- 172.21.15.45:0/2163412954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 msgr2=0x55f337f19b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 0x55f337f19b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 -- 172.21.15.45:0/2163412954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 msgr2=0x55f337f17210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f17210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8ffff640 1 -- 172.21.15.45:0/2163412954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f337e38060 con 0x55f337f0e250 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8effd640 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 0x55f337f19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.822+0000 7f8e8cff9640 1 -- 172.21.15.45:0/1293140346 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8e880f00a0 con 0x7f8e90101380 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.823+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8e80013880 tx=0x7f8e80013d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5994c64e067c9f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.823+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 0x7f8e780a4120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e80014010 tx=0x7f8e80020000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.823+0000 7f8e75ffb640 1 -- 172.21.15.45:0/2163412954 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80009d20 con 0x55f337f0e250 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.823+0000 7f8e75ffb640 1 -- 172.21.15.45:0/2163412954 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e8000fe30 con 0x55f337f0e250 2024-03-12T05:14:02.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.823+0000 7f8e75ffb640 1 -- 172.21.15.45:0/2163412954 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80010820 con 0x55f337f0e250 2024-03-12T05:14:02.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.824+0000 7f8e98657740 1 -- 172.21.15.45:0/2163412954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 msgr2=0x55f337f0e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.824+0000 7f8e98657740 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8e80013880 tx=0x7f8e80013d40 comp rx=0 tx=0).stop 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 -- 172.21.15.45:0/2163412954 shutdown_connections 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f17750 0x55f337f19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f17210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 --2- 172.21.15.45:0/2163412954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f0e630 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 -- 172.21.15.45:0/2163412954 >> 172.21.15.45:0/2163412954 conn(0x55f337f08980 msgr2=0x55f337f08d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 -- 172.21.15.45:0/2163412954 shutdown_connections 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 -- 172.21.15.45:0/2163412954 wait complete. 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 -- start start 2024-03-12T05:14:02.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.825+0000 7f8e98657740 1 Processor -- start 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 0x55f337f14c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f17750 0x55f337f15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337e76020 con 0x55f337f0eb70 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337b49430 con 0x55f337f17750 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337b05e00 con 0x55f337f0e250 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e98657740 10 client.0 _ll_register_callbacks cb 0x55f337e1dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54222/0 (socket says 172.21.15.45:54222) 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1796198390 learned_addr learned my addr 172.21.15.45:0/1796198390 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f17750 0x55f337f15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1796198390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f17750 msgr2=0x55f337f15160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f17750 0x55f337f15160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 0x55f337f14c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.826+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1796198390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 msgr2=0x55f337f14c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 0x55f337f14c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/1796198390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e80013460 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 0x55f337f14c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e880ed900 tx=0x7f8e8802c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbc6a2e96ccc6864 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e6effd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880f22d0 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8e880bd480 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.827+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f337b757e0 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e880bcd80 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 10 client.25861 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f337e746a0 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f337b757e0 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8e880bcfe0 con 0x55f337f0e250 2024-03-12T05:14:02.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 10 client.25861 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 10 client.25861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 20 client.25861 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f277f0 0x55f337f29c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 4 client.25861 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f337bb2720 con 0x55f337f277f0 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.828+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f277f0 0x55f337f29c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.829+0000 7f8e8effd640 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f277f0 0x55f337f29c00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bad562f8b356d3d0 server_cookie=69cea4a5d3fdfd1d in_seq=0 out_seq=0 2024-03-12T05:14:02.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.829+0000 7f8e6effd640 10 client.25861 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:02.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e6effd640 1 -- 172.21.15.45:0/1796198390 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f337bb2720 con 0x55f337f277f0 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e6effd640 10 client.25861 handle_command_reply: tid=0 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 client.25861 shutdown 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 msgr2=0x55f337f146e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e880ed900 tx=0x7f8e8802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 shutdown_connections 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0eb70 0x55f337f14c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f277f0 0x55f337f29c00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f17750 0x55f337f15160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 --2- 172.21.15.45:0/1796198390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f0e250 0x55f337f146e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 >> 172.21.15.45:0/1796198390 conn(0x55f337f08980 msgr2=0x55f337f0a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.830+0000 7f8e6effd640 0 client.25861 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:02.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.831+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 shutdown_connections 2024-03-12T05:14:02.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.831+0000 7f8e98657740 1 -- 172.21.15.45:0/1796198390 wait complete. 2024-03-12T05:14:02.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.831+0000 7f8e98657740 20 client.25861 trim_cache size 0 max 16384 2024-03-12T05:14:02.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.851+0000 7f8e98657740 1 Processor -- start 2024-03-12T05:14:02.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.851+0000 7f8e98657740 1 -- start start 2024-03-12T05:14:02.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.851+0000 7f8e98657740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e98657740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e98657740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f11e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e98657740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337dfb130 con 0x55f337f0e250 2024-03-12T05:14:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e98657740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337bb2720 con 0x55f337f0eb70 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e98657740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337b757e0 con 0x55f337f08980 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54230/0 (socket says 172.21.15.45:54230) 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/40698359 learned_addr learned my addr 172.21.15.45:0/40698359 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f11e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8effd640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/40698359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 msgr2=0x55f337f11e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f11e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/40698359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 msgr2=0x55f337f11940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f11940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/40698359 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f337e10c20 con 0x55f337f08980 2024-03-12T05:14:02.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8effd640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f11940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:02.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.852+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8e880bca80 tx=0x7f8e880c50e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9c003ca5393ff server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.853+0000 7f8e757fa640 1 -- 172.21.15.45:0/40698359 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880bc8c0 con 0x55f337f08980 2024-03-12T05:14:02.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.853+0000 7f8e757fa640 1 -- 172.21.15.45:0/40698359 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e880bd480 con 0x55f337f08980 2024-03-12T05:14:02.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.853+0000 7f8e757fa640 1 -- 172.21.15.45:0/40698359 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e880ed980 con 0x55f337f08980 2024-03-12T05:14:02.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- 172.21.15.45:0/40698359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 msgr2=0x55f337f16e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8e880bca80 tx=0x7f8e880c50e0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- 172.21.15.45:0/40698359 shutdown_connections 2024-03-12T05:14:02.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f11940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f11e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 --2- 172.21.15.45:0/40698359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f16e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- 172.21.15.45:0/40698359 >> 172.21.15.45:0/40698359 conn(0x55f337f03e80 msgr2=0x55f337f2e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- 172.21.15.45:0/40698359 shutdown_connections 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- 172.21.15.45:0/40698359 wait complete. 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 -- start start 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.855+0000 7f8e98657740 1 Processor -- start 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f1cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f1da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337f1ef30 con 0x55f337f0e250 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337f7dd30 con 0x55f337f0eb70 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f337f7de70 con 0x55f337f08980 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e8f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f1cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e8effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39408/0 (socket says 172.21.15.45:39408) 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e8effd640 1 -- 172.21.15.45:0/2070101918 learned_addr learned my addr 172.21.15.45:0/2070101918 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e8ffff640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f1da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.856+0000 7f8e98657740 10 client.0 _ll_register_callbacks cb 0x7f8e80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/2070101918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 msgr2=0x55f337f1da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f1da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/2070101918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 msgr2=0x55f337f1d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 -- 172.21.15.45:0/2070101918 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f337e746a0 con 0x55f337f08980 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8effd640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e8f7fe640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f1cf90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55f337f15980 tx=0x7f8e8802cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b09e83302afe730 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e6effd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88004d40 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.857+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8e880c0ed0 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e8000fe30 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e88004ea0 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 10 client.25867 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f337e368c0 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e8000fe30 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8e880d5ab0 con 0x55f337f08980 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 10 client.25867 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 10 client.25867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 20 client.25867 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:02.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f17750 0x55f337f81100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 4 client.25867 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f337c01890 con 0x55f337f17750 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.858+0000 7f8e8effd640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f17750 0x55f337f81100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.859+0000 7f8e6effd640 10 client.25867 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.859+0000 7f8e8effd640 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f17750 0x55f337f81100 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e959cada66c63428 server_cookie=8ad54232c287540c in_seq=0 out_seq=0 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.859+0000 7f8e6effd640 1 -- 172.21.15.45:0/2070101918 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f337c01890 con 0x55f337f17750 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.859+0000 7f8e6effd640 10 client.25867 handle_command_reply: tid=0 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.859+0000 7f8e98657740 1 client.25867 shutdown 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 msgr2=0x55f337f1cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f1cf90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55f337f15980 tx=0x7f8e8802cf40 comp rx=0 tx=0).stop 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 shutdown_connections 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f337f0e250 0x55f337f1d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f337f17750 0x55f337f81100 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e6effd640 0 client.25867 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f337f0eb70 0x55f337f1da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 --2- 172.21.15.45:0/2070101918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f337f08980 0x55f337f1cf90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 >> 172.21.15.45:0/2070101918 conn(0x55f337f03e80 msgr2=0x55f337f072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.860+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 shutdown_connections 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.861+0000 7f8e98657740 1 -- 172.21.15.45:0/2070101918 wait complete. 2024-03-12T05:14:02.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.861+0000 7f8e98657740 20 client.25867 trim_cache size 0 max 16384 2024-03-12T05:14:02.875 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 msgr2=0x7f8e780a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 0x7f8e780a4120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e80014010 tx=0x7f8e80020000 comp rx=0 tx=0).stop 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 msgr2=0x7f8e90115560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e8802c680 tx=0x7f8e88035ca0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 shutdown_connections 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e780a1c60 0x7f8e780a4120 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e90106450 0x7f8e90115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e90110670 0x7f8e90110b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.862+0000 7f8e95a8a640 1 --2- 172.21.15.45:0/1293140346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e90101380 0x7f8e90115560 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:02.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.863+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 >> 172.21.15.45:0/1293140346 conn(0x7f8e900fcbe0 msgr2=0x7f8e900fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:02.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.863+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 shutdown_connections 2024-03-12T05:14:02.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:02.863+0000 7f8e95a8a640 1 -- 172.21.15.45:0/1293140346 wait complete. 2024-03-12T05:14:02.879 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:02.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:14:02.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:02 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:14:02.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:02 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:14:02.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:02 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:14:02.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:02 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3945899841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:02 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:14:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:02 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:14:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:02 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:14:02.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:02 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3945899841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 -- 172.21.15.45:0/594713490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc10b580 msgr2=0x7f72fc10d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 --2- 172.21.15.45:0/594713490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc10b580 0x7f72fc10d9f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009900 tx=0x7f72f002bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 -- 172.21.15.45:0/594713490 shutdown_connections 2024-03-12T05:14:03.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 --2- 172.21.15.45:0/594713490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc10b580 0x7f72fc10d9f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 --2- 172.21.15.45:0/594713490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc100a50 0x7f72fc10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 --2- 172.21.15.45:0/594713490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 -- 172.21.15.45:0/594713490 >> 172.21.15.45:0/594713490 conn(0x7f72fc076c30 msgr2=0x7f72fc0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 -- 172.21.15.45:0/594713490 shutdown_connections 2024-03-12T05:14:03.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.095+0000 7f72fbfff640 1 -- 172.21.15.45:0/594713490 wait complete. 2024-03-12T05:14:03.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.096+0000 7f72fbfff640 1 Processor -- start 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.096+0000 7f72fbfff640 1 -- start start 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 0x7f72fc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 0x7f72fc19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc19e990 con 0x7f72fc1000a0 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc19eb00 con 0x7f72fc10b580 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fbfff640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc19ec70 con 0x7f72fc19dfa0 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc1996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39424/0 (socket says 172.21.15.45:39424) 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72faffd640 1 -- 172.21.15.45:0/1533243263 learned_addr learned my addr 172.21.15.45:0/1533243263 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:03.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 0x7f72fc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 0x7f72fc19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.097+0000 7f72fa7fc640 1 -- 172.21.15.45:0/1533243263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 msgr2=0x7f72fc19e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 0x7f72fc19e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fa7fc640 1 -- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 msgr2=0x7f72fc1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fa7fc640 1 -- 172.21.15.45:0/1533243263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f0009560 con 0x7f72fc10b580 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 0x7f72fc19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 0x7f72fc199c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00e9c0 tx=0x7f72ec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.098+0000 7f72e3fff640 1 -- 172.21.15.45:0/1533243263 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00ccc0 con 0x7f72fc10b580 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.099+0000 7f72e3fff640 1 -- 172.21.15.45:0/1533243263 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f72ec00ce20 con 0x7f72fc10b580 2024-03-12T05:14:03.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.099+0000 7f72e3fff640 1 -- 172.21.15.45:0/1533243263 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec018710 con 0x7f72fc10b580 2024-03-12T05:14:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.099+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc1a6250 con 0x7f72fc10b580 2024-03-12T05:14:03.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.099+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1a6750 con 0x7f72fc10b580 2024-03-12T05:14:03.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.102+0000 7f72e3fff640 1 -- 172.21.15.45:0/1533243263 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f72ec018870 con 0x7f72fc10b580 2024-03-12T05:14:03.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.102+0000 7f73041b7740 1 Processor -- start 2024-03-12T05:14:03.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.102+0000 7f73041b7740 1 -- start start 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 0x558fad2ef840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 0x558fad2fad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2aa960 con 0x558fad2ef460 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad25e310 con 0x558fad2efd80 2024-03-12T05:14:03.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f73041b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad25b840 con 0x558fad2f8960 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72fa7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 0x558fad2fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72fb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 0x558fad2ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72fa7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 0x558fad2fad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54266/0 (socket says 172.21.15.45:54266) 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72fa7fc640 1 -- 172.21.15.45:0/1817364541 learned_addr learned my addr 172.21.15.45:0/1817364541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72faffd640 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72e3fff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 0x7f72e40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72faffd640 1 -- 172.21.15.45:0/1817364541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 msgr2=0x558fad2fad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.103+0000 7f72faffd640 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 0x558fad2fad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72faffd640 1 -- 172.21.15.45:0/1817364541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 msgr2=0x558fad2ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72faffd640 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 0x558fad2ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72faffd640 1 -- 172.21.15.45:0/1817364541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fad219060 con 0x558fad2efd80 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 0x558fad2ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 0x7f72e40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72faffd640 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f8420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f72f40113e0 tx=0x7f72f40118a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f89caf508b04afeb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72e17fa640 1 -- 172.21.15.45:0/1817364541 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4012830 con 0x558fad2efd80 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72e17fa640 1 -- 172.21.15.45:0/1817364541 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f72f4012e40 con 0x558fad2efd80 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72e17fa640 1 -- 172.21.15.45:0/1817364541 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f401adb0 con 0x558fad2efd80 2024-03-12T05:14:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.104+0000 7f72e3fff640 1 -- 172.21.15.45:0/1533243263 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f72ec014070 con 0x7f72fc10b580 2024-03-12T05:14:03.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 -- 172.21.15.45:0/1817364541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 msgr2=0x558fad2f8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f8420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f72f40113e0 tx=0x7f72f40118a0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 0x7f72e40a4160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72f0005ab0 tx=0x7f72f0009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 -- 172.21.15.45:0/1817364541 shutdown_connections 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2f8960 0x558fad2fad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f8420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 --2- 172.21.15.45:0/1817364541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2ef460 0x558fad2ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 -- 172.21.15.45:0/1817364541 >> 172.21.15.45:0/1817364541 conn(0x558fad2e9b90 msgr2=0x558fad2e9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.105+0000 7f73041b7740 1 -- 172.21.15.45:0/1817364541 shutdown_connections 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.106+0000 7f73041b7740 1 -- 172.21.15.45:0/1817364541 wait complete. 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.106+0000 7f73041b7740 1 -- start start 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.106+0000 7f73041b7740 1 Processor -- start 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 0x558fad2f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 0x558fad2f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad257020 con 0x558fad2f8960 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558facf2a430 con 0x558fad2ef460 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558facee6e00 con 0x558fad2efd80 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f73041b7740 10 client.0 _ll_register_callbacks cb 0x558fad1fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 0x558fad2f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72fb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72fb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39452/0 (socket says 172.21.15.45:39452) 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72fb7fe640 1 -- 172.21.15.45:0/2082327205 learned_addr learned my addr 172.21.15.45:0/2082327205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 0x558fad2f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 -- 172.21.15.45:0/2082327205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 msgr2=0x558fad2f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 0x558fad2f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 -- 172.21.15.45:0/2082327205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 msgr2=0x558fad2f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72faffd640 1 -- 172.21.15.45:0/2082327205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f4009c10 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.107+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 0x558fad2f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.108+0000 7f72faffd640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 0x558fad2f59b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72f4000f80 tx=0x7f72f401ad60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e22bcdcbcafc5974 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.108+0000 7f72deff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.108+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f401a3f0 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.108+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f72f401a550 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.108+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558facf567e0 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72f401a710 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 10 client.25871 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fad2556a0 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558facf567e0 con 0x558fad2ef460 2024-03-12T05:14:03.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f72f4022040 con 0x558fad2ef460 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 10 client.25871 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 20 client.25871 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad311630 0x558fad313a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 4 client.25871 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.109+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558facf93720 con 0x558fad311630 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.110+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad311630 0x558fad313a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.110+0000 7f72deff5640 10 client.25871 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.110+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad311630 0x558fad313a70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1faa8bb6eccb6a08 server_cookie=f8dcfc9d05292dca in_seq=0 out_seq=0 2024-03-12T05:14:03.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f72deff5640 1 -- 172.21.15.45:0/2082327205 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558facf93720 con 0x558fad311630 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f72deff5640 10 client.25871 handle_command_reply: tid=0 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 client.25871 shutdown 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 msgr2=0x558fad2f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 0x558fad2f59b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72f4000f80 tx=0x7f72f401ad60 comp rx=0 tx=0).stop 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 shutdown_connections 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad311630 0x558fad313a70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2efd80 0x558fad2f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.111+0000 7f73041b7740 1 --2- 172.21.15.45:0/2082327205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2ef460 0x558fad2f59b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.112+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 >> 172.21.15.45:0/2082327205 conn(0x558fad2e9b90 msgr2=0x558fad2eb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.112+0000 7f72deff5640 0 client.25871 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.112+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 shutdown_connections 2024-03-12T05:14:03.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.112+0000 7f73041b7740 1 -- 172.21.15.45:0/2082327205 wait complete. 2024-03-12T05:14:03.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.112+0000 7f73041b7740 20 client.25871 trim_cache size 0 max 16384 2024-03-12T05:14:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.133+0000 7f73041b7740 1 Processor -- start 2024-03-12T05:14:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.133+0000 7f73041b7740 1 -- start start 2024-03-12T05:14:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f72faffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f72faffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54286/0 (socket says 172.21.15.45:54286) 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad1dc130 con 0x558fad2f8960 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f72faffd640 1 -- 172.21.15.45:0/2348176276 learned_addr learned my addr 172.21.15.45:0/2348176276 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2f3f60 con 0x558fad2efd80 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.134+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2efaa0 con 0x558fad2ef460 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 -- 172.21.15.45:0/2348176276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 msgr2=0x558fad2f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 -- 172.21.15.45:0/2348176276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 msgr2=0x558fad2f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 -- 172.21.15.45:0/2348176276 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fad1f1c20 con 0x558fad2ef460 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72faffd640 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f72f4011da0 tx=0x7f72f4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fe876eeae0f1078 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72e0ff9640 1 -- 172.21.15.45:0/2348176276 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4023dc0 con 0x558fad2ef460 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72e0ff9640 1 -- 172.21.15.45:0/2348176276 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f72f403a070 con 0x558fad2ef460 2024-03-12T05:14:03.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.135+0000 7f72e0ff9640 1 -- 172.21.15.45:0/2348176276 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4003970 con 0x558fad2ef460 2024-03-12T05:14:03.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 msgr2=0x558fad2f80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f72f4011da0 tx=0x7f72f4009e30 comp rx=0 tx=0).stop 2024-03-12T05:14:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 shutdown_connections 2024-03-12T05:14:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 --2- 172.21.15.45:0/2348176276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2f80e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.136+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 >> 172.21.15.45:0/2348176276 conn(0x558fad2e9b90 msgr2=0x558fad2e9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.137+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 shutdown_connections 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.137+0000 7f73041b7740 1 -- 172.21.15.45:0/2348176276 wait complete. 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.137+0000 7f73041b7740 1 -- start start 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.137+0000 7f73041b7740 1 Processor -- start 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2fddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2fe840 con 0x558fad2f8960 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2fe980 con 0x558fad2efd80 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fad2feb40 con 0x558fad2ef460 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f73041b7740 10 client.0 _ll_register_callbacks cb 0x7f72f4009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f72fb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f72fb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39472/0 (socket says 172.21.15.45:39472) 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f72fb7fe640 1 -- 172.21.15.45:0/1419219949 learned_addr learned my addr 172.21.15.45:0/1419219949 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.138+0000 7f72faffd640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2fddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 -- 172.21.15.45:0/1419219949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 msgr2=0x558fad2ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 -- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 msgr2=0x558fad2fddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:03.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2fddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 -- 172.21.15.45:0/1419219949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fad2556a0 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fa7fc640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2fddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72fb7fe640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009230 tx=0x7f72f002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5654efb2eb3741e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72deff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0004dc0 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f72f002de40 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.139+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72f4012e40 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72f0004f20 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 10 client.16410 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fad2178c0 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f72f0005530 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f72f0049040 con 0x558fad2f8960 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 10 client.16410 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:03.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 20 client.16410 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad368f60 0x558fad36b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 4 client.16410 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.140+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f72f4012e40 con 0x558fad368f60 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.141+0000 7f72faffd640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad368f60 0x558fad36b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f72faffd640 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad368f60 0x558fad36b380 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=657e6d8c889c4077 server_cookie=7143ad365f34b1ae in_seq=0 out_seq=0 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f72deff5640 10 client.16410 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f72deff5640 1 -- 172.21.15.45:0/1419219949 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f72f4012e40 con 0x558fad368f60 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f72deff5640 10 client.16410 handle_command_reply: tid=0 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f73041b7740 1 client.16410 shutdown 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 msgr2=0x558fad2fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009230 tx=0x7f72f002bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 shutdown_connections 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.142+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fad368f60 0x558fad36b380 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fad2f8960 0x558fad2fe300 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f72deff5640 0 client.16410 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fad2efd80 0x558fad2fddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 --2- 172.21.15.45:0/1419219949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fad2ef460 0x558fad2ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 >> 172.21.15.45:0/1419219949 conn(0x558fad2e9b90 msgr2=0x558fad2f0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 shutdown_connections 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 1 -- 172.21.15.45:0/1419219949 wait complete. 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.143+0000 7f73041b7740 20 client.16410 trim_cache size 0 max 16384 2024-03-12T05:14:03.147 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 msgr2=0x7f72e40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 0x7f72e40a4160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72f0005ab0 tx=0x7f72f0009260 comp rx=0 tx=0).stop 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 msgr2=0x7f72fc199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 0x7f72fc199c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00e9c0 tx=0x7f72ec00ee80 comp rx=0 tx=0).stop 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 shutdown_connections 2024-03-12T05:14:03.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f72e40a1ca0 0x7f72e40a4160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72fc19dfa0 0x7f72fc19e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72fc10b580 0x7f72fc199c20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 --2- 172.21.15.45:0/1533243263 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72fc1000a0 0x7f72fc1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 >> 172.21.15.45:0/1533243263 conn(0x7f72fc076c30 msgr2=0x7f72fc108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 shutdown_connections 2024-03-12T05:14:03.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:03.144+0000 7f72fbfff640 1 -- 172.21.15.45:0/1533243263 wait complete. 2024-03-12T05:14:03.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:03 smithi045 ceph-mon[32174]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:03 smithi130 ceph-mon[35398]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:03.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:03 smithi116 ceph-mon[35486]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:14:04.161 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:14:04.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:14:04.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:14:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- 172.21.15.45:0/1509781747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc654100c10 msgr2=0x7fc654104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 --2- 172.21.15.45:0/1509781747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc654100c10 0x7fc654104c50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc644009930 tx=0x7fc64402bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:04.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- 172.21.15.45:0/1509781747 shutdown_connections 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 --2- 172.21.15.45:0/1509781747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc654100c10 0x7fc654104c50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 --2- 172.21.15.45:0/1509781747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc654105330 0x7fc65410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 --2- 172.21.15.45:0/1509781747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654100260 0x7fc654100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- 172.21.15.45:0/1509781747 >> 172.21.15.45:0/1509781747 conn(0x7fc654077180 msgr2=0x7fc654077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- 172.21.15.45:0/1509781747 shutdown_connections 2024-03-12T05:14:04.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- 172.21.15.45:0/1509781747 wait complete. 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 Processor -- start 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.301+0000 7fc65b977640 1 -- start start 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc654100260 0x7fc6541965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654105330 0x7fc654196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419b870 con 0x7fc65419ae80 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419b9e0 con 0x7fc654100260 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc65b977640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419bb50 con 0x7fc654105330 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39502/0 (socket says 172.21.15.45:39502) 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 -- 172.21.15.45:0/1734395060 learned_addr learned my addr 172.21.15.45:0/1734395060 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc658f12640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654105330 0x7fc654196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 -- 172.21.15.45:0/1734395060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654105330 msgr2=0x7fc654196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654105330 0x7fc654196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 -- 172.21.15.45:0/1734395060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc654100260 msgr2=0x7fc6541965c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc654100260 0x7fc6541965c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 -- 172.21.15.45:0/1734395060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc644009560 con 0x7fc65419ae80 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.302+0000 7fc659f14640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00b970 tx=0x7fc64c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.303+0000 7fc64a7fc640 1 -- 172.21.15.45:0/1734395060 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c00cb40 con 0x7fc65419ae80 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.303+0000 7fc64a7fc640 1 -- 172.21.15.45:0/1734395060 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc64c00cca0 con 0x7fc65419ae80 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.303+0000 7fc64a7fc640 1 -- 172.21.15.45:0/1734395060 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0076e0 con 0x7fc65419ae80 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.303+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6541a3130 con 0x7fc65419ae80 2024-03-12T05:14:04.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.303+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6541a3560 con 0x7fc65419ae80 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.305+0000 7fc64a7fc640 1 -- 172.21.15.45:0/1734395060 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc64c014440 con 0x7fc65419ae80 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.305+0000 7fc65e627740 1 Processor -- start 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.305+0000 7fc65e627740 1 -- start start 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc64a7fc640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 0x7fc6400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf251450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc64a7fc640 1 -- 172.21.15.45:0/1734395060 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc64c0ccf50 con 0x7fc65419ae80 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf1bff90 con 0x55c7cf251990 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf1bd4c0 con 0x55c7cf251070 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc65e627740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf1b8ca0 con 0x55c7cf25ee20 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc658f12640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 0x7fc6400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659f14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659f14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54322/0 (socket says 172.21.15.45:54322) 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659f14640 1 -- 172.21.15.45:0/1554861028 learned_addr learned my addr 172.21.15.45:0/1554861028 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf251450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 -- 172.21.15.45:0/1554861028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 msgr2=0x55c7cf261210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 -- 172.21.15.45:0/1554861028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 msgr2=0x55c7cf25e8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc658f12640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 0x7fc6400a40e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc644000c00 tx=0x7fc644005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 -- 172.21.15.45:0/1554861028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7cf17b0b0 con 0x55c7cf251070 2024-03-12T05:14:04.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659f14640 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.306+0000 7fc659713640 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf251450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc65000a960 tx=0x7fc65000ae20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76f6053747caaad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc63f7fe640 1 -- 172.21.15.45:0/1554861028 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000cd80 con 0x55c7cf251070 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc63f7fe640 1 -- 172.21.15.45:0/1554861028 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc65000f900 con 0x55c7cf251070 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc63f7fe640 1 -- 172.21.15.45:0/1554861028 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500188a0 con 0x55c7cf251070 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 -- 172.21.15.45:0/1554861028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 msgr2=0x55c7cf251450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf251450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc65000a960 tx=0x7fc65000ae20 comp rx=0 tx=0).stop 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 -- 172.21.15.45:0/1554861028 shutdown_connections 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf261210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 --2- 172.21.15.45:0/1554861028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf251450 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 -- 172.21.15.45:0/1554861028 >> 172.21.15.45:0/1554861028 conn(0x55c7cf24b840 msgr2=0x55c7cf24bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.307+0000 7fc65e627740 1 -- 172.21.15.45:0/1554861028 shutdown_connections 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 -- 172.21.15.45:0/1554861028 wait complete. 2024-03-12T05:14:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 -- start start 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 Processor -- start 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf25bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf15df40 con 0x55c7cf251990 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cee8c4b0 con 0x55c7cf251070 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf13e670 con 0x55c7cf25ee20 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc65e627740 10 client.0 _ll_register_callbacks cb 0x55c7cf1abf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc658f12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc659713640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf25bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc659f14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc658f12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39506/0 (socket says 172.21.15.45:39506) 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.308+0000 7fc658f12640 1 -- 172.21.15.45:0/1018739101 learned_addr learned my addr 172.21.15.45:0/1018739101 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 -- 172.21.15.45:0/1018739101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 msgr2=0x55c7cf25c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 -- 172.21.15.45:0/1018739101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 msgr2=0x55c7cf25c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 -- 172.21.15.45:0/1018739101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65000a590 con 0x55c7cf251070 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659f14640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc659713640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf25bed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc650018440 tx=0x7fc6500053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20fa94b9c2be9df server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc63cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc650004220 con 0x55c7cf251070 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7cf192f40 con 0x55c7cf251070 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc650004380 con 0x55c7cf251070 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6500046c0 con 0x55c7cf251070 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc65e627740 10 client.25880 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:04.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c7cf1b7320 con 0x55c7cf251070 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.309+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c7cf192f40 con 0x55c7cf251070 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc650027070 con 0x55c7cf251070 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 10 client.25880 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 10 client.25880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 20 client.25880 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 4 client.25880 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7cf1af310 con 0x55c7cf26f0e0 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc658f12640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc658f12640 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2714d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f25e1b8c38f223 server_cookie=84146aeca0a040e3 in_seq=0 out_seq=0 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.310+0000 7fc63cff9640 10 client.25880 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc63cff9640 1 -- 172.21.15.45:0/1018739101 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c7cf1af310 con 0x55c7cf26f0e0 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc63cff9640 10 client.25880 handle_command_reply: tid=0 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 client.25880 shutdown 2024-03-12T05:14:04.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 msgr2=0x55c7cf25bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf25bed0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc650018440 tx=0x7fc6500053b0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 shutdown_connections 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2714d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251990 0x55c7cf25c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc63cff9640 0 client.25880 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 --2- 172.21.15.45:0/1018739101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251070 0x55c7cf25bed0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 >> 172.21.15.45:0/1018739101 conn(0x55c7cf24b840 msgr2=0x55c7cf24d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 shutdown_connections 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 1 -- 172.21.15.45:0/1018739101 wait complete. 2024-03-12T05:14:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.311+0000 7fc65e627740 20 client.25880 trim_cache size 0 max 16384 2024-03-12T05:14:04.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 Processor -- start 2024-03-12T05:14:04.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 -- start start 2024-03-12T05:14:04.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 0x55c7cf25e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf2598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf261ae0 con 0x55c7cf251070 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf260740 con 0x55c7cf251990 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc65e627740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf192f40 con 0x55c7cf25ee20 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc659713640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 0x55c7cf25e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf2598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc659713640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 0x55c7cf25e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39522/0 (socket says 172.21.15.45:39522) 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc659713640 1 -- 172.21.15.45:0/761382269 learned_addr learned my addr 172.21.15.45:0/761382269 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc659f14640 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 -- 172.21.15.45:0/761382269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 msgr2=0x55c7cf25c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 -- 172.21.15.45:0/761382269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 msgr2=0x55c7cf25e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 0x55c7cf25e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.331+0000 7fc658f12640 1 -- 172.21.15.45:0/761382269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7cf153c00 con 0x55c7cf251990 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.332+0000 7fc658f12640 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf2598b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6440047d0 tx=0x7fc64403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f36d490fc9fe1a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.332+0000 7fc63effd640 1 -- 172.21.15.45:0/761382269 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644004100 con 0x55c7cf251990 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.332+0000 7fc63effd640 1 -- 172.21.15.45:0/761382269 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc64402de90 con 0x55c7cf251990 2024-03-12T05:14:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.332+0000 7fc63effd640 1 -- 172.21.15.45:0/761382269 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644003e70 con 0x55c7cf251990 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- 172.21.15.45:0/761382269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 msgr2=0x55c7cf2598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf2598b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6440047d0 tx=0x7fc64403f000 comp rx=0 tx=0).stop 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- 172.21.15.45:0/761382269 shutdown_connections 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf25ee20 0x55c7cf25c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf2598b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 --2- 172.21.15.45:0/761382269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf251070 0x55c7cf25e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- 172.21.15.45:0/761382269 >> 172.21.15.45:0/761382269 conn(0x55c7cf24b840 msgr2=0x55c7cf25a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- 172.21.15.45:0/761382269 shutdown_connections 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- 172.21.15.45:0/761382269 wait complete. 2024-03-12T05:14:04.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.333+0000 7fc65e627740 1 -- start start 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.334+0000 7fc65e627740 1 Processor -- start 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.334+0000 7fc65e627740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf264e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 0x55c7cf265350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf2c5650 con 0x55c7cf25ee20 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf2c57c0 con 0x55c7cf251990 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7cf2c5980 con 0x55c7cf251070 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54346/0 (socket says 172.21.15.45:54346) 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 -- 172.21.15.45:0/2368365118 learned_addr learned my addr 172.21.15.45:0/2368365118 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659f14640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 0x55c7cf265350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc658f12640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf264e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 -- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 msgr2=0x55c7cf264e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf264e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 -- 172.21.15.45:0/2368365118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 msgr2=0x55c7cf265350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 10 client.0 _ll_register_callbacks cb 0x7fc650005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 0x55c7cf265350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 -- 172.21.15.45:0/2368365118 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65410d710 con 0x55c7cf251070 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659f14640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 0x55c7cf265350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7cf1de900 con 0x55c7cf251070 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.335+0000 7fc659713640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc65000ae60 tx=0x7fc6500040d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b06bceb229f1979b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc63cff9640 10 client.25897 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc63cff9640 1 -- 172.21.15.45:0/2368365118 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65000c750 con 0x55c7cf251070 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc63cff9640 1 -- 172.21.15.45:0/2368365118 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc65002e070 con 0x55c7cf251070 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc63cff9640 1 -- 172.21.15.45:0/2368365118 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc650005d40 con 0x55c7cf251070 2024-03-12T05:14:04.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 10 client.25897 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55c7cf179910 con 0x55c7cf251070 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc63cff9640 1 -- 172.21.15.45:0/2368365118 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc650016020 con 0x55c7cf251070 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 10 client.25897 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 10 client.25897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 20 client.25897 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.336+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.337+0000 7fc65e627740 4 client.25897 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.337+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c7ceeb8860 con 0x55c7cf26f0e0 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.337+0000 7fc658f12640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.337+0000 7fc63cff9640 10 client.25897 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.337+0000 7fc658f12640 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2c8bc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318b971140a855ae server_cookie=c284bd2aa72ec5cd in_seq=0 out_seq=0 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc63cff9640 1 -- 172.21.15.45:0/2368365118 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c7ceeb8860 con 0x55c7cf26f0e0 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc63cff9640 10 client.25897 handle_command_reply: tid=0 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 client.25897 shutdown 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 msgr2=0x55c7cf2648d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc65000ae60 tx=0x7fc6500040d0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 shutdown_connections 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7cf26f0e0 0x55c7cf2c8bc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7cf25ee20 0x55c7cf265350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc63cff9640 0 client.25897 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7cf251990 0x55c7cf264e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.339+0000 7fc65e627740 1 --2- 172.21.15.45:0/2368365118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7cf251070 0x55c7cf2648d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.340+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 >> 172.21.15.45:0/2368365118 conn(0x55c7cf24b840 msgr2=0x55c7cf24a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.340+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 shutdown_connections 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.340+0000 7fc65e627740 1 -- 172.21.15.45:0/2368365118 wait complete. 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.340+0000 7fc65e627740 20 client.25897 trim_cache size 0 max 16384 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "fc6a0a1f-ee43-4135-9832-595b54d59c3e", 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:14:04.340 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 msgr2=0x7fc6400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 0x7fc6400a40e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc644000c00 tx=0x7fc644005fd0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 msgr2=0x7fc65419b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00b970 tx=0x7fc64c00be30 comp rx=0 tx=0).stop 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 shutdown_connections 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc6400a1c20 0x7fc6400a40e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc65419ae80 0x7fc65419b330 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc654105330 0x7fc654196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 --2- 172.21.15.45:0/1734395060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc654100260 0x7fc6541965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 >> 172.21.15.45:0/1734395060 conn(0x7fc654077180 msgr2=0x7fc6540fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 shutdown_connections 2024-03-12T05:14:04.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.341+0000 7fc65b977640 1 -- 172.21.15.45:0/1734395060 wait complete. 2024-03-12T05:14:04.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:14:04.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:14:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 -- 172.21.15.45:0/174270885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c106150 msgr2=0x7f350c106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 --2- 172.21.15.45:0/174270885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c106150 0x7f350c106530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3508009900 tx=0x7f350802bc80 comp rx=0 tx=0).stop 2024-03-12T05:14:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 -- 172.21.15.45:0/174270885 shutdown_connections 2024-03-12T05:14:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 --2- 172.21.15.45:0/174270885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c100c90 0x7f350c10ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 --2- 172.21.15.45:0/174270885 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 --2- 172.21.15.45:0/174270885 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c106150 0x7f350c106530 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 -- 172.21.15.45:0/174270885 >> 172.21.15.45:0/174270885 conn(0x7f350c076e50 msgr2=0x7f350c077260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 -- 172.21.15.45:0/174270885 shutdown_connections 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.534+0000 7f351459a640 1 -- 172.21.15.45:0/174270885 wait complete. 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 Processor -- start 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 -- start start 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c100c90 0x7f350c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c106150 0x7f350c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c19c560 con 0x7f350c100c90 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c19c6d0 con 0x7f350c106150 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.535+0000 7f351459a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c19c890 con 0x7f350c1002d0 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3511b35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c100c90 0x7f350c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54354/0 (socket says 172.21.15.45:54354) 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 -- 172.21.15.45:0/830580951 learned_addr learned my addr 172.21.15.45:0/830580951 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512b37640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c106150 0x7f350c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 -- 172.21.15.45:0/830580951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c106150 msgr2=0x7f350c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c106150 0x7f350c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 -- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c100c90 msgr2=0x7f350c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c100c90 0x7f350c197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 -- 172.21.15.45:0/830580951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508009560 con 0x7f350c1002d0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f3512336640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3508005ab0 tx=0x7f3508004060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.536+0000 7f35037fe640 1 -- 172.21.15.45:0/830580951 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35080043e0 con 0x7f350c1002d0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.537+0000 7f35037fe640 1 -- 172.21.15.45:0/830580951 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3508003aa0 con 0x7f350c1002d0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.537+0000 7f35037fe640 1 -- 172.21.15.45:0/830580951 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35080054a0 con 0x7f350c1002d0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.537+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c19cb10 con 0x7f350c1002d0 2024-03-12T05:14:04.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.537+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c1a43f0 con 0x7f350c1002d0 2024-03-12T05:14:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.540+0000 7f35037fe640 1 -- 172.21.15.45:0/830580951 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f350803e450 con 0x7f350c1002d0 2024-03-12T05:14:04.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.540+0000 7f3517175740 1 Processor -- start 2024-03-12T05:14:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 -- start start 2024-03-12T05:14:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 0x5561a6771690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a677a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a66e0280 con 0x5561a67712b0 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a66dd7b0 con 0x5561a6771bd0 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3517175740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a66d8f90 con 0x5561a677a7b0 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3512b37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 0x5561a6771690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3511b35640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3511b35640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54366/0 (socket says 172.21.15.45:54366) 2024-03-12T05:14:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3512b37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 0x5561a6771690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39540/0 (socket says 172.21.15.45:39540) 2024-03-12T05:14:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3511b35640 1 -- 172.21.15.45:0/2975355810 learned_addr learned my addr 172.21.15.45:0/2975355810 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.541+0000 7f3512336640 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a677a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f3511b35640 1 -- 172.21.15.45:0/2975355810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 msgr2=0x5561a677a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f3511b35640 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a677a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f3511b35640 1 -- 172.21.15.45:0/2975355810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 msgr2=0x5561a6771690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f3511b35640 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 0x5561a6771690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f3511b35640 1 -- 172.21.15.45:0/2975355810 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561a6697570 con 0x5561a677a7b0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.542+0000 7f35037fe640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 0x7f34f80a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f3511b35640 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ca0 tx=0x7f34fc0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6128139d17a0ae49 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f3512b37640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 0x7f34f80a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f3500bfe640 1 -- 172.21.15.45:0/2975355810 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc019640 con 0x5561a677a7b0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f3500bfe640 1 -- 172.21.15.45:0/2975355810 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f34fc009e50 con 0x5561a677a7b0 2024-03-12T05:14:04.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f3500bfe640 1 -- 172.21.15.45:0/2975355810 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc011ab0 con 0x5561a677a7b0 2024-03-12T05:14:04.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.543+0000 7f35037fe640 1 -- 172.21.15.45:0/830580951 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f35080efe90 con 0x7f350c1002d0 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 -- 172.21.15.45:0/2975355810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 msgr2=0x5561a677cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3512b37640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 0x7f34f80a4120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3504005fd0 tx=0x7f3504009600 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ca0 tx=0x7f34fc0107a0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 -- 172.21.15.45:0/2975355810 shutdown_connections 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a677cba0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a677a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 --2- 172.21.15.45:0/2975355810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a67712b0 0x5561a6771690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.544+0000 7f3517175740 1 -- 172.21.15.45:0/2975355810 >> 172.21.15.45:0/2975355810 conn(0x5561a676b990 msgr2=0x5561a676bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.545+0000 7f3517175740 1 -- 172.21.15.45:0/2975355810 shutdown_connections 2024-03-12T05:14:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.545+0000 7f3517175740 1 -- 172.21.15.45:0/2975355810 wait complete. 2024-03-12T05:14:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.545+0000 7f3517175740 1 -- start start 2024-03-12T05:14:04.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 Processor -- start 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a677a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 0x5561a67775c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a6777c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a63ac430 con 0x5561a6771bd0 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a6368e00 con 0x5561a67712b0 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a665dfd0 con 0x5561a677a7b0 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3517175740 10 client.0 _ll_register_callbacks cb 0x5561a66b3fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:04.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3511b35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 0x5561a67775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3511b35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 0x5561a67775c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39550/0 (socket says 172.21.15.45:39550) 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3511b35640 1 -- 172.21.15.45:0/3086992237 learned_addr learned my addr 172.21.15.45:0/3086992237 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3512b37640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a6777c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.546+0000 7f3512336640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a677a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 -- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 msgr2=0x5561a677a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a677a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 -- 172.21.15.45:0/3086992237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 msgr2=0x5561a67775c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 0x5561a67775c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 -- 172.21.15.45:0/3086992237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc0098d0 con 0x5561a677a7b0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3512b37640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a6777c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3504011cf0 tx=0x7f3504004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9401579ef83c1452 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f34f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:04.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3504020620 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3504020780 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561a63d87e0 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.547+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f350401fe80 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f3517175740 10 client.25909 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5561a66d7610 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5561a63d87e0 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f350402f020 con 0x5561a677a7b0 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f3517175740 10 client.25909 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f3517175740 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.548+0000 7f3517175740 20 client.25909 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a678ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f3517175740 4 client.25909 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:04.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561a6415720 con 0x5561a678a890 2024-03-12T05:14:04.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f3512336640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a678ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f3512336640 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a678ccb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c16b17d7b9aabd server_cookie=f18f75933ca6410e in_seq=0 out_seq=0 2024-03-12T05:14:04.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.549+0000 7f34f5ffb640 10 client.25909 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f34f5ffb640 1 -- 172.21.15.45:0/3086992237 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5561a6415720 con 0x5561a678a890 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f34f5ffb640 10 client.25909 handle_command_reply: tid=0 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f3517175740 1 client.25909 shutdown 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 msgr2=0x5561a6777c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a6777c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3504011cf0 tx=0x7f3504004a40 comp rx=0 tx=0).stop 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 shutdown_connections 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.550+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a678ccb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a6771bd0 0x5561a67775c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a677a7b0 0x5561a6777c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f34f5ffb640 0 client.25909 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 --2- 172.21.15.45:0/3086992237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a677a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 >> 172.21.15.45:0/3086992237 conn(0x5561a676b990 msgr2=0x5561a676d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 shutdown_connections 2024-03-12T05:14:04.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 1 -- 172.21.15.45:0/3086992237 wait complete. 2024-03-12T05:14:04.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.551+0000 7f3517175740 20 client.25909 trim_cache size 0 max 16384 2024-03-12T05:14:04.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.577+0000 7f3517175740 1 Processor -- start 2024-03-12T05:14:04.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.577+0000 7f3517175740 1 -- start start 2024-03-12T05:14:04.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 0x5561a6791610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a6772d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a67874b0 con 0x5561a677a7b0 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a6778a50 con 0x5561a6771bd0 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3517175740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a67670d0 con 0x5561a67712b0 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3511b35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3511b35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41532/0 (socket says 172.21.15.45:41532) 2024-03-12T05:14:04.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3511b35640 1 -- 172.21.15.45:0/2831069451 learned_addr learned my addr 172.21.15.45:0/2831069451 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3512336640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 0x5561a6791610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.578+0000 7f3512b37640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a6772d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 -- 172.21.15.45:0/2831069451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 msgr2=0x5561a6791610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 0x5561a6791610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 -- 172.21.15.45:0/2831069451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 msgr2=0x5561a6772d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a6772d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 -- 172.21.15.45:0/2831069451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561a6673ac0 con 0x5561a6771bd0 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3512336640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 0x5561a6791610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3512b37640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a6772d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f3511b35640 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34fc005b90 tx=0x7f34fc009830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e9d8e7139a55bc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f34f7fff640 1 -- 172.21.15.45:0/2831069451 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc0054d0 con 0x5561a6771bd0 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f34f7fff640 1 -- 172.21.15.45:0/2831069451 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f34fc005630 con 0x5561a6771bd0 2024-03-12T05:14:04.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.579+0000 7f34f7fff640 1 -- 172.21.15.45:0/2831069451 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc024af0 con 0x5561a6771bd0 2024-03-12T05:14:04.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.580+0000 7f3517175740 1 -- 172.21.15.45:0/2831069451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 msgr2=0x5561a6772850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.580+0000 7f3517175740 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34fc005b90 tx=0x7f34fc009830 comp rx=0 tx=0).stop 2024-03-12T05:14:04.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 -- 172.21.15.45:0/2831069451 shutdown_connections 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a6772d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a6771bd0 0x5561a6772850 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 --2- 172.21.15.45:0/2831069451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a67712b0 0x5561a6791610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 -- 172.21.15.45:0/2831069451 >> 172.21.15.45:0/2831069451 conn(0x5561a676b990 msgr2=0x5561a6776b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 -- 172.21.15.45:0/2831069451 shutdown_connections 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 -- 172.21.15.45:0/2831069451 wait complete. 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 -- start start 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.581+0000 7f3517175740 1 Processor -- start 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 0x5561a677de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a677e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3512336640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3512336640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41540/0 (socket says 172.21.15.45:41540) 2024-03-12T05:14:04.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3512336640 1 -- 172.21.15.45:0/2364156864 learned_addr learned my addr 172.21.15.45:0/2364156864 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a677ea80 con 0x5561a677a7b0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a677ebf0 con 0x5561a67712b0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561a67e0d90 con 0x5561a6771bd0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.582+0000 7f3517175740 10 client.0 _ll_register_callbacks cb 0x7f34fc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3511b35640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 0x5561a677de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512b37640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a677e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 -- 172.21.15.45:0/2364156864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 msgr2=0x5561a677de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 0x5561a677de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 -- 172.21.15.45:0/2364156864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 msgr2=0x5561a677e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a677e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561a6697570 con 0x5561a67712b0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3511b35640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 0x5561a677de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512b37640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a677e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f3512336640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35080b8fd0 tx=0x7f350802bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8015509bee6b2e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.583+0000 7f34f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35080bdea0 con 0x5561a67712b0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f35080bcb70 con 0x5561a67712b0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34fc009e50 con 0x5561a67712b0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35080c0cc0 con 0x5561a67712b0 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f3517175740 10 client.25901 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:04.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5561a66996a0 con 0x5561a67712b0 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.584+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f35080c5310 con 0x5561a67712b0 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f350803c070 con 0x5561a67712b0 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 10 client.25901 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 10 client.25901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 20 client.25901 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a67e4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 4 client.25901 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f34fc009e50 con 0x5561a678a890 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3511b35640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a67e4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f34f5ffb640 10 client.25901 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.585+0000 7f3511b35640 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a67e4120 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d0a69bdd9569ee server_cookie=8c82b42cfffd77ae in_seq=0 out_seq=0 2024-03-12T05:14:04.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f34f5ffb640 1 -- 172.21.15.45:0/2364156864 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f34fc009e50 con 0x5561a678a890 2024-03-12T05:14:04.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f34f5ffb640 10 client.25901 handle_command_reply: tid=0 2024-03-12T05:14:04.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 client.25901 shutdown 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 msgr2=0x5561a6781a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f35080b8fd0 tx=0x7f350802bc80 comp rx=0 tx=0).stop 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 shutdown_connections 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5561a678a890 0x5561a67e4120 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561a677a7b0 0x5561a677e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.586+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561a6771bd0 0x5561a677de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f34f5ffb640 0 client.25901 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f3517175740 1 --2- 172.21.15.45:0/2364156864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561a67712b0 0x5561a6781a80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 >> 172.21.15.45:0/2364156864 conn(0x5561a676b990 msgr2=0x5561a6776b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 shutdown_connections 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f3517175740 1 -- 172.21.15.45:0/2364156864 wait complete. 2024-03-12T05:14:04.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.587+0000 7f3517175740 20 client.25901 trim_cache size 0 max 16384 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "fc6a0a1f-ee43-4135-9832-595b54d59c3e": { 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "fc6a0a1f-ee43-4135-9832-595b54d59c3e", 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:14:04.587 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:04.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 msgr2=0x7f34f80a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 0x7f34f80a4120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3504005fd0 tx=0x7f3504009600 comp rx=0 tx=0).stop 2024-03-12T05:14:04.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 msgr2=0x7f350c1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:04.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3508005ab0 tx=0x7f3508004060 comp rx=0 tx=0).stop 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 shutdown_connections 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f34f80a1c60 0x7f34f80a4120 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f350c100c90 0x7f350c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c106150 0x7f350c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 --2- 172.21.15.45:0/830580951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f350c1002d0 0x7f350c1976a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 >> 172.21.15.45:0/830580951 conn(0x7f350c076e50 msgr2=0x7f350c104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 shutdown_connections 2024-03-12T05:14:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:04.588+0000 7f351459a640 1 -- 172.21.15.45:0/830580951 wait complete. 2024-03-12T05:14:04.601 INFO:tasks.cephfs.filesystem:scrub status for tag:fc6a0a1f-ee43-4135-9832-595b54d59c3e - {'path': '/', 'tag': 'fc6a0a1f-ee43-4135-9832-595b54d59c3e', 'options': 'recursive,force'} 2024-03-12T05:14:04.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:04 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:14:04.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:04 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:04.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:04 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:04.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:04 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:04.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:04 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 120.0 s, 1186.0 tps, lat 84.255 ms stddev 74.464 2024-03-12T05:14:04.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:04 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:14:04.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:04 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:04.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:04 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:04.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:04 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:04.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:04 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:04.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:04 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:14:04.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:04 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:04.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:04 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:04.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:04 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:04.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:04 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:05.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:05.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:05 smithi045 ceph-mon[32174]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:05 smithi130 ceph-mon[35398]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:05.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:05 smithi116 ceph-mon[35486]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:07.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:07 smithi130 ceph-mon[35398]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:14:07.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:07 smithi116 ceph-mon[35486]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:14:08.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:07 smithi045 ceph-mon[32174]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:14:09.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 125.0 s, 1504.8 tps, lat 66.476 ms stddev 17.065 2024-03-12T05:14:09.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:09 smithi130 ceph-mon[35398]: pgmap v101: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.89k op/s 2024-03-12T05:14:09.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:09 smithi116 ceph-mon[35486]: pgmap v101: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.89k op/s 2024-03-12T05:14:10.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:09 smithi045 ceph-mon[32174]: pgmap v101: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 1.89k op/s 2024-03-12T05:14:11.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:11 smithi130 ceph-mon[35398]: pgmap v102: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:11 smithi116 ceph-mon[35486]: pgmap v102: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:12.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:11 smithi045 ceph-mon[32174]: pgmap v102: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:13.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:13 smithi130 ceph-mon[35398]: pgmap v103: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:13.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:13 smithi116 ceph-mon[35486]: pgmap v103: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:13 smithi045 ceph-mon[32174]: pgmap v103: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:14:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 130.0 s, 1596.8 tps, lat 62.638 ms stddev 15.854 2024-03-12T05:14:15.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:15 smithi130 ceph-mon[35398]: pgmap v104: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:14:15.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:15 smithi116 ceph-mon[35486]: pgmap v104: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:14:16.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:15 smithi045 ceph-mon[32174]: pgmap v104: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:14:17.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:17 smithi130 ceph-mon[35398]: pgmap v105: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:14:17.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:17 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:17.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:17 smithi116 ceph-mon[35486]: pgmap v105: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:14:17.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:17 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:18.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:17 smithi045 ceph-mon[32174]: pgmap v105: 97 pgs: 97 active+clean; 1.9 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:14:18.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:17 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:18.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:18.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:18.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:19.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:19.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:19.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:19.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 135.0 s, 1616.4 tps, lat 61.903 ms stddev 35.044 2024-03-12T05:14:19.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:19 smithi130 ceph-mon[35398]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:14:19.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:19 smithi116 ceph-mon[35486]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:14:20.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:19 smithi045 ceph-mon[32174]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:14:21.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:21 smithi130 ceph-mon[35398]: pgmap v107: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:21.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:21 smithi116 ceph-mon[35486]: pgmap v107: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:22.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:21 smithi045 ceph-mon[32174]: pgmap v107: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:23.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:23 smithi130 ceph-mon[35398]: pgmap v108: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:23.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:23 smithi116 ceph-mon[35486]: pgmap v108: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:24.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:23 smithi045 ceph-mon[32174]: pgmap v108: 97 pgs: 97 active+clean; 2.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:14:24.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 140.0 s, 1747.8 tps, lat 56.989 ms stddev 15.425 2024-03-12T05:14:25.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:25 smithi130 ceph-mon[35398]: pgmap v109: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 2.15k op/s 2024-03-12T05:14:25.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:25 smithi116 ceph-mon[35486]: pgmap v109: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 2.15k op/s 2024-03-12T05:14:26.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:25 smithi045 ceph-mon[32174]: pgmap v109: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 2.15k op/s 2024-03-12T05:14:27.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:27 smithi130 ceph-mon[35398]: pgmap v110: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.46k op/s 2024-03-12T05:14:27.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:27 smithi116 ceph-mon[35486]: pgmap v110: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.46k op/s 2024-03-12T05:14:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:27 smithi045 ceph-mon[32174]: pgmap v110: 97 pgs: 97 active+clean; 2.0 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.46k op/s 2024-03-12T05:14:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 145.0 s, 1456.2 tps, lat 68.910 ms stddev 64.909 2024-03-12T05:14:29.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:29 smithi130 ceph-mon[35398]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 2.22k op/s 2024-03-12T05:14:29.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:29 smithi116 ceph-mon[35486]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 2.22k op/s 2024-03-12T05:14:30.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:29 smithi045 ceph-mon[32174]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 2.22k op/s 2024-03-12T05:14:31.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:31 smithi130 ceph-mon[35398]: pgmap v112: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:31.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:31 smithi116 ceph-mon[35486]: pgmap v112: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:32.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:31 smithi045 ceph-mon[32174]: pgmap v112: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:33.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:33 smithi130 ceph-mon[35398]: pgmap v113: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:33.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:33 smithi116 ceph-mon[35486]: pgmap v113: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:34.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:33 smithi045 ceph-mon[32174]: pgmap v113: 97 pgs: 97 active+clean; 2.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:14:34.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:14:34.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.777+0000 7f029cfde640 1 -- 172.21.15.45:0/3238781015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298075fd0 msgr2=0x7f0298111960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.777+0000 7f029cfde640 1 --2- 172.21.15.45:0/3238781015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298075fd0 0x7f0298111960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0288009890 tx=0x7f028802bb70 comp rx=0 tx=0).stop 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 -- 172.21.15.45:0/3238781015 shutdown_connections 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 --2- 172.21.15.45:0/3238781015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0298111ea0 0x7f0298114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 --2- 172.21.15.45:0/3238781015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0298075fd0 0x7f0298111960 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0288009890 tx=0x7f028802bb70 comp rx=0 tx=0).stop 2024-03-12T05:14:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 --2- 172.21.15.45:0/3238781015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02980756b0 0x7f0298075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 -- 172.21.15.45:0/3238781015 >> 172.21.15.45:0/3238781015 conn(0x7f02980fec90 msgr2=0x7f02981010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 -- 172.21.15.45:0/3238781015 shutdown_connections 2024-03-12T05:14:34.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.778+0000 7f029cfde640 1 -- 172.21.15.45:0/3238781015 wait complete. 2024-03-12T05:14:34.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 Processor -- start 2024-03-12T05:14:34.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 -- start start 2024-03-12T05:14:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02980756b0 0x7f029819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 0x7f029819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02981a30c0 con 0x7f0298111ea0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02981a3230 con 0x7f02980756b0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.779+0000 7f029cfde640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02981a33a0 con 0x7f02981a26d0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47212/0 (socket says 172.21.15.45:47212) 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 -- 172.21.15.45:0/672998123 learned_addr learned my addr 172.21.15.45:0/672998123 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 -- 172.21.15.45:0/672998123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02980756b0 msgr2=0x7f029819de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02980756b0 0x7f029819de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 -- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 msgr2=0x7f029819e3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0296ffd640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 0x7f029819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 0x7f029819e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.780+0000 7f0297fff640 1 -- 172.21.15.45:0/672998123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288009490 con 0x7f02981a26d0 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.781+0000 7f0296ffd640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 0x7f029819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.781+0000 7f0297fff640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f029000eb40 tx=0x7f029000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.781+0000 7f0294ff9640 1 -- 172.21.15.45:0/672998123 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0290004b20 con 0x7f02981a26d0 2024-03-12T05:14:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.781+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02981aa980 con 0x7f02981a26d0 2024-03-12T05:14:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.781+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02981aae30 con 0x7f02981a26d0 2024-03-12T05:14:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.782+0000 7f0294ff9640 1 -- 172.21.15.45:0/672998123 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0290004c80 con 0x7f02981a26d0 2024-03-12T05:14:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.782+0000 7f0294ff9640 1 -- 172.21.15.45:0/672998123 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0290010690 con 0x7f02981a26d0 2024-03-12T05:14:34.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.785+0000 7f0294ff9640 1 -- 172.21.15.45:0/672998123 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0290005020 con 0x7f02981a26d0 2024-03-12T05:14:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 Processor -- start 2024-03-12T05:14:34.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 -- start start 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d3208f5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320900d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d320864280 con 0x55d3208f5db0 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3208617b0 con 0x55d3208f5490 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f02a0e14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d32085cf90 con 0x55d3208fe990 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.786+0000 7f0294ff9640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 0x7f02800a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57868/0 (socket says 172.21.15.45:57868) 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 -- 172.21.15.45:0/1633396108 learned_addr learned my addr 172.21.15.45:0/1633396108 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f0297fff640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 0x7f02800a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 -- 172.21.15.45:0/1633396108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 msgr2=0x55d320900d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320900d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f0294ff9640 1 -- 172.21.15.45:0/672998123 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0290014070 con 0x7f02981a26d0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 -- 172.21.15.45:0/1633396108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 msgr2=0x55d3208f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d3208f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 -- 172.21.15.45:0/1633396108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d32081b570 con 0x55d3208f5db0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f0297fff640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 0x7f02800a40e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02900a2dd0 tx=0x7f02900c5220 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.787+0000 7f02977fe640 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f028c009ac0 tx=0x7f028c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe045284a37dc024 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.788+0000 7f027dffb640 1 -- 172.21.15.45:0/1633396108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00ed10 con 0x55d3208f5db0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.788+0000 7f027dffb640 1 -- 172.21.15.45:0/1633396108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f028c00ee70 con 0x55d3208f5db0 2024-03-12T05:14:34.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.788+0000 7f027dffb640 1 -- 172.21.15.45:0/1633396108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c010430 con 0x55d3208f5db0 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 -- 172.21.15.45:0/1633396108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 msgr2=0x55d3208fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f028c009ac0 tx=0x7f028c009f80 comp rx=0 tx=0).stop 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 -- 172.21.15.45:0/1633396108 shutdown_connections 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d3208fe450 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320900d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 --2- 172.21.15.45:0/1633396108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d3208f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 -- 172.21.15.45:0/1633396108 >> 172.21.15.45:0/1633396108 conn(0x55d3208efbc0 msgr2=0x55d3208effa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.789+0000 7f02a0e14740 1 -- 172.21.15.45:0/1633396108 shutdown_connections 2024-03-12T05:14:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.790+0000 7f02a0e14740 1 -- 172.21.15.45:0/1633396108 wait complete. 2024-03-12T05:14:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.791+0000 7f02a0e14740 1 -- start start 2024-03-12T05:14:34.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.791+0000 7f02a0e14740 1 Processor -- start 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54638/0 (socket says 172.21.15.45:54638) 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02977fe640 1 -- 172.21.15.45:0/2292311319 learned_addr learned my addr 172.21.15.45:0/2292311319 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:34.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d320530430 con 0x55d3208fe990 2024-03-12T05:14:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3204ece00 con 0x55d3208f5db0 2024-03-12T05:14:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3207e1fd0 con 0x55d3208f5490 2024-03-12T05:14:34.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.792+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 -- 172.21.15.45:0/2292311319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 msgr2=0x55d3208fe3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fe3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 -- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 msgr2=0x55d3208fbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028c009770 con 0x55d3208f5db0 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02a0e14740 10 client.0 _ll_register_callbacks cb 0x55d320837fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:34.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d32055c7e0 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.793+0000 7f02977fe640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f028c004650 tx=0x7f028c00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee87f99b880c8042 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f0276ffd640 10 client.25907 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f0276ffd640 1 -- 172.21.15.45:0/2292311319 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c005200 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f0276ffd640 1 -- 172.21.15.45:0/2292311319 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f028c014070 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f0276ffd640 1 -- 172.21.15.45:0/2292311319 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f028c005360 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f02a0e14740 10 client.25907 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.794+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55d32085b610 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f0276ffd640 1 -- 172.21.15.45:0/2292311319 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f028c0054c0 con 0x55d3208f5db0 2024-03-12T05:14:34.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 10 client.25907 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 20 client.25907 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320910de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 4 client.25907 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d320599720 con 0x55d32090e920 2024-03-12T05:14:34.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320910de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.795+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320910de0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ace8f49d30a61ab server_cookie=94b27ad85076b027 in_seq=0 out_seq=0 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f0276ffd640 10 client.25907 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f0276ffd640 1 -- 172.21.15.45:0/2292311319 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d320599720 con 0x55d32090e920 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f0276ffd640 10 client.25907 handle_command_reply: tid=0 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 client.25907 shutdown 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 msgr2=0x55d3208fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f028c004650 tx=0x7f028c00ed40 comp rx=0 tx=0).stop 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 shutdown_connections 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320910de0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f0276ffd640 0 client.25907 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fb770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2292311319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.796+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 >> 172.21.15.45:0/2292311319 conn(0x55d3208efbc0 msgr2=0x55d3208f1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.797+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 shutdown_connections 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.797+0000 7f02a0e14740 1 -- 172.21.15.45:0/2292311319 wait complete. 2024-03-12T05:14:34.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.797+0000 7f02a0e14740 20 client.25907 trim_cache size 0 max 16384 2024-03-12T05:14:34.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.826+0000 7f02a0e14740 1 Processor -- start 2024-03-12T05:14:34.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.826+0000 7f02a0e14740 1 -- start start 2024-03-12T05:14:34.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3208eb300 con 0x55d3208fe990 2024-03-12T05:14:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3208ef450 con 0x55d3208f5db0 2024-03-12T05:14:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02a0e14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3208fcac0 con 0x55d3208f5490 2024-03-12T05:14:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54642/0 (socket says 172.21.15.45:54642) 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02977fe640 1 -- 172.21.15.45:0/447242188 learned_addr learned my addr 172.21.15.45:0/447242188 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f0296ffd640 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f0296ffd640 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02977fe640 1 -- 172.21.15.45:0/447242188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 msgr2=0x55d3208fdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.827+0000 7f02977fe640 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f02977fe640 1 -- 172.21.15.45:0/447242188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 msgr2=0x55d3208fb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f02977fe640 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f02977fe640 1 -- 172.21.15.45:0/447242188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3207f7ac0 con 0x55d3208f5db0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f02977fe640 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f028c00f8a0 tx=0x7f028c01e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7a3845541fecac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f027d7fa640 1 -- 172.21.15.45:0/447242188 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c009990 con 0x55d3208f5db0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f027d7fa640 1 -- 172.21.15.45:0/447242188 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d32055c7e0 con 0x55d3208f5db0 2024-03-12T05:14:34.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.828+0000 7f027d7fa640 1 -- 172.21.15.45:0/447242188 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c00e890 con 0x55d3208f5db0 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 -- 172.21.15.45:0/447242188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 msgr2=0x55d3208fe3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f028c00f8a0 tx=0x7f028c01e000 comp rx=0 tx=0).stop 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 -- 172.21.15.45:0/447242188 shutdown_connections 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208fe990 0x55d3208fb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5db0 0x55d3208fe3e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 --2- 172.21.15.45:0/447242188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208f5490 0x55d3208fdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.829+0000 7f02a0e14740 1 -- 172.21.15.45:0/447242188 >> 172.21.15.45:0/447242188 conn(0x55d3208efbc0 msgr2=0x55d3208f6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.830+0000 7f02a0e14740 1 -- 172.21.15.45:0/447242188 shutdown_connections 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.830+0000 7f02a0e14740 1 -- 172.21.15.45:0/447242188 wait complete. 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.830+0000 7f02a0e14740 1 -- start start 2024-03-12T05:14:34.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 Processor -- start 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d320901fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f0296ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f0296ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54650/0 (socket says 172.21.15.45:54650) 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320963ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3209026a0 con 0x55d3208f5db0 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d320964430 con 0x55d3208f5490 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3209645a0 con 0x55d3208fe990 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.831+0000 7f02a0e14740 10 client.0 _ll_register_callbacks cb 0x7f028c009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 -- 172.21.15.45:0/2210447205 learned_addr learned my addr 172.21.15.45:0/2210447205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f02977fe640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d320901fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320963ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 -- 172.21.15.45:0/2210447205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 msgr2=0x55d320963ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320963ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 -- 172.21.15.45:0/2210447205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 msgr2=0x55d320901fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d320901fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 -- 172.21.15.45:0/2210447205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d32085b610 con 0x55d3208f5490 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0296ffd640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f028802c0b0 tx=0x7f028c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b90dbfa78721a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.832+0000 7f0276ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288046070 con 0x55d3208f5490 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f0288048b70 con 0x55d3208f5490 2024-03-12T05:14:34.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f028c00ee70 con 0x55d3208f5490 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0288004040 con 0x55d3208f5490 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f02a0e14740 10 client.25913 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d32081d6a0 con 0x55d3208f5490 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.833+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f028804f030 con 0x55d3208f5490 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f028803e040 con 0x55d3208f5490 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 10 client.25913 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 10 client.25913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 20 client.25913 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320967e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 4 client.25913 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f028c00ee70 con 0x55d32090e920 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f02977fe640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320967e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.834+0000 7f0276ffd640 10 client.25913 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:34.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.835+0000 7f02977fe640 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320967e50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bb89a467a55319 server_cookie=1b1af7dc22a1d8b2 in_seq=0 out_seq=0 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.835+0000 7f0276ffd640 1 -- 172.21.15.45:0/2210447205 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f028c00ee70 con 0x55d32090e920 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.835+0000 7f0276ffd640 10 client.25913 handle_command_reply: tid=0 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.835+0000 7f02a0e14740 1 client.25913 shutdown 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 msgr2=0x55d320901a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f028802c0b0 tx=0x7f028c0094a0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 shutdown_connections 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d32090e920 0x55d320967e50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3208f5db0 0x55d320901fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3208fe990 0x55d320963ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 --2- 172.21.15.45:0/2210447205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3208f5490 0x55d320901a60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 >> 172.21.15.45:0/2210447205 conn(0x55d3208efbc0 msgr2=0x55d3208ee3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f0276ffd640 0 client.25913 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 shutdown_connections 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 1 -- 172.21.15.45:0/2210447205 wait complete. 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.836+0000 7f02a0e14740 20 client.25913 trim_cache size 0 max 16384 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:34.834 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 msgr2=0x7f02800a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 0x7f02800a40e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02900a2dd0 tx=0x7f02900c5220 comp rx=0 tx=0).stop 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 msgr2=0x7f02981a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f029000eb40 tx=0x7f029000c2d0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 shutdown_connections 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02800a1c20 0x7f02800a40e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0298111ea0 0x7f029819e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02981a26d0 0x7f02981a2b80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 --2- 172.21.15.45:0/672998123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02980756b0 0x7f029819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 >> 172.21.15.45:0/672998123 conn(0x7f02980fec90 msgr2=0x7f0298112620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 shutdown_connections 2024-03-12T05:14:34.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:34.837+0000 7f029cfde640 1 -- 172.21.15.45:0/672998123 wait complete. 2024-03-12T05:14:34.853 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:14:34.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:14:34.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 150.0 s, 1528.2 tps, lat 65.360 ms stddev 20.106 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.027+0000 7fef63623640 1 -- 172.21.15.45:0/1196024436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 msgr2=0x7fef5c109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.027+0000 7fef63623640 1 --2- 172.21.15.45:0/1196024436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c109df0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef54009ca0 tx=0x7fef5402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.027+0000 7fef63623640 1 -- 172.21.15.45:0/1196024436 shutdown_connections 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.027+0000 7fef63623640 1 --2- 172.21.15.45:0/1196024436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1167f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.028+0000 7fef63623640 1 --2- 172.21.15.45:0/1196024436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c109df0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.028+0000 7fef63623640 1 --2- 172.21.15.45:0/1196024436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.028+0000 7fef63623640 1 -- 172.21.15.45:0/1196024436 >> 172.21.15.45:0/1196024436 conn(0x7fef5c072740 msgr2=0x7fef5c072b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.028+0000 7fef63623640 1 -- 172.21.15.45:0/1196024436 shutdown_connections 2024-03-12T05:14:35.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.029+0000 7fef63623640 1 -- 172.21.15.45:0/1196024436 wait complete. 2024-03-12T05:14:35.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.030+0000 7fef63623640 1 Processor -- start 2024-03-12T05:14:35.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.030+0000 7fef63623640 1 -- start start 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c1a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1a4ec0 con 0x7fef5c10f930 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef60bbe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef60bbe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47264/0 (socket says 172.21.15.45:47264) 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef60bbe640 1 -- 172.21.15.45:0/1633933717 learned_addr learned my addr 172.21.15.45:0/1633933717 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1a5030 con 0x7fef5c109970 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.031+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1a51f0 con 0x7fef5c10a330 2024-03-12T05:14:35.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.032+0000 7fef61bc0640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.032+0000 7fef613bf640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c1a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.032+0000 7fef60bbe640 1 -- 172.21.15.45:0/1633933717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 msgr2=0x7fef5c1a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.032+0000 7fef60bbe640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c1a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.032+0000 7fef60bbe640 1 -- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 msgr2=0x7fef5c1a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef60bbe640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c1a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef60bbe640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c009490 con 0x7fef5c10a330 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef61bc0640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c1a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef60bbe640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef5402c680 tx=0x7fef54032f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54032820 con 0x7fef5c10a330 2024-03-12T05:14:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef58009490 con 0x7fef5c10a330 2024-03-12T05:14:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.033+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef54009900 con 0x7fef5c10a330 2024-03-12T05:14:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.034+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef54003e20 con 0x7fef5c10a330 2024-03-12T05:14:35.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.034+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54032820 con 0x7fef5c10a330 2024-03-12T05:14:35.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.036+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef40005290 con 0x7fef5c10a330 2024-03-12T05:14:35.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.037+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fef5400a040 con 0x7fef5c10a330 2024-03-12T05:14:35.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.037+0000 7fef527fc640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 0x7fef480a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.038+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fef54004040 con 0x7fef5c10a330 2024-03-12T05:14:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.038+0000 7fef613bf640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 0x7fef480a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.039+0000 7fef613bf640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 0x7fef480a4160 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef58009ce0 tx=0x7fef58005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.044+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef540b93a0 con 0x7fef5c10a330 2024-03-12T05:14:35.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.457+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef40005810 con 0x7fef5c10a330 2024-03-12T05:14:35.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.458+0000 7fef527fc640 1 -- 172.21.15.45:0/1633933717 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fef540b93a0 con 0x7fef5c10a330 2024-03-12T05:14:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:14:35.455 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:14:35.455 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 msgr2=0x7fef480a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 0x7fef480a4160 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fef58009ce0 tx=0x7fef58005f90 comp rx=0 tx=0).stop 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 msgr2=0x7fef5c1a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef5402c680 tx=0x7fef54032f80 comp rx=0 tx=0).stop 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 shutdown_connections 2024-03-12T05:14:35.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef480a1ca0 0x7fef480a4160 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef5c10f930 0x7fef5c1a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fef5c10a330 0x7fef5c1a0540 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 --2- 172.21.15.45:0/1633933717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef5c109970 0x7fef5c1a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 >> 172.21.15.45:0/1633933717 conn(0x7fef5c072740 msgr2=0x7fef5c06df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 shutdown_connections 2024-03-12T05:14:35.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.462+0000 7fef63623640 1 -- 172.21.15.45:0/1633933717 wait complete. 2024-03-12T05:14:35.476 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:35.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:14:35.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 -- 172.21.15.45:0/1357429122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c09d220 msgr2=0x7fa24c0a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 --2- 172.21.15.45:0/1357429122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c09d220 0x7fa24c0a4df0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa254066e50 tx=0x7fa254096e50 comp rx=0 tx=0).stop 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 -- 172.21.15.45:0/1357429122 shutdown_connections 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 --2- 172.21.15.45:0/1357429122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c09d220 0x7fa24c0a4df0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 --2- 172.21.15.45:0/1357429122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c098b00 0x7fa24c09cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 --2- 172.21.15.45:0/1357429122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c098530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 -- 172.21.15.45:0/1357429122 >> 172.21.15.45:0/1357429122 conn(0x7fa24c0939b0 msgr2=0x7fa24c095dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 -- 172.21.15.45:0/1357429122 shutdown_connections 2024-03-12T05:14:35.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.649+0000 7fa258a5e640 1 -- 172.21.15.45:0/1357429122 wait complete. 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 Processor -- start 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 -- start start 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 0x7fa24c12e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 0x7fa24c1330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57938/0 (socket says 172.21.15.45:57938) 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa2537fe640 1 -- 172.21.15.45:0/2711975237 learned_addr learned my addr 172.21.15.45:0/2711975237 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa252ffd640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 0x7fa24c12e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa253fff640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 0x7fa24c1330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c133600 con 0x7fa24c098150 2024-03-12T05:14:35.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c133770 con 0x7fa24c09d220 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.650+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c1338e0 con 0x7fa24c132c10 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 -- 172.21.15.45:0/2711975237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 msgr2=0x7fa24c1330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 0x7fa24c1330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 -- 172.21.15.45:0/2711975237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 msgr2=0x7fa24c12e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 0x7fa24c12e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25404f750 con 0x7fa24c098150 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa253fff640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 0x7fa24c1330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa252ffd640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 0x7fa24c12e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.651+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa24800c890 tx=0x7fa24800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.652+0000 7fa250ff9640 1 -- 172.21.15.45:0/2711975237 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248004220 con 0x7fa24c098150 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.652+0000 7fa250ff9640 1 -- 172.21.15.45:0/2711975237 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa248007de0 con 0x7fa24c098150 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.652+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c009bd0 con 0x7fa24c098150 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.652+0000 7fa250ff9640 1 -- 172.21.15.45:0/2711975237 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480047a0 con 0x7fa24c098150 2024-03-12T05:14:35.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.652+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c00a0d0 con 0x7fa24c098150 2024-03-12T05:14:35.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.656+0000 7fa250ff9640 1 -- 172.21.15.45:0/2711975237 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa248007930 con 0x7fa24c098150 2024-03-12T05:14:35.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.656+0000 7fa25c96a740 1 Processor -- start 2024-03-12T05:14:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 -- start start 2024-03-12T05:14:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 0x560b738dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738ddd80 0x560b738e6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 0x560b738e8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b73898960 con 0x560b738ddd80 2024-03-12T05:14:35.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b7384c310 con 0x560b738dd460 2024-03-12T05:14:35.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.657+0000 7fa25c96a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b73849840 con 0x560b738e6960 2024-03-12T05:14:35.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 0x560b738dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 0x560b738e8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 0x560b738dd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54686/0 (socket says 172.21.15.45:54686) 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa253fff640 1 -- 172.21.15.45:0/769227667 learned_addr learned my addr 172.21.15.45:0/769227667 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa250ff9640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 0x7fa2440a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 -- 172.21.15.45:0/769227667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 msgr2=0x560b738dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 0x560b738dd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 -- 172.21.15.45:0/769227667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738ddd80 msgr2=0x560b738e6420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738ddd80 0x560b738e6420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa252ffd640 1 -- 172.21.15.45:0/769227667 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b73807060 con 0x560b738e6960 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa253fff640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 0x7fa2440a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.658+0000 7fa250ff9640 1 -- 172.21.15.45:0/2711975237 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa2480cd3a0 con 0x7fa24c098150 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.659+0000 7fa252ffd640 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 0x560b738e8d50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007b10 tx=0x7fa240011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd3bac8f5c90b2fd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.659+0000 7fa253fff640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 0x7fa2440a40e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa254097460 tx=0x7fa25404ff00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.659+0000 7fa23dffb640 1 -- 172.21.15.45:0/769227667 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240004230 con 0x560b738e6960 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.659+0000 7fa23dffb640 1 -- 172.21.15.45:0/769227667 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa240004390 con 0x560b738e6960 2024-03-12T05:14:35.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.659+0000 7fa23dffb640 1 -- 172.21.15.45:0/769227667 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240004e40 con 0x560b738e6960 2024-03-12T05:14:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.660+0000 7fa25c96a740 1 -- 172.21.15.45:0/769227667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 msgr2=0x560b738e8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.660+0000 7fa25c96a740 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 0x560b738e8d50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa24c007b10 tx=0x7fa240011810 comp rx=0 tx=0).stop 2024-03-12T05:14:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 -- 172.21.15.45:0/769227667 shutdown_connections 2024-03-12T05:14:35.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738ddd80 0x560b738e6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738e6960 0x560b738e8d50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 --2- 172.21.15.45:0/769227667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738dd460 0x560b738dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 -- 172.21.15.45:0/769227667 >> 172.21.15.45:0/769227667 conn(0x560b738d7b90 msgr2=0x560b738d7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 -- 172.21.15.45:0/769227667 shutdown_connections 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 -- 172.21.15.45:0/769227667 wait complete. 2024-03-12T05:14:35.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.661+0000 7fa25c96a740 1 -- start start 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.662+0000 7fa25c96a740 1 Processor -- start 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.662+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738e3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.662+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738e43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa252ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54702/0 (socket says 172.21.15.45:54702) 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738e3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47292/0 (socket says 172.21.15.45:47292) 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa252ffd640 1 -- 172.21.15.45:0/1832254929 learned_addr learned my addr 172.21.15.45:0/1832254929 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b73845020 con 0x560b738e6960 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa253fff640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738e43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b73518430 con 0x560b738ddd80 2024-03-12T05:14:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b734d4e00 con 0x560b738dd460 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa25c96a740 10 client.0 _ll_register_callbacks cb 0x560b737eccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa253fff640 1 -- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 msgr2=0x560b738e3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa253fff640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738e3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.663+0000 7fa253fff640 1 -- 172.21.15.45:0/1832254929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 msgr2=0x560b738e3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa253fff640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa253fff640 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa240009c00 con 0x560b738e6960 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa252ffd640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b735447e0 con 0x560b738e6960 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa253fff640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738e43a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa254050150 tx=0x7fa2540aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5ee292bf623d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.664+0000 7fa232ffd640 10 client.16461 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa232ffd640 1 -- 172.21.15.45:0/1832254929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2540ad040 con 0x560b738e6960 2024-03-12T05:14:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa232ffd640 1 -- 172.21.15.45:0/1832254929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa254074e10 con 0x560b738e6960 2024-03-12T05:14:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa232ffd640 1 -- 172.21.15.45:0/1832254929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2540b5470 con 0x560b738e6960 2024-03-12T05:14:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa25c96a740 10 client.16461 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:35.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x560b738436a0 con 0x560b738e6960 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.667+0000 7fa232ffd640 1 -- 172.21.15.45:0/1832254929 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa2540b3070 con 0x560b738e6960 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 10 client.16461 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 10 client.16461 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 20 client.16461 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b738f6c80 0x560b738f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 4 client.16461 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b73581720 con 0x560b738f6c80 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.668+0000 7fa2537fe640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b738f6c80 0x560b738f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.669+0000 7fa232ffd640 10 client.16461 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:35.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.669+0000 7fa2537fe640 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b738f6c80 0x560b738f90a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5c465db6ae77b2d server_cookie=db48268d4e98fed2 in_seq=0 out_seq=0 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.669+0000 7fa232ffd640 1 -- 172.21.15.45:0/1832254929 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560b73581720 con 0x560b738f6c80 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.669+0000 7fa232ffd640 10 client.16461 handle_command_reply: tid=0 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.669+0000 7fa25c96a740 1 client.16461 shutdown 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 msgr2=0x560b738e43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738e43a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa254050150 tx=0x7fa2540aa040 comp rx=0 tx=0).stop 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 shutdown_connections 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b738f6c80 0x560b738f90a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738e43a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738e3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 --2- 172.21.15.45:0/1832254929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738e3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 >> 172.21.15.45:0/1832254929 conn(0x560b738d7b90 msgr2=0x560b738d98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa232ffd640 0 client.16461 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:35.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 shutdown_connections 2024-03-12T05:14:35.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 1 -- 172.21.15.45:0/1832254929 wait complete. 2024-03-12T05:14:35.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.670+0000 7fa25c96a740 20 client.16461 trim_cache size 0 max 16384 2024-03-12T05:14:35.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.697+0000 7fa25c96a740 1 Processor -- start 2024-03-12T05:14:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa25c96a740 1 -- start start 2024-03-12T05:14:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 0x560b738e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738e6960 0x560b738fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57956/0 (socket says 172.21.15.45:57956) 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa2537fe640 1 -- 172.21.15.45:0/3562297815 learned_addr learned my addr 172.21.15.45:0/3562297815 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa252ffd640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 0x560b738e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa253fff640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738e6960 0x560b738fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.698+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b737ca130 con 0x560b738dd460 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b738e9630 con 0x560b738e6960 2024-03-12T05:14:35.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b738e4d10 con 0x560b738ddd80 2024-03-12T05:14:35.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 -- 172.21.15.45:0/3562297815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 msgr2=0x560b738e41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 0x560b738e41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 -- 172.21.15.45:0/3562297815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738e6960 msgr2=0x560b738fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738e6960 0x560b738fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 -- 172.21.15.45:0/3562297815 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b737dfc20 con 0x560b738dd460 2024-03-12T05:14:35.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa252ffd640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 0x560b738e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.699+0000 7fa2537fe640 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa248005fd0 tx=0x7fa2480045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b8c06db9e0a84e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.700+0000 7fa23d7fa640 1 -- 172.21.15.45:0/3562297815 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248098120 con 0x560b738dd460 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.700+0000 7fa23d7fa640 1 -- 172.21.15.45:0/3562297815 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa248098380 con 0x560b738dd460 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.700+0000 7fa23d7fa640 1 -- 172.21.15.45:0/3562297815 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24807ee00 con 0x560b738dd460 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 msgr2=0x560b738e6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa248005fd0 tx=0x7fa2480045a0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 shutdown_connections 2024-03-12T05:14:35.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738e6960 0x560b738fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738ddd80 0x560b738e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 --2- 172.21.15.45:0/3562297815 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738dd460 0x560b738e6050 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 >> 172.21.15.45:0/3562297815 conn(0x560b738d7b90 msgr2=0x560b738deca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 shutdown_connections 2024-03-12T05:14:35.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.701+0000 7fa25c96a740 1 -- 172.21.15.45:0/3562297815 wait complete. 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.702+0000 7fa25c96a740 1 -- start start 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.702+0000 7fa25c96a740 1 Processor -- start 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.702+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.702+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738ec6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa25c96a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa25c96a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b738ee140 con 0x560b738e6960 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa25c96a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b7394d170 con 0x560b738ddd80 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47316/0 (socket says 172.21.15.45:47316) 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 -- 172.21.15.45:0/2127988592 learned_addr learned my addr 172.21.15.45:0/2127988592 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b7394d2b0 con 0x560b738dd460 2024-03-12T05:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa25c96a740 10 client.0 _ll_register_callbacks cb 0x7fa240009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa252ffd640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738ec6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 -- 172.21.15.45:0/2127988592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 msgr2=0x560b738ec6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738ec6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 -- 172.21.15.45:0/2127988592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 msgr2=0x560b738ecc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa253fff640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738ecc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 -- 172.21.15.45:0/2127988592 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24001d040 con 0x560b738dd460 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa253fff640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa2537fe640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa248096490 tx=0x7fa248098560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f725e1f1d3abc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.703+0000 7fa232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24807eba0 con 0x560b738dd460 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa24807ed00 con 0x560b738dd460 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa240004390 con 0x560b738dd460 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2480b8250 con 0x560b738dd460 2024-03-12T05:14:35.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa25c96a740 10 client.25951 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.704+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fa24c00a0d0 con 0x560b738dd460 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa240004390 con 0x560b738dd460 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa2480cd040 con 0x560b738dd460 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 10 client.25951 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 10 client.25951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 20 client.25951 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b7394e580 0x560b73950970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 4 client.25951 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560b735d0890 con 0x560b7394e580 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.705+0000 7fa252ffd640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b7394e580 0x560b73950970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa232ffd640 10 client.25951 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa252ffd640 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b7394e580 0x560b73950970 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2aacf161d5839b67 server_cookie=27d666163a37f3e in_seq=0 out_seq=0 2024-03-12T05:14:35.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa232ffd640 1 -- 172.21.15.45:0/2127988592 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560b735d0890 con 0x560b7394e580 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa232ffd640 10 client.25951 handle_command_reply: tid=0 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 client.25951 shutdown 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 msgr2=0x560b738ec1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa248096490 tx=0x7fa248098560 comp rx=0 tx=0).stop 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 shutdown_connections 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x560b7394e580 0x560b73950970 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa232ffd640 0 client.25951 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560b738e6960 0x560b738ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560b738ddd80 0x560b738ec6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 --2- 172.21.15.45:0/2127988592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560b738dd460 0x560b738ec1a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.706+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 >> 172.21.15.45:0/2127988592 conn(0x560b738d7b90 msgr2=0x560b738fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.707+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 shutdown_connections 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.707+0000 7fa25c96a740 1 -- 172.21.15.45:0/2127988592 wait complete. 2024-03-12T05:14:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.707+0000 7fa25c96a740 20 client.25951 trim_cache size 0 max 16384 2024-03-12T05:14:35.708 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 msgr2=0x7fa2440a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 0x7fa2440a40e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa254097460 tx=0x7fa25404ff00 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 msgr2=0x7fa24c12e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa24800c890 tx=0x7fa24800cd50 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 shutdown_connections 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2440a1c20 0x7fa2440a40e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24c132c10 0x7fa24c1330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa24c09d220 0x7fa24c12e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 --2- 172.21.15.45:0/2711975237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c098150 0x7fa24c12e350 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 >> 172.21.15.45:0/2711975237 conn(0x7fa24c0939b0 msgr2=0x7fa24c0944f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 shutdown_connections 2024-03-12T05:14:35.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.708+0000 7fa258a5e640 1 -- 172.21.15.45:0/2711975237 wait complete. 2024-03-12T05:14:35.721 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:35.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:14:35.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:35 smithi045 ceph-mon[32174]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:14:35.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1633933717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:35.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 -- 172.21.15.45:0/1079885967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 msgr2=0x7f4f2c109f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 --2- 172.21.15.45:0/1079885967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c109f00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009900 tx=0x7f4f1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 -- 172.21.15.45:0/1079885967 shutdown_connections 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 --2- 172.21.15.45:0/1079885967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c1154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 --2- 172.21.15.45:0/1079885967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c112b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 --2- 172.21.15.45:0/1079885967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c109f00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 -- 172.21.15.45:0/1079885967 >> 172.21.15.45:0/1079885967 conn(0x7f4f2c06c260 msgr2=0x7f4f2c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 -- 172.21.15.45:0/1079885967 shutdown_connections 2024-03-12T05:14:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.896+0000 7f4f32977640 1 -- 172.21.15.45:0/1079885967 wait complete. 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 Processor -- start 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 -- start start 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1ae340 con 0x7f4f2c1130d0 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1ae4b0 con 0x7f4f2c10a440 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f32977640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c1ae670 con 0x7f4f2c109b20 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.897+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54738/0 (socket says 172.21.15.45:54738) 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 -- 172.21.15.45:0/1015495196 learned_addr learned my addr 172.21.15.45:0/1015495196 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f30f14640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 -- 172.21.15.45:0/1015495196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 msgr2=0x7f4f2c19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 -- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 msgr2=0x7f4f2c19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f23fff640 1 -- 172.21.15.45:0/1015495196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c009560 con 0x7f4f2c10a440 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.898+0000 7f4f30f14640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400e9f0 tx=0x7f4f1400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f29ffb640 1 -- 172.21.15.45:0/1015495196 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14009700 con 0x7f4f2c10a440 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c1a0f30 con 0x7f4f2c10a440 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f29ffb640 1 -- 172.21.15.45:0/1015495196 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f1400cd80 con 0x7f4f2c10a440 2024-03-12T05:14:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f29ffb640 1 -- 172.21.15.45:0/1015495196 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14018840 con 0x7f4f2c10a440 2024-03-12T05:14:35.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.899+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c1a1490 con 0x7f4f2c10a440 2024-03-12T05:14:35.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.901+0000 7f4f29ffb640 1 -- 172.21.15.45:0/1015495196 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4f140189a0 con 0x7f4f2c10a440 2024-03-12T05:14:35.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.902+0000 7f4f29ffb640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 0x7f4f180a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.902+0000 7f4f29ffb640 1 -- 172.21.15.45:0/1015495196 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4f14014070 con 0x7f4f2c10a440 2024-03-12T05:14:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.902+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 0x7f4f180a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 Processor -- start 2024-03-12T05:14:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 0x7f4f180a4160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c005ab0 tx=0x7f4f1c005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 -- start start 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b91785a250 0x55b91785a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b917863210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917815960 con 0x55b91785ab70 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9177c9310 con 0x55b91785a250 2024-03-12T05:14:35.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.903+0000 7f4f35621740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9177c6840 con 0x55b917863750 2024-03-12T05:14:35.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47344/0 (socket says 172.21.15.45:47344) 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 -- 172.21.15.45:0/2764101438 learned_addr learned my addr 172.21.15.45:0/2764101438 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f23fff640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b917863210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b91785a250 0x55b91785a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 -- 172.21.15.45:0/2764101438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b91785a250 msgr2=0x55b91785a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b91785a250 0x55b91785a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 -- 172.21.15.45:0/2764101438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 msgr2=0x55b917863210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b917863210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f30f14640 1 -- 172.21.15.45:0/2764101438 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b917784060 con 0x55b917863750 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.904+0000 7f4f23fff640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b917863210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.905+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f20 tx=0x7f4f24010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27c3f97cc784046 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.905+0000 7f4f227fc640 1 -- 172.21.15.45:0/2764101438 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24011700 con 0x55b917863750 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.905+0000 7f4f227fc640 1 -- 172.21.15.45:0/2764101438 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f24015070 con 0x55b917863750 2024-03-12T05:14:35.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.905+0000 7f4f227fc640 1 -- 172.21.15.45:0/2764101438 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2401bbf0 con 0x55b917863750 2024-03-12T05:14:35.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 -- 172.21.15.45:0/2764101438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 msgr2=0x55b917865b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f20 tx=0x7f4f24010860 comp rx=0 tx=0).stop 2024-03-12T05:14:35.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 -- 172.21.15.45:0/2764101438 shutdown_connections 2024-03-12T05:14:35.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b917863210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b917863750 0x55b917865b40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 --2- 172.21.15.45:0/2764101438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b91785a250 0x55b91785a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 -- 172.21.15.45:0/2764101438 >> 172.21.15.45:0/2764101438 conn(0x55b917854980 msgr2=0x55b917854d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 -- 172.21.15.45:0/2764101438 shutdown_connections 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.906+0000 7f4f35621740 1 -- 172.21.15.45:0/2764101438 wait complete. 2024-03-12T05:14:35.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.907+0000 7f4f35621740 1 -- start start 2024-03-12T05:14:35.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.907+0000 7f4f35621740 1 Processor -- start 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917860c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917861190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9177c2020 con 0x55b91785a250 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917860c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58004/0 (socket says 172.21.15.45:58004) 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f2bfff640 1 -- 172.21.15.45:0/2327209643 learned_addr learned my addr 172.21.15.45:0/2327209643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f30f14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917861190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917495430 con 0x55b917863750 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.908+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917451e00 con 0x55b91785ab70 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f35621740 10 client.0 _ll_register_callbacks cb 0x55b917769cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:35.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 -- 172.21.15.45:0/2327209643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 msgr2=0x55b917860c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917860c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 -- 172.21.15.45:0/2327209643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 msgr2=0x55b917860710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f24009b80 con 0x55b917863750 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f23fff640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917860c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917861190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4f24010d60 tx=0x7f4f2401cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5e84cead485f93f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24029070 con 0x55b917863750 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9174c17e0 con 0x55b917863750 2024-03-12T05:14:35.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.909+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4f24017070 con 0x55b917863750 2024-03-12T05:14:35.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.910+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f2401f050 con 0x55b917863750 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.910+0000 7f4f35621740 10 client.25949 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.910+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b9177c06a0 con 0x55b917863750 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.910+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b9174c17e0 con 0x55b917863750 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f24005040 con 0x55b917863750 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 10 client.25949 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 10 client.25949 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 20 client.25949 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b917873a40 0x55b917875e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 4 client.25949 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9174fe720 con 0x55b917873a40 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.911+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b917873a40 0x55b917875e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.912+0000 7f4f0b7fe640 10 client.25949 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:35.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.912+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b917873a40 0x55b917875e50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c062324ddc2c827 server_cookie=630fc7acb5beab38 in_seq=0 out_seq=0 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/2327209643 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b9174fe720 con 0x55b917873a40 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f0b7fe640 10 client.25949 handle_command_reply: tid=0 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 client.25949 shutdown 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 msgr2=0x55b917861190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917861190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4f24010d60 tx=0x7f4f2401cb30 comp rx=0 tx=0).stop 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 shutdown_connections 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b917873a40 0x55b917875e50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917861190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917860c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 --2- 172.21.15.45:0/2327209643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b917860710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 >> 172.21.15.45:0/2327209643 conn(0x55b917854980 msgr2=0x55b9178566d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.913+0000 7f4f0b7fe640 0 client.25949 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:35.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.914+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 shutdown_connections 2024-03-12T05:14:35.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.914+0000 7f4f35621740 1 -- 172.21.15.45:0/2327209643 wait complete. 2024-03-12T05:14:35.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.914+0000 7f4f35621740 20 client.25949 trim_cache size 0 max 16384 2024-03-12T05:14:35.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 Processor -- start 2024-03-12T05:14:35.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 -- start start 2024-03-12T05:14:35.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b91785af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b91787a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917747130 con 0x55b91785ab70 2024-03-12T05:14:35.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9178706a0 con 0x55b917863750 2024-03-12T05:14:35.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f35621740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917866420 con 0x55b91785a250 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47364/0 (socket says 172.21.15.45:47364) 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f2bfff640 1 -- 172.21.15.45:0/2822944794 learned_addr learned my addr 172.21.15.45:0/2822944794 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f23fff640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b91785af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.939+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b91787a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 -- 172.21.15.45:0/2822944794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 msgr2=0x55b917862e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 -- 172.21.15.45:0/2822944794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 msgr2=0x55b91785af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b91785af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 -- 172.21.15.45:0/2822944794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b91775cc20 con 0x55b917863750 2024-03-12T05:14:35.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f23fff640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b91785af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f30f14640 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b91787a7f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4f2402f040 tx=0x7f4f240099c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15f43d5dd4012f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f21ffb640 1 -- 172.21.15.45:0/2822944794 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24029070 con 0x55b917863750 2024-03-12T05:14:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f21ffb640 1 -- 172.21.15.45:0/2822944794 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f24020040 con 0x55b917863750 2024-03-12T05:14:35.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.940+0000 7f4f21ffb640 1 -- 172.21.15.45:0/2822944794 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2401f070 con 0x55b917863750 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 -- 172.21.15.45:0/2822944794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 msgr2=0x55b91787a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b91787a7f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4f2402f040 tx=0x7f4f240099c0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 -- 172.21.15.45:0/2822944794 shutdown_connections 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785ab70 0x55b91785af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b91787a7f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 --2- 172.21.15.45:0/2822944794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785a250 0x55b917862e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 -- 172.21.15.45:0/2822944794 >> 172.21.15.45:0/2822944794 conn(0x55b917854980 msgr2=0x55b917854d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 -- 172.21.15.45:0/2822944794 shutdown_connections 2024-03-12T05:14:35.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.942+0000 7f4f35621740 1 -- 172.21.15.45:0/2822944794 wait complete. 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.943+0000 7f4f35621740 1 -- start start 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.943+0000 7f4f35621740 1 Processor -- start 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.943+0000 7f4f35621740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917868e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917869350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917869a30 con 0x55b91785a250 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58018/0 (socket says 172.21.15.45:58018) 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 -- 172.21.15.45:0/1734530729 learned_addr learned my addr 172.21.15.45:0/1734530729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b917869ba0 con 0x55b917863750 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9178c9f80 con 0x55b91785ab70 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f35621740 10 client.0 _ll_register_callbacks cb 0x7f4f2c109a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917868e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f30f14640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917869350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 -- 172.21.15.45:0/1734530729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 msgr2=0x55b917868e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917868e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 -- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 msgr2=0x55b917869350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917869350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f2bfff640 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c115ef0 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.944+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917868e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f2bfff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c02c170 tx=0x7f4f1c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a98d875c72bab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c054040 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4f1c047040 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b917860540 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f1c0035f0 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f35621740 10 client.16482 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.945+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f4f2c1a1490 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f1c066020 con 0x55b91785a250 2024-03-12T05:14:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f1c005200 con 0x55b91785a250 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 10 client.16482 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 10 client.16482 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 20 client.16482 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b9178cb700 0x55b9178cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 4 client.16482 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b917860540 con 0x55b9178cb700 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.946+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b9178cb700 0x55b9178cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.947+0000 7f4f23fff640 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b9178cb700 0x55b9178cdb20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fdac26f06bb5300 server_cookie=3f075c017c3620e2 in_seq=0 out_seq=0 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.947+0000 7f4f0b7fe640 10 client.16482 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.947+0000 7f4f0b7fe640 1 -- 172.21.15.45:0/1734530729 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b917860540 con 0x55b9178cb700 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.947+0000 7f4f0b7fe640 10 client.16482 handle_command_reply: tid=0 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.947+0000 7f4f35621740 1 client.16482 shutdown 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 msgr2=0x55b91786aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c02c170 tx=0x7f4f1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 shutdown_connections 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55b9178cb700 0x55b9178cdb20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b917863750 0x55b917869350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f0b7fe640 0 client.16482 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b91785ab70 0x55b917868e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 --2- 172.21.15.45:0/1734530729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b91785a250 0x55b91786aa70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 >> 172.21.15.45:0/1734530729 conn(0x55b917854980 msgr2=0x55b91785af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 shutdown_connections 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 1 -- 172.21.15.45:0/1734530729 wait complete. 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.948+0000 7f4f35621740 20 client.16482 trim_cache size 0 max 16384 2024-03-12T05:14:35.947 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 msgr2=0x7f4f180a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 0x7f4f180a4160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c005ab0 tx=0x7f4f1c005a20 comp rx=0 tx=0).stop 2024-03-12T05:14:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 msgr2=0x7f4f2c19f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400e9f0 tx=0x7f4f1400eeb0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 shutdown_connections 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f180a1ca0 0x7f4f180a4160 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f2c1130d0 0x7f4f2c19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f2c10a440 0x7f4f2c19f140 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 --2- 172.21.15.45:0/1015495196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f2c109b20 0x7f4f2c19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.950+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 >> 172.21.15.45:0/1015495196 conn(0x7f4f2c06c260 msgr2=0x7f4f2c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.951+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 shutdown_connections 2024-03-12T05:14:35.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:35.951+0000 7f4f32977640 1 -- 172.21.15.45:0/1015495196 wait complete. 2024-03-12T05:14:35.964 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:35.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:14:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:35 smithi130 ceph-mon[35398]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:14:35.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:35 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1633933717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:35.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:35 smithi116 ceph-mon[35486]: pgmap v114: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 2.08k op/s 2024-03-12T05:14:35.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:35 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1633933717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:14:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 -- 172.21.15.45:0/1436620410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f0105370 msgr2=0x7fc4f010cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/1436620410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f0105370 0x7fc4f010cf40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4e801ace0 tx=0x7fc4e803d110 comp rx=0 tx=0).stop 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 -- 172.21.15.45:0/1436620410 shutdown_connections 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/1436620410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f0105370 0x7fc4f010cf40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/1436620410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f0100c50 0x7fc4f0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.181+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/1436620410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f01002a0 0x7fc4f0100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.182+0000 7fc4f7f25640 1 -- 172.21.15.45:0/1436620410 >> 172.21.15.45:0/1436620410 conn(0x7fc4f0076400 msgr2=0x7fc4f0076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.182+0000 7fc4f7f25640 1 -- 172.21.15.45:0/1436620410 shutdown_connections 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.182+0000 7fc4f7f25640 1 -- 172.21.15.45:0/1436620410 wait complete. 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 Processor -- start 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 -- start start 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 0x7fc4f019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f0105370 0x7fc4f019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f006e6b0 con 0x7fc4f0105370 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f006e820 con 0x7fc4f006dc00 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f7f25640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f006e990 con 0x7fc4f01002a0 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f0105370 0x7fc4f019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 0x7fc4f019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f64c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 0x7fc4f019b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47386/0 (socket says 172.21.15.45:47386) 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f64c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54788/0 (socket says 172.21.15.45:54788) 2024-03-12T05:14:36.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.183+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/507619617 learned_addr learned my addr 172.21.15.45:0/507619617 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 -- 172.21.15.45:0/507619617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 msgr2=0x7fc4f019b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 0x7fc4f019b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 -- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f0105370 msgr2=0x7fc4f019b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f0105370 0x7fc4f019b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 -- 172.21.15.45:0/507619617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e801a910 con 0x7fc4f006dc00 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4f64c2640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc4e803d650 tx=0x7fc4e8004660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.184+0000 7fc4e6ffd640 1 -- 172.21.15.45:0/507619617 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e803dcc0 con 0x7fc4f006dc00 2024-03-12T05:14:36.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.185+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f01a2020 con 0x7fc4f006dc00 2024-03-12T05:14:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.185+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f01a24d0 con 0x7fc4f006dc00 2024-03-12T05:14:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.185+0000 7fc4e6ffd640 1 -- 172.21.15.45:0/507619617 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4e8004850 con 0x7fc4f006dc00 2024-03-12T05:14:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.185+0000 7fc4e6ffd640 1 -- 172.21.15.45:0/507619617 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e804e6c0 con 0x7fc4f006dc00 2024-03-12T05:14:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.187+0000 7fc4e6ffd640 1 -- 172.21.15.45:0/507619617 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc4e804c070 con 0x7fc4f006dc00 2024-03-12T05:14:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.188+0000 7fc4fabd4740 1 Processor -- start 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.188+0000 7fc4fabd4740 1 -- start start 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d39210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d3bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68ceb960 con 0x55ef68d39750 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68c9f310 con 0x55ef68d30250 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68c9c840 con 0x55ef68d30b70 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4e6ffd640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 0x7fc4dc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54796/0 (socket says 172.21.15.45:54796) 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 -- 172.21.15.45:0/2958570681 learned_addr learned my addr 172.21.15.45:0/2958570681 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d3bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d39210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f64c2640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 0x7fc4dc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 -- 172.21.15.45:0/2958570681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 msgr2=0x55ef68d39210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d39210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 -- 172.21.15.45:0/2958570681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 msgr2=0x55ef68d3bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d3bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4f54c0640 1 -- 172.21.15.45:0/2958570681 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef68c5a060 con 0x55ef68d30250 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.189+0000 7fc4e6ffd640 1 -- 172.21.15.45:0/507619617 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc4e8100340 con 0x7fc4f006dc00 2024-03-12T05:14:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.190+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009330 tx=0x7fc4e00094d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c656017f5b0bad server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.190+0000 7fc4dbfff640 1 -- 172.21.15.45:0/2958570681 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0017e60 con 0x55ef68d30250 2024-03-12T05:14:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.190+0000 7fc4dbfff640 1 -- 172.21.15.45:0/2958570681 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4e0015c10 con 0x55ef68d30250 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.190+0000 7fc4dbfff640 1 -- 172.21.15.45:0/2958570681 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0018b30 con 0x55ef68d30250 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 -- 172.21.15.45:0/2958570681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 msgr2=0x55ef68d30630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009330 tx=0x7fc4e00094d0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4f64c2640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 0x7fc4dc0a40e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80ccd80 tx=0x7fc4e809f280 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 -- 172.21.15.45:0/2958570681 shutdown_connections 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d3bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d39210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/2958570681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d30630 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.191+0000 7fc4fabd4740 1 -- 172.21.15.45:0/2958570681 >> 172.21.15.45:0/2958570681 conn(0x55ef68d2a980 msgr2=0x55ef68d2ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.192+0000 7fc4fabd4740 1 -- 172.21.15.45:0/2958570681 shutdown_connections 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.192+0000 7fc4fabd4740 1 -- 172.21.15.45:0/2958570681 wait complete. 2024-03-12T05:14:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.192+0000 7fc4fabd4740 1 -- start start 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.192+0000 7fc4fabd4740 1 Processor -- start 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d36660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68c98020 con 0x55ef68d39750 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6896b430 con 0x55ef68d30250 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68927e00 con 0x55ef68d30b70 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4fabd4740 10 client.0 _ll_register_callbacks cb 0x55ef68c3fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d36660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47418/0 (socket says 172.21.15.45:47418) 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/161725619 learned_addr learned my addr 172.21.15.45:0/161725619 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.193+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/161725619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 msgr2=0x55ef68d36660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d36660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/161725619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 msgr2=0x55ef68d370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/161725619 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e00173e0 con 0x55ef68d30b70 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d36660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00bb00 tx=0x7fc4ec0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b86ccde37cd1b8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec016070 con 0x55ef68d30b70 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef689977e0 con 0x55ef68d30b70 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.194+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc4ec005020 con 0x55ef68d30b70 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.195+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4ec016070 con 0x55ef68d30b70 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.195+0000 7fc4fabd4740 10 client.25978 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.195+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef68c966a0 con 0x55ef68d30b70 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.195+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ef689977e0 con 0x55ef68d30b70 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc4ec01b470 con 0x55ef68d30b70 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 10 client.25978 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 10 client.25978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 20 client.25978 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68d49b90 0x55ef68d4bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 4 client.25978 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef689d4720 con 0x55ef68d49b90 2024-03-12T05:14:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.196+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68d49b90 0x55ef68d4bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.197+0000 7fc4d97fa640 10 client.25978 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.197+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68d49b90 0x55ef68d4bfb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c912122bde892375 server_cookie=4c8c61de76aa7aef in_seq=0 out_seq=0 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4d97fa640 1 -- 172.21.15.45:0/161725619 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ef689d4720 con 0x55ef68d49b90 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4d97fa640 10 client.25978 handle_command_reply: tid=0 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 client.25978 shutdown 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 msgr2=0x55ef68d36ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00bb00 tx=0x7fc4ec0035f0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 shutdown_connections 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68d49b90 0x55ef68d4bfb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4d97fa640 0 client.25978 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d39750 0x55ef68d370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d30b70 0x55ef68d36ba0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.198+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/161725619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d36660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.199+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 >> 172.21.15.45:0/161725619 conn(0x55ef68d2a980 msgr2=0x55ef68d2c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.199+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 shutdown_connections 2024-03-12T05:14:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.199+0000 7fc4fabd4740 1 -- 172.21.15.45:0/161725619 wait complete. 2024-03-12T05:14:36.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.199+0000 7fc4fabd4740 20 client.25978 trim_cache size 0 max 16384 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.218+0000 7fc4fabd4740 1 Processor -- start 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.218+0000 7fc4fabd4740 1 -- start start 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 0x55ef68d36f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d35640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68c1d130 con 0x55ef68d30250 2024-03-12T05:14:36.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68d265c0 con 0x55ef68d30b70 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68d2a7b0 con 0x55ef68d39750 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58050/0 (socket says 172.21.15.45:58050) 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4f54c0640 1 -- 172.21.15.45:0/4167139499 learned_addr learned my addr 172.21.15.45:0/4167139499 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.219+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 0x55ef68d36f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d35640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/4167139499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 msgr2=0x55ef68d35640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d35640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/4167139499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 msgr2=0x55ef68d38d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/4167139499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef68c32c20 con 0x55ef68d30b70 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:36.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 0x55ef68d36f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec01b880 tx=0x7fc4ec003b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=874fba59e96a425b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4db7fe640 1 -- 172.21.15.45:0/4167139499 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec016040 con 0x55ef68d30b70 2024-03-12T05:14:36.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4db7fe640 1 -- 172.21.15.45:0/4167139499 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4ec00b5d0 con 0x55ef68d30b70 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.220+0000 7fc4db7fe640 1 -- 172.21.15.45:0/4167139499 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec01bb50 con 0x55ef68d30b70 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.221+0000 7fc4fabd4740 1 -- 172.21.15.45:0/4167139499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 msgr2=0x55ef68d36f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.221+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 0x55ef68d36f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec01b880 tx=0x7fc4ec003b70 comp rx=0 tx=0).stop 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 -- 172.21.15.45:0/4167139499 shutdown_connections 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d35640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30b70 0x55ef68d36f00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/4167139499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30250 0x55ef68d38d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 -- 172.21.15.45:0/4167139499 >> 172.21.15.45:0/4167139499 conn(0x55ef68d2a980 msgr2=0x55ef68d34f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 -- 172.21.15.45:0/4167139499 shutdown_connections 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 -- 172.21.15.45:0/4167139499 wait complete. 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 -- start start 2024-03-12T05:14:36.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.222+0000 7fc4fabd4740 1 Processor -- start 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 0x55ef68d3f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d3fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68da00a0 con 0x55ef68d30b70 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68da01e0 con 0x55ef68d30250 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef68da0320 con 0x55ef68d39750 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4fabd4740 10 client.0 _ll_register_callbacks cb 0x7fc4e0009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 0x55ef68d3f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f5cc1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 0x55ef68d3f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58058/0 (socket says 172.21.15.45:58058) 2024-03-12T05:14:36.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f54c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54820/0 (socket says 172.21.15.45:54820) 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f5cc1640 1 -- 172.21.15.45:0/3235674559 learned_addr learned my addr 172.21.15.45:0/3235674559 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.223+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d3fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 -- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 msgr2=0x55ef68d3fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d3fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 -- 172.21.15.45:0/3235674559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 msgr2=0x55ef68d3f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 0x55ef68d3f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 -- 172.21.15.45:0/3235674559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef68c5a060 con 0x55ef68d30250 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4f54c0640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0006aa0 tx=0x7fc4e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=996fffe9aa572093 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0018650 con 0x55ef68d30250 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4e0015c10 con 0x55ef68d30250 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fc4e00187b0 con 0x55ef68d30250 2024-03-12T05:14:36.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4e0018970 con 0x55ef68d30250 2024-03-12T05:14:36.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4fabd4740 10 client.25973 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.224+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef68c966a0 con 0x55ef68d30250 2024-03-12T05:14:36.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4e0015c10 con 0x55ef68d30250 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc4e0022400 con 0x55ef68d30250 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 10 client.25973 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 10 client.25973 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 20 client.25973 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68da1930 0x55ef68da3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 4 client.25973 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.225+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef68a23890 con 0x55ef68da1930 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.226+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68da1930 0x55ef68da3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.226+0000 7fc4d97fa640 10 client.25973 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.226+0000 7fc4f5cc1640 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68da1930 0x55ef68da3d20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b93c27dd53df63 server_cookie=16146ec6d69832d3 in_seq=0 out_seq=0 2024-03-12T05:14:36.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.226+0000 7fc4d97fa640 1 -- 172.21.15.45:0/3235674559 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef68a23890 con 0x55ef68da1930 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.226+0000 7fc4d97fa640 10 client.25973 handle_command_reply: tid=0 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 client.25973 shutdown 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 msgr2=0x55ef68d3f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0006aa0 tx=0x7fc4e00046c0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 shutdown_connections 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55ef68da1930 0x55ef68da3d20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4d97fa640 0 client.25973 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef68d30b70 0x55ef68d3f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef68d39750 0x55ef68d3fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 --2- 172.21.15.45:0/3235674559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef68d30250 0x55ef68d3f150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 >> 172.21.15.45:0/3235674559 conn(0x55ef68d2a980 msgr2=0x55ef68d292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 shutdown_connections 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 1 -- 172.21.15.45:0/3235674559 wait complete. 2024-03-12T05:14:36.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.227+0000 7fc4fabd4740 20 client.25973 trim_cache size 0 max 16384 2024-03-12T05:14:36.228 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 msgr2=0x7fc4dc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 0x7fc4dc0a40e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80ccd80 tx=0x7fc4e809f280 comp rx=0 tx=0).stop 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 msgr2=0x7fc4f006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc4e803d650 tx=0x7fc4e8004660 comp rx=0 tx=0).stop 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 shutdown_connections 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4dc0a1c20 0x7fc4dc0a40e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f0105370 0x7fc4f019b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4f006dc00 0x7fc4f006e0b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 --2- 172.21.15.45:0/507619617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4f01002a0 0x7fc4f019b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 >> 172.21.15.45:0/507619617 conn(0x7fc4f0076400 msgr2=0x7fc4f00fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 shutdown_connections 2024-03-12T05:14:36.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.228+0000 7fc4f7f25640 1 -- 172.21.15.45:0/507619617 wait complete. 2024-03-12T05:14:36.241 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:36.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2401994133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38105330 msgr2=0x7f6a3810cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2401994133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38105330 0x7f6a3810cf00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009cd0 tx=0x7f6a3002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2401994133 shutdown_connections 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2401994133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38105330 0x7f6a3810cf00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2401994133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38100c10 0x7f6a38104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2401994133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38100260 0x7f6a38100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2401994133 >> 172.21.15.45:0/2401994133 conn(0x7f6a38077180 msgr2=0x7f6a38077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2401994133 shutdown_connections 2024-03-12T05:14:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.415+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2401994133 wait complete. 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.416+0000 7f6a3f5e2640 1 Processor -- start 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.416+0000 7f6a3f5e2640 1 -- start start 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38105330 0x7f6a38195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 0x7f6a3819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3819a7b0 con 0x7f6a38199dc0 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3819a920 con 0x7f6a38100260 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3f5e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3819aa90 con 0x7f6a38105330 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54838/0 (socket says 172.21.15.45:54838) 2024-03-12T05:14:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 -- 172.21.15.45:0/2571761965 learned_addr learned my addr 172.21.15.45:0/2571761965 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 0x7f6a3819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 -- 172.21.15.45:0/2571761965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38105330 msgr2=0x7f6a38195a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38105330 0x7f6a38195a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.417+0000 7f6a3d37e640 1 -- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 msgr2=0x7f6a3819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 0x7f6a3819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a3d37e640 1 -- 172.21.15.45:0/2571761965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a34009490 con 0x7f6a38100260 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 0x7f6a3819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400ea60 tx=0x7f6a3400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a2e7fc640 1 -- 172.21.15.45:0/2571761965 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400cde0 con 0x7f6a38100260 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a2e7fc640 1 -- 172.21.15.45:0/2571761965 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6a340044d0 con 0x7f6a38100260 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.418+0000 7f6a2e7fc640 1 -- 172.21.15.45:0/2571761965 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34010470 con 0x7f6a38100260 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.419+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a30009900 con 0x7f6a38100260 2024-03-12T05:14:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.419+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a381a2340 con 0x7f6a38100260 2024-03-12T05:14:36.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.422+0000 7f6a2e7fc640 1 -- 172.21.15.45:0/2571761965 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6a34010610 con 0x7f6a38100260 2024-03-12T05:14:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.423+0000 7f6a421bb740 1 Processor -- start 2024-03-12T05:14:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 -- start start 2024-03-12T05:14:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a2e7fc640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 0x7f6a240a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a4953630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 0x55b7a495c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a495c750 0x55b7a495eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a490e960 con 0x55b7a4953250 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a48c2310 con 0x55b7a495c750 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a421bb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a48bf840 con 0x55b7a4953b70 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 0x7f6a240a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a3cb7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 0x55b7a495c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a3cb7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 0x55b7a495c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47456/0 (socket says 172.21.15.45:47456) 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a3cb7d640 1 -- 172.21.15.45:0/627474695 learned_addr learned my addr 172.21.15.45:0/627474695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a4953630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.424+0000 7f6a2e7fc640 1 -- 172.21.15.45:0/2571761965 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6a34014070 con 0x7f6a38100260 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 -- 172.21.15.45:0/627474695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 msgr2=0x55b7a495c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 0x55b7a495c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 -- 172.21.15.45:0/627474695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a495c750 msgr2=0x55b7a495eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a495c750 0x55b7a495eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 -- 172.21.15.45:0/627474695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7a487d060 con 0x55b7a4953250 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 0x7f6a240a85b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006010 tx=0x7f6a30005d80 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.425+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a4953630 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6a340cb490 tx=0x7f6a3400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98d583ed7073f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.426+0000 7f6a237fe640 1 -- 172.21.15.45:0/627474695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3409a970 con 0x55b7a4953250 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.426+0000 7f6a237fe640 1 -- 172.21.15.45:0/627474695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6a3409eac0 con 0x55b7a4953250 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.426+0000 7f6a237fe640 1 -- 172.21.15.45:0/627474695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a340cb710 con 0x55b7a4953250 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 -- 172.21.15.45:0/627474695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 msgr2=0x55b7a4953630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a4953630 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6a340cb490 tx=0x7f6a3400ef90 comp rx=0 tx=0).stop 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 -- 172.21.15.45:0/627474695 shutdown_connections 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a495c750 0x55b7a495eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a4953b70 0x55b7a495c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 --2- 172.21.15.45:0/627474695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a4953630 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 -- 172.21.15.45:0/627474695 >> 172.21.15.45:0/627474695 conn(0x55b7a494d980 msgr2=0x55b7a494dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.427+0000 7f6a421bb740 1 -- 172.21.15.45:0/627474695 shutdown_connections 2024-03-12T05:14:36.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.428+0000 7f6a421bb740 1 -- 172.21.15.45:0/627474695 wait complete. 2024-03-12T05:14:36.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.428+0000 7f6a421bb740 1 -- start start 2024-03-12T05:14:36.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.428+0000 7f6a421bb740 1 Processor -- start 2024-03-12T05:14:36.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.429+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58108/0 (socket says 172.21.15.45:58108) 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a3d37e640 1 -- 172.21.15.45:0/3282755142 learned_addr learned my addr 172.21.15.45:0/3282755142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a4959cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a495a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.430+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a48bb020 con 0x55b7a4953250 2024-03-12T05:14:36.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a458e430 con 0x55b7a4953b70 2024-03-12T05:14:36.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a454ae00 con 0x55b7a495c750 2024-03-12T05:14:36.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a495a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3cb7d640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a4959cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a421bb740 10 client.0 _ll_register_callbacks cb 0x55b7a4862cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 -- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 msgr2=0x55b7a4959cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a4959cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 -- 172.21.15.45:0/3282755142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 msgr2=0x55b7a49597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3db7f640 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a381a2340 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.431+0000 7f6a3cb7d640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a4959cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7a45ba7e0 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a495a230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c700 tx=0x7f6a3002c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c4751fbcbcb873 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a20ff9640 10 client.25993 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a20ff9640 1 -- 172.21.15.45:0/3282755142 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003b70 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a20ff9640 1 -- 172.21.15.45:0/3282755142 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6a30047050 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.432+0000 7f6a20ff9640 1 -- 172.21.15.45:0/3282755142 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a30032eb0 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 10 client.25993 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55b7a48b96a0 con 0x55b7a495c750 2024-03-12T05:14:36.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a20ff9640 1 -- 172.21.15.45:0/3282755142 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6a3003e040 con 0x55b7a495c750 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 10 client.25993 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 10 client.25993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 20 client.25993 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a496cac0 0x55b7a496eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 4 client.25993 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7a45f7720 con 0x55b7a496cac0 2024-03-12T05:14:36.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.433+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a496cac0 0x55b7a496eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.434+0000 7f6a20ff9640 10 client.25993 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:36.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.434+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a496cac0 0x55b7a496eee0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48524a3e83c8c259 server_cookie=74c291ca50da840c in_seq=0 out_seq=0 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.435+0000 7f6a20ff9640 1 -- 172.21.15.45:0/3282755142 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b7a45f7720 con 0x55b7a496cac0 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.435+0000 7f6a20ff9640 10 client.25993 handle_command_reply: tid=0 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.435+0000 7f6a421bb740 1 client.25993 shutdown 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.435+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 msgr2=0x55b7a495a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.435+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a495a230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c700 tx=0x7f6a3002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 shutdown_connections 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a496cac0 0x55b7a496eee0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a495a230 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a4959cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 --2- 172.21.15.45:0/3282755142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a49597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 >> 172.21.15.45:0/3282755142 conn(0x55b7a494d980 msgr2=0x55b7a494f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a20ff9640 0 client.25993 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:36.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 shutdown_connections 2024-03-12T05:14:36.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 1 -- 172.21.15.45:0/3282755142 wait complete. 2024-03-12T05:14:36.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.436+0000 7f6a421bb740 20 client.25993 trim_cache size 0 max 16384 2024-03-12T05:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.459+0000 7f6a421bb740 1 Processor -- start 2024-03-12T05:14:36.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.459+0000 7f6a421bb740 1 -- start start 2024-03-12T05:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.459+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a495bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.459+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a49580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a421bb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a4840130 con 0x55b7a4953250 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a421bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a4949410 con 0x55b7a4953b70 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a421bb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a496c8c0 con 0x55b7a495c750 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47468/0 (socket says 172.21.15.45:47468) 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 -- 172.21.15.45:0/1007330473 learned_addr learned my addr 172.21.15.45:0/1007330473 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a495bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3cb7d640 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a49580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 -- 172.21.15.45:0/1007330473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 msgr2=0x55b7a49580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a49580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 -- 172.21.15.45:0/1007330473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 msgr2=0x55b7a495bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a495bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.460+0000 7f6a3db7f640 1 -- 172.21.15.45:0/1007330473 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7a4855c20 con 0x55b7a495c750 2024-03-12T05:14:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.461+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a30004af0 tx=0x7f6a3004a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a25c287819bf29 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.461+0000 7f6a22ffd640 1 -- 172.21.15.45:0/1007330473 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30003970 con 0x55b7a495c750 2024-03-12T05:14:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.461+0000 7f6a22ffd640 1 -- 172.21.15.45:0/1007330473 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b7a45ba7e0 con 0x55b7a495c750 2024-03-12T05:14:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.461+0000 7f6a22ffd640 1 -- 172.21.15.45:0/1007330473 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30004130 con 0x55b7a495c750 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 -- 172.21.15.45:0/1007330473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 msgr2=0x55b7a49585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a30004af0 tx=0x7f6a3004a000 comp rx=0 tx=0).stop 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 -- 172.21.15.45:0/1007330473 shutdown_connections 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a49585e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953b70 0x55b7a49580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 --2- 172.21.15.45:0/1007330473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953250 0x55b7a495bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 -- 172.21.15.45:0/1007330473 >> 172.21.15.45:0/1007330473 conn(0x55b7a494d980 msgr2=0x55b7a4954ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 -- 172.21.15.45:0/1007330473 shutdown_connections 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.462+0000 7f6a421bb740 1 -- 172.21.15.45:0/1007330473 wait complete. 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a421bb740 1 -- start start 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a421bb740 1 Processor -- start 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a421bb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953b70 0x55b7a49627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.463+0000 7f6a3d37e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54868/0 (socket says 172.21.15.45:54868) 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3d37e640 1 -- 172.21.15.45:0/588325451 learned_addr learned my addr 172.21.15.45:0/588325451 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a4962d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3cb7d640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953b70 0x55b7a49627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a49c3180 con 0x55b7a4953b70 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a49c32f0 con 0x55b7a4953250 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a49c34b0 con 0x55b7a495c750 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a421bb740 10 client.0 _ll_register_callbacks cb 0x7f6a34003a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a4962d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 -- 172.21.15.45:0/588325451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 msgr2=0x55b7a4962290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 -- 172.21.15.45:0/588325451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953b70 msgr2=0x55b7a49627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953b70 0x55b7a49627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b7a48b96a0 con 0x55b7a495c750 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a3db7f640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a4962d10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c700 tx=0x7f6a3004bfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=913c777ceca1e1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.464+0000 7f6a20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:14:36.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30004d40 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6a30047040 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a3409eac0 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a30004ea0 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a421bb740 10 client.25999 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b7a487b8c0 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6a30076020 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6a3003e050 con 0x55b7a495c750 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a421bb740 10 client.25999 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.465+0000 7f6a421bb740 10 client.25999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.466+0000 7f6a421bb740 20 client.25999 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.466+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a49c47b0 0x55b7a49c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.466+0000 7f6a421bb740 4 client.25999 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.466+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b7a4646890 con 0x55b7a49c47b0 2024-03-12T05:14:36.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.466+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a49c47b0 0x55b7a49c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.467+0000 7f6a3d37e640 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a49c47b0 0x55b7a49c6ba0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e085772b32932bde server_cookie=24a7cbf5f5e08002 in_seq=0 out_seq=0 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.467+0000 7f6a20ff9640 10 client.25999 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.467+0000 7f6a20ff9640 1 -- 172.21.15.45:0/588325451 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b7a4646890 con 0x55b7a49c47b0 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.467+0000 7f6a20ff9640 10 client.25999 handle_command_reply: tid=0 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.467+0000 7f6a421bb740 1 client.25999 shutdown 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 msgr2=0x55b7a4962d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a4962d10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002c700 tx=0x7f6a3004bfe0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 shutdown_connections 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b7a4953b70 0x55b7a49627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b7a49c47b0 0x55b7a49c6ba0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b7a495c750 0x55b7a4962d10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a20ff9640 0 client.25999 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 --2- 172.21.15.45:0/588325451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b7a4953250 0x55b7a4962290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 >> 172.21.15.45:0/588325451 conn(0x55b7a494d980 msgr2=0x55b7a4958060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 shutdown_connections 2024-03-12T05:14:36.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 1 -- 172.21.15.45:0/588325451 wait complete. 2024-03-12T05:14:36.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.468+0000 7f6a421bb740 20 client.25999 trim_cache size 0 max 16384 2024-03-12T05:14:36.469 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 msgr2=0x7f6a240a85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 0x7f6a240a85b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006010 tx=0x7f6a30005d80 comp rx=0 tx=0).stop 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 msgr2=0x7f6a38195500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400ea60 tx=0x7f6a3400ef20 comp rx=0 tx=0).stop 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 shutdown_connections 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6a240a60f0 0x7f6a240a85b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a38199dc0 0x7f6a3819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6a38105330 0x7f6a38195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 --2- 172.21.15.45:0/2571761965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a38100260 0x7f6a38195500 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.469+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 >> 172.21.15.45:0/2571761965 conn(0x7f6a38077180 msgr2=0x7f6a380fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.470+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 shutdown_connections 2024-03-12T05:14:36.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.470+0000 7f6a3f5e2640 1 -- 172.21.15.45:0/2571761965 wait complete. 2024-03-12T05:14:36.483 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:14:36.483 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:14:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.657+0000 7fbe667fc640 1 -- 172.21.15.45:0/933830095 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600329d0 con 0x7fbe6811a960 2024-03-12T05:14:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.657+0000 7fbe6df25640 1 -- 172.21.15.45:0/933830095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6811a960 msgr2=0x7fbe6811cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.657+0000 7fbe6df25640 1 --2- 172.21.15.45:0/933830095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6811a960 0x7fbe6811cd50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009ca0 tx=0x7fbe6002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 -- 172.21.15.45:0/933830095 shutdown_connections 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 --2- 172.21.15.45:0/933830095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6811a960 0x7fbe6811cd50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 --2- 172.21.15.45:0/933830095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6810d9d0 0x7fbe6810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 --2- 172.21.15.45:0/933830095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810f9a0 0x7fbe6810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 -- 172.21.15.45:0/933830095 >> 172.21.15.45:0/933830095 conn(0x7fbe6806c260 msgr2=0x7fbe6806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 -- 172.21.15.45:0/933830095 shutdown_connections 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.658+0000 7fbe6df25640 1 -- 172.21.15.45:0/933830095 wait complete. 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.659+0000 7fbe6df25640 1 Processor -- start 2024-03-12T05:14:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.659+0000 7fbe6df25640 1 -- start start 2024-03-12T05:14:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe6df25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 0x7fbe68119d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe6df25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe6df25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6811a960 0x7fbe68112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 0x7fbe68119d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 0x7fbe68119d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58150/0 (socket says 172.21.15.45:58150) 2024-03-12T05:14:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54878/0 (socket says 172.21.15.45:54878) 2024-03-12T05:14:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.660+0000 7fbe677fe640 1 -- 172.21.15.45:0/15989283 learned_addr learned my addr 172.21.15.45:0/15989283 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68113640 con 0x7fbe6810d9d0 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681137b0 con 0x7fbe6810f9a0 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68113970 con 0x7fbe6811a960 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe67fff640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6811a960 0x7fbe68112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe66ffd640 1 -- 172.21.15.45:0/15989283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6811a960 msgr2=0x7fbe68112eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6811a960 0x7fbe68112eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe66ffd640 1 -- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 msgr2=0x7fbe68119d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 0x7fbe68119d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.661+0000 7fbe66ffd640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe60009900 con 0x7fbe6810f9a0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.662+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9f0 tx=0x7fbe5c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.662+0000 7fbe64ff9640 1 -- 172.21.15.45:0/15989283 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00ccc0 con 0x7fbe6810f9a0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.662+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68113c50 con 0x7fbe6810f9a0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.662+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68117b70 con 0x7fbe6810f9a0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.662+0000 7fbe64ff9640 1 -- 172.21.15.45:0/15989283 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe5c00ce20 con 0x7fbe6810f9a0 2024-03-12T05:14:36.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.663+0000 7fbe64ff9640 1 -- 172.21.15.45:0/15989283 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0187a0 con 0x7fbe6810f9a0 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe64ff9640 1 -- 172.21.15.45:0/15989283 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbe5c010430 con 0x7fbe6810f9a0 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 Processor -- start 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 -- start start 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75b70 0x5560e9f7e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f80b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9f30960 con 0x5560e9f7e750 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9ee4310 con 0x5560e9f75b70 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.666+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9ee1840 con 0x5560e9f75250 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47500/0 (socket says 172.21.15.45:47500) 2024-03-12T05:14:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 -- 172.21.15.45:0/8182009 learned_addr learned my addr 172.21.15.45:0/8182009 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f80b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe677fe640 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75b70 0x5560e9f7e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 -- 172.21.15.45:0/8182009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75b70 msgr2=0x5560e9f7e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75b70 0x5560e9f7e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 -- 172.21.15.45:0/8182009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 msgr2=0x5560e9f80b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f80b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.667+0000 7fbe67fff640 1 -- 172.21.15.45:0/8182009 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560e9e9f060 con 0x5560e9f75250 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.668+0000 7fbe67fff640 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbe6002c700 tx=0x7fbe60005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95db7d0f22733898 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.668+0000 7fbe4dffb640 1 -- 172.21.15.45:0/8182009 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60032440 con 0x5560e9f75250 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.668+0000 7fbe4dffb640 1 -- 172.21.15.45:0/8182009 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe60032c90 con 0x5560e9f75250 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.668+0000 7fbe4dffb640 1 -- 172.21.15.45:0/8182009 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe600329f0 con 0x5560e9f75250 2024-03-12T05:14:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.668+0000 7fbe64ff9640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 0x7fbe500a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe64ff9640 1 -- 172.21.15.45:0/15989283 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbe5c014070 con 0x7fbe6810f9a0 2024-03-12T05:14:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe677fe640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 0x7fbe500a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 -- 172.21.15.45:0/8182009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 msgr2=0x5560e9f75630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbe6002c700 tx=0x7fbe60005ef0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 -- 172.21.15.45:0/8182009 shutdown_connections 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f80b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75b70 0x5560e9f7e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/8182009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75250 0x5560e9f75630 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 -- 172.21.15.45:0/8182009 >> 172.21.15.45:0/8182009 conn(0x5560e9f6f980 msgr2=0x5560e9f6fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 -- 172.21.15.45:0/8182009 shutdown_connections 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe70bd2740 1 -- 172.21.15.45:0/8182009 wait complete. 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.669+0000 7fbe677fe640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 0x7fbe500a4230 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5560e9f74d20 tx=0x7fbe58005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.670+0000 7fbe70bd2740 1 -- start start 2024-03-12T05:14:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.670+0000 7fbe70bd2740 1 Processor -- start 2024-03-12T05:14:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f7bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f7c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54898/0 (socket says 172.21.15.45:54898) 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 -- 172.21.15.45:0/3018153880 learned_addr learned my addr 172.21.15.45:0/3018153880 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f7bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9edd020 con 0x5560e9f7e750 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9bb0430 con 0x5560e9f75250 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9b6ce00 con 0x5560e9f75b70 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe67fff640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f7c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe70bd2740 10 client.0 _ll_register_callbacks cb 0x5560e9e84cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 -- 172.21.15.45:0/3018153880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 msgr2=0x5560e9f7bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f7bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 -- 172.21.15.45:0/3018153880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 msgr2=0x5560e9f7c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f7c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe677fe640 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6811d680 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.671+0000 7fbe67fff640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f7c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe677fe640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe58020a10 tx=0x7fbe58004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd867bc91a3bbaf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58017070 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbe580119c0 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560e9bdc7e0 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe58011b80 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe70bd2740 10 client.25997 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.672+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5560e9edb6a0 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe58035080 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe5802f030 con 0x5560e9f75250 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 10 client.25997 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 20 client.25997 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9f90df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 4 client.25997 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560e9bdc7e0 con 0x5560e9f8e950 2024-03-12T05:14:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.673+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9f90df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.674+0000 7fbe46ffd640 10 client.25997 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.674+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9f90df0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be119b48f3900e41 server_cookie=fbdab55bb80a5512 in_seq=0 out_seq=0 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.674+0000 7fbe46ffd640 1 -- 172.21.15.45:0/3018153880 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5560e9bdc7e0 con 0x5560e9f8e950 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.674+0000 7fbe46ffd640 10 client.25997 handle_command_reply: tid=0 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.674+0000 7fbe70bd2740 1 client.25997 shutdown 2024-03-12T05:14:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 msgr2=0x5560e9f7b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe58020a10 tx=0x7fbe58004730 comp rx=0 tx=0).stop 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 shutdown_connections 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9f90df0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f7c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe46ffd640 0 client.25997 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f7bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/3018153880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7b7c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 >> 172.21.15.45:0/3018153880 conn(0x5560e9f6f980 msgr2=0x5560e9f716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 shutdown_connections 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.675+0000 7fbe70bd2740 1 -- 172.21.15.45:0/3018153880 wait complete. 2024-03-12T05:14:36.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.676+0000 7fbe70bd2740 20 client.25997 trim_cache size 0 max 16384 2024-03-12T05:14:36.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 Processor -- start 2024-03-12T05:14:36.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 -- start start 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f6ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f855a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9e62130 con 0x5560e9f7e750 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9f8a5b0 con 0x5560e9f75250 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe70bd2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9f70980 con 0x5560e9f75b70 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54910/0 (socket says 172.21.15.45:54910) 2024-03-12T05:14:36.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f6ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.695+0000 7fbe677fe640 1 -- 172.21.15.45:0/854607706 learned_addr learned my addr 172.21.15.45:0/854607706 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe67fff640 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f855a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 -- 172.21.15.45:0/854607706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 msgr2=0x5560e9f7def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 -- 172.21.15.45:0/854607706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 msgr2=0x5560e9f855a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f855a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 -- 172.21.15.45:0/854607706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560e9e77c20 con 0x5560e9f75b70 2024-03-12T05:14:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe67fff640 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f855a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:36.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.696+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f6ea00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0cb310 tx=0x7fbe5c0d2040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f56cedcbaf2feeda server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.697+0000 7fbe4d7fa640 1 -- 172.21.15.45:0/854607706 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c09a330 con 0x5560e9f75b70 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.697+0000 7fbe4d7fa640 1 -- 172.21.15.45:0/854607706 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbe5c09a150 con 0x5560e9f75b70 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.697+0000 7fbe4d7fa640 1 -- 172.21.15.45:0/854607706 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c098c10 con 0x5560e9f75b70 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- 172.21.15.45:0/854607706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 msgr2=0x5560e9f6ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f6ea00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0cb310 tx=0x7fbe5c0d2040 comp rx=0 tx=0).stop 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- 172.21.15.45:0/854607706 shutdown_connections 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f7e750 0x5560e9f855a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f75b70 0x5560e9f6ea00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/854607706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9f7def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- 172.21.15.45:0/854607706 >> 172.21.15.45:0/854607706 conn(0x5560e9f6f980 msgr2=0x5560e9f6e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- 172.21.15.45:0/854607706 shutdown_connections 2024-03-12T05:14:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- 172.21.15.45:0/854607706 wait complete. 2024-03-12T05:14:36.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.698+0000 7fbe70bd2740 1 -- start start 2024-03-12T05:14:36.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.699+0000 7fbe70bd2740 1 Processor -- start 2024-03-12T05:14:36.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.700+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.700+0000 7fbe70bd2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 0x5560e9fdb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.700+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.700+0000 7fbe677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54922/0 (socket says 172.21.15.45:54922) 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 0x5560e9fdb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 0x5560e9fdb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58196/0 (socket says 172.21.15.45:58196) 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe677fe640 1 -- 172.21.15.45:0/996461886 learned_addr learned my addr 172.21.15.45:0/996461886 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f7e750 0x5560e9fdb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9fe5fb0 con 0x5560e9f75b70 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe67fff640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f7e750 0x5560e9fdb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9fe6120 con 0x5560e9f75250 2024-03-12T05:14:36.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.701+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560e9fe62e0 con 0x5560e9f7e750 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 -- 172.21.15.45:0/996461886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f7e750 msgr2=0x5560e9fdb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f7e750 0x5560e9fdb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 -- 172.21.15.45:0/996461886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 msgr2=0x5560e9fdb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 0x5560e9fdb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68117b70 con 0x5560e9f75250 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe677fe640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe58031500 tx=0x7fbe580296f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c101ff29d045cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe70bd2740 10 client.0 _ll_register_callbacks cb 0x7fbe60037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe46ffd640 1 -- 172.21.15.45:0/996461886 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58017040 con 0x5560e9f75250 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.702+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe60032c90 con 0x5560e9f75250 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe46ffd640 1 -- 172.21.15.45:0/996461886 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fbe580207d0 con 0x5560e9f75250 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe46ffd640 1 -- 172.21.15.45:0/996461886 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe58011e50 con 0x5560e9f75250 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 10 client.26003 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5560e9e9d8c0 con 0x5560e9f75250 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe46ffd640 1 -- 172.21.15.45:0/996461886 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe58029cf0 con 0x5560e9f75250 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 10 client.26003 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 10 client.26003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 20 client.26003 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9fe98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 4 client.26003 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.703+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5560e9c19720 con 0x5560e9f8e950 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9fe98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe66ffd640 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9fe98c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a742b2e17aeaa53 server_cookie=a0905126c340da84 in_seq=0 out_seq=0 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe46ffd640 10 client.26003 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe46ffd640 1 -- 172.21.15.45:0/996461886 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5560e9c19720 con 0x5560e9f8e950 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe46ffd640 10 client.26003 handle_command_reply: tid=0 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.704+0000 7fbe70bd2740 1 client.26003 shutdown 2024-03-12T05:14:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 msgr2=0x5560e9fdaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe58031500 tx=0x7fbe580296f0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 shutdown_connections 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5560e9f8e950 0x5560e9fe98c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5560e9f75b70 0x5560e9fdb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5560e9f7e750 0x5560e9fdb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 --2- 172.21.15.45:0/996461886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560e9f75250 0x5560e9fdaeb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 >> 172.21.15.45:0/996461886 conn(0x5560e9f6f980 msgr2=0x5560e9f7bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.705+0000 7fbe46ffd640 0 client.26003 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.706+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 shutdown_connections 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.706+0000 7fbe70bd2740 1 -- 172.21.15.45:0/996461886 wait complete. 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.706+0000 7fbe70bd2740 20 client.26003 trim_cache size 0 max 16384 2024-03-12T05:14:36.704 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:14:36.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 msgr2=0x7fbe500a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 0x7fbe500a4230 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5560e9f74d20 tx=0x7fbe58005f90 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 msgr2=0x7fbe6811a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9f0 tx=0x7fbe5c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 shutdown_connections 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbe500a1d70 0x7fbe500a4230 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe6811a960 0x7fbe68112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe6810f9a0 0x7fbe6811a2d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 --2- 172.21.15.45:0/15989283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe6810d9d0 0x7fbe68119d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.707+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 >> 172.21.15.45:0/15989283 conn(0x7fbe6806c260 msgr2=0x7fbe6811c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.708+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 shutdown_connections 2024-03-12T05:14:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:36.708+0000 7fbe6df25640 1 -- 172.21.15.45:0/15989283 wait complete. 2024-03-12T05:14:37.723 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:14:37.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:14:37.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:14:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 -- 172.21.15.45:0/2023147703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010f7c0 msgr2=0x7efc4010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 --2- 172.21.15.45:0/2023147703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010f7c0 0x7efc4010fba0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009900 tx=0x7efc2c02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 -- 172.21.15.45:0/2023147703 shutdown_connections 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 --2- 172.21.15.45:0/2023147703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc4010a100 0x7efc40116610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 --2- 172.21.15.45:0/2023147703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc40109760 0x7efc40109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 --2- 172.21.15.45:0/2023147703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010f7c0 0x7efc4010fba0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 -- 172.21.15.45:0/2023147703 >> 172.21.15.45:0/2023147703 conn(0x7efc4006cdf0 msgr2=0x7efc4006f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 -- 172.21.15.45:0/2023147703 shutdown_connections 2024-03-12T05:14:37.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.909+0000 7efc45f67640 1 -- 172.21.15.45:0/2023147703 wait complete. 2024-03-12T05:14:37.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 Processor -- start 2024-03-12T05:14:37.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 -- start start 2024-03-12T05:14:37.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 0x7efc401ab450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010f7c0 0x7efc401156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40115d20 con 0x7efc4010a100 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40115e90 con 0x7efc4010f7c0 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.910+0000 7efc45f67640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40116050 con 0x7efc40109760 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc44f65640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc44f65640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46378/0 (socket says 172.21.15.45:46378) 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc44f65640 1 -- 172.21.15.45:0/2741077019 learned_addr learned my addr 172.21.15.45:0/2741077019 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc3ffff640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 0x7efc401ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc45766640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010f7c0 0x7efc401156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc44f65640 1 -- 172.21.15.45:0/2741077019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010f7c0 msgr2=0x7efc401156d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.911+0000 7efc44f65640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010f7c0 0x7efc401156d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.912+0000 7efc44f65640 1 -- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 msgr2=0x7efc401ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.912+0000 7efc44f65640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 0x7efc401ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.912+0000 7efc44f65640 1 -- 172.21.15.45:0/2741077019 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c009560 con 0x7efc40109760 2024-03-12T05:14:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.912+0000 7efc3ffff640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 0x7efc401ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.912+0000 7efc44f65640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc2c005940 tx=0x7efc2c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.913+0000 7efc3dffb640 1 -- 172.21.15.45:0/2741077019 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c004440 con 0x7efc40109760 2024-03-12T05:14:37.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.913+0000 7efc3dffb640 1 -- 172.21.15.45:0/2741077019 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc2c003aa0 con 0x7efc40109760 2024-03-12T05:14:37.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.913+0000 7efc3dffb640 1 -- 172.21.15.45:0/2741077019 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0053f0 con 0x7efc40109760 2024-03-12T05:14:37.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.913+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc401162d0 con 0x7efc40109760 2024-03-12T05:14:37.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.913+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc401b3010 con 0x7efc40109760 2024-03-12T05:14:37.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.916+0000 7efc3dffb640 1 -- 172.21.15.45:0/2741077019 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efc2c03e450 con 0x7efc40109760 2024-03-12T05:14:37.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.916+0000 7efc49d95740 1 Processor -- start 2024-03-12T05:14:37.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 -- start start 2024-03-12T05:14:37.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 0x55d122fa89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 0x55d122fb1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122f16f90 con 0x55d122fb1b00 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122f144c0 con 0x55d122fa8f80 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc49d95740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122f0fca0 con 0x55d122fa85d0 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc3ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc45766640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 0x55d122fa89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc3ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38494/0 (socket says 172.21.15.45:38494) 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc3ffff640 1 -- 172.21.15.45:0/3487502598 learned_addr learned my addr 172.21.15.45:0/3487502598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.917+0000 7efc44f65640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 0x55d122fb1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 -- 172.21.15.45:0/3487502598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 msgr2=0x55d122fa89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 0x55d122fa89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 -- 172.21.15.45:0/3487502598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 msgr2=0x55d122fb1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 0x55d122fb1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 -- 172.21.15.45:0/3487502598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d122ed20b0 con 0x55d122fb1b00 2024-03-12T05:14:37.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc45766640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 0x55d122fa89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:37.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc44f65640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 0x55d122fb1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:14:37.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3dffb640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 0x7efc240a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3ffff640 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efc380117e0 tx=0x7efc38011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e091163b2bf35e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc45766640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 0x7efc240a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.918+0000 7efc3dffb640 1 -- 172.21.15.45:0/2741077019 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efc2c02de40 con 0x7efc40109760 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.919+0000 7efc45766640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 0x7efc240a4160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc30005fd0 tx=0x7efc30005cc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.919+0000 7efc36ffd640 1 -- 172.21.15.45:0/3487502598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38003c30 con 0x55d122fb1b00 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.919+0000 7efc36ffd640 1 -- 172.21.15.45:0/3487502598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc3800fe50 con 0x55d122fb1b00 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.919+0000 7efc36ffd640 1 -- 172.21.15.45:0/3487502598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380193e0 con 0x55d122fb1b00 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 -- 172.21.15.45:0/3487502598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 msgr2=0x55d122fb3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efc380117e0 tx=0x7efc38011ca0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 -- 172.21.15.45:0/3487502598 shutdown_connections 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fb3f50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa8f80 0x55d122fb1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 --2- 172.21.15.45:0/3487502598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa85d0 0x55d122fa89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.920+0000 7efc49d95740 1 -- 172.21.15.45:0/3487502598 >> 172.21.15.45:0/3487502598 conn(0x55d122fa2a90 msgr2=0x55d122fa2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.921+0000 7efc49d95740 1 -- 172.21.15.45:0/3487502598 shutdown_connections 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.921+0000 7efc49d95740 1 -- 172.21.15.45:0/3487502598 wait complete. 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.921+0000 7efc49d95740 1 -- start start 2024-03-12T05:14:37.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.922+0000 7efc49d95740 1 Processor -- start 2024-03-12T05:14:37.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.922+0000 7efc49d95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.922+0000 7efc49d95740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.922+0000 7efc49d95740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122faf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc49d95740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122eb4f40 con 0x55d122fb1b00 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc3ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33884/0 (socket says 172.21.15.45:33884) 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 -- 172.21.15.45:0/4026214653 learned_addr learned my addr 172.21.15.45:0/4026214653 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122be34b0 con 0x55d122fa85d0 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122e95670 con 0x55d122fa8f80 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc45766640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122faf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc49d95740 10 client.0 _ll_register_callbacks cb 0x55d122f02f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:37.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 -- 172.21.15.45:0/4026214653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 msgr2=0x55d122faf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 -- 172.21.15.45:0/4026214653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 msgr2=0x55d122faf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122faf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc38011440 con 0x55d122fa85d0 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc44f65640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0bcb70 tx=0x7efc2c0bc720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7721893398e8c5d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.923+0000 7efc23fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0bcbe0 con 0x55d122fa85d0 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efc2c0c0d30 con 0x55d122fa85d0 2024-03-12T05:14:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d122ee9f40 con 0x55d122fa85d0 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc2c0c5380 con 0x55d122fa85d0 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc49d95740 10 client.26015 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d122f0e320 con 0x55d122fa85d0 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d122ee9f40 con 0x55d122fa85d0 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.924+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efc2c03c070 con 0x55d122fa85d0 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 10 client.26015 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 10 client.26015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 20 client.26015 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d122fc40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 4 client.26015 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:37.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d122f06310 con 0x55d122fc1cd0 2024-03-12T05:14:37.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc3ffff640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d122fc40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc3ffff640 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d122fc40c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259941bacaf01946 server_cookie=ff917f05d412a205 in_seq=0 out_seq=0 2024-03-12T05:14:37.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.925+0000 7efc23fff640 10 client.26015 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:37.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.926+0000 7efc23fff640 1 -- 172.21.15.45:0/4026214653 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d122f06310 con 0x55d122fc1cd0 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.926+0000 7efc23fff640 10 client.26015 handle_command_reply: tid=0 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.926+0000 7efc49d95740 1 client.26015 shutdown 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 msgr2=0x55d122faeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0bcb70 tx=0x7efc2c0bc720 comp rx=0 tx=0).stop 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 shutdown_connections 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d122fc40c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122faf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc23fff640 0 client.26015 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 --2- 172.21.15.45:0/4026214653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122faeae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.927+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 >> 172.21.15.45:0/4026214653 conn(0x55d122fa2a90 msgr2=0x55d122fa49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.928+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 shutdown_connections 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.928+0000 7efc49d95740 1 -- 172.21.15.45:0/4026214653 wait complete. 2024-03-12T05:14:37.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.928+0000 7efc49d95740 20 client.26015 trim_cache size 0 max 16384 2024-03-12T05:14:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 Processor -- start 2024-03-12T05:14:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 -- start start 2024-03-12T05:14:37.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.954+0000 7efc49d95740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122ee9f40 con 0x55d122fb1b00 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc49d95740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122f9e6e0 con 0x55d122fa85d0 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc49d95740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122fa7660 con 0x55d122fa8f80 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc3ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc45766640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc45766640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38512/0 (socket says 172.21.15.45:38512) 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc45766640 1 -- 172.21.15.45:0/2183440586 learned_addr learned my addr 172.21.15.45:0/2183440586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 -- 172.21.15.45:0/2183440586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 msgr2=0x55d122faf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 -- 172.21.15.45:0/2183440586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 msgr2=0x55d122fabd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 -- 172.21.15.45:0/2183440586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d122eaac00 con 0x55d122fa85d0 2024-03-12T05:14:37.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc45766640 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.955+0000 7efc44f65640 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb1210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0b9040 tx=0x7efc2c0bc850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a64e562c28d5af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.956+0000 7efc367fc640 1 -- 172.21.15.45:0/2183440586 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0bc540 con 0x55d122fa85d0 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.956+0000 7efc367fc640 1 -- 172.21.15.45:0/2183440586 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc2c0c5050 con 0x55d122fa85d0 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.956+0000 7efc367fc640 1 -- 172.21.15.45:0/2183440586 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0bce40 con 0x55d122fa85d0 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 -- 172.21.15.45:0/2183440586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 msgr2=0x55d122fb1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb1210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0b9040 tx=0x7efc2c0bc850 comp rx=0 tx=0).stop 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 -- 172.21.15.45:0/2183440586 shutdown_connections 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d122fabd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122faf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 --2- 172.21.15.45:0/2183440586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb1210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.957+0000 7efc49d95740 1 -- 172.21.15.45:0/2183440586 >> 172.21.15.45:0/2183440586 conn(0x55d122fa2a90 msgr2=0x55d122fc8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.958+0000 7efc49d95740 1 -- 172.21.15.45:0/2183440586 shutdown_connections 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.958+0000 7efc49d95740 1 -- 172.21.15.45:0/2183440586 wait complete. 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.958+0000 7efc49d95740 1 -- start start 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.958+0000 7efc49d95740 1 Processor -- start 2024-03-12T05:14:37.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.959+0000 7efc49d95740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.959+0000 7efc49d95740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122fb7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.959+0000 7efc44f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.959+0000 7efc44f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33910/0 (socket says 172.21.15.45:33910) 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.959+0000 7efc44f65640 1 -- 172.21.15.45:0/1932236307 learned_addr learned my addr 172.21.15.45:0/1932236307 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc3ffff640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122fb7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d1230181c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d122fb8720 con 0x55d122fb1b00 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d123018700 con 0x55d122fa85d0 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc45766640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d1230181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d123018840 con 0x55d122fa8f80 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.960+0000 7efc49d95740 10 client.0 _ll_register_callbacks cb 0x7efc38009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 -- 172.21.15.45:0/1932236307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 msgr2=0x55d122fb7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122fb7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 -- 172.21.15.45:0/1932236307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 msgr2=0x55d1230181c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d1230181c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c0d4050 con 0x55d122fa85d0 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc44f65640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0c5020 tx=0x7efc2c0bd070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae2b0e4460d7360 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:37.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc23fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:37.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0bc540 con 0x55d122fa85d0 2024-03-12T05:14:37.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efc2c0c0d30 con 0x55d122fa85d0 2024-03-12T05:14:37.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.961+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d122f35900 con 0x55d122fa85d0 2024-03-12T05:14:37.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.962+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc2c0bcbe0 con 0x55d122fa85d0 2024-03-12T05:14:37.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.963+0000 7efc49d95740 10 client.26021 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:37.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.963+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d122ed0910 con 0x55d122fa85d0 2024-03-12T05:14:37.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.963+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc2c0ca080 con 0x55d122fa85d0 2024-03-12T05:14:37.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.963+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efc2c0a3070 con 0x55d122fa85d0 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 10 client.26021 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 10 client.26021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 20 client.26021 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d12301bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 4 client.26021 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d122c0f860 con 0x55d122fc1cd0 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.964+0000 7efc3ffff640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d12301bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.965+0000 7efc23fff640 10 client.26021 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:37.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.965+0000 7efc3ffff640 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d12301bb50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75b2878de0fa9a05 server_cookie=8604c525ae45825a in_seq=0 out_seq=0 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc23fff640 1 -- 172.21.15.45:0/1932236307 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d122c0f860 con 0x55d122fc1cd0 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc23fff640 10 client.26021 handle_command_reply: tid=0 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc49d95740 1 client.26021 shutdown 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 msgr2=0x55d122fb7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0c5020 tx=0x7efc2c0bd070 comp rx=0 tx=0).stop 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.966+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 shutdown_connections 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d122fc1cd0 0x55d12301bb50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d122fb1b00 0x55d1230181c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d122fa8f80 0x55d122fb7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 --2- 172.21.15.45:0/1932236307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d122fa85d0 0x55d122fb7930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 >> 172.21.15.45:0/1932236307 conn(0x55d122fa2a90 msgr2=0x55d122fa1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc23fff640 0 client.26021 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 shutdown_connections 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 1 -- 172.21.15.45:0/1932236307 wait complete. 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.967+0000 7efc49d95740 20 client.26021 trim_cache size 0 max 16384 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "5525cd23-cc34-4e8a-b8fe-b4506193e976", 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:14:37.966 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 msgr2=0x7efc240a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 0x7efc240a4160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc30005fd0 tx=0x7efc30005cc0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 msgr2=0x7efc401aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc2c005940 tx=0x7efc2c004040 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 shutdown_connections 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc240a1ca0 0x7efc240a4160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc4010a100 0x7efc401ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc4010f7c0 0x7efc401156d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 --2- 172.21.15.45:0/2741077019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc40109760 0x7efc401aaf10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 >> 172.21.15.45:0/2741077019 conn(0x7efc4006cdf0 msgr2=0x7efc4006d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:37.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 shutdown_connections 2024-03-12T05:14:37.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:37.968+0000 7efc45f67640 1 -- 172.21.15.45:0/2741077019 wait complete. 2024-03-12T05:14:37.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:37 smithi130 ceph-mon[35398]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:14:37.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:37 smithi045 ceph-mon[32174]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:14:37.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:14:37.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:14:37.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:37 smithi116 ceph-mon[35486]: pgmap v115: 97 pgs: 97 active+clean; 2.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 -- 172.21.15.45:0/2951626459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 msgr2=0x7fe92c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 --2- 172.21.15.45:0/2951626459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c102780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009860 tx=0x7fe91c02bc20 comp rx=0 tx=0).stop 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 -- 172.21.15.45:0/2951626459 shutdown_connections 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 --2- 172.21.15.45:0/2951626459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c10b6e0 0x7fe92c10db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 --2- 172.21.15.45:0/2951626459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c102cc0 0x7fe92c10b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 --2- 172.21.15.45:0/2951626459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c102780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.168+0000 7fe9317bc640 1 -- 172.21.15.45:0/2951626459 >> 172.21.15.45:0/2951626459 conn(0x7fe92c077230 msgr2=0x7fe92c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.169+0000 7fe9317bc640 1 -- 172.21.15.45:0/2951626459 shutdown_connections 2024-03-12T05:14:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.169+0000 7fe9317bc640 1 -- 172.21.15.45:0/2951626459 wait complete. 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.169+0000 7fe9317bc640 1 Processor -- start 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.170+0000 7fe9317bc640 1 -- start start 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.170+0000 7fe9317bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.170+0000 7fe9317bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 0x7fe92c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe9317bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c10b6e0 0x7fe92c19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe9317bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c19c5d0 con 0x7fe92c102cc0 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe9317bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c19c740 con 0x7fe92c10b6e0 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe9317bc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c19c900 con 0x7fe92c1023a0 2024-03-12T05:14:38.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46418/0 (socket says 172.21.15.45:46418) 2024-03-12T05:14:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 -- 172.21.15.45:0/612599922 learned_addr learned my addr 172.21.15.45:0/612599922 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92a7fc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 0x7fe92c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c10b6e0 0x7fe92c19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 -- 172.21.15.45:0/612599922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c10b6e0 msgr2=0x7fe92c19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c10b6e0 0x7fe92c19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 -- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 msgr2=0x7fe92c197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 0x7fe92c197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92affd640 1 -- 172.21.15.45:0/612599922 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920009560 con 0x7fe92c1023a0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.171+0000 7fe92a7fc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 0x7fe92c197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe92affd640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe91c037040 tx=0x7fe91c034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe913fff640 1 -- 172.21.15.45:0/612599922 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c02c7c0 con 0x7fe92c1023a0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe913fff640 1 -- 172.21.15.45:0/612599922 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe91c0044f0 con 0x7fe92c1023a0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe924009840 con 0x7fe92c1023a0 2024-03-12T05:14:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe913fff640 1 -- 172.21.15.45:0/612599922 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c03d3f0 con 0x7fe92c1023a0 2024-03-12T05:14:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.172+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c009490 con 0x7fe92c1023a0 2024-03-12T05:14:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.175+0000 7fe913fff640 1 -- 172.21.15.45:0/612599922 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe91c03b040 con 0x7fe92c1023a0 2024-03-12T05:14:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.176+0000 7fe913fff640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 0x7fe9140a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.176+0000 7fe92a7fc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 0x7fe9140a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe913fff640 1 -- 172.21.15.45:0/612599922 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe91c0ef030 con 0x7fe92c1023a0 2024-03-12T05:14:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe934393740 1 Processor -- start 2024-03-12T05:14:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe92a7fc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 0x7fe9140a40e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0ff260 tx=0x7fe920009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe934393740 1 -- start start 2024-03-12T05:14:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe934393740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe934393740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e87270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.177+0000 7fe934393740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e89ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe934393740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5ded280 con 0x564bf5e7ebd0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe934393740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5dea7b0 con 0x564bf5e877b0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe934393740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5de5f90 con 0x564bf5e7e2b0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e87270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e87270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38558/0 (socket says 172.21.15.45:38558) 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92a7fc640 1 -- 172.21.15.45:0/11485806 learned_addr learned my addr 172.21.15.45:0/11485806 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46434/0 (socket says 172.21.15.45:46434) 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e89ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 -- 172.21.15.45:0/11485806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 msgr2=0x564bf5e89ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e89ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 -- 172.21.15.45:0/11485806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 msgr2=0x564bf5e87270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e87270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 -- 172.21.15.45:0/11485806 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bf5da4570 con 0x564bf5e7e2b0 2024-03-12T05:14:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.178+0000 7fe92affd640 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0ec830 tx=0x7fe91c0bb850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac7e09735b479403 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.179+0000 7fe9117fa640 1 -- 172.21.15.45:0/11485806 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0bbd10 con 0x564bf5e7e2b0 2024-03-12T05:14:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.179+0000 7fe9117fa640 1 -- 172.21.15.45:0/11485806 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe91c0bfe60 con 0x564bf5e7e2b0 2024-03-12T05:14:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.179+0000 7fe9117fa640 1 -- 172.21.15.45:0/11485806 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c02c920 con 0x564bf5e7e2b0 2024-03-12T05:14:38.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.180+0000 7fe934393740 1 -- 172.21.15.45:0/11485806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 msgr2=0x564bf5e7e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.180+0000 7fe934393740 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0ec830 tx=0x7fe91c0bb850 comp rx=0 tx=0).stop 2024-03-12T05:14:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 -- 172.21.15.45:0/11485806 shutdown_connections 2024-03-12T05:14:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e87270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e89ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 --2- 172.21.15.45:0/11485806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e7e690 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 -- 172.21.15.45:0/11485806 >> 172.21.15.45:0/11485806 conn(0x564bf5e78990 msgr2=0x564bf5e78d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 -- 172.21.15.45:0/11485806 shutdown_connections 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 -- 172.21.15.45:0/11485806 wait complete. 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 -- start start 2024-03-12T05:14:38.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.181+0000 7fe934393740 1 Processor -- start 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e87210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e845a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e84cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e87210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e87210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38566/0 (socket says 172.21.15.45:38566) 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e845a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e845a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46444/0 (socket says 172.21.15.45:46444) 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92a7fc640 1 -- 172.21.15.45:0/832567225 learned_addr learned my addr 172.21.15.45:0/832567225 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5ab9430 con 0x564bf5e7e2b0 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5a75e00 con 0x564bf5e877b0 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5d6afd0 con 0x564bf5e7ebd0 2024-03-12T05:14:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e84cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.182+0000 7fe934393740 10 client.0 _ll_register_callbacks cb 0x564bf5dc0fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 -- 172.21.15.45:0/832567225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 msgr2=0x564bf5e845a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e845a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 -- 172.21.15.45:0/832567225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 msgr2=0x564bf5e87210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e87210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c009490 con 0x564bf5e877b0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e84cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe924005fd0 tx=0x7fe92400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a0c011bcc0a12e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe90eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924015040 con 0x564bf5e877b0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564bf5ae57e0 con 0x564bf5e877b0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.183+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe924003b70 con 0x564bf5e877b0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.184+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe924003d30 con 0x564bf5e877b0 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.184+0000 7fe934393740 10 client.26033 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.184+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564bf5de4610 con 0x564bf5e877b0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.184+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x564bf5ae57e0 con 0x564bf5e877b0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.184+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe92401b470 con 0x564bf5e877b0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 10 client.26033 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 10 client.26033 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 20 client.26033 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5e99ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 4 client.26033 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe92affd640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5e99ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.185+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bf5b22720 con 0x564bf5e976b0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.186+0000 7fe92affd640 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5e99ad0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8718f726d3b27abf server_cookie=c4158f65b69fbd76 in_seq=0 out_seq=0 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.186+0000 7fe90eff5640 10 client.26033 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:38.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.187+0000 7fe90eff5640 1 -- 172.21.15.45:0/832567225 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564bf5b22720 con 0x564bf5e976b0 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.187+0000 7fe90eff5640 10 client.26033 handle_command_reply: tid=0 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.187+0000 7fe934393740 1 client.26033 shutdown 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.187+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 msgr2=0x564bf5e84cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.187+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e84cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe924005fd0 tx=0x7fe92400b690 comp rx=0 tx=0).stop 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 shutdown_connections 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5e99ad0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e84cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe90eff5640 0 client.26033 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e845a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 --2- 172.21.15.45:0/832567225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e87210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 >> 172.21.15.45:0/832567225 conn(0x564bf5e78990 msgr2=0x564bf5e7a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 shutdown_connections 2024-03-12T05:14:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 1 -- 172.21.15.45:0/832567225 wait complete. 2024-03-12T05:14:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.188+0000 7fe934393740 20 client.26033 trim_cache size 0 max 16384 2024-03-12T05:14:38.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.216+0000 7fe934393740 1 Processor -- start 2024-03-12T05:14:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.216+0000 7fe934393740 1 -- start start 2024-03-12T05:14:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e86cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5e740d0 con 0x564bf5e7e2b0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5e95eb0 con 0x564bf5e877b0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe934393740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5e79c10 con 0x564bf5e7ebd0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe92affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe92affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38576/0 (socket says 172.21.15.45:38576) 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe92affd640 1 -- 172.21.15.45:0/3257997087 learned_addr learned my addr 172.21.15.45:0/3257997087 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe92a7fc640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.217+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e86cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 -- 172.21.15.45:0/3257997087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 msgr2=0x564bf5e8e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 -- 172.21.15.45:0/3257997087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 msgr2=0x564bf5e86cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e86cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 -- 172.21.15.45:0/3257997087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bf5d80ac0 con 0x564bf5e7e2b0 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e86cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:14:38.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe92affd640 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0b8170 tx=0x7fe91c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b8ef01e7f85ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe910ff9640 1 -- 172.21.15.45:0/3257997087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0eca20 con 0x564bf5e7e2b0 2024-03-12T05:14:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe910ff9640 1 -- 172.21.15.45:0/3257997087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe91c0c44b0 con 0x564bf5e7e2b0 2024-03-12T05:14:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.218+0000 7fe910ff9640 1 -- 172.21.15.45:0/3257997087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0c4020 con 0x564bf5e7e2b0 2024-03-12T05:14:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.219+0000 7fe934393740 1 -- 172.21.15.45:0/3257997087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 msgr2=0x564bf5e8e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.219+0000 7fe934393740 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0b8170 tx=0x7fe91c005f90 comp rx=0 tx=0).stop 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.219+0000 7fe934393740 1 -- 172.21.15.45:0/3257997087 shutdown_connections 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.219+0000 7fe934393740 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5e86cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.219+0000 7fe934393740 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.220+0000 7fe934393740 1 --2- 172.21.15.45:0/3257997087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8e330 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.220+0000 7fe934393740 1 -- 172.21.15.45:0/3257997087 >> 172.21.15.45:0/3257997087 conn(0x564bf5e78990 msgr2=0x564bf5e9e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.220+0000 7fe934393740 1 -- 172.21.15.45:0/3257997087 shutdown_connections 2024-03-12T05:14:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.220+0000 7fe934393740 1 -- 172.21.15.45:0/3257997087 wait complete. 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.221+0000 7fe934393740 1 -- start start 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.221+0000 7fe934393740 1 Processor -- start 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.221+0000 7fe934393740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.221+0000 7fe934393740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.221+0000 7fe934393740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5ee6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe934393740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5ee7130 con 0x564bf5e7ebd0 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe934393740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5ee72a0 con 0x564bf5e877b0 2024-03-12T05:14:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe934393740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bf5ee7460 con 0x564bf5e7e2b0 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe934393740 10 client.0 _ll_register_callbacks cb 0x7fe91c034a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46456/0 (socket says 172.21.15.45:46456) 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92affd640 1 -- 172.21.15.45:0/2264822563 learned_addr learned my addr 172.21.15.45:0/2264822563 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5ee6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38582/0 (socket says 172.21.15.45:38582) 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 -- 172.21.15.45:0/2264822563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 msgr2=0x564bf5e8c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 -- 172.21.15.45:0/2264822563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 msgr2=0x564bf5e8cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.222+0000 7fe92b7fe640 1 -- 172.21.15.45:0/2264822563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bf5de4610 con 0x564bf5e877b0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe92b7fe640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5ee6bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe924009d50 tx=0x7fe924009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f5f9021987bebe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe90eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924015070 con 0x564bf5e877b0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fe92401b470 con 0x564bf5e877b0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe91c0bfe60 con 0x564bf5e877b0 2024-03-12T05:14:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe92401b630 con 0x564bf5e877b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe934393740 10 client.26039 fetch_fsmap learned FSMap version 33 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.223+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564bf5da4570 con 0x564bf5e877b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe92403a020 con 0x564bf5e877b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe924005320 con 0x564bf5e877b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 10 client.26039 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 10 client.26039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 20 client.26039 populate_metadata read hostname 'smithi045' 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5ef9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 4 client.26039 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564bf5b71890 con 0x564bf5e976b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.224+0000 7fe92affd640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5ef9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.225+0000 7fe90eff5640 10 client.26039 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.226+0000 7fe92affd640 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5ef9470 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ff18e27b7b6cf0 server_cookie=bbed2a6e2f479d66 in_seq=0 out_seq=0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.226+0000 7fe90eff5640 1 -- 172.21.15.45:0/2264822563 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564bf5b71890 con 0x564bf5e976b0 2024-03-12T05:14:38.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.226+0000 7fe90eff5640 10 client.26039 handle_command_reply: tid=0 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.226+0000 7fe934393740 1 client.26039 shutdown 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 msgr2=0x564bf5ee6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5ee6bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe924009d50 tx=0x7fe924009f00 comp rx=0 tx=0).stop 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 shutdown_connections 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564bf5e976b0 0x564bf5ef9470 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564bf5e7ebd0 0x564bf5e8cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564bf5e877b0 0x564bf5ee6bf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 --2- 172.21.15.45:0/2264822563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bf5e7e2b0 0x564bf5e8c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 >> 172.21.15.45:0/2264822563 conn(0x564bf5e78990 msgr2=0x564bf5e83e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe90eff5640 0 client.26039 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.227+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 shutdown_connections 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.228+0000 7fe934393740 1 -- 172.21.15.45:0/2264822563 wait complete. 2024-03-12T05:14:38.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.228+0000 7fe934393740 20 client.26039 trim_cache size 0 max 16384 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "5525cd23-cc34-4e8a-b8fe-b4506193e976": { 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "5525cd23-cc34-4e8a-b8fe-b4506193e976", 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:14:38.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:14:38.228 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:14:38.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.230+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 msgr2=0x7fe9140a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.230+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 0x7fe9140a40e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0ff260 tx=0x7fe920009260 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 msgr2=0x7fe92c197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe91c037040 tx=0x7fe91c034670 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 shutdown_connections 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe9140a1c20 0x7fe9140a40e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe92c102cc0 0x7fe92c197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe92c10b6e0 0x7fe92c19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 --2- 172.21.15.45:0/612599922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe92c1023a0 0x7fe92c197710 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 >> 172.21.15.45:0/612599922 conn(0x7fe92c077230 msgr2=0x7fe92c0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 shutdown_connections 2024-03-12T05:14:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:14:38.231+0000 7fe9317bc640 1 -- 172.21.15.45:0/612599922 wait complete. 2024-03-12T05:14:38.245 INFO:tasks.cephfs.filesystem:scrub status for tag:5525cd23-cc34-4e8a-b8fe-b4506193e976 - {'path': '/', 'tag': '5525cd23-cc34-4e8a-b8fe-b4506193e976', 'options': 'recursive,force'} 2024-03-12T05:14:38.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:14:38.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:38.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:38.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:38.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:38 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:38.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:38 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:38 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 155.0 s, 1517.0 tps, lat 65.922 ms stddev 18.550 2024-03-12T05:14:39.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:39 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:14:39.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:39 smithi130 ceph-mon[35398]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 18 MiB/s wr, 1.98k op/s 2024-03-12T05:14:39.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:39 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:14:39.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:39.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:39 smithi116 ceph-mon[35486]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 18 MiB/s wr, 1.98k op/s 2024-03-12T05:14:40.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:39 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:14:40.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:14:40.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:39 smithi045 ceph-mon[32174]: pgmap v116: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 18 MiB/s wr, 1.98k op/s 2024-03-12T05:14:41.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:41 smithi130 ceph-mon[35398]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:41.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:41 smithi116 ceph-mon[35486]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:42.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:41 smithi045 ceph-mon[32174]: pgmap v117: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:43.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:43 smithi130 ceph-mon[35398]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:43.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:43 smithi116 ceph-mon[35486]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:44.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:43 smithi045 ceph-mon[32174]: pgmap v118: 97 pgs: 97 active+clean; 2.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:14:44.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 160.0 s, 1606.0 tps, lat 62.308 ms stddev 18.144 2024-03-12T05:14:45.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:45 smithi130 ceph-mon[35398]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:14:45.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:45 smithi116 ceph-mon[35486]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:14:46.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:45 smithi045 ceph-mon[32174]: pgmap v119: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:14:47.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:47 smithi130 ceph-mon[35398]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:14:47.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:47 smithi116 ceph-mon[35486]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:14:48.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:47 smithi045 ceph-mon[32174]: pgmap v120: 97 pgs: 97 active+clean; 2.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:14:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 165.0 s, 1540.4 tps, lat 64.777 ms stddev 17.232 2024-03-12T05:14:49.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:49 smithi130 ceph-mon[35398]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:49.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:49 smithi116 ceph-mon[35486]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:50.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:49 smithi045 ceph-mon[32174]: pgmap v121: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:51.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:51 smithi130 ceph-mon[35398]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:51.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:51 smithi116 ceph-mon[35486]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:52.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:51 smithi045 ceph-mon[32174]: pgmap v122: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:52.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:52 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:52.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:52 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:52.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:52 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:52.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:52 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:53.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:52 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:53.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:52 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:53.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:53 smithi130 ceph-mon[35398]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:53.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:53.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:14:53.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:53 smithi116 ceph-mon[35486]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:53.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:53.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:14:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:53 smithi045 ceph-mon[32174]: pgmap v123: 97 pgs: 97 active+clean; 2.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:14:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:14:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 170.0 s, 1443.0 tps, lat 69.384 ms stddev 64.403 2024-03-12T05:14:55.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:55 smithi130 ceph-mon[35398]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:55.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:55 smithi116 ceph-mon[35486]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:56.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:55 smithi045 ceph-mon[32174]: pgmap v124: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:14:57.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:57 smithi130 ceph-mon[35398]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:14:57.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:57 smithi116 ceph-mon[35486]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:14:58.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:57 smithi045 ceph-mon[32174]: pgmap v125: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:14:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 175.0 s, 1699.8 tps, lat 58.680 ms stddev 15.573 2024-03-12T05:14:59.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:14:59 smithi130 ceph-mon[35398]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 1.99k op/s 2024-03-12T05:14:59.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:14:59 smithi116 ceph-mon[35486]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 1.99k op/s 2024-03-12T05:15:00.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:14:59 smithi045 ceph-mon[32174]: pgmap v126: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 1.99k op/s 2024-03-12T05:15:01.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:01 smithi130 ceph-mon[35398]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:01 smithi116 ceph-mon[35486]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:01 smithi045 ceph-mon[32174]: pgmap v127: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:02.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:02 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:15:02.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:02 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:15:03.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:02 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:15:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:03 smithi130 ceph-mon[35398]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:03 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:15:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:03 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:15:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:03 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:15:03.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:03 smithi116 ceph-mon[35486]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:03.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:03 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:15:03.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:03 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:15:03.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:03 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:15:04.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:03 smithi045 ceph-mon[32174]: pgmap v128: 97 pgs: 97 active+clean; 2.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 1.33k op/s 2024-03-12T05:15:04.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:03 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:15:04.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:03 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:15:04.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:03 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:15:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 180.0 s, 1407.0 tps, lat 71.162 ms stddev 35.390 2024-03-12T05:15:05.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:05 smithi116 ceph-mon[35486]: pgmap v129: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:15:06.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:05 smithi045 ceph-mon[32174]: pgmap v129: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:15:06.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:05 smithi130 ceph-mon[35398]: pgmap v129: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.01k op/s 2024-03-12T05:15:07.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:07 smithi116 ceph-mon[35486]: pgmap v130: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:15:08.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:07 smithi045 ceph-mon[32174]: pgmap v130: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:15:08.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:07 smithi130 ceph-mon[35398]: pgmap v130: 97 pgs: 97 active+clean; 2.2 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.37k op/s 2024-03-12T05:15:08.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:15:08.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:15:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.419+0000 7fde928ab640 1 -- 172.21.15.45:0/2375964009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 msgr2=0x7fde8c1119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.419+0000 7fde928ab640 1 --2- 172.21.15.45:0/2375964009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1119f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009900 tx=0x7fde7c02bcc0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 -- 172.21.15.45:0/2375964009 shutdown_connections 2024-03-12T05:15:08.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 --2- 172.21.15.45:0/2375964009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1119f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 --2- 172.21.15.45:0/2375964009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c111f30 0x7fde8c114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 --2- 172.21.15.45:0/2375964009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c073520 0x7fde8c073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 -- 172.21.15.45:0/2375964009 >> 172.21.15.45:0/2375964009 conn(0x7fde8c0fca50 msgr2=0x7fde8c0fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 -- 172.21.15.45:0/2375964009 shutdown_connections 2024-03-12T05:15:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.420+0000 7fde928ab640 1 -- 172.21.15.45:0/2375964009 wait complete. 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 Processor -- start 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 -- start start 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 0x7fde8c1a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19df10 con 0x7fde8c073e40 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.421+0000 7fde928ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19e080 con 0x7fde8c111f30 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde928ab640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19e240 con 0x7fde8c073520 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34082/0 (socket says 172.21.15.45:34082) 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 -- 172.21.15.45:0/2295808077 learned_addr learned my addr 172.21.15.45:0/2295808077 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde910a8640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde918a9640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 0x7fde8c1a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 -- 172.21.15.45:0/2295808077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 msgr2=0x7fde8c1a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 0x7fde8c1a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 -- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 msgr2=0x7fde8c1a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.422+0000 7fde920aa640 1 -- 172.21.15.45:0/2295808077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c009560 con 0x7fde8c111f30 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.423+0000 7fde920aa640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e9d0 tx=0x7fde8400ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.423+0000 7fde82ffd640 1 -- 172.21.15.45:0/2295808077 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8400cd80 con 0x7fde8c111f30 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.423+0000 7fde82ffd640 1 -- 172.21.15.45:0/2295808077 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde84005020 con 0x7fde8c111f30 2024-03-12T05:15:08.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.423+0000 7fde82ffd640 1 -- 172.21.15.45:0/2295808077 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840116d0 con 0x7fde8c111f30 2024-03-12T05:15:08.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.423+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c19e520 con 0x7fde8c111f30 2024-03-12T05:15:08.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.424+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c1aa5f0 con 0x7fde8c111f30 2024-03-12T05:15:08.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.424+0000 7fde918a9640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 0x7fde8c1a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:08.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.427+0000 7fde82ffd640 1 -- 172.21.15.45:0/2295808077 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fde84005190 con 0x7fde8c111f30 2024-03-12T05:15:08.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.428+0000 7fde967bc740 1 Processor -- start 2024-03-12T05:15:08.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.428+0000 7fde967bc740 1 -- start start 2024-03-12T05:15:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed20a280 con 0x5596ed29b4d0 2024-03-12T05:15:08.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2077b0 con 0x5596ed2a49d0 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde967bc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed202f90 con 0x5596ed29bdf0 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde910a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde910a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48496/0 (socket says 172.21.15.45:48496) 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde910a8640 1 -- 172.21.15.45:0/3473656523 learned_addr learned my addr 172.21.15.45:0/3473656523 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde82ffd640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 0x7fde780a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.429+0000 7fde910a8640 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 -- 172.21.15.45:0/3473656523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 msgr2=0x5596ed2a4490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a4490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 -- 172.21.15.45:0/3473656523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 msgr2=0x5596ed2a6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 -- 172.21.15.45:0/3473656523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ed1c1570 con 0x5596ed29b4d0 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde920aa640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 0x7fde780a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde910a8640 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fde7c005c40 tx=0x7fde7c005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1fc9423b30b664 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.430+0000 7fde82ffd640 1 -- 172.21.15.45:0/2295808077 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fde840cd970 con 0x7fde8c111f30 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.431+0000 7fde77fff640 1 -- 172.21.15.45:0/3473656523 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0042d0 con 0x5596ed29b4d0 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.431+0000 7fde77fff640 1 -- 172.21.15.45:0/3473656523 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde7c004430 con 0x5596ed29b4d0 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.431+0000 7fde77fff640 1 -- 172.21.15.45:0/3473656523 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0053c0 con 0x5596ed29b4d0 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.432+0000 7fde967bc740 1 -- 172.21.15.45:0/3473656523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 msgr2=0x5596ed29b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.432+0000 7fde920aa640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 0x7fde780a40e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde840cb170 tx=0x7fde840658e0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.432+0000 7fde967bc740 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fde7c005c40 tx=0x7fde7c005b90 comp rx=0 tx=0).stop 2024-03-12T05:15:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 -- 172.21.15.45:0/3473656523 shutdown_connections 2024-03-12T05:15:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 --2- 172.21.15.45:0/3473656523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed29b8b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 -- 172.21.15.45:0/3473656523 >> 172.21.15.45:0/3473656523 conn(0x5596ed295c00 msgr2=0x5596ed295fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 -- 172.21.15.45:0/3473656523 shutdown_connections 2024-03-12T05:15:08.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.433+0000 7fde967bc740 1 -- 172.21.15.45:0/3473656523 wait complete. 2024-03-12T05:15:08.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.434+0000 7fde967bc740 1 -- start start 2024-03-12T05:15:08.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.434+0000 7fde967bc740 1 Processor -- start 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed2a1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596eced6430 con 0x5596ed29b4d0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ece92e00 con 0x5596ed2a49d0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed187fd0 con 0x5596ed29bdf0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.435+0000 7fde967bc740 10 client.0 _ll_register_callbacks cb 0x5596ed1ddfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde910a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde910a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed2a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde918a9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde910a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed2a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48508/0 (socket says 172.21.15.45:48508) 2024-03-12T05:15:08.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde910a8640 1 -- 172.21.15.45:0/1503755303 learned_addr learned my addr 172.21.15.45:0/1503755303 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde918a9640 1 -- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 msgr2=0x5596ed2a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.436+0000 7fde918a9640 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde918a9640 1 -- 172.21.15.45:0/1503755303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 msgr2=0x5596ed2a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde918a9640 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed2a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde918a9640 1 -- 172.21.15.45:0/1503755303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c114c50 con 0x5596ed29bdf0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde918a9640 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a1d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde880105f0 tx=0x7fde88010ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0fb9788874945b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde757fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88004230 con 0x5596ed29bdf0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.437+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fde88016070 con 0x5596ed29bdf0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.438+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5596ecf027e0 con 0x5596ed29bdf0 2024-03-12T05:15:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.438+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde88004390 con 0x5596ed29bdf0 2024-03-12T05:15:08.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.439+0000 7fde967bc740 10 client.26059 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:08.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.439+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5596ed201610 con 0x5596ed29bdf0 2024-03-12T05:15:08.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde880044f0 con 0x5596ed29bdf0 2024-03-12T05:15:08.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fde88003ac0 con 0x5596ed29bdf0 2024-03-12T05:15:08.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 10 client.26059 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 10 client.26059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 20 client.26059 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed2b4a60 0x5596ed2b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 4 client.26059 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596ecf027e0 con 0x5596ed2b4a60 2024-03-12T05:15:08.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.440+0000 7fde910a8640 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed2b4a60 0x5596ed2b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.441+0000 7fde910a8640 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed2b4a60 0x5596ed2b6e80 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52dc5e99b81d821c server_cookie=ba66f3b8107c6794 in_seq=0 out_seq=0 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.441+0000 7fde757fa640 10 client.26059 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde757fa640 1 -- 172.21.15.45:0/1503755303 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5596ecf027e0 con 0x5596ed2b4a60 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde757fa640 10 client.26059 handle_command_reply: tid=0 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 client.26059 shutdown 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 msgr2=0x5596ed2a1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a1d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde880105f0 tx=0x7fde88010ab0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 shutdown_connections 2024-03-12T05:15:08.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed2b4a60 0x5596ed2b6e80 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed2a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29bdf0 0x5596ed2a1d80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 --2- 172.21.15.45:0/1503755303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29b4d0 0x5596ed2a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 >> 172.21.15.45:0/1503755303 conn(0x5596ed295c00 msgr2=0x5596ed297950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.442+0000 7fde757fa640 0 client.26059 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.443+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 shutdown_connections 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.443+0000 7fde967bc740 1 -- 172.21.15.45:0/1503755303 wait complete. 2024-03-12T05:15:08.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.443+0000 7fde967bc740 20 client.26059 trim_cache size 0 max 16384 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.471+0000 7fde967bc740 1 Processor -- start 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 -- start start 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 0x5596ed2a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed29eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2b0760 con 0x5596ed29bdf0 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2a2b90 con 0x5596ed2a49d0 2024-03-12T05:15:08.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde967bc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed29d4b0 con 0x5596ed29b4d0 2024-03-12T05:15:08.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde918a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 0x5596ed2a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde910a8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde918a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 0x5596ed2a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48510/0 (socket says 172.21.15.45:48510) 2024-03-12T05:15:08.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde918a9640 1 -- 172.21.15.45:0/419912713 learned_addr learned my addr 172.21.15.45:0/419912713 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.472+0000 7fde910a8640 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed29eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 -- 172.21.15.45:0/419912713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 msgr2=0x5596ed29eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed29eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 -- 172.21.15.45:0/419912713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 msgr2=0x5596ed2a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 0x5596ed2a20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 -- 172.21.15.45:0/419912713 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ed19dac0 con 0x5596ed29b4d0 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde910a8640 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2a3f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5596ed2b7770 tx=0x7fde7c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2833c97f1f14e83f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde777fe640 1 -- 172.21.15.45:0/419912713 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0054e0 con 0x5596ed29b4d0 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde777fe640 1 -- 172.21.15.45:0/419912713 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde7c005640 con 0x5596ed29b4d0 2024-03-12T05:15:08.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.473+0000 7fde777fe640 1 -- 172.21.15.45:0/419912713 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c003ba0 con 0x5596ed29b4d0 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 -- 172.21.15.45:0/419912713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 msgr2=0x5596ed2a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2a3f70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5596ed2b7770 tx=0x7fde7c005850 comp rx=0 tx=0).stop 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 -- 172.21.15.45:0/419912713 shutdown_connections 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed29bdf0 0x5596ed2a20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed2a49d0 0x5596ed29eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 --2- 172.21.15.45:0/419912713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2a3f70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.475+0000 7fde967bc740 1 -- 172.21.15.45:0/419912713 >> 172.21.15.45:0/419912713 conn(0x5596ed295c00 msgr2=0x5596ed290bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.476+0000 7fde967bc740 1 -- 172.21.15.45:0/419912713 shutdown_connections 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.476+0000 7fde967bc740 1 -- 172.21.15.45:0/419912713 wait complete. 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.476+0000 7fde967bc740 1 -- start start 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.476+0000 7fde967bc740 1 Processor -- start 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2ab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 0x5596ed2a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed2a49d0 0x5596ed2aa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2aa780 con 0x5596ed2a49d0 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2aa8f0 con 0x5596ed29bdf0 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ed2aaab0 con 0x5596ed29b4d0 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde967bc740 10 client.0 _ll_register_callbacks cb 0x7fde7c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde910a8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde910a8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2ab960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33452/0 (socket says 172.21.15.45:33452) 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde910a8640 1 -- 172.21.15.45:0/1289602055 learned_addr learned my addr 172.21.15.45:0/1289602055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 0x5596ed2a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde910a8640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed2a49d0 0x5596ed2aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 -- 172.21.15.45:0/1289602055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 msgr2=0x5596ed2ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 -- 172.21.15.45:0/1289602055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed2a49d0 msgr2=0x5596ed2aa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed2a49d0 0x5596ed2aa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.477+0000 7fde918a9640 1 -- 172.21.15.45:0/1289602055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c05c040 con 0x5596ed29bdf0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.478+0000 7fde918a9640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 0x5596ed2a9d00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde88009c70 tx=0x7fde88009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87373c9259f88e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.478+0000 7fde757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.479+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88011bc0 con 0x5596ed29bdf0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.479+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fde88000e40 con 0x5596ed29bdf0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.479+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde7c004430 con 0x5596ed29bdf0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.480+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde88011d20 con 0x5596ed29bdf0 2024-03-12T05:15:08.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.480+0000 7fde967bc740 10 client.26057 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:08.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.480+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5596ed1c36a0 con 0x5596ed29bdf0 2024-03-12T05:15:08.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fde88011e80 con 0x5596ed29bdf0 2024-03-12T05:15:08.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fde88033070 con 0x5596ed29bdf0 2024-03-12T05:15:08.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 10 client.26057 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:08.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 20 client.26057 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed30c480 0x5596ed30e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 4 client.26057 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:15:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fde7c004430 con 0x5596ed30c480 2024-03-12T05:15:08.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.481+0000 7fde910a8640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed30c480 0x5596ed30e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde757fa640 10 client.26057 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde910a8640 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed30c480 0x5596ed30e8a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d9e7361e4ec3ce server_cookie=b5439c0b7e67c4f2 in_seq=0 out_seq=0 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde757fa640 1 -- 172.21.15.45:0/1289602055 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fde7c004430 con 0x5596ed30c480 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde757fa640 10 client.26057 handle_command_reply: tid=0 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde967bc740 1 client.26057 shutdown 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 msgr2=0x5596ed2a9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.482+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 0x5596ed2a9d00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde88009c70 tx=0x7fde88009ed0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 shutdown_connections 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5596ed30c480 0x5596ed30e8a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5596ed2a49d0 0x5596ed2aa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5596ed29bdf0 0x5596ed2a9d00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 --2- 172.21.15.45:0/1289602055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5596ed29b4d0 0x5596ed2ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 >> 172.21.15.45:0/1289602055 conn(0x5596ed295c00 msgr2=0x5596ed2a2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde757fa640 0 client.26057 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 shutdown_connections 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 1 -- 172.21.15.45:0/1289602055 wait complete. 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.483+0000 7fde967bc740 20 client.26057 trim_cache size 0 max 16384 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:15:08.481 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:08.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 msgr2=0x7fde780a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 0x7fde780a40e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde840cb170 tx=0x7fde840658e0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 msgr2=0x7fde8c19d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e9d0 tx=0x7fde8400ee90 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 shutdown_connections 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fde780a1c20 0x7fde780a40e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde8c073e40 0x7fde8c1a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8c111f30 0x7fde8c19d780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 --2- 172.21.15.45:0/2295808077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde8c073520 0x7fde8c1a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 >> 172.21.15.45:0/2295808077 conn(0x7fde8c0fca50 msgr2=0x7fde8c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 shutdown_connections 2024-03-12T05:15:08.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.484+0000 7fde928ab640 1 -- 172.21.15.45:0/2295808077 wait complete. 2024-03-12T05:15:08.500 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:15:08.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:15:08.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.673+0000 7f3c78a8e640 1 -- 172.21.15.45:0/3319766401 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70032340 con 0x7f3c6c0a5700 2024-03-12T05:15:08.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 -- 172.21.15.45:0/3319766401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0a5700 msgr2=0x7f3c6c0b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/3319766401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c0b7530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009860 tx=0x7f3c7002bc10 comp rx=0 tx=0).stop 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 -- 172.21.15.45:0/3319766401 shutdown_connections 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/3319766401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c0b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/3319766401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c0b7530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/3319766401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a4de0 0x7f3c6c0a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 -- 172.21.15.45:0/3319766401 >> 172.21.15.45:0/3319766401 conn(0x7f3c6c01a640 msgr2=0x7f3c6c01aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:08.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 -- 172.21.15.45:0/3319766401 shutdown_connections 2024-03-12T05:15:08.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.674+0000 7f3c7aa92640 1 -- 172.21.15.45:0/3319766401 wait complete. 2024-03-12T05:15:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.675+0000 7f3c7aa92640 1 Processor -- start 2024-03-12T05:15:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.675+0000 7f3c7aa92640 1 -- start start 2024-03-12T05:15:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.675+0000 7f3c7aa92640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c6c0a4de0 0x7f3c6c1495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.675+0000 7f3c7aa92640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7aa92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7aa92640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c143df0 con 0x7f3c6c0a5700 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7aa92640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c143f60 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7aa92640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c144120 con 0x7f3c6c0a4de0 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7928f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7928f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48528/0 (socket says 172.21.15.45:48528) 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7928f640 1 -- 172.21.15.45:0/975107805 learned_addr learned my addr 172.21.15.45:0/975107805 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 -- 172.21.15.45:0/975107805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c6c0a4de0 msgr2=0x7f3c6c1495a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c6c0a4de0 0x7f3c6c1495a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 -- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 msgr2=0x7f3c6c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.676+0000 7f3c7a291640 1 -- 172.21.15.45:0/975107805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70009490 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c7a291640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c143660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3c74066e50 tx=0x7f3c74072b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74062220 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c144400 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c74062380 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c150520 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.677+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74061c40 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.679+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58005290 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.680+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3c74074430 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.681+0000 7f3c6affd640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 0x7f3c600a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.681+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3c74130550 con 0x7f3c6c0b7a70 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.681+0000 7f3c79a90640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 0x7f3c600a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:08.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.682+0000 7f3c79a90640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 0x7f3c600a4160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c018920 tx=0x7f3c64009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:08.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:08.685+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c740f93c0 con 0x7f3c6c0b7a70 2024-03-12T05:15:09.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.095+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c58005810 con 0x7f3c6c0b7a70 2024-03-12T05:15:09.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.096+0000 7f3c6affd640 1 -- 172.21.15.45:0/975107805 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f3c74114ca0 con 0x7f3c6c0b7a70 2024-03-12T05:15:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:15:09.093 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:15:09.093 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:15:09.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.100+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 msgr2=0x7f3c600a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.100+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 0x7f3c600a4160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c018920 tx=0x7f3c64009260 comp rx=0 tx=0).stop 2024-03-12T05:15:09.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.100+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 msgr2=0x7f3c6c143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.100+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c143660 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3c74066e50 tx=0x7f3c74072b50 comp rx=0 tx=0).stop 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 shutdown_connections 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c600a1ca0 0x7f3c600a4160 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c6c0a5700 0x7f3c6c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c6c0b7a70 0x7f3c6c143660 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 --2- 172.21.15.45:0/975107805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c6c0a4de0 0x7f3c6c1495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.101+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 >> 172.21.15.45:0/975107805 conn(0x7f3c6c01a640 msgr2=0x7f3c6c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.102+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 shutdown_connections 2024-03-12T05:15:09.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.102+0000 7f3c7aa92640 1 -- 172.21.15.45:0/975107805 wait complete. 2024-03-12T05:15:09.117 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:09.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 -- 172.21.15.45:0/3293742047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8830100040 msgr2=0x7f88301004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 --2- 172.21.15.45:0/3293742047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8830100040 0x7f88301004a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8824009860 tx=0x7f882402bc10 comp rx=0 tx=0).stop 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 -- 172.21.15.45:0/3293742047 shutdown_connections 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 --2- 172.21.15.45:0/3293742047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88301009e0 0x7f883010cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 --2- 172.21.15.45:0/3293742047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8830100040 0x7f88301004a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 --2- 172.21.15.45:0/3293742047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f88301064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 -- 172.21.15.45:0/3293742047 >> 172.21.15.45:0/3293742047 conn(0x7f8830077110 msgr2=0x7f8830077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 -- 172.21.15.45:0/3293742047 shutdown_connections 2024-03-12T05:15:09.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.291+0000 7f8836993640 1 -- 172.21.15.45:0/3293742047 wait complete. 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 Processor -- start 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 -- start start 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8830100040 0x7f8830195220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f8830195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88301a9e70 con 0x7f8830106100 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88301a9fb0 con 0x7f8830100040 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.292+0000 7f8836993640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88301aa0f0 con 0x7f88301009e0 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33494/0 (socket says 172.21.15.45:33494) 2024-03-12T05:15:09.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 -- 172.21.15.45:0/203388304 learned_addr learned my addr 172.21.15.45:0/203388304 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882ffff640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8830100040 0x7f8830195220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f8834f30640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f8830195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 -- 172.21.15.45:0/203388304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8830100040 msgr2=0x7f8830195220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8830100040 0x7f8830195220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 -- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 msgr2=0x7f8830195ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f8830195ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f882f7fe640 1 -- 172.21.15.45:0/203388304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8820009560 con 0x7f88301009e0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.293+0000 7f8834f30640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f8830195ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f882f7fe640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88240057c0 tx=0x7f88240346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f882d7fa640 1 -- 172.21.15.45:0/203388304 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882402c7b0 con 0x7f88301009e0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f882d7fa640 1 -- 172.21.15.45:0/203388304 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88240044d0 con 0x7f88301009e0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f882d7fa640 1 -- 172.21.15.45:0/203388304 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882403d3f0 con 0x7f88301009e0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8824009490 con 0x7f88301009e0 2024-03-12T05:15:09.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.294+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88301aa610 con 0x7f88301009e0 2024-03-12T05:15:09.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.297+0000 7f882d7fa640 1 -- 172.21.15.45:0/203388304 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f882403b040 con 0x7f88301009e0 2024-03-12T05:15:09.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.297+0000 7f883956a740 1 Processor -- start 2024-03-12T05:15:09.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 -- start start 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d92630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d9b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d9db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213d4d960 con 0x562213d92b70 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213d01310 con 0x562213d9b750 2024-03-12T05:15:09.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f883956a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213cfe840 con 0x562213d92250 2024-03-12T05:15:09.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f882d7fa640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 0x7f88180a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f882ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d9b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f882ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d9b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48546/0 (socket says 172.21.15.45:48546) 2024-03-12T05:15:09.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f882ffff640 1 -- 172.21.15.45:0/4067210402 learned_addr learned my addr 172.21.15.45:0/4067210402 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.298+0000 7f8834f30640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d92630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f882f7fe640 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d9db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f882d7fa640 1 -- 172.21.15.45:0/203388304 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f88240ef240 con 0x7f88301009e0 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 0x7f88180a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 -- 172.21.15.45:0/4067210402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 msgr2=0x562213d9db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d9db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 -- 172.21.15.45:0/4067210402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 msgr2=0x562213d9b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d9b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 -- 172.21.15.45:0/4067210402 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562213cbc060 con 0x562213d92250 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f882f7fe640 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d9db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.299+0000 7f8834f30640 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d92630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8828004460 tx=0x7f8828004860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=714fa891044aa963 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.300+0000 7f88167fc640 1 -- 172.21.15.45:0/4067210402 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828013070 con 0x562213d92250 2024-03-12T05:15:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.300+0000 7f88167fc640 1 -- 172.21.15.45:0/4067210402 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f882801cd80 con 0x562213d92250 2024-03-12T05:15:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.300+0000 7f88167fc640 1 -- 172.21.15.45:0/4067210402 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828025bb0 con 0x562213d92250 2024-03-12T05:15:09.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.300+0000 7f8834f30640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 0x7f88180a40e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8828022000 tx=0x7f8828020000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- 172.21.15.45:0/4067210402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 msgr2=0x562213d92630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d92630 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8828004460 tx=0x7f8828004860 comp rx=0 tx=0).stop 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- 172.21.15.45:0/4067210402 shutdown_connections 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d9b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d9db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 --2- 172.21.15.45:0/4067210402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d92630 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- 172.21.15.45:0/4067210402 >> 172.21.15.45:0/4067210402 conn(0x562213d8c980 msgr2=0x562213d8cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- 172.21.15.45:0/4067210402 shutdown_connections 2024-03-12T05:15:09.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- 172.21.15.45:0/4067210402 wait complete. 2024-03-12T05:15:09.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.302+0000 7f883956a740 1 -- start start 2024-03-12T05:15:09.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 Processor -- start 2024-03-12T05:15:09.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d98780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d99200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213cfa020 con 0x562213d92b70 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622139cd430 con 0x562213d9b750 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213989e00 con 0x562213d92250 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f883956a740 10 client.0 _ll_register_callbacks cb 0x562213ca1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48548/0 (socket says 172.21.15.45:48548) 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f882f7fe640 1 -- 172.21.15.45:0/463382586 learned_addr learned my addr 172.21.15.45:0/463382586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f882ffff640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d98780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.303+0000 7f8834f30640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d99200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 -- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 msgr2=0x562213d98780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d98780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 -- 172.21.15.45:0/463382586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 msgr2=0x562213d99200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d99200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 -- 172.21.15.45:0/463382586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8828004040 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f8834f30640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d99200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882ffff640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d98780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f882f7fe640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f88240ecf90 tx=0x7f882404c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563a9c496aec28cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.304+0000 7f880f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882407e990 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f88240c00d0 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622139f97e0 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88240039d0 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 10 client.16608 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562213cf86a0 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5622139f97e0 con 0x562213d92b70 2024-03-12T05:15:09.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f88240be020 con 0x562213d92b70 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 10 client.16608 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 10 client.16608 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.305+0000 7f883956a740 20 client.16608 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.306+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213dab770 0x562213dadb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.306+0000 7f883956a740 4 client.16608 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:09.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.306+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562213a36720 con 0x562213dab770 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.307+0000 7f882ffff640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213dab770 0x562213dadb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.307+0000 7f880f7fe640 10 client.16608 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.307+0000 7f882ffff640 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213dab770 0x562213dadb60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e25b629431dff51a server_cookie=dea96a0c8e40c3a5 in_seq=0 out_seq=0 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f880f7fe640 1 -- 172.21.15.45:0/463382586 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562213a36720 con 0x562213dab770 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f880f7fe640 10 client.16608 handle_command_reply: tid=0 2024-03-12T05:15:09.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 client.16608 shutdown 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 msgr2=0x562213d98cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f88240ecf90 tx=0x7f882404c880 comp rx=0 tx=0).stop 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 shutdown_connections 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213dab770 0x562213dadb60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d98cc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d99200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 --2- 172.21.15.45:0/463382586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d98780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 >> 172.21.15.45:0/463382586 conn(0x562213d8c980 msgr2=0x562213d8e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f880f7fe640 0 client.16608 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 shutdown_connections 2024-03-12T05:15:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.308+0000 7f883956a740 1 -- 172.21.15.45:0/463382586 wait complete. 2024-03-12T05:15:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.309+0000 7f883956a740 20 client.16608 trim_cache size 0 max 16384 2024-03-12T05:15:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.328+0000 7f883956a740 1 Processor -- start 2024-03-12T05:15:09.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f883956a740 1 -- start start 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f883956a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f883956a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f883956a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d959e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f882f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48562/0 (socket says 172.21.15.45:48562) 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f882ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.329+0000 7f882ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33536/0 (socket says 172.21.15.45:33536) 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 -- 172.21.15.45:0/2944690802 learned_addr learned my addr 172.21.15.45:0/2944690802 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213c7f130 con 0x562213d92b70 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213a36750 con 0x562213d9b750 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213d943c0 con 0x562213d92250 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f8834f30640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 -- 172.21.15.45:0/2944690802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 msgr2=0x562213d9aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 -- 172.21.15.45:0/2944690802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 msgr2=0x562213d959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f882f7fe640 1 -- 172.21.15.45:0/2944690802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562213c94c20 con 0x562213d92b70 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.330+0000 7f8834f30640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.331+0000 7f882ffff640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.331+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8824032040 tx=0x7f882402c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4588d64c1f3823e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.331+0000 7f8815ffb640 1 -- 172.21.15.45:0/2944690802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240bc050 con 0x562213d92b70 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.331+0000 7f8815ffb640 1 -- 172.21.15.45:0/2944690802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88240c4290 con 0x562213d92b70 2024-03-12T05:15:09.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.331+0000 7f8815ffb640 1 -- 172.21.15.45:0/2944690802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240bc4f0 con 0x562213d92b70 2024-03-12T05:15:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.332+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 msgr2=0x562213d99020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.332+0000 7f883956a740 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8824032040 tx=0x7f882402c960 comp rx=0 tx=0).stop 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 shutdown_connections 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213d99020 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d9b750 0x562213d959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 --2- 172.21.15.45:0/2944690802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d92250 0x562213d9aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 >> 172.21.15.45:0/2944690802 conn(0x562213d8c980 msgr2=0x562213db24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 shutdown_connections 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 -- 172.21.15.45:0/2944690802 wait complete. 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.333+0000 7f883956a740 1 -- start start 2024-03-12T05:15:09.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.334+0000 7f883956a740 1 Processor -- start 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.334+0000 7f883956a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 0x562213da2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.334+0000 7f883956a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213da2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.334+0000 7f882ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 0x562213da2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.334+0000 7f882ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 0x562213da2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34198/0 (socket says 172.21.15.45:34198) 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f882ffff640 1 -- 172.21.15.45:0/2453922973 learned_addr learned my addr 172.21.15.45:0/2453922973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d9b750 0x562213da0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213da2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213da1270 con 0x562213d92b70 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f8834f30640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d9b750 0x562213da0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213da13e0 con 0x562213d92250 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562213da15a0 con 0x562213d9b750 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.335+0000 7f883956a740 10 client.0 _ll_register_callbacks cb 0x7f8828003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.336+0000 7f882f7fe640 1 -- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d9b750 msgr2=0x562213da0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.336+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d9b750 0x562213da0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.336+0000 7f882f7fe640 1 -- 172.21.15.45:0/2453922973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 msgr2=0x562213da2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.336+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 0x562213da2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.336+0000 7f882f7fe640 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8820005d10 con 0x562213d92b70 2024-03-12T05:15:09.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.337+0000 7f882f7fe640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213da2b40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8824032040 tx=0x7f88240bc4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367381135871054a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.337+0000 7f880f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:15:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.337+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240c00d0 con 0x562213d92b70 2024-03-12T05:15:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.337+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f88240bc050 con 0x562213d92b70 2024-03-12T05:15:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.338+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f882801cd80 con 0x562213d92b70 2024-03-12T05:15:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.338+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88240c4720 con 0x562213d92b70 2024-03-12T05:15:09.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.338+0000 7f883956a740 10 client.16614 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.338+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f882802a040 con 0x562213d92b70 2024-03-12T05:15:09.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f882409f080 con 0x562213d92b70 2024-03-12T05:15:09.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f88240be070 con 0x562213d92b70 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 10 client.16614 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 10 client.16614 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 20 client.16614 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213e02ba0 0x562213e04f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 4 client.16614 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f882801cd80 con 0x562213e02ba0 2024-03-12T05:15:09.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.339+0000 7f882ffff640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213e02ba0 0x562213e04f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.340+0000 7f880f7fe640 10 client.16614 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.340+0000 7f882ffff640 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213e02ba0 0x562213e04f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b484dba6aad077fc server_cookie=cbfb93ad334d98bb in_seq=0 out_seq=0 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.340+0000 7f880f7fe640 1 -- 172.21.15.45:0/2453922973 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f882801cd80 con 0x562213e02ba0 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.340+0000 7f880f7fe640 10 client.16614 handle_command_reply: tid=0 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.340+0000 7f883956a740 1 client.16614 shutdown 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 msgr2=0x562213da2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213da2b40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8824032040 tx=0x7f88240bc4c0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 shutdown_connections 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x562213e02ba0 0x562213e04f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f880f7fe640 0 client.16614 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562213d92b70 0x562213da2b40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562213d9b750 0x562213da0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 --2- 172.21.15.45:0/2453922973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562213d92250 0x562213da2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 >> 172.21.15.45:0/2453922973 conn(0x562213d8c980 msgr2=0x562213d8b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.341+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 shutdown_connections 2024-03-12T05:15:09.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.342+0000 7f883956a740 1 -- 172.21.15.45:0/2453922973 wait complete. 2024-03-12T05:15:09.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.342+0000 7f883956a740 20 client.16614 trim_cache size 0 max 16384 2024-03-12T05:15:09.339 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 msgr2=0x7f88180a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 0x7f88180a40e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8828022000 tx=0x7f8828020000 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 msgr2=0x7f8830195760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88240057c0 tx=0x7f88240346f0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 shutdown_connections 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88180a1c20 0x7f88180a40e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8830106100 0x7f8830195ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88301009e0 0x7f8830195760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.344+0000 7f8836993640 1 --2- 172.21.15.45:0/203388304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8830100040 0x7f8830195220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.345+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 >> 172.21.15.45:0/203388304 conn(0x7f8830077110 msgr2=0x7f88300fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.345+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 shutdown_connections 2024-03-12T05:15:09.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.345+0000 7f8836993640 1 -- 172.21.15.45:0/203388304 wait complete. 2024-03-12T05:15:09.358 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:09.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.533+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/3071091278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0101390 msgr2=0x7fcdd0101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.533+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/3071091278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0101390 0x7fcdd0101770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0009890 tx=0x7fcdc002bb70 comp rx=0 tx=0).stop 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/3071091278 shutdown_connections 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/3071091278 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0106460 0x7fcdd010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/3071091278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101d40 0x7fcdd0105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/3071091278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0101390 0x7fcdd0101770 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/3071091278 >> 172.21.15.45:0/3071091278 conn(0x7fcdd00fcbb0 msgr2=0x7fcdd00fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/3071091278 shutdown_connections 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.534+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/3071091278 wait complete. 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 Processor -- start 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 -- start start 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 0x7fcdd01145a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0110db0 0x7fcdd0111230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd01b2ab0 con 0x7fcdd0110db0 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd01b2c20 con 0x7fcdd0106460 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdd4bf4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd01b2de0 con 0x7fcdd0101390 2024-03-12T05:15:09.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34204/0 (socket says 172.21.15.45:34204) 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 0x7fcdd01145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.535+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2201325033 learned_addr learned my addr 172.21.15.45:0/2201325033 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdced9d640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0110db0 0x7fcdd0111230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2201325033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 msgr2=0x7fcdd01145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 0x7fcdd01145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0110db0 msgr2=0x7fcdd0111230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0110db0 0x7fcdd0111230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2201325033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4009490 con 0x7fcdd0106460 2024-03-12T05:15:09.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdce59c640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 0x7fcdd01145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9a0 tx=0x7fcdc400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdbf7fe640 1 -- 172.21.15.45:0/2201325033 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400cd80 con 0x7fcdd0106460 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdbf7fe640 1 -- 172.21.15.45:0/2201325033 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcdc40044d0 con 0x7fcdd0106460 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc8009900 con 0x7fcdd0106460 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdbf7fe640 1 -- 172.21.15.45:0/2201325033 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4010690 con 0x7fcdd0106460 2024-03-12T05:15:09.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.536+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc0009490 con 0x7fcdd0106460 2024-03-12T05:15:09.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.539+0000 7fcdbf7fe640 1 -- 172.21.15.45:0/2201325033 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fcdc40107f0 con 0x7fcdd0106460 2024-03-12T05:15:09.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 Processor -- start 2024-03-12T05:15:09.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 -- start start 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 0x55cd564ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56466960 con 0x55cd564ab250 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd5641a310 con 0x55cd564abb70 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.540+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56417840 con 0x55cd564b4750 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdced9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 0x55cd564ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdced9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 0x55cd564ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48612/0 (socket says 172.21.15.45:48612) 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34216/0 (socket says 172.21.15.45:34216) 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdced9d640 1 -- 172.21.15.45:0/2863919744 learned_addr learned my addr 172.21.15.45:0/2863919744 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdbf7fe640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 0x7fcdb80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 -- 172.21.15.45:0/2863919744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 msgr2=0x55cd564b6b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564b6b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 -- 172.21.15.45:0/2863919744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 msgr2=0x55cd564ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 0x55cd564ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 -- 172.21.15.45:0/2863919744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd563d5060 con 0x55cd564abb70 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdbf7fe640 1 -- 172.21.15.45:0/2201325033 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcdc4014070 con 0x7fcdd0106460 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdced9d640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 0x7fcdb80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.541+0000 7fcdce59c640 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0035fd0 tx=0x7fcdc0035f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c231544ae067d4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.542+0000 7fcdbcff9640 1 -- 172.21.15.45:0/2863919744 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0004400 con 0x55cd564abb70 2024-03-12T05:15:09.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.542+0000 7fcdbcff9640 1 -- 172.21.15.45:0/2863919744 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcdc002ce90 con 0x55cd564abb70 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.542+0000 7fcdbcff9640 1 -- 172.21.15.45:0/2863919744 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0003850 con 0x55cd564abb70 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdced9d640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 0x7fcdb80a4230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005c40 tx=0x7fcdc8004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2863919744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 msgr2=0x55cd564b4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0035fd0 tx=0x7fcdc0035f60 comp rx=0 tx=0).stop 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2863919744 shutdown_connections 2024-03-12T05:15:09.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564b4210 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2863919744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564ab250 0x55cd564ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2863919744 >> 172.21.15.45:0/2863919744 conn(0x55cd564a5980 msgr2=0x55cd564a5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2863919744 shutdown_connections 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.543+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2863919744 wait complete. 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdd77ca740 1 -- start start 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdd77ca740 1 Processor -- start 2024-03-12T05:15:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 0x55cd564b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.544+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564b1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48614/0 (socket says 172.21.15.45:48614) 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33568/0 (socket says 172.21.15.45:33568) 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/3364988191 learned_addr learned my addr 172.21.15.45:0/3364988191 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56413020 con 0x55cd564abb70 2024-03-12T05:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd560e6430 con 0x55cd564b4750 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd560a2e00 con 0x55cd564ab250 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 0x55cd564b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdd77ca740 10 client.0 _ll_register_callbacks cb 0x55cd563bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 -- 172.21.15.45:0/3364988191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 msgr2=0x55cd564b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 -- 172.21.15.45:0/3364988191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 msgr2=0x55cd564b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd010e840 con 0x55cd564b4750 2024-03-12T05:15:09.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.545+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 0x55cd564b2220 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800ffd0 tx=0x7fcdc8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43aa5dbbbd3756b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3364988191 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8013030 con 0x55cd564b4750 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3364988191 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcdc8014070 con 0x55cd564b4750 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd561127e0 con 0x55cd564b4750 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3364988191 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc8004820 con 0x55cd564b4750 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdd77ca740 10 client.26093 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55cd564116a0 con 0x55cd564b4750 2024-03-12T05:15:09.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.546+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3364988191 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcdc802f030 con 0x55cd564b4750 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 10 client.26093 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 10 client.26093 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 20 client.26093 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd564c4a10 0x55cd564c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 4 client.26093 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd5614f720 con 0x55cd564c4a10 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.547+0000 7fcdce59c640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd564c4a10 0x55cd564c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.548+0000 7fcdb5ffb640 10 client.26093 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.548+0000 7fcdce59c640 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd564c4a10 0x55cd564c6e30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb7c1c84947ee35 server_cookie=847822bec483d9fd in_seq=0 out_seq=0 2024-03-12T05:15:09.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.548+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3364988191 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cd5614f720 con 0x55cd564c4a10 2024-03-12T05:15:09.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.548+0000 7fcdb5ffb640 10 client.26093 handle_command_reply: tid=0 2024-03-12T05:15:09.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.548+0000 7fcdd77ca740 1 client.26093 shutdown 2024-03-12T05:15:09.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 msgr2=0x55cd564b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 0x55cd564b2220 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800ffd0 tx=0x7fcdc8004a40 comp rx=0 tx=0).stop 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 shutdown_connections 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd564c4a10 0x55cd564c6e30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564b4750 0x55cd564b2220 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3364988191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 >> 172.21.15.45:0/3364988191 conn(0x55cd564a5980 msgr2=0x55cd564a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdb5ffb640 0 client.26093 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:09.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 shutdown_connections 2024-03-12T05:15:09.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3364988191 wait complete. 2024-03-12T05:15:09.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.549+0000 7fcdd77ca740 20 client.26093 trim_cache size 0 max 16384 2024-03-12T05:15:09.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 Processor -- start 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 -- start start 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564ab250 0x55cd564b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56398130 con 0x55cd564abb70 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd564ad3e0 con 0x55cd564ab250 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd564a6bb0 con 0x55cd564b4750 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.572+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564ab250 0x55cd564b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48622/0 (socket says 172.21.15.45:48622) 2024-03-12T05:15:09.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2524953598 learned_addr learned my addr 172.21.15.45:0/2524953598 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdced9d640 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2524953598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 msgr2=0x55cd564aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2524953598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564ab250 msgr2=0x55cd564b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564ab250 0x55cd564b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 -- 172.21.15.45:0/2524953598 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd563adc20 con 0x55cd564abb70 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c6a0 tx=0x7fcdc4003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7480a95b840487e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.573+0000 7fcdb7fff640 1 -- 172.21.15.45:0/2524953598 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc40b3270 con 0x55cd564abb70 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.574+0000 7fcdb7fff640 1 -- 172.21.15.45:0/2524953598 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcdc40b8e90 con 0x55cd564abb70 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.574+0000 7fcdb7fff640 1 -- 172.21.15.45:0/2524953598 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4003810 con 0x55cd564abb70 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2524953598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 msgr2=0x55cd564cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c6a0 tx=0x7fcdc4003ae0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2524953598 shutdown_connections 2024-03-12T05:15:09.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564abb70 0x55cd564cb7b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564b4750 0x55cd564aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/2524953598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564ab250 0x55cd564b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2524953598 >> 172.21.15.45:0/2524953598 conn(0x55cd564a5980 msgr2=0x55cd564a0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2524953598 shutdown_connections 2024-03-12T05:15:09.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.575+0000 7fcdd77ca740 1 -- 172.21.15.45:0/2524953598 wait complete. 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.576+0000 7fcdd77ca740 1 -- start start 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.576+0000 7fcdd77ca740 1 Processor -- start 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56514440 con 0x55cd564b4750 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd565145b0 con 0x55cd564abb70 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdced9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdced9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48632/0 (socket says 172.21.15.45:48632) 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdcdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34240/0 (socket says 172.21.15.45:34240) 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdced9d640 1 -- 172.21.15.45:0/3602366116 learned_addr learned my addr 172.21.15.45:0/3602366116 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdce59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd56514770 con 0x55cd564ab250 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.577+0000 7fcdd77ca740 10 client.0 _ll_register_callbacks cb 0x7fcdc8009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 -- 172.21.15.45:0/3602366116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 msgr2=0x55cd564baa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564baa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 -- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 msgr2=0x55cd564bafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 -- 172.21.15.45:0/3602366116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc0009490 con 0x55cd564b4750 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdcdd9b640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdce59c640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdced9d640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8011010 tx=0x7fcdc8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea3d87b93251d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.578+0000 7fcdb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:15:09.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.579+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8004820 con 0x55cd564b4750 2024-03-12T05:15:09.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.579+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fcdc801ee70 con 0x55cd564b4750 2024-03-12T05:15:09.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.580+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcdc002ce90 con 0x55cd564b4750 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.581+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdc8005eb0 con 0x55cd564b4750 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.581+0000 7fcdd77ca740 10 client.16632 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.581+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cd563d38c0 con 0x55cd564b4750 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.581+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fcdc002ce90 con 0x55cd564b4750 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcdc803b9a0 con 0x55cd564b4750 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 10 client.16632 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 10 client.16632 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 20 client.16632 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd5651d0b0 0x55cd5651f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 4 client.16632 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cd5619e890 con 0x55cd5651d0b0 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.582+0000 7fcdce59c640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd5651d0b0 0x55cd5651f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.583+0000 7fcdb5ffb640 10 client.16632 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.583+0000 7fcdce59c640 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd5651d0b0 0x55cd5651f4d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe1558a05e8996a8 server_cookie=fdf5ff204aad3788 in_seq=0 out_seq=0 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.583+0000 7fcdb5ffb640 1 -- 172.21.15.45:0/3602366116 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cd5619e890 con 0x55cd5651d0b0 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.583+0000 7fcdb5ffb640 10 client.16632 handle_command_reply: tid=0 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.583+0000 7fcdd77ca740 1 client.16632 shutdown 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 msgr2=0x55cd56513f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8011010 tx=0x7fcdc8014000 comp rx=0 tx=0).stop 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 shutdown_connections 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55cd5651d0b0 0x55cd5651f4d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cd564b4750 0x55cd56513f00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd564abb70 0x55cd564bafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 --2- 172.21.15.45:0/3602366116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cd564ab250 0x55cd564baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdb5ffb640 0 client.16632 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 >> 172.21.15.45:0/3602366116 conn(0x55cd564a5980 msgr2=0x55cd564afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 shutdown_connections 2024-03-12T05:15:09.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.584+0000 7fcdd77ca740 1 -- 172.21.15.45:0/3602366116 wait complete. 2024-03-12T05:15:09.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.585+0000 7fcdd77ca740 20 client.16632 trim_cache size 0 max 16384 2024-03-12T05:15:09.585 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 msgr2=0x7fcdb80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 0x7fcdb80a4230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8005c40 tx=0x7fcdc8004ab0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 msgr2=0x7fcdd0110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e9a0 tx=0x7fcdc400ee60 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 shutdown_connections 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fcdb80a1d70 0x7fcdb80a4230 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdd0110db0 0x7fcdd0111230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd0106460 0x7fcdd0110870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 --2- 172.21.15.45:0/2201325033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdd0101390 0x7fcdd01145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 >> 172.21.15.45:0/2201325033 conn(0x7fcdd00fcbb0 msgr2=0x7fcdd0104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 shutdown_connections 2024-03-12T05:15:09.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.586+0000 7fcdd4bf4640 1 -- 172.21.15.45:0/2201325033 wait complete. 2024-03-12T05:15:09.600 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:09.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.780+0000 7f335f615640 1 -- 172.21.15.45:0/978734801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 msgr2=0x7f3358100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.780+0000 7f335f615640 1 --2- 172.21.15.45:0/978734801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f3358100640 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3354009ca0 tx=0x7f335402c100 comp rx=0 tx=0).stop 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 -- 172.21.15.45:0/978734801 shutdown_connections 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 --2- 172.21.15.45:0/978734801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 0x7f33580682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 --2- 172.21.15.45:0/978734801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358100c10 0x7f3358104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 --2- 172.21.15.45:0/978734801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f3358100640 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 -- 172.21.15.45:0/978734801 >> 172.21.15.45:0/978734801 conn(0x7f3358077180 msgr2=0x7f3358077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 -- 172.21.15.45:0/978734801 shutdown_connections 2024-03-12T05:15:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.781+0000 7f335f615640 1 -- 172.21.15.45:0/978734801 wait complete. 2024-03-12T05:15:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.782+0000 7f335f615640 1 Processor -- start 2024-03-12T05:15:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.782+0000 7f335f615640 1 -- start start 2024-03-12T05:15:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 0x7f3358110420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 0x7f3358114ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3358111190 con 0x7f3358100260 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3358115400 con 0x7f3358110ac0 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335f615640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581155c0 con 0x7f3358105330 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48660/0 (socket says 172.21.15.45:48660) 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335d3b1640 1 -- 172.21.15.45:0/2452181699 learned_addr learned my addr 172.21.15.45:0/2452181699 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335cbb0640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 0x7f3358110420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.783+0000 7f335dbb2640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 0x7f3358114ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 -- 172.21.15.45:0/2452181699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 msgr2=0x7f3358110420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 0x7f3358110420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 -- 172.21.15.45:0/2452181699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 msgr2=0x7f3358114ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 0x7f3358114ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 -- 172.21.15.45:0/2452181699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3354009900 con 0x7f3358100260 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335dbb2640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 0x7f3358114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.784+0000 7f335d3b1640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f335402c710 tx=0x7f33540347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.785+0000 7f334e7fc640 1 -- 172.21.15.45:0/2452181699 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354032a00 con 0x7f3358100260 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.785+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3358115840 con 0x7f3358100260 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.785+0000 7f334e7fc640 1 -- 172.21.15.45:0/2452181699 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3354003e20 con 0x7f3358100260 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.785+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3358115cf0 con 0x7f3358100260 2024-03-12T05:15:09.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.785+0000 7f334e7fc640 1 -- 172.21.15.45:0/2452181699 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335403f440 con 0x7f3358100260 2024-03-12T05:15:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.789+0000 7f334e7fc640 1 -- 172.21.15.45:0/2452181699 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f335403b040 con 0x7f3358100260 2024-03-12T05:15:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.789+0000 7f33621ec740 1 Processor -- start 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.789+0000 7f33621ec740 1 -- start start 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe21630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe2cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebddc960 con 0x5591ebe21250 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebd90310 con 0x5591ebe21b70 2024-03-12T05:15:09.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f33621ec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebd8d840 con 0x5591ebe2a750 2024-03-12T05:15:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335cbb0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe2cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34260/0 (socket says 172.21.15.45:34260) 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335d3b1640 1 -- 172.21.15.45:0/1272430397 learned_addr learned my addr 172.21.15.45:0/1272430397 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335cbb0640 1 -- 172.21.15.45:0/1272430397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 msgr2=0x5591ebe2a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335cbb0640 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335cbb0640 1 -- 172.21.15.45:0/1272430397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 msgr2=0x5591ebe21630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.790+0000 7f335dbb2640 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe21630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f335cbb0640 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe21630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f335cbb0640 1 -- 172.21.15.45:0/1272430397 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591ebd4b060 con 0x5591ebe2a750 2024-03-12T05:15:09.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f335dbb2640 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe21630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f334e7fc640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 0x7f33440a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f335cbb0640 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe2cb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3348005fd0 tx=0x7f334800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe5e7eb2e22a0fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f33437fe640 1 -- 172.21.15.45:0/1272430397 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334800dea0 con 0x5591ebe2a750 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f33437fe640 1 -- 172.21.15.45:0/1272430397 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f33480054d0 con 0x5591ebe2a750 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f33437fe640 1 -- 172.21.15.45:0/1272430397 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33480116a0 con 0x5591ebe2a750 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f334e7fc640 1 -- 172.21.15.45:0/2452181699 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f33540f0410 con 0x7f3358100260 2024-03-12T05:15:09.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.791+0000 7f335dbb2640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 0x7f33440a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 -- 172.21.15.45:0/1272430397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 msgr2=0x5591ebe2cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe2cb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3348005fd0 tx=0x7f334800f730 comp rx=0 tx=0).stop 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f335dbb2640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 0x7f33440a4120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f335000bc00 tx=0x7f3350005f30 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 -- 172.21.15.45:0/1272430397 shutdown_connections 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe2cb40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 --2- 172.21.15.45:0/1272430397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe21630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.792+0000 7f33621ec740 1 -- 172.21.15.45:0/1272430397 >> 172.21.15.45:0/1272430397 conn(0x5591ebe1b980 msgr2=0x5591ebe1bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.793+0000 7f33621ec740 1 -- 172.21.15.45:0/1272430397 shutdown_connections 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.793+0000 7f33621ec740 1 -- 172.21.15.45:0/1272430397 wait complete. 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.793+0000 7f33621ec740 1 -- start start 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 Processor -- start 2024-03-12T05:15:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe27c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe281b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebd89020 con 0x5591ebe2a750 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591eba5c430 con 0x5591ebe21250 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f33621ec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591eba18e00 con 0x5591ebe21b70 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.794+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34270/0 (socket says 172.21.15.45:34270) 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 -- 172.21.15.45:0/3606890074 learned_addr learned my addr 172.21.15.45:0/3606890074 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335cbb0640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe27c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335dbb2640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe281b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f33621ec740 10 client.0 _ll_register_callbacks cb 0x5591ebd30cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 -- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 msgr2=0x5591ebe27c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe27c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 -- 172.21.15.45:0/3606890074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 msgr2=0x5591ebe281b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe281b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 -- 172.21.15.45:0/3606890074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334800d840 con 0x5591ebe21250 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f335d3b1640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f33540edd20 tx=0x7f33540f6160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202cbeabb2818460 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591eba887e0 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.795+0000 7f3340ff9640 10 client.26111 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f3340ff9640 1 -- 172.21.15.45:0/3606890074 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354003b70 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f3340ff9640 1 -- 172.21.15.45:0/3606890074 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f33540333a0 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f3340ff9640 1 -- 172.21.15.45:0/3606890074 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33540bcf20 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 10 client.26111 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5591ebd876a0 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f3340ff9640 1 -- 172.21.15.45:0/3606890074 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f33540ef070 con 0x5591ebe21250 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 10 client.26111 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 10 client.26111 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:15:09.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 20 client.26111 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe3a7e0 0x5591ebe3cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 4 client.26111 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.796+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591eba887e0 con 0x5591ebe3a7e0 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.797+0000 7f335cbb0640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe3a7e0 0x5591ebe3cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.797+0000 7f335cbb0640 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe3a7e0 0x5591ebe3cca0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6ef5d98de192210 server_cookie=30b73383d453af6 in_seq=0 out_seq=0 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.797+0000 7f3340ff9640 10 client.26111 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.798+0000 7f3340ff9640 1 -- 172.21.15.45:0/3606890074 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5591eba887e0 con 0x5591ebe3a7e0 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.798+0000 7f3340ff9640 10 client.26111 handle_command_reply: tid=0 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.798+0000 7f33621ec740 1 client.26111 shutdown 2024-03-12T05:15:09.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.798+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 msgr2=0x5591ebe27730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.798+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f33540edd20 tx=0x7f33540f6160 comp rx=0 tx=0).stop 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 shutdown_connections 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe3a7e0 0x5591ebe3cca0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f3340ff9640 0 client.26111 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe281b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe27c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 --2- 172.21.15.45:0/3606890074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe27730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 >> 172.21.15.45:0/3606890074 conn(0x5591ebe1b980 msgr2=0x5591ebe1d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 shutdown_connections 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 1 -- 172.21.15.45:0/3606890074 wait complete. 2024-03-12T05:15:09.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.799+0000 7f33621ec740 20 client.26111 trim_cache size 0 max 16384 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.823+0000 7f33621ec740 1 Processor -- start 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.823+0000 7f33621ec740 1 -- start start 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe29e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe24990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe24ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebd0e130 con 0x5591ebe2a750 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebe20f10 con 0x5591ebe21250 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f33621ec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebe37680 con 0x5591ebe21b70 2024-03-12T05:15:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f335dbb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe24ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f335dbb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe24ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48688/0 (socket says 172.21.15.45:48688) 2024-03-12T05:15:09.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f335dbb2640 1 -- 172.21.15.45:0/240983550 learned_addr learned my addr 172.21.15.45:0/240983550 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f335d3b1640 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe29e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.824+0000 7f335cbb0640 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe24990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 -- 172.21.15.45:0/240983550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 msgr2=0x5591ebe24990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe24990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 -- 172.21.15.45:0/240983550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 msgr2=0x5591ebe24ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe24ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 -- 172.21.15.45:0/240983550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591ebd23c20 con 0x5591ebe21250 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f335d3b1640 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe29e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33540bb400 tx=0x7f33540349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d484df896ccf64ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.825+0000 7f3342ffd640 1 -- 172.21.15.45:0/240983550 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540bd5b0 con 0x5591ebe21250 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.826+0000 7f3342ffd640 1 -- 172.21.15.45:0/240983550 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3354003b80 con 0x5591ebe21250 2024-03-12T05:15:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.826+0000 7f3342ffd640 1 -- 172.21.15.45:0/240983550 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540c56c0 con 0x5591ebe21250 2024-03-12T05:15:09.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- 172.21.15.45:0/240983550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 msgr2=0x5591ebe29e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe29e60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33540bb400 tx=0x7f33540349e0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- 172.21.15.45:0/240983550 shutdown_connections 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe2a750 0x5591ebe24ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe21b70 0x5591ebe24990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 --2- 172.21.15.45:0/240983550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21250 0x5591ebe29e60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- 172.21.15.45:0/240983550 >> 172.21.15.45:0/240983550 conn(0x5591ebe1b980 msgr2=0x5591ebe26000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- 172.21.15.45:0/240983550 shutdown_connections 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- 172.21.15.45:0/240983550 wait complete. 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.827+0000 7f33621ec740 1 -- start start 2024-03-12T05:15:09.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.828+0000 7f33621ec740 1 Processor -- start 2024-03-12T05:15:09.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.828+0000 7f33621ec740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.828+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.828+0000 7f33621ec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.828+0000 7f335d3b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48704/0 (socket says 172.21.15.45:48704) 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 -- 172.21.15.45:0/923938208 learned_addr learned my addr 172.21.15.45:0/923938208 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe30460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebe30b30 con 0x5591ebe21250 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebe30ca0 con 0x5591ebe21b70 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591ebe90dd0 con 0x5591ebe2a750 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335cbb0640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335dbb2640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe30460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f33621ec740 10 client.0 _ll_register_callbacks cb 0x7f334800d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 -- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 msgr2=0x5591ebe30460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe30460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 -- 172.21.15.45:0/923938208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 msgr2=0x5591ebe2fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335d3b1640 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591ebd876a0 con 0x5591ebe21250 2024-03-12T05:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.829+0000 7f335cbb0640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.830+0000 7f335d3b1640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33540c11a0 tx=0x7f3354004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316fde658924efd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.830+0000 7f3340ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.830+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540bcc90 con 0x5591ebe21250 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.830+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f33540333a0 con 0x5591ebe21250 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.830+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33480054d0 con 0x5591ebe21250 2024-03-12T05:15:09.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33540bcf20 con 0x5591ebe21250 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f33621ec740 10 client.16647 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5591ebd498c0 con 0x5591ebe21250 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f33540bd850 con 0x5591ebe21250 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f33540ef040 con 0x5591ebe21250 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.831+0000 7f33621ec740 10 client.16647 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f33621ec740 10 client.16647 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f33621ec740 20 client.16647 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe92060 0x5591ebe94450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f33621ec740 4 client.16647 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:09.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f33480054d0 con 0x5591ebe92060 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.832+0000 7f335cbb0640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe92060 0x5591ebe94450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f3340ff9640 10 client.16647 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f335cbb0640 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe92060 0x5591ebe94450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b940dd85be1e65f3 server_cookie=55cf15c74602c0c9 in_seq=0 out_seq=0 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f3340ff9640 1 -- 172.21.15.45:0/923938208 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f33480054d0 con 0x5591ebe92060 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f3340ff9640 10 client.16647 handle_command_reply: tid=0 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f33621ec740 1 client.16647 shutdown 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 msgr2=0x5591ebe31ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.833+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33540c11a0 tx=0x7f3354004270 comp rx=0 tx=0).stop 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 shutdown_connections 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5591ebe92060 0x5591ebe94450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5591ebe2a750 0x5591ebe30460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5591ebe21b70 0x5591ebe2fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f3340ff9640 0 client.16647 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 --2- 172.21.15.45:0/923938208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5591ebe21250 0x5591ebe31ab0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.834+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 >> 172.21.15.45:0/923938208 conn(0x5591ebe1b980 msgr2=0x5591ebe1a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.835+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 shutdown_connections 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.835+0000 7f33621ec740 1 -- 172.21.15.45:0/923938208 wait complete. 2024-03-12T05:15:09.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.835+0000 7f33621ec740 20 client.16647 trim_cache size 0 max 16384 2024-03-12T05:15:09.835 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 msgr2=0x7f33440a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 0x7f33440a4120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f335000bc00 tx=0x7f3350005f30 comp rx=0 tx=0).stop 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 msgr2=0x7f33581173a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f335402c710 tx=0x7f33540347b0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 shutdown_connections 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f33440a1c60 0x7f33440a4120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3358110ac0 0x7f3358114ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3358105330 0x7f3358110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 --2- 172.21.15.45:0/2452181699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3358100260 0x7f33581173a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:09.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.836+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 >> 172.21.15.45:0/2452181699 conn(0x7f3358077180 msgr2=0x7f33580fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:09.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.837+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 shutdown_connections 2024-03-12T05:15:09.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:09.837+0000 7f335f615640 1 -- 172.21.15.45:0/2452181699 wait complete. 2024-03-12T05:15:09.851 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:09.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:15:09.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 185.0 s, 1482.0 tps, lat 67.468 ms stddev 19.591 2024-03-12T05:15:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:09 smithi116 ceph-mon[35486]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.02k op/s 2024-03-12T05:15:09.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:09 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/975107805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.023+0000 7f4559ffb640 1 -- 172.21.15.45:0/715531409 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c02cb70 con 0x7f455c105330 2024-03-12T05:15:10.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.023+0000 7f4561363640 1 -- 172.21.15.45:0/715531409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c105330 msgr2=0x7f455c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.023+0000 7f4561363640 1 --2- 172.21.15.45:0/715531409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c105330 0x7f455c10cf00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f454c009930 tx=0x7f454c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.023+0000 7f4561363640 1 -- 172.21.15.45:0/715531409 shutdown_connections 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.023+0000 7f4561363640 1 --2- 172.21.15.45:0/715531409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c100c10 0x7f455c104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 --2- 172.21.15.45:0/715531409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c105330 0x7f455c10cf00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 --2- 172.21.15.45:0/715531409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c100260 0x7f455c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 -- 172.21.15.45:0/715531409 >> 172.21.15.45:0/715531409 conn(0x7f455c077180 msgr2=0x7f455c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 -- 172.21.15.45:0/715531409 shutdown_connections 2024-03-12T05:15:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 -- 172.21.15.45:0/715531409 wait complete. 2024-03-12T05:15:10.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.024+0000 7f4561363640 1 Processor -- start 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 -- start start 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 0x7f455c106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c107a00 0x7f455c1a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c10b160 con 0x7f455c105330 2024-03-12T05:15:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c1aa420 con 0x7f455c100260 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f4561363640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c1aa5e0 con 0x7f455c107a00 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f455affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 0x7f455c106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f455a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f455a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48710/0 (socket says 172.21.15.45:48710) 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.025+0000 7f455a7fc640 1 -- 172.21.15.45:0/2180215405 learned_addr learned my addr 172.21.15.45:0/2180215405 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455b7fe640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c107a00 0x7f455c1a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455a7fc640 1 -- 172.21.15.45:0/2180215405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c107a00 msgr2=0x7f455c1a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c107a00 0x7f455c1a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455a7fc640 1 -- 172.21.15.45:0/2180215405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 msgr2=0x7f455c106f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 0x7f455c106f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455a7fc640 1 -- 172.21.15.45:0/2180215405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c009560 con 0x7f455c105330 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.026+0000 7f455affd640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 0x7f455c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9c0 tx=0x7f455000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f4543fff640 1 -- 172.21.15.45:0/2180215405 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550009700 con 0x7f455c105330 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f4543fff640 1 -- 172.21.15.45:0/2180215405 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f455000cd80 con 0x7f455c105330 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f4543fff640 1 -- 172.21.15.45:0/2180215405 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550010430 con 0x7f455c105330 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c1aa8c0 con 0x7f455c105330 2024-03-12T05:15:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.027+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c1aadd0 con 0x7f455c105330 2024-03-12T05:15:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f456400b740 1 Processor -- start 2024-03-12T05:15:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f4543fff640 1 -- 172.21.15.45:0/2180215405 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4550004f10 con 0x7f455c105330 2024-03-12T05:15:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f456400b740 1 -- start start 2024-03-12T05:15:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f456400b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f456400b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205bc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.031+0000 7f456400b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205beb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f456400b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56542056e960 con 0x5654205bc750 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f456400b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565420522310 con 0x5654205b3250 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f456400b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56542051f840 con 0x5654205b3b70 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34320/0 (socket says 172.21.15.45:34320) 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 -- 172.21.15.45:0/963973591 learned_addr learned my addr 172.21.15.45:0/963973591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455a7fc640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455affd640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205bc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 -- 172.21.15.45:0/963973591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 msgr2=0x5654205bc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205bc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 -- 172.21.15.45:0/963973591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 msgr2=0x5654205beb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205beb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455b7fe640 1 -- 172.21.15.45:0/963973591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654204dd060 con 0x5654205b3250 2024-03-12T05:15:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455a7fc640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.032+0000 7f455affd640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.033+0000 7f455b7fe640 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f454c005ab0 tx=0x7f454c005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f916bbcaa2c54ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.033+0000 7f45417fa640 1 -- 172.21.15.45:0/963973591 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c004880 con 0x5654205b3250 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.033+0000 7f45417fa640 1 -- 172.21.15.45:0/963973591 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f454c02cc20 con 0x5654205b3250 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.033+0000 7f45417fa640 1 -- 172.21.15.45:0/963973591 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c0038e0 con 0x5654205b3250 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.033+0000 7f4543fff640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 0x7f45440a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f455affd640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 0x7f45440a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 -- 172.21.15.45:0/963973591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 msgr2=0x5654205b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f454c005ab0 tx=0x7f454c005db0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f4543fff640 1 -- 172.21.15.45:0/2180215405 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4550014070 con 0x7f455c105330 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 -- 172.21.15.45:0/963973591 shutdown_connections 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 --2- 172.21.15.45:0/963973591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205b3630 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.034+0000 7f456400b740 1 -- 172.21.15.45:0/963973591 >> 172.21.15.45:0/963973591 conn(0x5654205ad980 msgr2=0x5654205add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.035+0000 7f456400b740 1 -- 172.21.15.45:0/963973591 shutdown_connections 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.035+0000 7f455affd640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 0x7f45440a41b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4554009fd0 tx=0x7f4554009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.035+0000 7f456400b740 1 -- 172.21.15.45:0/963973591 wait complete. 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.035+0000 7f456400b740 1 -- start start 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.035+0000 7f456400b740 1 Processor -- start 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205ba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f455affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f455affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48734/0 (socket says 172.21.15.45:48734) 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56542051b020 con 0x5654205b3250 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654201ee430 con 0x5654205b3b70 2024-03-12T05:15:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654201aae00 con 0x5654205bc750 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f455a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f456400b740 10 client.0 _ll_register_callbacks cb 0x5654204c2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f455a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34330/0 (socket says 172.21.15.45:34330) 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.036+0000 7f455a7fc640 1 -- 172.21.15.45:0/3180376789 learned_addr learned my addr 172.21.15.45:0/3180376789 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455b7fe640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205ba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 -- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 msgr2=0x5654205ba250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205ba250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 -- 172.21.15.45:0/3180376789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 msgr2=0x5654205b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 -- 172.21.15.45:0/3180376789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c10d710 con 0x5654205b3b70 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455affd640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f455a7fc640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f455009a300 tx=0x7f4550096960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=809183502044348c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f453eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.037+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45500cb5a0 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f455009ad00 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56542021a7e0 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f455009a600 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f456400b740 10 client.26129 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5654205196a0 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.038+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45500d2070 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f455001f060 con 0x5654205b3b70 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 10 client.26129 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 10 client.26129 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:15:10.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 20 client.26129 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654205dd870 0x5654205dfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 4 client.26129 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56542021a7e0 con 0x5654205dd870 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.039+0000 7f455affd640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654205dd870 0x5654205dfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.040+0000 7f453eff5640 10 client.26129 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.040+0000 7f455affd640 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654205dd870 0x5654205dfc90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8de85704e9ef56f server_cookie=89d1858e0dea9df5 in_seq=0 out_seq=0 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.041+0000 7f453eff5640 1 -- 172.21.15.45:0/3180376789 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56542021a7e0 con 0x5654205dd870 2024-03-12T05:15:10.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.041+0000 7f453eff5640 10 client.26129 handle_command_reply: tid=0 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 client.26129 shutdown 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 msgr2=0x5654205b9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f455009a300 tx=0x7f4550096960 comp rx=0 tx=0).stop 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 shutdown_connections 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654205dd870 0x5654205dfc90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205ba250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205b9d10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.042+0000 7f456400b740 1 --2- 172.21.15.45:0/3180376789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.043+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 >> 172.21.15.45:0/3180376789 conn(0x5654205ad980 msgr2=0x5654205af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.043+0000 7f453eff5640 0 client.26129 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.043+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 shutdown_connections 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.043+0000 7f456400b740 1 -- 172.21.15.45:0/3180376789 wait complete. 2024-03-12T05:15:10.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.043+0000 7f456400b740 20 client.26129 trim_cache size 0 max 16384 2024-03-12T05:15:10.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 Processor -- start 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 -- start start 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205ba070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f455affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f455affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34340/0 (socket says 172.21.15.45:34340) 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48744/0 (socket says 172.21.15.45:48744) 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f455affd640 1 -- 172.21.15.45:0/1809769436 learned_addr learned my addr 172.21.15.45:0/1809769436 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654204a0130 con 0x5654205bc750 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654205b25d0 con 0x5654205b3250 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.063+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654205ae980 con 0x5654205b3b70 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455a7fc640 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205ba070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 -- 172.21.15.45:0/1809769436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 msgr2=0x5654205ba070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205ba070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 -- 172.21.15.45:0/1809769436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 msgr2=0x5654205b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 -- 172.21.15.45:0/1809769436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654204b5c20 con 0x5654205b3250 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f455affd640 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5654205e05e0 tx=0x7f4554015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aac0c12afe9393c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f4540ff9640 1 -- 172.21.15.45:0/1809769436 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554026040 con 0x5654205b3250 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.064+0000 7f4540ff9640 1 -- 172.21.15.45:0/1809769436 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f455402d070 con 0x5654205b3250 2024-03-12T05:15:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.065+0000 7f4540ff9640 1 -- 172.21.15.45:0/1809769436 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554021400 con 0x5654205b3250 2024-03-12T05:15:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 msgr2=0x5654205bbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x5654205e05e0 tx=0x7f4554015040 comp rx=0 tx=0).stop 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 shutdown_connections 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205bc750 0x5654205b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205b3b70 0x5654205ba070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 --2- 172.21.15.45:0/1809769436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3250 0x5654205bbe90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.066+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 >> 172.21.15.45:0/1809769436 conn(0x5654205ad980 msgr2=0x5654205a8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.067+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 shutdown_connections 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.067+0000 7f456400b740 1 -- 172.21.15.45:0/1809769436 wait complete. 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.067+0000 7f456400b740 1 -- start start 2024-03-12T05:15:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 Processor -- start 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205d3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x565420633dc0 con 0x5654205b3250 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565420633f30 con 0x5654205b3b70 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.068+0000 7f456400b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654206340f0 con 0x5654205bc750 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f456400b740 10 client.0 _ll_register_callbacks cb 0x7f45540093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48758/0 (socket says 172.21.15.45:48758) 2024-03-12T05:15:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455affd640 1 -- 172.21.15.45:0/2236236854 learned_addr learned my addr 172.21.15.45:0/2236236854 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455b7fe640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205d3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 -- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 msgr2=0x5654205d3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205d3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 -- 172.21.15.45:0/2236236854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 msgr2=0x5654205d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 -- 172.21.15.45:0/2236236854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4554019070 con 0x5654205b3b70 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455affd640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f455a7fc640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205d36b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4550096fd0 tx=0x7f45500055b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29ca3b4e10cebcb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.069+0000 7f453eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550099c50 con 0x5654205b3b70 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f455009ad00 con 0x5654205b3b70 2024-03-12T05:15:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f454c02cc20 con 0x5654205b3b70 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4550099db0 con 0x5654205b3b70 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f456400b740 10 client.26135 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5654204db8c0 con 0x5654205b3b70 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f454c02cc20 con 0x5654205b3b70 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.070+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f455001f040 con 0x5654205b3b70 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 10 client.26135 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 10 client.26135 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 20 client.26135 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654206353e0 0x565420637800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 4 client.26135 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565420257720 con 0x5654206353e0 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.071+0000 7f455affd640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654206353e0 0x565420637800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.072+0000 7f453eff5640 10 client.26135 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.072+0000 7f455affd640 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654206353e0 0x565420637800 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b9641dbdd240223 server_cookie=b7d622afb7aaede in_seq=0 out_seq=0 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f453eff5640 1 -- 172.21.15.45:0/2236236854 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565420257720 con 0x5654206353e0 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f453eff5640 10 client.26135 handle_command_reply: tid=0 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 client.26135 shutdown 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 msgr2=0x5654205d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205d36b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4550096fd0 tx=0x7f45500055b0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 shutdown_connections 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5654206353e0 0x565420637800 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654205bc750 0x5654205d3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654205b3b70 0x5654205d36b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 --2- 172.21.15.45:0/2236236854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5654205b3250 0x5654205d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 >> 172.21.15.45:0/2236236854 conn(0x5654205ad980 msgr2=0x5654205b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f453eff5640 0 client.26135 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 shutdown_connections 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 1 -- 172.21.15.45:0/2236236854 wait complete. 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.073+0000 7f456400b740 20 client.26135 trim_cache size 0 max 16384 2024-03-12T05:15:10.072 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:10.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.074+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 msgr2=0x7f45440a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.074+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 0x7f45440a41b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4554009fd0 tx=0x7f4554009500 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 msgr2=0x7f455c1074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9c0 tx=0x7f455000ee80 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 shutdown_connections 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45440a1cf0 0x7f45440a41b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f455c105330 0x7f455c1074c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f455c107a00 0x7f455c1a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 --2- 172.21.15.45:0/2180215405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f455c100260 0x7f455c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.075+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 >> 172.21.15.45:0/2180215405 conn(0x7f455c077180 msgr2=0x7f455c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.076+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 shutdown_connections 2024-03-12T05:15:10.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.076+0000 7f4561363640 1 -- 172.21.15.45:0/2180215405 wait complete. 2024-03-12T05:15:10.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:09 smithi045 ceph-mon[32174]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.02k op/s 2024-03-12T05:15:10.089 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:09 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/975107805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:10.090 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:10.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:15:10.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:09 smithi130 ceph-mon[35398]: pgmap v131: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 18 MiB/s wr, 2.02k op/s 2024-03-12T05:15:10.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:09 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/975107805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 -- 172.21.15.45:0/1627214213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 msgr2=0x7fefa010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 --2- 172.21.15.45:0/1627214213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa010e020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef98009be0 tx=0x7fef9802bef0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 -- 172.21.15.45:0/1627214213 shutdown_connections 2024-03-12T05:15:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 --2- 172.21.15.45:0/1627214213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa010e020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 --2- 172.21.15.45:0/1627214213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101d30 0x7fefa0105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 --2- 172.21.15.45:0/1627214213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa0101380 0x7fefa0101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 -- 172.21.15.45:0/1627214213 >> 172.21.15.45:0/1627214213 conn(0x7fefa0078540 msgr2=0x7fefa0078950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 -- 172.21.15.45:0/1627214213 shutdown_connections 2024-03-12T05:15:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.271+0000 7fefa7cae640 1 -- 172.21.15.45:0/1627214213 wait complete. 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 Processor -- start 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 -- start start 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101380 0x7fefa019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa010d070 0x7fefa010d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa010dcb0 con 0x7fefa010d070 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa010de20 con 0x7fefa0101380 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa7cae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa019def0 con 0x7fefa0106450 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa5249640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.272+0000 7fefa5249640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33706/0 (socket says 172.21.15.45:33706) 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 -- 172.21.15.45:0/2968990383 learned_addr learned my addr 172.21.15.45:0/2968990383 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101380 0x7fefa019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 -- 172.21.15.45:0/2968990383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101380 msgr2=0x7fefa019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101380 0x7fefa019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 -- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa010d070 msgr2=0x7fefa010d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa010d070 0x7fefa010d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 -- 172.21.15.45:0/2968990383 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c009560 con 0x7fefa0106450 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.273+0000 7fefa5249640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.274+0000 7fef96ffd640 1 -- 172.21.15.45:0/2968990383 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000ccc0 con 0x7fefa0106450 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.274+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef98009840 con 0x7fefa0106450 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.274+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa019e550 con 0x7fefa0106450 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.274+0000 7fef96ffd640 1 -- 172.21.15.45:0/2968990383 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef9000ce20 con 0x7fefa0106450 2024-03-12T05:15:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.274+0000 7fef96ffd640 1 -- 172.21.15.45:0/2968990383 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90010430 con 0x7fefa0106450 2024-03-12T05:15:10.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.277+0000 7fefaa882740 1 Processor -- start 2024-03-12T05:15:10.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.277+0000 7fef96ffd640 1 -- 172.21.15.45:0/2968990383 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fef90010590 con 0x7fefa0106450 2024-03-12T05:15:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 -- start start 2024-03-12T05:15:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e52f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e53ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e4ea960 con 0x560a5e52f250 2024-03-12T05:15:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e49e310 con 0x560a5e538750 2024-03-12T05:15:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefaa882740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e49b840 con 0x560a5e52fb70 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.278+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33720/0 (socket says 172.21.15.45:33720) 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 -- 172.21.15.45:0/251959560 learned_addr learned my addr 172.21.15.45:0/251959560 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5249640 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e53ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 -- 172.21.15.45:0/251959560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 msgr2=0x560a5e53ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e53ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 -- 172.21.15.45:0/251959560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 msgr2=0x560a5e52f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e52f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 -- 172.21.15.45:0/251959560 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a5e459060 con 0x560a5e52fb70 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009e60 tx=0x7fef9c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba53b7fab35b6bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.279+0000 7fef96ffd640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 0x7fef8c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.280+0000 7fef8bfff640 1 -- 172.21.15.45:0/251959560 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c005020 con 0x560a5e52fb70 2024-03-12T05:15:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.280+0000 7fef96ffd640 1 -- 172.21.15.45:0/2968990383 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fef90014070 con 0x7fefa0106450 2024-03-12T05:15:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.280+0000 7fef8bfff640 1 -- 172.21.15.45:0/251959560 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef9c005630 con 0x560a5e52fb70 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.280+0000 7fefa624b640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 0x7fef8c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.280+0000 7fef8bfff640 1 -- 172.21.15.45:0/251959560 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c019700 con 0x560a5e52fb70 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 -- 172.21.15.45:0/251959560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 msgr2=0x560a5e538210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009e60 tx=0x7fef9c011810 comp rx=0 tx=0).stop 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 -- 172.21.15.45:0/251959560 shutdown_connections 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e53ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e538210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 --2- 172.21.15.45:0/251959560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e52f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.281+0000 7fefaa882740 1 -- 172.21.15.45:0/251959560 >> 172.21.15.45:0/251959560 conn(0x560a5e529980 msgr2=0x560a5e529d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.282+0000 7fefa624b640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 0x7fef8c0a4160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef98036040 tx=0x7fef98009330 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.282+0000 7fefaa882740 1 -- 172.21.15.45:0/251959560 shutdown_connections 2024-03-12T05:15:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.282+0000 7fefaa882740 1 -- 172.21.15.45:0/251959560 wait complete. 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.282+0000 7fefaa882740 1 -- start start 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.282+0000 7fefaa882740 1 Processor -- start 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e535b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e536070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e497020 con 0x560a5e52f250 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e16a430 con 0x560a5e538750 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e126e00 con 0x560a5e52fb70 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.283+0000 7fefaa882740 10 client.0 _ll_register_callbacks cb 0x560a5e43ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48810/0 (socket says 172.21.15.45:48810) 2024-03-12T05:15:10.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5a4a640 1 -- 172.21.15.45:0/1484017758 learned_addr learned my addr 172.21.15.45:0/1484017758 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e536070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5249640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e535b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 -- 172.21.15.45:0/1484017758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 msgr2=0x560a5e535b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e535b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 -- 172.21.15.45:0/1484017758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 msgr2=0x560a5e5355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa624b640 1 -- 172.21.15.45:0/1484017758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c009a90 con 0x560a5e538750 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.284+0000 7fefa5249640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e535b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fefa624b640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e536070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fef98034fd0 tx=0x7fef98003d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7eb5b553a03c19 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fef897fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9802c9b0 con 0x560a5e538750 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fef98033c70 con 0x560a5e538750 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a5e1967e0 con 0x560a5e538750 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef98033e30 con 0x560a5e538750 2024-03-12T05:15:10.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.285+0000 7fefaa882740 10 client.26147 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560a5e4956a0 con 0x560a5e538750 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x560a5e1967e0 con 0x560a5e538750 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fef9803c070 con 0x560a5e538750 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 10 client.26147 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 20 client.26147 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e559b40 0x560a5e55bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 4 client.26147 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a5e1d3720 con 0x560a5e559b40 2024-03-12T05:15:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.286+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e559b40 0x560a5e55bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.287+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e559b40 0x560a5e55bf60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10212df4a41bb85f server_cookie=7e6e2704e9d0a060 in_seq=0 out_seq=0 2024-03-12T05:15:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.287+0000 7fef897fa640 10 client.26147 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fef897fa640 1 -- 172.21.15.45:0/1484017758 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560a5e1d3720 con 0x560a5e559b40 2024-03-12T05:15:10.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fef897fa640 10 client.26147 handle_command_reply: tid=0 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 client.26147 shutdown 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 msgr2=0x560a5e536070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e536070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fef98034fd0 tx=0x7fef98003d40 comp rx=0 tx=0).stop 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 shutdown_connections 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e559b40 0x560a5e55bf60 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e536070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52fb70 0x560a5e535b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 --2- 172.21.15.45:0/1484017758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e5355f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 >> 172.21.15.45:0/1484017758 conn(0x560a5e529980 msgr2=0x560a5e52b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fef897fa640 0 client.26147 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.288+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 shutdown_connections 2024-03-12T05:15:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.289+0000 7fefaa882740 1 -- 172.21.15.45:0/1484017758 wait complete. 2024-03-12T05:15:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.289+0000 7fefaa882740 20 client.26147 trim_cache size 0 max 16384 2024-03-12T05:15:10.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.310+0000 7fefaa882740 1 Processor -- start 2024-03-12T05:15:10.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.310+0000 7fefaa882740 1 -- start start 2024-03-12T05:15:10.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52fb70 0x560a5e5345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefa5a4a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33750/0 (socket says 172.21.15.45:33750) 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefa5a4a640 1 -- 172.21.15.45:0/1385919523 learned_addr learned my addr 172.21.15.45:0/1385919523 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e535e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e41c130 con 0x560a5e52fb70 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e52ef10 con 0x560a5e538750 2024-03-12T05:15:10.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.311+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e529830 con 0x560a5e52f250 2024-03-12T05:15:10.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa624b640 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e535e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 -- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 msgr2=0x560a5e535e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e535e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 -- 172.21.15.45:0/1385919523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52fb70 msgr2=0x560a5e5345d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52fb70 0x560a5e5345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 -- 172.21.15.45:0/1385919523 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a5e431c20 con 0x560a5e52f250 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa624b640 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e535e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef9c011d10 tx=0x7fef9c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e54a891d6bd3a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fef8b7fe640 1 -- 172.21.15.45:0/1385919523 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c005b30 con 0x560a5e52f250 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fef8b7fe640 1 -- 172.21.15.45:0/1385919523 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef9c02f070 con 0x560a5e52f250 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.312+0000 7fef8b7fe640 1 -- 172.21.15.45:0/1385919523 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0051c0 con 0x560a5e52f250 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 msgr2=0x560a5e537d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef9c011d10 tx=0x7fef9c009260 comp rx=0 tx=0).stop 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 shutdown_connections 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52fb70 0x560a5e5345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e538750 0x560a5e535e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 --2- 172.21.15.45:0/1385919523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e52f250 0x560a5e537d20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 >> 172.21.15.45:0/1385919523 conn(0x560a5e529980 msgr2=0x560a5e533ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.313+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 shutdown_connections 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.314+0000 7fefaa882740 1 -- 172.21.15.45:0/1385919523 wait complete. 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 -- start start 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 Processor -- start 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e550af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 0x560a5e54f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e54fb10 con 0x560a5e52f250 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e54fc80 con 0x560a5e52fb70 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.315+0000 7fefaa882740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5e5b0050 con 0x560a5e538750 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefaa882740 10 client.0 _ll_register_callbacks cb 0x7fef9c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa624b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 0x560a5e54f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34414/0 (socket says 172.21.15.45:34414) 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 -- 172.21.15.45:0/384764281 learned_addr learned my addr 172.21.15.45:0/384764281 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e550af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 -- 172.21.15.45:0/384764281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 msgr2=0x560a5e54f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 0x560a5e54f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 -- 172.21.15.45:0/384764281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 msgr2=0x560a5e550af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e550af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 -- 172.21.15.45:0/384764281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a5e459060 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa624b640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 0x560a5e54f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fefa5249640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fef90096b60 tx=0x7fef9009a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d370f585ec4df41c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.316+0000 7fef897fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef900a2dc0 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fef900046c0 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fef9c005630 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef90004880 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fefaa882740 10 client.26153 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560a5e4578c0 con 0x560a5e52fb70 2024-03-12T05:15:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.317+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fef9c005630 con 0x560a5e52fb70 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fef9001f060 con 0x560a5e52fb70 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 10 client.26153 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 10 client.26153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 20 client.26153 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e5b1920 0x560a5e5b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 4 client.26153 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560a5e222890 con 0x560a5e5b1920 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e5b1920 0x560a5e5b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fefa5a4a640 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e5b1920 0x560a5e5b3d40 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d671c5563759db server_cookie=837ca8d3cc6f5a0d in_seq=0 out_seq=0 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.318+0000 7fef897fa640 10 client.26153 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fef897fa640 1 -- 172.21.15.45:0/384764281 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560a5e222890 con 0x560a5e5b1920 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fef897fa640 10 client.26153 handle_command_reply: tid=0 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 client.26153 shutdown 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 msgr2=0x560a5e54ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fef90096b60 tx=0x7fef9009a300 comp rx=0 tx=0).stop 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 shutdown_connections 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560a5e5b1920 0x560a5e5b3d40 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a5e538750 0x560a5e54f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560a5e52fb70 0x560a5e54ee90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fef897fa640 0 client.26153 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 --2- 172.21.15.45:0/384764281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560a5e52f250 0x560a5e550af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 >> 172.21.15.45:0/384764281 conn(0x560a5e529980 msgr2=0x560a5e52b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.319+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 shutdown_connections 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.320+0000 7fefaa882740 1 -- 172.21.15.45:0/384764281 wait complete. 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.320+0000 7fefaa882740 20 client.26153 trim_cache size 0 max 16384 2024-03-12T05:15:10.319 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 msgr2=0x7fef8c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 0x7fef8c0a4160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef98036040 tx=0x7fef98009330 comp rx=0 tx=0).stop 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 msgr2=0x7fefa019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 shutdown_connections 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fef8c0a1ca0 0x7fef8c0a4160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa010d070 0x7fefa010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefa0106450 0x7fefa019a730 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 --2- 172.21.15.45:0/2968990383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa0101380 0x7fefa019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 >> 172.21.15.45:0/2968990383 conn(0x7fefa0078540 msgr2=0x7fefa00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.321+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 shutdown_connections 2024-03-12T05:15:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:10.322+0000 7fefa7cae640 1 -- 172.21.15.45:0/2968990383 wait complete. 2024-03-12T05:15:11.336 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:15:11.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:15:11.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:15:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.509+0000 7fb67c2c0640 1 -- 172.21.15.45:0/229993908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674108660 msgr2=0x7fb674110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.509+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/229993908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674108660 0x7fb674110230 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009c10 tx=0x7fb66c02bef0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.509+0000 7fb67c2c0640 1 -- 172.21.15.45:0/229993908 shutdown_connections 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.509+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/229993908 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674103f40 0x7fb674107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/229993908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674108660 0x7fb674110230 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/229993908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674103590 0x7fb674103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 -- 172.21.15.45:0/229993908 >> 172.21.15.45:0/229993908 conn(0x7fb6740fedd0 msgr2=0x7fb6741011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 -- 172.21.15.45:0/229993908 shutdown_connections 2024-03-12T05:15:11.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 -- 172.21.15.45:0/229993908 wait complete. 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.510+0000 7fb67c2c0640 1 Processor -- start 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67c2c0640 1 -- start start 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67c2c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67c2c0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674108660 0x7fb674075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34426/0 (socket says 172.21.15.45:34426) 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.511+0000 7fb67a05c640 1 -- 172.21.15.45:0/336697375 learned_addr learned my addr 172.21.15.45:0/336697375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674076110 0x7fb6741a9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb674079870 con 0x7fb674108660 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67985b640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674108660 0x7fb674075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741aa400 con 0x7fb674103590 2024-03-12T05:15:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67a85d640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674076110 0x7fb6741a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.512+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741aa5c0 con 0x7fb674076110 2024-03-12T05:15:11.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 -- 172.21.15.45:0/336697375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674076110 msgr2=0x7fb6741a9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674076110 0x7fb6741a9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 -- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674108660 msgr2=0x7fb674075bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674108660 0x7fb674075bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c009840 con 0x7fb674103590 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67a05c640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb66b7fe640 1 -- 172.21.15.45:0/336697375 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67000cc60 con 0x7fb674103590 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6741aa8a0 con 0x7fb674103590 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb66b7fe640 1 -- 172.21.15.45:0/336697375 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb67000cdc0 con 0x7fb674103590 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb66b7fe640 1 -- 172.21.15.45:0/336697375 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67001a660 con 0x7fb674103590 2024-03-12T05:15:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.513+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6741aae00 con 0x7fb674103590 2024-03-12T05:15:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.516+0000 7fb66b7fe640 1 -- 172.21.15.45:0/336697375 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb670011470 con 0x7fb674103590 2024-03-12T05:15:11.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.518+0000 7fb66b7fe640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 0x7fb6600a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.518+0000 7fb66b7fe640 1 -- 172.21.15.45:0/336697375 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb6700cdbe0 con 0x7fb674103590 2024-03-12T05:15:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.518+0000 7fb67ef6f740 1 Processor -- start 2024-03-12T05:15:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.518+0000 7fb67985b640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 0x7fb6600a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.518+0000 7fb67ef6f740 1 -- start start 2024-03-12T05:15:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.519+0000 7fb67985b640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 0x7fb6600a4160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb674076e50 tx=0x7fb664009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.519+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.520+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.520+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.520+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33792/0 (socket says 172.21.15.45:33792) 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.520+0000 7fb67a05c640 1 -- 172.21.15.45:0/725789226 learned_addr learned my addr 172.21.15.45:0/725789226 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.520+0000 7fb67985b640 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.521+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.521+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b26df90 con 0x55cf8b2ffd30 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.521+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b26b4c0 con 0x55cf8b3088b0 2024-03-12T05:15:11.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.521+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b266ca0 con 0x55cf8b2ff380 2024-03-12T05:15:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 -- 172.21.15.45:0/725789226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 msgr2=0x55cf8b30ad00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:15:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 -- 172.21.15.45:0/725789226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 msgr2=0x55cf8b3082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 -- 172.21.15.45:0/725789226 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf8b2290b0 con 0x55cf8b2ff380 2024-03-12T05:15:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.522+0000 7fb67a05c640 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb670096c10 tx=0x7fb67009a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5918fb8289eddd36 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.523+0000 7fb668bfe640 1 -- 172.21.15.45:0/725789226 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67009a7b0 con 0x55cf8b2ff380 2024-03-12T05:15:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.523+0000 7fb668bfe640 1 -- 172.21.15.45:0/725789226 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb67009e900 con 0x55cf8b2ff380 2024-03-12T05:15:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.523+0000 7fb668bfe640 1 -- 172.21.15.45:0/725789226 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6700a2ac0 con 0x55cf8b2ff380 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 msgr2=0x55cf8b2ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb670096c10 tx=0x7fb67009a6b0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 shutdown_connections 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/725789226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ff380 0x55cf8b2ff760 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.525+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 >> 172.21.15.45:0/725789226 conn(0x55cf8b2f9840 msgr2=0x55cf8b2f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.526+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 shutdown_connections 2024-03-12T05:15:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.526+0000 7fb67ef6f740 1 -- 172.21.15.45:0/725789226 wait complete. 2024-03-12T05:15:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.526+0000 7fb67ef6f740 1 -- start start 2024-03-12T05:15:11.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 Processor -- start 2024-03-12T05:15:11.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b3058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b306360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67985b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67985b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34444/0 (socket says 172.21.15.45:34444) 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b3058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b3058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48884/0 (socket says 172.21.15.45:48884) 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67985b640 1 -- 172.21.15.45:0/2100666204 learned_addr learned my addr 172.21.15.45:0/2100666204 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b20bf40 con 0x55cf8b2ff380 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8af3a4b0 con 0x55cf8b2ffd30 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b1ec670 con 0x55cf8b3088b0 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.527+0000 7fb67ef6f740 10 client.0 _ll_register_callbacks cb 0x55cf8b259f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:11.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67a85d640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b306360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 -- 172.21.15.45:0/2100666204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 msgr2=0x55cf8b306360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b306360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 -- 172.21.15.45:0/2100666204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 msgr2=0x55cf8b3058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b3058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6741aae00 con 0x55cf8b2ffd30 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67985b640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb67406a7d0 tx=0x7fb664004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43fa09329926992 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb65dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66400ed50 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.528+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf8b240f40 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.529+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb664005020 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.529+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb664005220 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.529+0000 7fb67ef6f740 10 client.26162 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.529+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cf8b265320 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.529+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cf8b240f40 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb664030430 con 0x55cf8b2ffd30 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 10 client.26162 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 10 client.26162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 20 client.26162 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b318ee0 0x55cf8b31b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 4 client.26162 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf8b25d310 con 0x55cf8b318ee0 2024-03-12T05:15:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.530+0000 7fb67a05c640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b318ee0 0x55cf8b31b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.531+0000 7fb65dffb640 10 client.26162 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.531+0000 7fb67a05c640 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b318ee0 0x55cf8b31b300 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=985090f9c17c3e0c server_cookie=eb26bea1ccc38286 in_seq=0 out_seq=0 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.531+0000 7fb65dffb640 1 -- 172.21.15.45:0/2100666204 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cf8b25d310 con 0x55cf8b318ee0 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.531+0000 7fb65dffb640 10 client.26162 handle_command_reply: tid=0 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.531+0000 7fb67ef6f740 1 client.26162 shutdown 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 msgr2=0x55cf8b305e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb67406a7d0 tx=0x7fb664004730 comp rx=0 tx=0).stop 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 shutdown_connections 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b318ee0 0x55cf8b31b300 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b306360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ffd30 0x55cf8b305e20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/2100666204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b3058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 >> 172.21.15.45:0/2100666204 conn(0x55cf8b2f9840 msgr2=0x55cf8b2fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb65dffb640 0 client.26162 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 shutdown_connections 2024-03-12T05:15:11.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 1 -- 172.21.15.45:0/2100666204 wait complete. 2024-03-12T05:15:11.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.532+0000 7fb67ef6f740 20 client.26162 trim_cache size 0 max 16384 2024-03-12T05:15:11.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.559+0000 7fb67ef6f740 1 Processor -- start 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.559+0000 7fb67ef6f740 1 -- start start 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.559+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b3046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b315a70 con 0x55cf8b2ffd30 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b306c30 con 0x55cf8b2ff380 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b304c30 con 0x55cf8b3088b0 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34460/0 (socket says 172.21.15.45:34460) 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 -- 172.21.15.45:0/1938138206 learned_addr learned my addr 172.21.15.45:0/1938138206 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67985b640 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a85d640 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b3046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 -- 172.21.15.45:0/1938138206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 msgr2=0x55cf8b3046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b3046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 -- 172.21.15.45:0/1938138206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 msgr2=0x55cf8b3041b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3041b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.560+0000 7fb67a05c640 1 -- 172.21.15.45:0/1938138206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf8b201c00 con 0x55cf8b2ff380 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.561+0000 7fb67a05c640 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb670000c00 tx=0x7fb670011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb3ecde9439fa20 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.561+0000 7fb65ffff640 1 -- 172.21.15.45:0/1938138206 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670096e20 con 0x55cf8b2ff380 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.561+0000 7fb65ffff640 1 -- 172.21.15.45:0/1938138206 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb67009a7b0 con 0x55cf8b2ff380 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.561+0000 7fb65ffff640 1 -- 172.21.15.45:0/1938138206 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6700cb550 con 0x55cf8b2ff380 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 -- 172.21.15.45:0/1938138206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 msgr2=0x55cf8b308010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb670000c00 tx=0x7fb670011cb0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 -- 172.21.15.45:0/1938138206 shutdown_connections 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ffd30 0x55cf8b3041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b3088b0 0x55cf8b3046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/1938138206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b2ff380 0x55cf8b308010 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 -- 172.21.15.45:0/1938138206 >> 172.21.15.45:0/1938138206 conn(0x55cf8b2f9840 msgr2=0x55cf8b300b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 -- 172.21.15.45:0/1938138206 shutdown_connections 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.562+0000 7fb67ef6f740 1 -- 172.21.15.45:0/1938138206 wait complete. 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 -- start start 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 Processor -- start 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b30e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b36f430 con 0x55cf8b2ff380 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b36f570 con 0x55cf8b3088b0 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf8b36f730 con 0x55cf8b2ffd30 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.563+0000 7fb67ef6f740 10 client.0 _ll_register_callbacks cb 0x7fb670096390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:11.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33836/0 (socket says 172.21.15.45:33836) 2024-03-12T05:15:11.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 -- 172.21.15.45:0/52955166 learned_addr learned my addr 172.21.15.45:0/52955166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67a85d640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67a05c640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b30e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 -- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 msgr2=0x55cf8b30ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 -- 172.21.15.45:0/52955166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 msgr2=0x55cf8b30e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b30e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 -- 172.21.15.45:0/52955166 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf8b2290b0 con 0x55cf8b2ffd30 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67a05c640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b30e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb67985b640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55cf8b305140 tx=0x7fb66401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c63b2eed16223355 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.564+0000 7fb65dffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66401e7e0 con 0x55cf8b2ffd30 2024-03-12T05:15:11.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb66400ed50 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf8b28c900 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb664004040 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb67ef6f740 10 client.26170 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cf8b227910 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb664046020 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.565+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb664038070 con 0x55cf8b2ffd30 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 10 client.26170 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 10 client.26170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 20 client.26170 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b371040 0x55cf8b373460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 4 client.26170 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cf8af66860 con 0x55cf8b371040 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.566+0000 7fb67a05c640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b371040 0x55cf8b373460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.567+0000 7fb67a05c640 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b371040 0x55cf8b373460 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9f556d44c887cc server_cookie=76ff4d798a7353ed in_seq=0 out_seq=0 2024-03-12T05:15:11.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.567+0000 7fb65dffb640 10 client.26170 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.568+0000 7fb65dffb640 1 -- 172.21.15.45:0/52955166 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cf8af66860 con 0x55cf8b371040 2024-03-12T05:15:11.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.568+0000 7fb65dffb640 10 client.26170 handle_command_reply: tid=0 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb67ef6f740 1 client.26170 shutdown 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 msgr2=0x55cf8b30ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55cf8b305140 tx=0x7fb66401eb90 comp rx=0 tx=0).stop 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 shutdown_connections 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cf8b371040 0x55cf8b373460 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.570+0000 7fb65dffb640 0 client.26170 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cf8b3088b0 0x55cf8b30ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf8b2ffd30 0x55cf8b30ea00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 --2- 172.21.15.45:0/52955166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cf8b2ff380 0x55cf8b30e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 >> 172.21.15.45:0/52955166 conn(0x55cf8b2f9840 msgr2=0x55cf8b303d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 shutdown_connections 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 1 -- 172.21.15.45:0/52955166 wait complete. 2024-03-12T05:15:11.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.571+0000 7fb67ef6f740 20 client.26170 trim_cache size 0 max 16384 2024-03-12T05:15:11.569 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:11.569 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:15:11.569 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "1ca182a4-cd7c-44dd-b4dc-385b861e1191", 2024-03-12T05:15:11.569 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:15:11.569 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 msgr2=0x7fb6600a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 0x7fb6600a4160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb674076e50 tx=0x7fb664009260 comp rx=0 tx=0).stop 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 msgr2=0x7fb674075690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000edc0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 shutdown_connections 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb6600a1ca0 0x7fb6600a4160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb674108660 0x7fb674075bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb674076110 0x7fb6741a9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 --2- 172.21.15.45:0/336697375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb674103590 0x7fb674075690 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 >> 172.21.15.45:0/336697375 conn(0x7fb6740fedd0 msgr2=0x7fb674100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 shutdown_connections 2024-03-12T05:15:11.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.573+0000 7fb67c2c0640 1 -- 172.21.15.45:0/336697375 wait complete. 2024-03-12T05:15:11.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:15:11.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:15:11.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.772+0000 7f9691c22640 1 -- 172.21.15.45:0/280010113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c108da0 msgr2=0x7f968c10c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.772+0000 7f9691c22640 1 --2- 172.21.15.45:0/280010113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c108da0 0x7f968c10c3b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f96740074e0 tx=0x7f967402c8d0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 -- 172.21.15.45:0/280010113 shutdown_connections 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 --2- 172.21.15.45:0/280010113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c108da0 0x7f968c10c3b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 --2- 172.21.15.45:0/280010113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c10cec0 0x7f968c079490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 --2- 172.21.15.45:0/280010113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c1083f0 0x7f968c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 -- 172.21.15.45:0/280010113 >> 172.21.15.45:0/280010113 conn(0x7f968c06c270 msgr2=0x7f968c06c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 -- 172.21.15.45:0/280010113 shutdown_connections 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.773+0000 7f9691c22640 1 -- 172.21.15.45:0/280010113 wait complete. 2024-03-12T05:15:11.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.774+0000 7f9691c22640 1 Processor -- start 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.774+0000 7f9691c22640 1 -- start start 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c10cec0 0x7f968c07dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c07dff0 0x7f968c07e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33852/0 (socket says 172.21.15.45:33852) 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f968b7fe640 1 -- 172.21.15.45:0/1778897020 learned_addr learned my addr 172.21.15.45:0/1778897020 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9683fff640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c10cec0 0x7f968c07dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f968bfff640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c07dff0 0x7f968c07e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c084240 con 0x7f968c10cec0 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c1bb310 con 0x7f968c07dff0 2024-03-12T05:15:11.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.775+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c1bb4a0 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 -- 172.21.15.45:0/1778897020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c07dff0 msgr2=0x7f968c07e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c07dff0 0x7f968c07e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 -- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c10cec0 msgr2=0x7f968c07dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c10cec0 0x7f968c07dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96740071c0 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f968b7fe640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f967c00b500 tx=0x7f967c00b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f96897fa640 1 -- 172.21.15.45:0/1778897020 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c010870 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f96897fa640 1 -- 172.21.15.45:0/1778897020 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f967c010e80 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f96897fa640 1 -- 172.21.15.45:0/1778897020 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c017360 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.776+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c1bb780 con 0x7f968c1083f0 2024-03-12T05:15:11.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.777+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c1bbbb0 con 0x7f968c1083f0 2024-03-12T05:15:11.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.780+0000 7f96897fa640 1 -- 172.21.15.45:0/1778897020 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f967c00f3c0 con 0x7f968c1083f0 2024-03-12T05:15:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.781+0000 7f96897fa640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 0x7f96780a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.781+0000 7f96947f0740 1 Processor -- start 2024-03-12T05:15:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 -- start start 2024-03-12T05:15:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f9683fff640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 0x7f96780a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96897fa640 1 -- 172.21.15.45:0/1778897020 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f967c0cd2c0 con 0x7f968c1083f0 2024-03-12T05:15:11.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34786d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786da70 0x55f34787ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3477dc280 con 0x55f34787efc0 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3477d97b0 con 0x55f34786d0c0 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f96947f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3477d4f90 con 0x55f34786da70 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f968bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f968bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48926/0 (socket says 172.21.15.45:48926) 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f968bfff640 1 -- 172.21.15.45:0/1060368992 learned_addr learned my addr 172.21.15.45:0/1060368992 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f968b7fe640 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34786d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.782+0000 7f9683fff640 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786da70 0x55f34787ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 -- 172.21.15.45:0/1060368992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786da70 msgr2=0x55f34787ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786da70 0x55f34787ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 -- 172.21.15.45:0/1060368992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 msgr2=0x55f34786d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34786d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 -- 172.21.15.45:0/1060368992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f347793570 con 0x55f34787efc0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f968bfff640 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f968400a4d0 tx=0x7f9684009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18ccae8de8bac42 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f9683fff640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 0x7f96780a8640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f967400d090 tx=0x7f9674004890 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f9681ffb640 1 -- 172.21.15.45:0/1060368992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684010040 con 0x55f34787efc0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f9681ffb640 1 -- 172.21.15.45:0/1060368992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9684013c40 con 0x55f34787efc0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.783+0000 7f9681ffb640 1 -- 172.21.15.45:0/1060368992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684014cb0 con 0x55f34787efc0 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.784+0000 7f96947f0740 1 -- 172.21.15.45:0/1060368992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 msgr2=0x55f3478813b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.784+0000 7f96947f0740 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f968400a4d0 tx=0x7f9684009650 comp rx=0 tx=0).stop 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 -- 172.21.15.45:0/1060368992 shutdown_connections 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f3478813b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786da70 0x55f34787ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 --2- 172.21.15.45:0/1060368992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34786d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 -- 172.21.15.45:0/1060368992 >> 172.21.15.45:0/1060368992 conn(0x55f347867990 msgr2=0x55f347867d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 -- 172.21.15.45:0/1060368992 shutdown_connections 2024-03-12T05:15:11.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 -- 172.21.15.45:0/1060368992 wait complete. 2024-03-12T05:15:11.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.785+0000 7f96947f0740 1 -- start start 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 Processor -- start 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 0x55f34787e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3474a8430 con 0x55f34786d0c0 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f347464e00 con 0x55f34786da70 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f347759fd0 con 0x55f34787efc0 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 0x55f34787e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f968bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f9683fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 0x55f34787e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48940/0 (socket says 172.21.15.45:48940) 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f9683fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34504/0 (socket says 172.21.15.45:34504) 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f968bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33878/0 (socket says 172.21.15.45:33878) 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f968b7fe640 1 -- 172.21.15.45:0/622189129 learned_addr learned my addr 172.21.15.45:0/622189129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.786+0000 7f96947f0740 10 client.0 _ll_register_callbacks cb 0x55f3477affd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 -- 172.21.15.45:0/622189129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 msgr2=0x55f34787c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 -- 172.21.15.45:0/622189129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 msgr2=0x55f34787e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 0x55f34787e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 -- 172.21.15.45:0/622189129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9684009d00 con 0x55f34786da70 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f968bfff640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f9683fff640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f967402ce10 tx=0x7f967402c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66d7bec22124e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f966affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674036040 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9674045070 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.787+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f3474d47e0 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96740089a0 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 10 client.26177 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f3477d3610 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f967403f900 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9674050020 con 0x55f34786da70 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 10 client.26177 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 10 client.26177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 20 client.26177 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f34788f180 0x55f3478915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 4 client.26177 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3474d47e0 con 0x55f34788f180 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.788+0000 7f968b7fe640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f34788f180 0x55f3478915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.789+0000 7f966affd640 10 client.26177 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.789+0000 7f968b7fe640 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f34788f180 0x55f3478915e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5f687a240db467 server_cookie=722ad6746c215887 in_seq=0 out_seq=0 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.789+0000 7f966affd640 1 -- 172.21.15.45:0/622189129 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f967c07ea90 con 0x55f34788f180 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f966affd640 10 client.26177 handle_command_reply: tid=0 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 client.26177 shutdown 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 msgr2=0x55f34787bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f967402ce10 tx=0x7f967402c8d0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 shutdown_connections 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f34788f180 0x55f3478915e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f34787c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f34787bd10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f96947f0740 1 --2- 172.21.15.45:0/622189129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786d0c0 0x55f34787e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.791+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 >> 172.21.15.45:0/622189129 conn(0x55f347867990 msgr2=0x55f347869730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.790+0000 7f966affd640 0 client.26177 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.791+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 shutdown_connections 2024-03-12T05:15:11.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.791+0000 7f96947f0740 1 -- 172.21.15.45:0/622189129 wait complete. 2024-03-12T05:15:11.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.791+0000 7f96947f0740 20 client.26177 trim_cache size 0 max 16384 2024-03-12T05:15:11.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.817+0000 7f96947f0740 1 Processor -- start 2024-03-12T05:15:11.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.817+0000 7f96947f0740 1 -- start start 2024-03-12T05:15:11.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.817+0000 7f96947f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34787e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.817+0000 7f96947f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 0x55f34787e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.817+0000 7f96947f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f347877020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34787e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f968b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34787e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34518/0 (socket says 172.21.15.45:34518) 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f968b7fe640 1 -- 172.21.15.45:0/3234680491 learned_addr learned my addr 172.21.15.45:0/3234680491 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f347868c10 con 0x55f34786da70 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f34786c3a0 con 0x55f34786d0c0 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f968bfff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f347877020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f34787c920 con 0x55f34787efc0 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.818+0000 7f9683fff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 0x55f34787e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 -- 172.21.15.45:0/3234680491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 msgr2=0x55f34787e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34787e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 -- 172.21.15.45:0/3234680491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 msgr2=0x55f34787e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 0x55f34787e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 -- 172.21.15.45:0/3234680491 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f34776fac0 con 0x55f34787efc0 2024-03-12T05:15:11.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f9683fff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 0x55f34787e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f968bfff640 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f347877020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f96840148d0 tx=0x7f9684007040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b7a36f93ba165e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.819+0000 7f96817fa640 1 -- 172.21.15.45:0/3234680491 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684009ba0 con 0x55f34787efc0 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.820+0000 7f96817fa640 1 -- 172.21.15.45:0/3234680491 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9684013590 con 0x55f34787efc0 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.820+0000 7f96817fa640 1 -- 172.21.15.45:0/3234680491 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684014c70 con 0x55f34787efc0 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.820+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 msgr2=0x55f347877020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.820+0000 7f96947f0740 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f347877020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f96840148d0 tx=0x7f9684007040 comp rx=0 tx=0).stop 2024-03-12T05:15:11.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 shutdown_connections 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34786da70 0x55f34787e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34787efc0 0x55f347877020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 --2- 172.21.15.45:0/3234680491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786d0c0 0x55f34787e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 >> 172.21.15.45:0/3234680491 conn(0x55f347867990 msgr2=0x55f347895f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 shutdown_connections 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 -- 172.21.15.45:0/3234680491 wait complete. 2024-03-12T05:15:11.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.821+0000 7f96947f0740 1 -- start start 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 Processor -- start 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786d0c0 0x55f347882980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f347883400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3478e5700 con 0x55f34787efc0 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3478e5840 con 0x55f34786da70 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3478e59b0 con 0x55f34786d0c0 2024-03-12T05:15:11.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f96947f0740 10 client.0 _ll_register_callbacks cb 0x7f9684009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f9683fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f9683fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34522/0 (socket says 172.21.15.45:34522) 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.822+0000 7f9683fff640 1 -- 172.21.15.45:0/2863083858 learned_addr learned my addr 172.21.15.45:0/2863083858 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f968bfff640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f347883400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 -- 172.21.15.45:0/2863083858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786d0c0 msgr2=0x55f347882980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786d0c0 0x55f347882980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 -- 172.21.15.45:0/2863083858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 msgr2=0x55f347883400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f347883400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 -- 172.21.15.45:0/2863083858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f347793570 con 0x55f34786da70 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f9683fff640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55f34787b570 tx=0x7f9674005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d241032c0b7dbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f966affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674036070 con 0x55f34786da70 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f34787bb40 con 0x55f34786da70 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9674045040 con 0x55f34786da70 2024-03-12T05:15:11.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.823+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9674008410 con 0x55f34786da70 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 10 client.26183 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f3477956a0 con 0x55f34786da70 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f34787bb40 con 0x55f34786da70 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9674050040 con 0x55f34786da70 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 10 client.26183 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 20 client.26183 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f3478e6f20 0x55f3478e9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 4 client.26183 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f9684013c40 con 0x55f3478e6f20 2024-03-12T05:15:11.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.824+0000 7f968b7fe640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f3478e6f20 0x55f3478e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:11.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.825+0000 7f966affd640 10 client.26183 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.825+0000 7f968b7fe640 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f3478e6f20 0x55f3478e9340 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4637a3d0c5152db server_cookie=3eaa136f251c704f in_seq=0 out_seq=0 2024-03-12T05:15:11.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.825+0000 7f966affd640 1 -- 172.21.15.45:0/2863083858 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f9684013c40 con 0x55f3478e6f20 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.825+0000 7f966affd640 10 client.26183 handle_command_reply: tid=0 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.825+0000 7f96947f0740 1 client.26183 shutdown 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 msgr2=0x55f347882ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55f34787b570 tx=0x7f9674005b50 comp rx=0 tx=0).stop 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 shutdown_connections 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f3478e6f20 0x55f3478e9340 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f34787efc0 0x55f347883400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f34786da70 0x55f347882ec0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 --2- 172.21.15.45:0/2863083858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f34786d0c0 0x55f347882980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 >> 172.21.15.45:0/2863083858 conn(0x55f347867990 msgr2=0x55f347877020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f966affd640 0 client.26183 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 shutdown_connections 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 1 -- 172.21.15.45:0/2863083858 wait complete. 2024-03-12T05:15:11.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.826+0000 7f96947f0740 20 client.26183 trim_cache size 0 max 16384 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "1ca182a4-cd7c-44dd-b4dc-385b861e1191": { 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "1ca182a4-cd7c-44dd-b4dc-385b861e1191", 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:15:11.825 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 msgr2=0x7f96780a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 0x7f96780a8640 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f967400d090 tx=0x7f9674004890 comp rx=0 tx=0).stop 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 msgr2=0x7f968c083a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f967c00b500 tx=0x7f967c00b9c0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 shutdown_connections 2024-03-12T05:15:11.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96780a6180 0x7f96780a8640 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968c10cec0 0x7f968c07dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f968c07dff0 0x7f968c07e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 --2- 172.21.15.45:0/1778897020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f968c1083f0 0x7f968c083a40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 >> 172.21.15.45:0/1778897020 conn(0x7f968c06c270 msgr2=0x7f968c070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.828+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 shutdown_connections 2024-03-12T05:15:11.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:11.829+0000 7f9691c22640 1 -- 172.21.15.45:0/1778897020 wait complete. 2024-03-12T05:15:11.842 INFO:tasks.cephfs.filesystem:scrub status for tag:1ca182a4-cd7c-44dd-b4dc-385b861e1191 - {'path': '/', 'tag': '1ca182a4-cd7c-44dd-b4dc-385b861e1191', 'options': 'recursive,force'} 2024-03-12T05:15:11.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:11.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:11 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:12.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:15:12.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:12.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:12.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:11 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: pgmap v132: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:11 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:12 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:12.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:12 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:12.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:12 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:12.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:12 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:13.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:12 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:13.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:12 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:13.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:12 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:13.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:12 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:13.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:12 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:13.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:12 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:13.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:12 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:13.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:12 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:13 smithi116 ceph-mon[35486]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:13 smithi045 ceph-mon[32174]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:14.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:13 smithi130 ceph-mon[35398]: pgmap v133: 97 pgs: 97 active+clean; 2.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 190.0 s, 1413.4 tps, lat 70.681 ms stddev 25.544 2024-03-12T05:15:15.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:15 smithi116 ceph-mon[35486]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 MiB/s wr, 1.89k op/s 2024-03-12T05:15:16.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:15 smithi045 ceph-mon[32174]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 MiB/s wr, 1.89k op/s 2024-03-12T05:15:16.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:15 smithi130 ceph-mon[35398]: pgmap v134: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 16 MiB/s wr, 1.89k op/s 2024-03-12T05:15:17.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:17 smithi116 ceph-mon[35486]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:15:18.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:17 smithi045 ceph-mon[32174]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:15:18.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:17 smithi130 ceph-mon[35398]: pgmap v135: 97 pgs: 97 active+clean; 2.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:15:19.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 195.0 s, 1441.2 tps, lat 67.356 ms stddev 16.879 2024-03-12T05:15:19.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:19 smithi116 ceph-mon[35486]: pgmap v136: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 1.83k op/s 2024-03-12T05:15:20.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:19 smithi045 ceph-mon[32174]: pgmap v136: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 1.83k op/s 2024-03-12T05:15:20.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:19 smithi130 ceph-mon[35398]: pgmap v136: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 1.83k op/s 2024-03-12T05:15:22.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:21 smithi045 ceph-mon[32174]: pgmap v137: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:22.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:21 smithi130 ceph-mon[35398]: pgmap v137: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:22.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:21 smithi116 ceph-mon[35486]: pgmap v137: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:23.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:22 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:23.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:22 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:23.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:22 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:23.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:22 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:23.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:22 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:23.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:22 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:24.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:23 smithi045 ceph-mon[32174]: pgmap v138: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:24.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:24.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:24.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:23 smithi130 ceph-mon[35398]: pgmap v138: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:24.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:24.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:24.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:23 smithi116 ceph-mon[35486]: pgmap v138: 97 pgs: 97 active+clean; 2.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:15:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 200.0 s, 1435.8 tps, lat 71.673 ms stddev 67.115 2024-03-12T05:15:26.019 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:25 smithi130 ceph-mon[35398]: pgmap v139: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 16 MiB/s wr, 1.76k op/s 2024-03-12T05:15:26.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:25 smithi045 ceph-mon[32174]: pgmap v139: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 16 MiB/s wr, 1.76k op/s 2024-03-12T05:15:26.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:25 smithi116 ceph-mon[35486]: pgmap v139: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 16 MiB/s wr, 1.76k op/s 2024-03-12T05:15:28.044 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:27 smithi116 ceph-mon[35486]: pgmap v140: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-03-12T05:15:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:27 smithi045 ceph-mon[32174]: pgmap v140: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-03-12T05:15:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:27 smithi130 ceph-mon[35398]: pgmap v140: 97 pgs: 97 active+clean; 2.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-03-12T05:15:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 205.0 s, 1456.6 tps, lat 68.653 ms stddev 22.026 2024-03-12T05:15:30.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:29 smithi045 ceph-mon[32174]: pgmap v141: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:30.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:29 smithi130 ceph-mon[35398]: pgmap v141: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:30.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:29 smithi116 ceph-mon[35486]: pgmap v141: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:32.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:31 smithi045 ceph-mon[32174]: pgmap v142: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:32.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:31 smithi130 ceph-mon[35398]: pgmap v142: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:32.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:31 smithi116 ceph-mon[35486]: pgmap v142: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:34.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:33 smithi045 ceph-mon[32174]: pgmap v143: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:34.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:33 smithi130 ceph-mon[35398]: pgmap v143: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:34.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:33 smithi116 ceph-mon[35486]: pgmap v143: 97 pgs: 97 active+clean; 2.5 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:34.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 210.0 s, 1628.8 tps, lat 61.408 ms stddev 15.921 2024-03-12T05:15:36.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:35 smithi045 ceph-mon[32174]: pgmap v144: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 17 MiB/s wr, 1.88k op/s 2024-03-12T05:15:36.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:35 smithi130 ceph-mon[35398]: pgmap v144: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 17 MiB/s wr, 1.88k op/s 2024-03-12T05:15:36.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:35 smithi116 ceph-mon[35486]: pgmap v144: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 17 MiB/s wr, 1.88k op/s 2024-03-12T05:15:38.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:37 smithi045 ceph-mon[32174]: pgmap v145: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:15:38.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:37 smithi130 ceph-mon[35398]: pgmap v145: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:15:38.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:37 smithi116 ceph-mon[35486]: pgmap v145: 97 pgs: 97 active+clean; 2.5 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:15:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 215.0 s, 1577.4 tps, lat 63.419 ms stddev 17.285 2024-03-12T05:15:40.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:39 smithi045 ceph-mon[32174]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 1.95k op/s 2024-03-12T05:15:40.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:39 smithi130 ceph-mon[35398]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 1.95k op/s 2024-03-12T05:15:40.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:39 smithi116 ceph-mon[35486]: pgmap v146: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 17 MiB/s wr, 1.95k op/s 2024-03-12T05:15:41.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:15:41.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:15:42.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 -- 172.21.15.45:0/3967162742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4108660 msgr2=0x7ff8a4110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 --2- 172.21.15.45:0/3967162742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4108660 0x7ff8a4110230 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009cd0 tx=0x7ff89c02c0e0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 -- 172.21.15.45:0/3967162742 shutdown_connections 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 --2- 172.21.15.45:0/3967162742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4108660 0x7ff8a4110230 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 --2- 172.21.15.45:0/3967162742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103f40 0x7ff8a4107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 --2- 172.21.15.45:0/3967162742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4103590 0x7ff8a4103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 -- 172.21.15.45:0/3967162742 >> 172.21.15.45:0/3967162742 conn(0x7ff8a40fee30 msgr2=0x7ff8a4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 -- 172.21.15.45:0/3967162742 shutdown_connections 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.052+0000 7ff8aa023640 1 -- 172.21.15.45:0/3967162742 wait complete. 2024-03-12T05:15:42.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.054+0000 7ff8aa023640 1 Processor -- start 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.054+0000 7ff8aa023640 1 -- start start 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.054+0000 7ff8aa023640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.054+0000 7ff8aa023640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 0x7ff8a4075bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8aa023640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 0x7ff8a41a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43908/0 (socket says 172.21.15.45:43908) 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a37fe640 1 -- 172.21.15.45:0/2204648334 learned_addr learned my addr 172.21.15.45:0/2204648334 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a4079820 con 0x7ff8a4108660 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41aa3b0 con 0x7ff8a4076110 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41aa540 con 0x7ff8a4103590 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 0x7ff8a4075bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 0x7ff8a41a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 -- 172.21.15.45:0/2204648334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 msgr2=0x7ff8a4075690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 -- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 msgr2=0x7ff8a4075bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 0x7ff8a4075bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a3fff640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c009900 con 0x7ff8a4076110 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.055+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 0x7ff8a4075bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 0x7ff8a41a9e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff89c02c620 tx=0x7ff89c035d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8a17fa640 1 -- 172.21.15.45:0/2204648334 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c0329e0 con 0x7ff8a4076110 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8a17fa640 1 -- 172.21.15.45:0/2204648334 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff89c032410 con 0x7ff8a4076110 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a41aa790 con 0x7ff8a4076110 2024-03-12T05:15:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8a17fa640 1 -- 172.21.15.45:0/2204648334 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c0047d0 con 0x7ff8a4076110 2024-03-12T05:15:42.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.056+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a41aac90 con 0x7ff8a4076110 2024-03-12T05:15:42.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.060+0000 7ff8a17fa640 1 -- 172.21.15.45:0/2204648334 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff89c03b040 con 0x7ff8a4076110 2024-03-12T05:15:42.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.060+0000 7ff8acbf6740 1 Processor -- start 2024-03-12T05:15:42.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.060+0000 7ff8acbf6740 1 -- start start 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a31f2b0 0x55713a31f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31fbd0 0x55713a328270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a28e280 con 0x55713a31fbd0 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a28b7b0 con 0x55713a3287b0 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a286f90 con 0x55713a31f2b0 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8a17fa640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 0x7ff8900a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54036/0 (socket says 172.21.15.45:54036) 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff89bfff640 1 -- 172.21.15.45:0/1882846718 learned_addr learned my addr 172.21.15.45:0/1882846718 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a31f2b0 0x55713a31f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31fbd0 0x55713a328270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.061+0000 7ff8a17fa640 1 -- 172.21.15.45:0/2204648334 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff89c0f0210 con 0x7ff8a4076110 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 -- 172.21.15.45:0/1882846718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a31f2b0 msgr2=0x55713a31f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a31f2b0 0x55713a31f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 -- 172.21.15.45:0/1882846718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31fbd0 msgr2=0x55713a328270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31fbd0 0x55713a328270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 -- 172.21.15.45:0/1882846718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55713a245570 con 0x55713a3287b0 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 0x7ff8900a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff88c012a90 tx=0x7ff88c00ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3dbd7f21d111eb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff899ffb640 1 -- 172.21.15.45:0/1882846718 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c004020 con 0x55713a3287b0 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff899ffb640 1 -- 172.21.15.45:0/1882846718 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff88c005020 con 0x55713a3287b0 2024-03-12T05:15:42.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.062+0000 7ff899ffb640 1 -- 172.21.15.45:0/1882846718 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c010cc0 con 0x55713a3287b0 2024-03-12T05:15:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1882846718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 msgr2=0x55713a32aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff88c012a90 tx=0x7ff88c00ba90 comp rx=0 tx=0).stop 2024-03-12T05:15:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8a3fff640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 0x7ff8900a4160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0ba160 tx=0x7ff89c0095c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1882846718 shutdown_connections 2024-03-12T05:15:42.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31fbd0 0x55713a328270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a3287b0 0x55713a32aba0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1882846718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a31f2b0 0x55713a31f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1882846718 >> 172.21.15.45:0/1882846718 conn(0x55713a319990 msgr2=0x55713a319d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1882846718 shutdown_connections 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.063+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1882846718 wait complete. 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.064+0000 7ff8acbf6740 1 -- start start 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.064+0000 7ff8acbf6740 1 Processor -- start 2024-03-12T05:15:42.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.064+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a328240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a3255d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a325cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557139f5a430 con 0x55713a31f2b0 2024-03-12T05:15:42.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557139f16e00 con 0x55713a31fbd0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a20bfd0 con 0x55713a3287b0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8acbf6740 10 client.0 _ll_register_callbacks cb 0x55713a261fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a328240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a3255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a328240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33642/0 (socket says 172.21.15.45:33642) 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff89bfff640 1 -- 172.21.15.45:0/1426296442 learned_addr learned my addr 172.21.15.45:0/1426296442 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a325cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 -- 172.21.15.45:0/1426296442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 msgr2=0x55713a325cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a325cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 -- 172.21.15.45:0/1426296442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 msgr2=0x55713a328240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a328240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.065+0000 7ff8a37fe640 1 -- 172.21.15.45:0/1426296442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c01a040 con 0x55713a31fbd0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.066+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a3255d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff894005fd0 tx=0x7ff8940039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa2bbf7993aab13 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.066+0000 7ff882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:42.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.066+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894016070 con 0x55713a31fbd0 2024-03-12T05:15:42.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.066+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff8940117d0 con 0x55713a31fbd0 2024-03-12T05:15:42.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.066+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557139f867e0 con 0x55713a31fbd0 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.067+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff894011990 con 0x55713a31fbd0 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.067+0000 7ff8acbf6740 10 client.26195 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.067+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55713a285610 con 0x55713a31fbd0 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557139f867e0 con 0x55713a31fbd0 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff894024c30 con 0x55713a31fbd0 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 10 client.26195 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:42.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 20 client.26195 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a338930 0x55713a33ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 4 client.26195 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.068+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557139fc3720 con 0x55713a338930 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.069+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a338930 0x55713a33ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.069+0000 7ff89bfff640 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a338930 0x55713a33ad50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c74a3de499252e76 server_cookie=67ff0ab492352898 in_seq=0 out_seq=0 2024-03-12T05:15:42.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.069+0000 7ff882ffd640 10 client.26195 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff882ffd640 1 -- 172.21.15.45:0/1426296442 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557139fc3720 con 0x55713a338930 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff882ffd640 10 client.26195 handle_command_reply: tid=0 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 client.26195 shutdown 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 msgr2=0x55713a3255d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a3255d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff894005fd0 tx=0x7ff8940039c0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 shutdown_connections 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a338930 0x55713a33ad50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a325cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a3255d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/1426296442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a328240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 >> 172.21.15.45:0/1426296442 conn(0x55713a319990 msgr2=0x55713a31b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff882ffd640 0 client.26195 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 shutdown_connections 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 1 -- 172.21.15.45:0/1426296442 wait complete. 2024-03-12T05:15:42.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.070+0000 7ff8acbf6740 20 client.26195 trim_cache size 0 max 16384 2024-03-12T05:15:42.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.102+0000 7ff8acbf6740 1 Processor -- start 2024-03-12T05:15:42.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.102+0000 7ff8acbf6740 1 -- start start 2024-03-12T05:15:42.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.102+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33650/0 (socket says 172.21.15.45:33650) 2024-03-12T05:15:42.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff89bfff640 1 -- 172.21.15.45:0/2015505377 learned_addr learned my addr 172.21.15.45:0/2015505377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a328240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a3197f0 con 0x55713a31f2b0 2024-03-12T05:15:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a31a9e0 con 0x55713a31fbd0 2024-03-12T05:15:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.103+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a33ff60 con 0x55713a3287b0 2024-03-12T05:15:42.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a328240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 -- 172.21.15.45:0/2015505377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 msgr2=0x55713a32f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 -- 172.21.15.45:0/2015505377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 msgr2=0x55713a328240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a328240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.104+0000 7ff89bfff640 1 -- 172.21.15.45:0/2015505377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55713a221ac0 con 0x55713a31f2b0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.105+0000 7ff89bfff640 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0109a0 tx=0x7ff88c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d51f114a824c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.105+0000 7ff8997fa640 1 -- 172.21.15.45:0/2015505377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0053f0 con 0x55713a31f2b0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.105+0000 7ff8997fa640 1 -- 172.21.15.45:0/2015505377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff88c0044d0 con 0x55713a31f2b0 2024-03-12T05:15:42.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.105+0000 7ff8997fa640 1 -- 172.21.15.45:0/2015505377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0303e0 con 0x55713a31f2b0 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 msgr2=0x55713a327d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0109a0 tx=0x7ff88c00bed0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 shutdown_connections 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a328240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/2015505377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a327d00 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 >> 172.21.15.45:0/2015505377 conn(0x55713a319990 msgr2=0x55713a3208d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 shutdown_connections 2024-03-12T05:15:42.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.106+0000 7ff8acbf6740 1 -- 172.21.15.45:0/2015505377 wait complete. 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 -- start start 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 Processor -- start 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a32e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a38ee60 con 0x55713a31f2b0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a38efd0 con 0x55713a31fbd0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55713a38f190 con 0x55713a3287b0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.107+0000 7ff8acbf6740 10 client.0 _ll_register_callbacks cb 0x7ff88c00bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a32e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff89bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a32e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33666/0 (socket says 172.21.15.45:33666) 2024-03-12T05:15:42.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54052/0 (socket says 172.21.15.45:54052) 2024-03-12T05:15:42.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff89bfff640 1 -- 172.21.15.45:0/419687888 learned_addr learned my addr 172.21.15.45:0/419687888 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 -- 172.21.15.45:0/419687888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 msgr2=0x55713a32ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 -- 172.21.15.45:0/419687888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 msgr2=0x55713a32e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a32e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 -- 172.21.15.45:0/419687888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55713a245570 con 0x55713a31fbd0 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.108+0000 7ff8a37fe640 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55713a337340 tx=0x7ff894022e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df1a88331160ed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:42.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894016040 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff89401bc10 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff88c005020 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff89401b850 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff8acbf6740 10 client.26201 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.109+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55713a2476a0 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff89403b080 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff894035030 con 0x55713a31fbd0 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 10 client.26201 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 10 client.26201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 20 client.26201 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a390970 0x55713a392d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 4 client.26201 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55713a012890 con 0x55713a390970 2024-03-12T05:15:42.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.110+0000 7ff89bfff640 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a390970 0x55713a392d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.111+0000 7ff882ffd640 10 client.26201 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.111+0000 7ff89bfff640 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a390970 0x55713a392d90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58516a08b0cca2cb server_cookie=99c88c3493b9cdf6 in_seq=0 out_seq=0 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.111+0000 7ff882ffd640 1 -- 172.21.15.45:0/419687888 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55713a012890 con 0x55713a390970 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.111+0000 7ff882ffd640 10 client.26201 handle_command_reply: tid=0 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 client.26201 shutdown 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 msgr2=0x55713a32e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55713a337340 tx=0x7ff894022e40 comp rx=0 tx=0).stop 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 shutdown_connections 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55713a390970 0x55713a392d90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55713a3287b0 0x55713a32ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55713a31fbd0 0x55713a32e6b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 --2- 172.21.15.45:0/419687888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55713a31f2b0 0x55713a32e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 >> 172.21.15.45:0/419687888 conn(0x55713a319990 msgr2=0x55713a324e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff882ffd640 0 client.26201 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.112+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 shutdown_connections 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.113+0000 7ff8acbf6740 1 -- 172.21.15.45:0/419687888 wait complete. 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.113+0000 7ff8acbf6740 20 client.26201 trim_cache size 0 max 16384 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:15:42.110 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:15:42.111 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 msgr2=0x7ff8900a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 0x7ff8900a4160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff89c0ba160 tx=0x7ff89c0095c0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 msgr2=0x7ff8a41a9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 0x7ff8a41a9e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff89c02c620 tx=0x7ff89c035d20 comp rx=0 tx=0).stop 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 shutdown_connections 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff8900a1ca0 0x7ff8900a4160 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8a4108660 0x7ff8a4075bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8a4076110 0x7ff8a41a9e70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 --2- 172.21.15.45:0/2204648334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8a4103590 0x7ff8a4075690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 >> 172.21.15.45:0/2204648334 conn(0x7ff8a40fee30 msgr2=0x7ff8a4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.114+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 shutdown_connections 2024-03-12T05:15:42.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.115+0000 7ff8aa023640 1 -- 172.21.15.45:0/2204648334 wait complete. 2024-03-12T05:15:42.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:41 smithi045 ceph-mon[32174]: pgmap v147: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:42.135 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:15:42.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:15:42.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:41 smithi130 ceph-mon[35398]: pgmap v147: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:42.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:41 smithi116 ceph-mon[35486]: pgmap v147: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 -- 172.21.15.45:0/3546345280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a410c9d0 msgr2=0x7f09a410cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 --2- 172.21.15.45:0/3546345280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a410c9d0 0x7f09a410cdb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0994007510 tx=0x7f099402c8b0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 -- 172.21.15.45:0/3546345280 shutdown_connections 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 --2- 172.21.15.45:0/3546345280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a4108180 0x7f09a4108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 --2- 172.21.15.45:0/3546345280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a4114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 --2- 172.21.15.45:0/3546345280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a410c9d0 0x7f09a410cdb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.311+0000 7f09aaf80640 1 -- 172.21.15.45:0/3546345280 >> 172.21.15.45:0/3546345280 conn(0x7f09a406c4a0 msgr2=0x7f09a406c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.313+0000 7f09aaf80640 1 -- 172.21.15.45:0/3546345280 shutdown_connections 2024-03-12T05:15:42.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.313+0000 7f09aaf80640 1 -- 172.21.15.45:0/3546345280 wait complete. 2024-03-12T05:15:42.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.315+0000 7f09aaf80640 1 Processor -- start 2024-03-12T05:15:42.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.315+0000 7f09aaf80640 1 -- start start 2024-03-12T05:15:42.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.315+0000 7f09aaf80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a4108180 0x7f09a41a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.315+0000 7f09aaf80640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aaf80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 0x7f09a419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aaf80640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41a0020 con 0x7f09a410c9d0 2024-03-12T05:15:42.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aaf80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41a0190 con 0x7f09a4108180 2024-03-12T05:15:42.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aaf80640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a41a0350 con 0x7f09a4108b40 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43966/0 (socket says 172.21.15.45:43966) 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 -- 172.21.15.45:0/4003752939 learned_addr learned my addr 172.21.15.45:0/4003752939 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aa77f640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 0x7f09a419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 -- 172.21.15.45:0/4003752939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a4108180 msgr2=0x7f09a41a5850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a4108180 0x7f09a41a5850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 -- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 msgr2=0x7f09a419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 0x7f09a419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09a977d640 1 -- 172.21.15.45:0/4003752939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940071c0 con 0x7f09a4108b40 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.316+0000 7f09aa77f640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 0x7f09a419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f09a977d640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9f0 tx=0x7f09a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a000cd80 con 0x7f09a4108b40 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f09a00108e0 con 0x7f09a4108b40 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a00188c0 con 0x7f09a4108b40 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a41a0630 con 0x7f09a4108b40 2024-03-12T05:15:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.317+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a406fbc0 con 0x7f09a4108b40 2024-03-12T05:15:42.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.318+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a41a0850 con 0x7f09a4108b40 2024-03-12T05:15:42.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.320+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f09a0010a50 con 0x7f09a4108b40 2024-03-12T05:15:42.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.321+0000 7f099affd640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 0x7f09900a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.321+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f09a0014070 con 0x7f09a4108b40 2024-03-12T05:15:42.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.321+0000 7f09a9f7e640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 0x7f09900a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.322+0000 7f09a9f7e640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 0x7f09900a4110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09a41a0b40 tx=0x7f0994005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.327+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f09a0096b00 con 0x7f09a4108b40 2024-03-12T05:15:42.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.729+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a4109190 con 0x7f09a4108b40 2024-03-12T05:15:42.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.730+0000 7f099affd640 1 -- 172.21.15.45:0/4003752939 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f09a009a1c0 con 0x7f09a4108b40 2024-03-12T05:15:42.726 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:15:42.726 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:15:42.727 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 msgr2=0x7f09900a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 0x7f09900a4110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f09a41a0b40 tx=0x7f0994005fd0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 msgr2=0x7f09a41a5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e9f0 tx=0x7f09a000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 shutdown_connections 2024-03-12T05:15:42.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f09900a1c50 0x7f09900a4110 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09a410c9d0 0x7f09a419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09a4108b40 0x7f09a41a5d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 --2- 172.21.15.45:0/4003752939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09a4108180 0x7f09a41a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 >> 172.21.15.45:0/4003752939 conn(0x7f09a406c4a0 msgr2=0x7f09a4071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 shutdown_connections 2024-03-12T05:15:42.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.734+0000 7f09aaf80640 1 -- 172.21.15.45:0/4003752939 wait complete. 2024-03-12T05:15:42.748 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:42.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/2985106479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c41057a0 msgr2=0x7ff4c4105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/2985106479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c41057a0 0x7ff4c4105b80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400a910 tx=0x7ff4b402be50 comp rx=0 tx=0).stop 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/2985106479 shutdown_connections 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/2985106479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c40ff790 0x7ff4c40ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/2985106479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/2985106479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c41057a0 0x7ff4c4105b80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/2985106479 >> 172.21.15.45:0/2985106479 conn(0x7ff4c40765a0 msgr2=0x7ff4c40fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/2985106479 shutdown_connections 2024-03-12T05:15:42.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.930+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/2985106479 wait complete. 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.931+0000 7ff4c8cfb640 1 Processor -- start 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.931+0000 7ff4c8cfb640 1 -- start start 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 0x7ff4c419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4199e30 con 0x7ff4c41057a0 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4199fa0 con 0x7ff4c40ff790 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c8cfb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c419a160 con 0x7ff4c4100150 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43994/0 (socket says 172.21.15.45:43994) 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33706/0 (socket says 172.21.15.45:33706) 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/1006368558 learned_addr learned my addr 172.21.15.45:0/1006368558 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4c259c640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 0x7ff4c419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 -- 172.21.15.45:0/1006368558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 msgr2=0x7ff4c419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 0x7ff4c419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 -- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 msgr2=0x7ff4c41996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.932+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4bbfff640 1 -- 172.21.15.45:0/1006368558 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac00b6a0 con 0x7ff4c4100150 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4c259c640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 0x7ff4c419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009800 tx=0x7ff4ac009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4bb7fe640 1 -- 172.21.15.45:0/1006368558 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac018070 con 0x7ff4c4100150 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b400a580 con 0x7ff4c4100150 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4bb7fe640 1 -- 172.21.15.45:0/1006368558 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4ac0044d0 con 0x7ff4c4100150 2024-03-12T05:15:42.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4bb7fe640 1 -- 172.21.15.45:0/1006368558 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac018070 con 0x7ff4c4100150 2024-03-12T05:15:42.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.933+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c41a6510 con 0x7ff4c4100150 2024-03-12T05:15:42.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.937+0000 7ff4cb9ad740 1 Processor -- start 2024-03-12T05:15:42.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.937+0000 7ff4bb7fe640 1 -- 172.21.15.45:0/1006368558 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff4ac01d410 con 0x7ff4c4100150 2024-03-12T05:15:42.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 -- start start 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536db70 0x55b175376210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175378b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b175328960 con 0x55b17536d250 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1752dc310 con 0x55b175376750 2024-03-12T05:15:42.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1752d9840 con 0x55b17536db70 2024-03-12T05:15:42.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33712/0 (socket says 172.21.15.45:33712) 2024-03-12T05:15:42.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/2144092394 learned_addr learned my addr 172.21.15.45:0/2144092394 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4bb7fe640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 0x7ff4b00a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.938+0000 7ff4c259c640 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536db70 0x55b175376210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.939+0000 7ff4bb7fe640 1 -- 172.21.15.45:0/1006368558 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff4ac0cce80 con 0x7ff4c4100150 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.939+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 0x7ff4b00a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.939+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175378b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/2144092394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536db70 msgr2=0x55b175376210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536db70 0x55b175376210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/2144092394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 msgr2=0x55b175378b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175378b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/2144092394 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b175297060 con 0x55b17536d250 2024-03-12T05:15:42.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.940+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c8d0 tx=0x7ff4bc00cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e681730ae2ff1f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.941+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 0x7ff4b00a40c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc027000 tx=0x7ff4bc025000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.941+0000 7ff4b8ff9640 1 -- 172.21.15.45:0/2144092394 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00bd70 con 0x55b17536d250 2024-03-12T05:15:42.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.941+0000 7ff4b8ff9640 1 -- 172.21.15.45:0/2144092394 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4bc0238f0 con 0x55b17536d250 2024-03-12T05:15:42.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.941+0000 7ff4b8ff9640 1 -- 172.21.15.45:0/2144092394 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0052c0 con 0x55b17536d250 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/2144092394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 msgr2=0x55b17536d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c8d0 tx=0x7ff4bc00cd90 comp rx=0 tx=0).stop 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/2144092394 shutdown_connections 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175378b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536db70 0x55b175376210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/2144092394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536d250 0x55b17536d630 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/2144092394 >> 172.21.15.45:0/2144092394 conn(0x55b175367980 msgr2=0x55b175367d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/2144092394 shutdown_connections 2024-03-12T05:15:42.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.943+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/2144092394 wait complete. 2024-03-12T05:15:42.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.944+0000 7ff4cb9ad740 1 -- start start 2024-03-12T05:15:42.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.944+0000 7ff4cb9ad740 1 Processor -- start 2024-03-12T05:15:42.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.944+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b1753737a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.944+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.944+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1752d5020 con 0x55b17536db70 2024-03-12T05:15:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b174fa8430 con 0x55b175376750 2024-03-12T05:15:42.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b174f64e00 con 0x55b17536d250 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b1753737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54112/0 (socket says 172.21.15.45:54112) 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/3592910839 learned_addr learned my addr 172.21.15.45:0/3592910839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33722/0 (socket says 172.21.15.45:33722) 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b1753737a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44014/0 (socket says 172.21.15.45:44014) 2024-03-12T05:15:42.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.945+0000 7ff4cb9ad740 10 client.0 _ll_register_callbacks cb 0x55b17527ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 -- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 msgr2=0x55b1753737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b1753737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 -- 172.21.15.45:0/3592910839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 msgr2=0x55b175374220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 -- 172.21.15.45:0/3592910839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc00c4b0 con 0x55b17536db70 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0ca540 tx=0x7ff4ac099650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fea7080b1a043 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.947+0000 7ff4a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.948+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac010040 con 0x55b17536db70 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.948+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b174fd47e0 con 0x55b17536db70 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.948+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff4ac099a50 con 0x55b17536db70 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.948+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4ac09dba0 con 0x55b17536db70 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.948+0000 7ff4cb9ad740 10 client.16746 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:42.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b1752d36a0 con 0x55b17536db70 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b174fd47e0 con 0x55b17536db70 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff4ac0b2200 con 0x55b17536db70 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 10 client.16746 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 10 client.16746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 20 client.16746 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b175386a10 0x55b175388e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 4 client.16746 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.949+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b175011720 con 0x55b175386a10 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.950+0000 7ff4c259c640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b175386a10 0x55b175388e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.950+0000 7ff4c259c640 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b175386a10 0x55b175388e00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82b99481c0a58391 server_cookie=a6141f95cdfd5942 in_seq=0 out_seq=0 2024-03-12T05:15:42.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.950+0000 7ff4a9ffb640 10 client.16746 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.951+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3592910839 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b175011720 con 0x55b175386a10 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.951+0000 7ff4a9ffb640 10 client.16746 handle_command_reply: tid=0 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.951+0000 7ff4cb9ad740 1 client.16746 shutdown 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.951+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 msgr2=0x55b175373ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.951+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0ca540 tx=0x7ff4ac099650 comp rx=0 tx=0).stop 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 shutdown_connections 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b175386a10 0x55b175388e00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17536db70 0x55b175373ce0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b175374220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3592910839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b1753737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 >> 172.21.15.45:0/3592910839 conn(0x55b175367980 msgr2=0x55b1753696d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4a9ffb640 0 client.16746 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:42.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 shutdown_connections 2024-03-12T05:15:42.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3592910839 wait complete. 2024-03-12T05:15:42.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.952+0000 7ff4cb9ad740 20 client.16746 trim_cache size 0 max 16384 2024-03-12T05:15:42.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.973+0000 7ff4cb9ad740 1 Processor -- start 2024-03-12T05:15:42.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4cb9ad740 1 -- start start 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b175375ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b175375ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33738/0 (socket says 172.21.15.45:33738) 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b175375ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44018/0 (socket says 172.21.15.45:44018) 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.974+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 0x55b17536ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4bbfff640 1 -- 172.21.15.45:0/1362786111 learned_addr learned my addr 172.21.15.45:0/1362786111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b17525a130 con 0x55b175376750 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1753674b0 con 0x55b17538dce0 2024-03-12T05:15:42.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b175011720 con 0x55b17536d250 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 0x55b17536ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/1362786111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 msgr2=0x55b175375ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b175375ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/1362786111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 msgr2=0x55b17538d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4c2d9d640 1 -- 172.21.15.45:0/1362786111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b17526fc20 con 0x55b17538dce0 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.975+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.976+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 0x55b17536ea90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc015fd0 tx=0x7ff4bc018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b57ba6743bee8286 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.976+0000 7ff4abfff640 1 -- 172.21.15.45:0/1362786111 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0051d0 con 0x55b17538dce0 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.976+0000 7ff4abfff640 1 -- 172.21.15.45:0/1362786111 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4bc00c330 con 0x55b17538dce0 2024-03-12T05:15:42.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.976+0000 7ff4abfff640 1 -- 172.21.15.45:0/1362786111 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc005790 con 0x55b17538dce0 2024-03-12T05:15:42.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 msgr2=0x55b17536ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 0x55b17536ea90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc015fd0 tx=0x7ff4bc018000 comp rx=0 tx=0).stop 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 shutdown_connections 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b175376750 0x55b17538d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b17538dce0 0x55b17536ea90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/1362786111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b175375ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 >> 172.21.15.45:0/1362786111 conn(0x55b175367980 msgr2=0x55b175362a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 shutdown_connections 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.977+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/1362786111 wait complete. 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.978+0000 7ff4cb9ad740 1 -- start start 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.978+0000 7ff4cb9ad740 1 Processor -- start 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.978+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.978+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b17537cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4cb9ad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17538dce0 0x55b1753dcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b17537d2d0 con 0x55b17538dce0 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44022/0 (socket says 172.21.15.45:44022) 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4cb9ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1753dd470 con 0x55b175376750 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1753dd630 con 0x55b17536d250 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 -- 172.21.15.45:0/3569551888 learned_addr learned my addr 172.21.15.45:0/3569551888 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4cb9ad740 10 client.0 _ll_register_callbacks cb 0x7ff4bc007d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:42.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b17537cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c2d9d640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17538dce0 0x55b1753dcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 -- 172.21.15.45:0/3569551888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 msgr2=0x55b17537cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b17537cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 -- 172.21.15.45:0/3569551888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17538dce0 msgr2=0x55b1753dcf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17538dce0 0x55b1753dcf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4c259c640 1 -- 172.21.15.45:0/3569551888 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b175297060 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.979+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b17537cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4c259c640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402c430 tx=0x7ff4b4049c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89e256292d863bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4046040 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff4b40048d0 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b175060890 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4b4049e80 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4cb9ad740 10 client.26218 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b1752958c0 con 0x55b17536d250 2024-03-12T05:15:42.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4b404d070 con 0x55b17536d250 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.980+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff4b4034700 con 0x55b17536d250 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 10 client.26218 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 10 client.26218 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 20 client.26218 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b1753dedb0 0x55b1753e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 4 client.26218 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b175060890 con 0x55b1753dedb0 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b1753dedb0 0x55b1753e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4a9ffb640 10 client.26218 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.981+0000 7ff4bbfff640 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b1753dedb0 0x55b1753e11d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=43bb2f1c5d383e2a server_cookie=3081d3d9f4f08012 in_seq=0 out_seq=0 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.982+0000 7ff4a9ffb640 1 -- 172.21.15.45:0/3569551888 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b175060890 con 0x55b1753dedb0 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.982+0000 7ff4a9ffb640 10 client.26218 handle_command_reply: tid=0 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.982+0000 7ff4cb9ad740 1 client.26218 shutdown 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.982+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 msgr2=0x55b17537c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402c430 tx=0x7ff4b4049c50 comp rx=0 tx=0).stop 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 shutdown_connections 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b1753dedb0 0x55b1753e11d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b17538dce0 0x55b1753dcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b175376750 0x55b17537cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 --2- 172.21.15.45:0/3569551888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b17536d250 0x55b17537c640 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 >> 172.21.15.45:0/3569551888 conn(0x55b175367980 msgr2=0x55b17536ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4a9ffb640 0 client.26218 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 shutdown_connections 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 1 -- 172.21.15.45:0/3569551888 wait complete. 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.983+0000 7ff4cb9ad740 20 client.26218 trim_cache size 0 max 16384 2024-03-12T05:15:42.982 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 msgr2=0x7ff4b00a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 0x7ff4b00a40c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc027000 tx=0x7ff4bc025000 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 msgr2=0x7ff4c419fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009800 tx=0x7ff4ac009cc0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 shutdown_connections 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4b00a1c00 0x7ff4b00a40c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4c41057a0 0x7ff4c41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c4100150 0x7ff4c419fad0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 --2- 172.21.15.45:0/1006368558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4c40ff790 0x7ff4c419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.984+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 >> 172.21.15.45:0/1006368558 conn(0x7ff4c40765a0 msgr2=0x7ff4c41026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.986+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 shutdown_connections 2024-03-12T05:15:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:42.986+0000 7ff4c8cfb640 1 -- 172.21.15.45:0/1006368558 wait complete. 2024-03-12T05:15:42.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:42 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4003752939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:43.000 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:43.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 -- 172.21.15.45:0/3248488654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018101d40 msgr2=0x7f3018105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 --2- 172.21.15.45:0/3248488654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018101d40 0x7f3018105d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3008009930 tx=0x7f300802bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 -- 172.21.15.45:0/3248488654 shutdown_connections 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 --2- 172.21.15.45:0/3248488654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018106460 0x7f301810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 --2- 172.21.15.45:0/3248488654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018101d40 0x7f3018105d80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 --2- 172.21.15.45:0/3248488654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018101390 0x7f3018101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 -- 172.21.15.45:0/3248488654 >> 172.21.15.45:0/3248488654 conn(0x7f30180fcbb0 msgr2=0x7f30180fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 -- 172.21.15.45:0/3248488654 shutdown_connections 2024-03-12T05:15:43.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.179+0000 7f301fa0e640 1 -- 172.21.15.45:0/3248488654 wait complete. 2024-03-12T05:15:43.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 Processor -- start 2024-03-12T05:15:43.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 -- start start 2024-03-12T05:15:43.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 0x7f3018195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018199e10 0x7f301819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301819a800 con 0x7f3018199e10 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301819a970 con 0x7f3018106460 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301fa0e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301819aae0 con 0x7f3018101390 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44042/0 (socket says 172.21.15.45:44042) 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.180+0000 7f301d7aa640 1 -- 172.21.15.45:0/16615602 learned_addr learned my addr 172.21.15.45:0/16615602 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301dfab640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018199e10 0x7f301819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301cfa9640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 0x7f3018195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 -- 172.21.15.45:0/16615602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 msgr2=0x7f3018195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 0x7f3018195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 -- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018199e10 msgr2=0x7f301819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018199e10 0x7f301819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 -- 172.21.15.45:0/16615602 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3008009560 con 0x7f3018101390 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301cfa9640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 0x7f3018195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f301d7aa640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3014009be0 tx=0x7f30140071c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.181+0000 7f300e7fc640 1 -- 172.21.15.45:0/16615602 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014007e00 con 0x7f3018101390 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.182+0000 7f300e7fc640 1 -- 172.21.15.45:0/16615602 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3014015040 con 0x7f3018101390 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.182+0000 7f300e7fc640 1 -- 172.21.15.45:0/16615602 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140186c0 con 0x7f3018101390 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.182+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30181a20c0 con 0x7f3018101390 2024-03-12T05:15:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.182+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30181a2520 con 0x7f3018101390 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.185+0000 7f300e7fc640 1 -- 172.21.15.45:0/16615602 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3014019ac0 con 0x7f3018101390 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.185+0000 7f30225e8740 1 Processor -- start 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 -- start start 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73edb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73e96960 con 0x559f73edb250 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73e4a310 con 0x559f73ee4750 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f30225e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73e47840 con 0x559f73edbb70 2024-03-12T05:15:43.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301dfab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73edb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44054/0 (socket says 172.21.15.45:44054) 2024-03-12T05:15:43.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301d7aa640 1 -- 172.21.15.45:0/1754222028 learned_addr learned my addr 172.21.15.45:0/1754222028 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301dfab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73edb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33760/0 (socket says 172.21.15.45:33760) 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.186+0000 7f301d7aa640 1 -- 172.21.15.45:0/1754222028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 msgr2=0x559f73ee6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f301d7aa640 1 -- 172.21.15.45:0/1754222028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 msgr2=0x559f73edb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73edb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f301d7aa640 1 -- 172.21.15.45:0/1754222028 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f73e05060 con 0x559f73edbb70 2024-03-12T05:15:43.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30140ce870 tx=0x7f30140d96f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30185295594b1459 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.187+0000 7f300e7fc640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 0x7f30040a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.188+0000 7f300e7fc640 1 -- 172.21.15.45:0/16615602 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f30140d1110 con 0x7f3018101390 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.188+0000 7f301dfab640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 0x7f30040a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.188+0000 7f30037fe640 1 -- 172.21.15.45:0/1754222028 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301409a080 con 0x559f73edbb70 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.188+0000 7f30037fe640 1 -- 172.21.15.45:0/1754222028 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f301409dbb0 con 0x559f73edbb70 2024-03-12T05:15:43.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.188+0000 7f30037fe640 1 -- 172.21.15.45:0/1754222028 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301409e4e0 con 0x559f73edbb70 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 -- 172.21.15.45:0/1754222028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 msgr2=0x559f73ee4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30140ce870 tx=0x7f30140d96f0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f301dfab640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 0x7f30040a41f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3010005fd0 tx=0x7f3010009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 -- 172.21.15.45:0/1754222028 shutdown_connections 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edbb70 0x559f73ee4210 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 --2- 172.21.15.45:0/1754222028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73edb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 -- 172.21.15.45:0/1754222028 >> 172.21.15.45:0/1754222028 conn(0x559f73ed5980 msgr2=0x559f73ed5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 -- 172.21.15.45:0/1754222028 shutdown_connections 2024-03-12T05:15:43.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.189+0000 7f30225e8740 1 -- 172.21.15.45:0/1754222028 wait complete. 2024-03-12T05:15:43.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.190+0000 7f30225e8740 1 -- start start 2024-03-12T05:15:43.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.190+0000 7f30225e8740 1 Processor -- start 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 0x559f73ee1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301cfa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 0x559f73ee1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301cfa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 0x559f73ee1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33770/0 (socket says 172.21.15.45:33770) 2024-03-12T05:15:43.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44056/0 (socket says 172.21.15.45:44056) 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301cfa9640 1 -- 172.21.15.45:0/4226609260 learned_addr learned my addr 172.21.15.45:0/4226609260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73e43020 con 0x559f73edbb70 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73b16430 con 0x559f73ee4750 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73ad2e00 con 0x559f73edb250 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f30225e8740 10 client.0 _ll_register_callbacks cb 0x559f73deacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.191+0000 7f301dfab640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 -- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 msgr2=0x559f73ee2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 -- 172.21.15.45:0/4226609260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 msgr2=0x559f73ee1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 0x559f73ee1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30181a2520 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f301d7aa640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3014007900 tx=0x7f30140d4820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f96393f60b83790e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f3000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301409a080 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f73b427e0 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.192+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f301409e4e0 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f301409d920 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 10 client.26230 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559f73e416a0 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f301409d740 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3014018070 con 0x559f73edb250 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 10 client.26230 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 10 client.26230 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 20 client.26230 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73ef4680 0x559f73ef6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 4 client.26230 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:43.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.193+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f73b427e0 con 0x559f73ef4680 2024-03-12T05:15:43.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.194+0000 7f301cfa9640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73ef4680 0x559f73ef6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.194+0000 7f301cfa9640 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73ef4680 0x559f73ef6ab0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1839e2d61fcfb002 server_cookie=b101bb328d4db6da in_seq=0 out_seq=0 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.194+0000 7f3000ff9640 10 client.26230 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.195+0000 7f3000ff9640 1 -- 172.21.15.45:0/4226609260 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559f73b427e0 con 0x559f73ef4680 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.195+0000 7f3000ff9640 10 client.26230 handle_command_reply: tid=0 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.195+0000 7f30225e8740 1 client.26230 shutdown 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.195+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 msgr2=0x559f73ee17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.195+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3014007900 tx=0x7f30140d4820 comp rx=0 tx=0).stop 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 shutdown_connections 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73ef4680 0x559f73ef6ab0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edbb70 0x559f73ee1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73ee4750 0x559f73ee2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 --2- 172.21.15.45:0/4226609260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73edb250 0x559f73ee17f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 >> 172.21.15.45:0/4226609260 conn(0x559f73ed5980 msgr2=0x559f73ed76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f3000ff9640 0 client.26230 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:43.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 shutdown_connections 2024-03-12T05:15:43.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 1 -- 172.21.15.45:0/4226609260 wait complete. 2024-03-12T05:15:43.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.196+0000 7f30225e8740 20 client.26230 trim_cache size 0 max 16384 2024-03-12T05:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.217+0000 7f30225e8740 1 Processor -- start 2024-03-12T05:15:43.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.217+0000 7f30225e8740 1 -- start start 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.217+0000 7f30225e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f30225e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73edcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f30225e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73edd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33778/0 (socket says 172.21.15.45:33778) 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 -- 172.21.15.45:0/1299497793 learned_addr learned my addr 172.21.15.45:0/1299497793 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73dc8130 con 0x559f73edb250 2024-03-12T05:15:43.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73ee4460 con 0x559f73edbb70 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73ed6c00 con 0x559f73ee4750 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301cfa9640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73edcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301dfab640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73edd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 -- 172.21.15.45:0/1299497793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 msgr2=0x559f73edd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73edd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 -- 172.21.15.45:0/1299497793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 msgr2=0x559f73edcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73edcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301d7aa640 1 -- 172.21.15.45:0/1299497793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f73dddc20 con 0x559f73edb250 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301dfab640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73edd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.218+0000 7f301cfa9640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73edcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.219+0000 7f301d7aa640 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f301409dd20 tx=0x7f301409c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effb942a9b8e5c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.219+0000 7f3002ffd640 1 -- 172.21.15.45:0/1299497793 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301409d920 con 0x559f73edb250 2024-03-12T05:15:43.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.219+0000 7f3002ffd640 1 -- 172.21.15.45:0/1299497793 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f301409d740 con 0x559f73edb250 2024-03-12T05:15:43.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.220+0000 7f3002ffd640 1 -- 172.21.15.45:0/1299497793 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140a6350 con 0x559f73edb250 2024-03-12T05:15:43.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:42 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4003752939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.222+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 msgr2=0x559f73ee3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.222+0000 7f30225e8740 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f301409dd20 tx=0x7f301409c2f0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.222+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 shutdown_connections 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73edd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73edcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 --2- 172.21.15.45:0/1299497793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee3f20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 >> 172.21.15.45:0/1299497793 conn(0x559f73ed5980 msgr2=0x559f73efb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 shutdown_connections 2024-03-12T05:15:43.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 -- 172.21.15.45:0/1299497793 wait complete. 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 -- start start 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.223+0000 7f30225e8740 1 Processor -- start 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73eea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73eebf20 con 0x559f73edb250 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73f53380 con 0x559f73edbb70 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f73f53540 con 0x559f73ee4750 2024-03-12T05:15:43.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f30225e8740 10 client.0 _ll_register_callbacks cb 0x7f30140d4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f301d7aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f301dfab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f301dfab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44074/0 (socket says 172.21.15.45:44074) 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.224+0000 7f301dfab640 1 -- 172.21.15.45:0/2132505142 learned_addr learned my addr 172.21.15.45:0/2132505142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301cfa9640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73eea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 -- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 msgr2=0x559f73eea4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73eea4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 -- 172.21.15.45:0/2132505142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 msgr2=0x559f73ee9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 -- 172.21.15.45:0/2132505142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301810e840 con 0x559f73ee4750 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f301dfab640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3010009e30 tx=0x7f3010004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0da3ef30d4ff1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.225+0000 7f3000ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:43.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010023070 con 0x559f73ee4750 2024-03-12T05:15:43.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f301001b940 con 0x559f73ee4750 2024-03-12T05:15:43.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f301409dbb0 con 0x559f73ee4750 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f301001bb00 con 0x559f73ee4750 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f30225e8740 10 client.26233 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.226+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559f73e416a0 con 0x559f73ee4750 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3010034080 con 0x559f73ee4750 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f301002e030 con 0x559f73ee4750 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 10 client.26233 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 10 client.26233 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 20 client.26233 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73f54de0 0x559f73f57200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 4 client.26233 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559f73b7f720 con 0x559f73f54de0 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.227+0000 7f301d7aa640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73f54de0 0x559f73f57200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f301d7aa640 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73f54de0 0x559f73f57200 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=666887e217c19e9e server_cookie=3df51b1ad5b969a8 in_seq=0 out_seq=0 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f3000ff9640 10 client.26233 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f3000ff9640 1 -- 172.21.15.45:0/2132505142 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559f73b7f720 con 0x559f73f54de0 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f3000ff9640 10 client.26233 handle_command_reply: tid=0 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f30225e8740 1 client.26233 shutdown 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 msgr2=0x559f73eeaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.228+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3010009e30 tx=0x7f3010004a40 comp rx=0 tx=0).stop 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 shutdown_connections 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559f73f54de0 0x559f73f57200 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559f73ee4750 0x559f73eeaa00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f73edbb70 0x559f73eea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 --2- 172.21.15.45:0/2132505142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559f73edb250 0x559f73ee9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 >> 172.21.15.45:0/2132505142 conn(0x559f73ed5980 msgr2=0x559f73edcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f3000ff9640 0 client.26233 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 shutdown_connections 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 1 -- 172.21.15.45:0/2132505142 wait complete. 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.229+0000 7f30225e8740 20 client.26233 trim_cache size 0 max 16384 2024-03-12T05:15:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 msgr2=0x7f30040a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 0x7f30040a41f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3010005fd0 tx=0x7f3010009500 comp rx=0 tx=0).stop 2024-03-12T05:15:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 msgr2=0x7f3018195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3014009be0 tx=0x7f30140071c0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 shutdown_connections 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30040a1d30 0x7f30040a41f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3018199e10 0x7f301819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3018106460 0x7f3018195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 --2- 172.21.15.45:0/16615602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3018101390 0x7f3018195550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 >> 172.21.15.45:0/16615602 conn(0x7f30180fcbb0 msgr2=0x7f30180fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 shutdown_connections 2024-03-12T05:15:43.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.230+0000 7f301fa0e640 1 -- 172.21.15.45:0/16615602 wait complete. 2024-03-12T05:15:43.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:42 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4003752939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:15:43.244 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:43.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 -- 172.21.15.45:0/2624178929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 msgr2=0x7f308c1005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2624178929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c1005e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3080009900 tx=0x7f308002bc40 comp rx=0 tx=0).stop 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 -- 172.21.15.45:0/2624178929 shutdown_connections 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2624178929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2624178929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 0x7f308c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2624178929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c1005e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.426+0000 7f3090b0c640 1 -- 172.21.15.45:0/2624178929 >> 172.21.15.45:0/2624178929 conn(0x7f308c0771e0 msgr2=0x7f308c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.427+0000 7f3090b0c640 1 -- 172.21.15.45:0/2624178929 shutdown_connections 2024-03-12T05:15:43.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.427+0000 7f3090b0c640 1 -- 172.21.15.45:0/2624178929 wait complete. 2024-03-12T05:15:43.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 Processor -- start 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 -- start start 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 0x7f308c06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c06b9a0 con 0x7f308c100200 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c06bb10 con 0x7f308c10db50 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3090b0c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c1b2ab0 con 0x7f308c100b20 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.428+0000 7f3089d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 0x7f308c06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54200/0 (socket says 172.21.15.45:54200) 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 -- 172.21.15.45:0/2726466334 learned_addr learned my addr 172.21.15.45:0/2726466334 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308a59c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 -- 172.21.15.45:0/2726466334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 msgr2=0x7f308c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 0x7f308c06acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 -- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 msgr2=0x7f308c072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 -- 172.21.15.45:0/2726466334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3080009560 con 0x7f308c10db50 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308a59c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.429+0000 7f308ad9d640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f308400eb70 tx=0x7f308400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.430+0000 7f307b7fe640 1 -- 172.21.15.45:0/2726466334 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084004d20 con 0x7f308c10db50 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.430+0000 7f307b7fe640 1 -- 172.21.15.45:0/2726466334 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3084004e80 con 0x7f308c10db50 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.430+0000 7f307b7fe640 1 -- 172.21.15.45:0/2726466334 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084010670 con 0x7f308c10db50 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.430+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c1b2d60 con 0x7f308c10db50 2024-03-12T05:15:43.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.430+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c1b3270 con 0x7f308c10db50 2024-03-12T05:15:43.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.433+0000 7f307b7fe640 1 -- 172.21.15.45:0/2726466334 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3084005020 con 0x7f308c10db50 2024-03-12T05:15:43.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.433+0000 7f30937bf740 1 Processor -- start 2024-03-12T05:15:43.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 -- start start 2024-03-12T05:15:43.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f3089d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f3089d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44104/0 (socket says 172.21.15.45:44104) 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f3089d9b640 1 -- 172.21.15.45:0/3120975848 learned_addr learned my addr 172.21.15.45:0/3120975848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 0x561c5d0e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d095960 con 0x561c5d0dab70 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d049310 con 0x561c5d0e3750 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.434+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d046840 con 0x561c5d0da250 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f307b7fe640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 0x7f30740a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f308a59c640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 0x561c5d0e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f307b7fe640 1 -- 172.21.15.45:0/2726466334 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f3084014070 con 0x7f308c10db50 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f308ad9d640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 0x7f30740a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 -- 172.21.15.45:0/3120975848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 msgr2=0x561c5d0e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 -- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 msgr2=0x561c5d0e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 0x561c5d0e3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f3089d9b640 1 -- 172.21.15.45:0/3120975848 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c5d004060 con 0x561c5d0da250 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.435+0000 7f308a59c640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 0x561c5d0e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.436+0000 7f3089d9b640 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f307c009730 tx=0x7f307c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65979adb7212c667 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.436+0000 7f3078ff9640 1 -- 172.21.15.45:0/3120975848 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c013130 con 0x561c5d0da250 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.436+0000 7f308ad9d640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 0x7f30740a4120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3084004af0 tx=0x7f3084004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.436+0000 7f3078ff9640 1 -- 172.21.15.45:0/3120975848 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f307c0044e0 con 0x561c5d0da250 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.436+0000 7f3078ff9640 1 -- 172.21.15.45:0/3120975848 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c0047a0 con 0x561c5d0da250 2024-03-12T05:15:43.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 msgr2=0x561c5d0da630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f307c009730 tx=0x7f307c005850 comp rx=0 tx=0).stop 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 shutdown_connections 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0dab70 0x561c5d0e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 --2- 172.21.15.45:0/3120975848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0da250 0x561c5d0da630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 >> 172.21.15.45:0/3120975848 conn(0x561c5d0d4980 msgr2=0x561c5d0d4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 shutdown_connections 2024-03-12T05:15:43.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.437+0000 7f30937bf740 1 -- 172.21.15.45:0/3120975848 wait complete. 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.438+0000 7f30937bf740 1 -- start start 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.438+0000 7f30937bf740 1 Processor -- start 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.438+0000 7f30937bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 0x561c5d0e0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.438+0000 7f30937bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f30937bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0e3750 0x561c5d0e12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f308a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f308a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44118/0 (socket says 172.21.15.45:44118) 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f308a59c640 1 -- 172.21.15.45:0/236912199 learned_addr learned my addr 172.21.15.45:0/236912199 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f3089d9b640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 0x561c5d0e0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d042020 con 0x561c5d0e3750 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5cd15430 con 0x561c5d0da250 2024-03-12T05:15:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5ccd1e00 con 0x561c5d0dab70 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f30937bf740 10 client.0 _ll_register_callbacks cb 0x561c5cfe9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.439+0000 7f3089d9b640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0e3750 0x561c5d0e12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 -- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 msgr2=0x561c5d0e0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 -- 172.21.15.45:0/236912199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0e3750 msgr2=0x561c5d0e12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0e3750 0x561c5d0e12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c01b040 con 0x561c5d0da250 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3089d9b640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 0x561c5d0e0820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f307c009cc0 tx=0x7f307c005440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c4a40f3ecc177c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c00f7c0 con 0x561c5d0da250 2024-03-12T05:15:43.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f307c00f920 con 0x561c5d0da250 2024-03-12T05:15:43.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c5cd417e0 con 0x561c5d0da250 2024-03-12T05:15:43.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.440+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f307c00fae0 con 0x561c5d0da250 2024-03-12T05:15:43.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 10 client.26243 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561c5d0406a0 con 0x561c5d0da250 2024-03-12T05:15:43.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f307c032080 con 0x561c5d0da250 2024-03-12T05:15:43.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f307c01f400 con 0x561c5d0da250 2024-03-12T05:15:43.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 10 client.26243 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 10 client.26243 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:15:43.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 20 client.26243 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d1049a0 0x561c5d106db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 4 client.26243 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.441+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c5cd417e0 con 0x561c5d1049a0 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.442+0000 7f308a59c640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d1049a0 0x561c5d106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.442+0000 7f3071ffb640 10 client.26243 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.442+0000 7f308a59c640 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d1049a0 0x561c5d106db0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=400ab1c9fc12c908 server_cookie=30dcc15f16c437f in_seq=0 out_seq=0 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.443+0000 7f3071ffb640 1 -- 172.21.15.45:0/236912199 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561c5cd417e0 con 0x561c5d1049a0 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.443+0000 7f3071ffb640 10 client.26243 handle_command_reply: tid=0 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.443+0000 7f30937bf740 1 client.26243 shutdown 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 msgr2=0x561c5d0e0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 0x561c5d0e0820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f307c009cc0 tx=0x7f307c005440 comp rx=0 tx=0).stop 2024-03-12T05:15:43.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 shutdown_connections 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d1049a0 0x561c5d106db0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0e3750 0x561c5d0e12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 --2- 172.21.15.45:0/236912199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0da250 0x561c5d0e0820 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f3071ffb640 0 client.26243 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.444+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 >> 172.21.15.45:0/236912199 conn(0x561c5d0d4980 msgr2=0x561c5d0d66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.445+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 shutdown_connections 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.445+0000 7f30937bf740 1 -- 172.21.15.45:0/236912199 wait complete. 2024-03-12T05:15:43.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.445+0000 7f30937bf740 20 client.26243 trim_cache size 0 max 16384 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 Processor -- start 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 -- start start 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5cfc7130 con 0x561c5d0da250 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d100570 con 0x561c5d0e3750 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f30937bf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d0d52c0 con 0x561c5d0dab70 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f308a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f308a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44132/0 (socket says 172.21.15.45:44132) 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.467+0000 7f308a59c640 1 -- 172.21.15.45:0/1935474629 learned_addr learned my addr 172.21.15.45:0/1935474629 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f3089d9b640 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 -- 172.21.15.45:0/1935474629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 msgr2=0x561c5d0e12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 -- 172.21.15.45:0/1935474629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 msgr2=0x561c5d0e2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0e2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 -- 172.21.15.45:0/1935474629 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c1076e0 con 0x561c5d0dab70 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.468+0000 7f308a59c640 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3080035000 tx=0x7f3080009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e8cf6d2770f7e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.469+0000 7f3073fff640 1 -- 172.21.15.45:0/1935474629 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080045040 con 0x561c5d0dab70 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.469+0000 7f3073fff640 1 -- 172.21.15.45:0/1935474629 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f308002cc30 con 0x561c5d0dab70 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.469+0000 7f3073fff640 1 -- 172.21.15.45:0/1935474629 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080047520 con 0x561c5d0dab70 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- 172.21.15.45:0/1935474629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 msgr2=0x561c5d0e0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3080035000 tx=0x7f3080009260 comp rx=0 tx=0).stop 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- 172.21.15.45:0/1935474629 shutdown_connections 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0e3750 0x561c5d0e12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0dab70 0x561c5d0e0d70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 --2- 172.21.15.45:0/1935474629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0e2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- 172.21.15.45:0/1935474629 >> 172.21.15.45:0/1935474629 conn(0x561c5d0d4980 msgr2=0x561c5d0df0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- 172.21.15.45:0/1935474629 shutdown_connections 2024-03-12T05:15:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- 172.21.15.45:0/1935474629 wait complete. 2024-03-12T05:15:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.470+0000 7f30937bf740 1 -- start start 2024-03-12T05:15:43.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.471+0000 7f30937bf740 1 Processor -- start 2024-03-12T05:15:43.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0edf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0e3750 0x561c5d0e9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d0e9a30 con 0x561c5d0da250 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d0e9ba0 con 0x561c5d0dab70 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5d0e9d60 con 0x561c5d0e3750 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f30937bf740 10 client.0 _ll_register_callbacks cb 0x7f307c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54250/0 (socket says 172.21.15.45:54250) 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 -- 172.21.15.45:0/43144435 learned_addr learned my addr 172.21.15.45:0/43144435 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f3089d9b640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0edf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f3089d9b640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0e3750 0x561c5d0e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 -- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0e3750 msgr2=0x561c5d0e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0e3750 0x561c5d0e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 -- 172.21.15.45:0/43144435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 msgr2=0x561c5d0edf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0edf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.472+0000 7f308a59c640 1 -- 172.21.15.45:0/43144435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3080030070 con 0x561c5d0dab70 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.473+0000 7f308a59c640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f308002e010 tx=0x7f308002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619ad8bbf3ac450f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.473+0000 7f3071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.473+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3080069d80 con 0x561c5d0dab70 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.473+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f308002ce40 con 0x561c5d0dab70 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.473+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c5d107180 con 0x561c5d0dab70 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3080004250 con 0x561c5d0dab70 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 10 client.26246 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561c5cfdcc20 con 0x561c5d0dab70 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3080076070 con 0x561c5d0dab70 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f308007a030 con 0x561c5d0dab70 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 10 client.26246 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 10 client.26246 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 20 client.26246 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.474+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d0f6ba0 0x561c5d0f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.475+0000 7f30937bf740 4 client.26246 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.475+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c5d107180 con 0x561c5d0f6ba0 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.475+0000 7f3089d9b640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d0f6ba0 0x561c5d0f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.475+0000 7f3071ffb640 10 client.26246 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.475+0000 7f3089d9b640 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d0f6ba0 0x561c5d0f8fc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df2189eca9499ae0 server_cookie=d3bbe139d8933271 in_seq=0 out_seq=0 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f3071ffb640 1 -- 172.21.15.45:0/43144435 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c5d107180 con 0x561c5d0f6ba0 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f3071ffb640 10 client.26246 handle_command_reply: tid=0 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 client.26246 shutdown 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 msgr2=0x561c5d0ee470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f308002e010 tx=0x7f308002c6c0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 shutdown_connections 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x561c5d0f6ba0 0x561c5d0f8fc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f3071ffb640 0 client.26246 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561c5d0e3750 0x561c5d0e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.476+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561c5d0dab70 0x561c5d0ee470 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.477+0000 7f30937bf740 1 --2- 172.21.15.45:0/43144435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561c5d0da250 0x561c5d0edf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.477+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 >> 172.21.15.45:0/43144435 conn(0x561c5d0d4980 msgr2=0x561c5d0d32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.477+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 shutdown_connections 2024-03-12T05:15:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.477+0000 7f30937bf740 1 -- 172.21.15.45:0/43144435 wait complete. 2024-03-12T05:15:43.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.477+0000 7f30937bf740 20 client.26246 trim_cache size 0 max 16384 2024-03-12T05:15:43.478 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:43.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 msgr2=0x7f30740a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 0x7f30740a4120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3084004af0 tx=0x7f3084004a40 comp rx=0 tx=0).stop 2024-03-12T05:15:43.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 msgr2=0x7f308c06b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f308400eb70 tx=0x7f308400c6a0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 shutdown_connections 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30740a1c60 0x7f30740a4120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f308c10db50 0x7f308c06b210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f308c100b20 0x7f308c06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 --2- 172.21.15.45:0/2726466334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f308c100200 0x7f308c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.478+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 >> 172.21.15.45:0/2726466334 conn(0x7f308c0771e0 msgr2=0x7f308c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.479+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 shutdown_connections 2024-03-12T05:15:43.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.479+0000 7f3090b0c640 1 -- 172.21.15.45:0/2726466334 wait complete. 2024-03-12T05:15:43.493 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:43.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 -- 172.21.15.45:0/2561973087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc10f9a0 msgr2=0x7fabcc10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 --2- 172.21.15.45:0/2561973087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc10f9a0 0x7fabcc10d490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009900 tx=0x7fabbc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 -- 172.21.15.45:0/2561973087 shutdown_connections 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 --2- 172.21.15.45:0/2561973087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc11a960 0x7fabcc11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 --2- 172.21.15.45:0/2561973087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc10d9d0 0x7fabcc10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 --2- 172.21.15.45:0/2561973087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc10f9a0 0x7fabcc10d490 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.624+0000 7fabd22f1640 1 -- 172.21.15.45:0/2561973087 >> 172.21.15.45:0/2561973087 conn(0x7fabcc06c260 msgr2=0x7fabcc06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.625+0000 7fabd22f1640 1 -- 172.21.15.45:0/2561973087 shutdown_connections 2024-03-12T05:15:43.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.625+0000 7fabd22f1640 1 -- 172.21.15.45:0/2561973087 wait complete. 2024-03-12T05:15:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 Processor -- start 2024-03-12T05:15:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 -- start start 2024-03-12T05:15:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc10d9d0 0x7fabcc1a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 0x7fabcc1a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 0x7fabcc1ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc1ab920 con 0x7fabcc10d9d0 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc1aba90 con 0x7fabcc11a960 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd22f1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc1abc00 con 0x7fabcc1aaf30 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd088e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 0x7fabcc1ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd088e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 0x7fabcc1ab3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44154/0 (socket says 172.21.15.45:44154) 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.626+0000 7fabd088e640 1 -- 172.21.15.45:0/3345016513 learned_addr learned my addr 172.21.15.45:0/3345016513 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 0x7fabcc1a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcbfff640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc10d9d0 0x7fabcc1a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 -- 172.21.15.45:0/3345016513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 msgr2=0x7fabcc1ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 0x7fabcc1ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 -- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc10d9d0 msgr2=0x7fabcc1a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc10d9d0 0x7fabcc1a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 -- 172.21.15.45:0/3345016513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc009560 con 0x7fabcc11a960 2024-03-12T05:15:43.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.627+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 0x7fabcc1a6bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabc400eb00 tx=0x7fabc400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.628+0000 7fabc97fa640 1 -- 172.21.15.45:0/3345016513 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4010430 con 0x7fabcc11a960 2024-03-12T05:15:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.628+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc1b31e0 con 0x7fabcc11a960 2024-03-12T05:15:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.628+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc1b3690 con 0x7fabcc11a960 2024-03-12T05:15:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.628+0000 7fabc97fa640 1 -- 172.21.15.45:0/3345016513 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fabc4010a40 con 0x7fabcc11a960 2024-03-12T05:15:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.628+0000 7fabc97fa640 1 -- 172.21.15.45:0/3345016513 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4018a30 con 0x7fabcc11a960 2024-03-12T05:15:43.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.632+0000 7fabc97fa640 1 -- 172.21.15.45:0/3345016513 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fabc4018c70 con 0x7fabcc11a960 2024-03-12T05:15:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 Processor -- start 2024-03-12T05:15:43.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 -- start start 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1879630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e1884b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1834960 con 0x5650e1882750 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e17e8310 con 0x5650e1879b70 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.633+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e17e5840 con 0x5650e1879250 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabd088e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1879630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54280/0 (socket says 172.21.15.45:54280) 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabd088e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1879630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44166/0 (socket says 172.21.15.45:44166) 2024-03-12T05:15:43.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 -- 172.21.15.45:0/3190996351 learned_addr learned my addr 172.21.15.45:0/3190996351 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 -- 172.21.15.45:0/3190996351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 msgr2=0x5650e1879630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1879630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 -- 172.21.15.45:0/3190996351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 msgr2=0x5650e1884b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:15:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e1884b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.634+0000 7fabcbfff640 1 -- 172.21.15.45:0/3190996351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650e17a3060 con 0x5650e1879b70 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.635+0000 7fabcbfff640 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02c240 tx=0x7fabbc005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38170b45016e4dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.635+0000 7fabc97fa640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 0x7fabb40a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.635+0000 7fabb27fc640 1 -- 172.21.15.45:0/3190996351 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc004840 con 0x5650e1879b70 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.635+0000 7fabb27fc640 1 -- 172.21.15.45:0/3190996351 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fabbc02cc20 con 0x5650e1879b70 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.635+0000 7fabb27fc640 1 -- 172.21.15.45:0/3190996351 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc003850 con 0x5650e1879b70 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.636+0000 7fabc97fa640 1 -- 172.21.15.45:0/3345016513 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fabc4014070 con 0x7fabcc11a960 2024-03-12T05:15:43.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.636+0000 7fabd088e640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 0x7fabb40a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- 172.21.15.45:0/3190996351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 msgr2=0x5650e1882210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02c240 tx=0x7fabbc005e10 comp rx=0 tx=0).stop 2024-03-12T05:15:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd088e640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 0x7fabb40a41b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009df0 tx=0x7fabc0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- 172.21.15.45:0/3190996351 shutdown_connections 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e1884b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e1882210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/3190996351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1879630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- 172.21.15.45:0/3190996351 >> 172.21.15.45:0/3190996351 conn(0x5650e1873980 msgr2=0x5650e1873d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- 172.21.15.45:0/3190996351 shutdown_connections 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- 172.21.15.45:0/3190996351 wait complete. 2024-03-12T05:15:43.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.637+0000 7fabd4f9f740 1 -- start start 2024-03-12T05:15:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.638+0000 7fabd4f9f740 1 Processor -- start 2024-03-12T05:15:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e18801b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e17e1020 con 0x5650e1882750 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e14b4430 con 0x5650e1879b70 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1470e00 con 0x5650e1879250 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44178/0 (socket says 172.21.15.45:44178) 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabcbfff640 1 -- 172.21.15.45:0/542121018 learned_addr learned my addr 172.21.15.45:0/542121018 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabd4f9f740 10 client.0 _ll_register_callbacks cb 0x5650e1788cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.639+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabd088e640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e18801b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 -- 172.21.15.45:0/542121018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 msgr2=0x5650e187fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 -- 172.21.15.45:0/542121018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 msgr2=0x5650e18801b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e18801b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 -- 172.21.15.45:0/542121018 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc11d680 con 0x5650e1879250 2024-03-12T05:15:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabcbfff640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabbc036040 tx=0x7fabbc02c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69867617cf4dddc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:15:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc0051d0 con 0x5650e1879250 2024-03-12T05:15:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.640+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650e14e07e0 con 0x5650e1879250 2024-03-12T05:15:43.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fabbc02de40 con 0x5650e1879250 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabbc005330 con 0x5650e1879250 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 10 client.26263 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5650e17df6a0 con 0x5650e1879250 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fabbc05e080 con 0x5650e1879250 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fabbc058030 con 0x5650e1879250 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 10 client.26263 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 10 client.26263 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 20 client.26263 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.641+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18926b0 0x5650e1894b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.642+0000 7fabd4f9f740 4 client.26263 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.642+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650e14e07e0 con 0x5650e18926b0 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.642+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18926b0 0x5650e1894b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.642+0000 7fabab7fe640 10 client.26263 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:43.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.642+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18926b0 0x5650e1894b20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa5afcc46da9713d server_cookie=104774614538eee in_seq=0 out_seq=0 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.643+0000 7fabab7fe640 1 -- 172.21.15.45:0/542121018 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5650e14e07e0 con 0x5650e18926b0 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.643+0000 7fabab7fe640 10 client.26263 handle_command_reply: tid=0 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.643+0000 7fabd4f9f740 1 client.26263 shutdown 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 msgr2=0x5650e187f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabbc036040 tx=0x7fabbc02c980 comp rx=0 tx=0).stop 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 shutdown_connections 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e18801b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18926b0 0x5650e1894b20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/542121018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e187f730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 >> 172.21.15.45:0/542121018 conn(0x5650e1873980 msgr2=0x5650e18756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabab7fe640 0 client.26263 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:43.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 shutdown_connections 2024-03-12T05:15:43.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 1 -- 172.21.15.45:0/542121018 wait complete. 2024-03-12T05:15:43.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.644+0000 7fabd4f9f740 20 client.26263 trim_cache size 0 max 16384 2024-03-12T05:15:43.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 Processor -- start 2024-03-12T05:15:43.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 -- start start 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1881e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e187c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1766130 con 0x5650e1882750 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1899d50 con 0x5650e1879b70 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.671+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e186f2b0 con 0x5650e1879250 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1881e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54296/0 (socket says 172.21.15.45:54296) 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 -- 172.21.15.45:0/4120025093 learned_addr learned my addr 172.21.15.45:0/4120025093 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabd088e640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e187c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 -- 172.21.15.45:0/4120025093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 msgr2=0x5650e1881e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1881e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 -- 172.21.15.45:0/4120025093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 msgr2=0x5650e187c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e187c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 -- 172.21.15.45:0/4120025093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650e177bc20 con 0x5650e1879b70 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcbfff640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1881e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabd088e640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e187c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.672+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fabc4097310 tx=0x7fabc4000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd10690e8cbf174 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabb1ffb640 1 -- 172.21.15.45:0/4120025093 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40b2070 con 0x5650e1879b70 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabb1ffb640 1 -- 172.21.15.45:0/4120025093 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fabc40a30e0 con 0x5650e1879b70 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabb1ffb640 1 -- 172.21.15.45:0/4120025093 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40b3270 con 0x5650e1879b70 2024-03-12T05:15:43.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 -- 172.21.15.45:0/4120025093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 msgr2=0x5650e187aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fabc4097310 tx=0x7fabc4000f40 comp rx=0 tx=0).stop 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 -- 172.21.15.45:0/4120025093 shutdown_connections 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1882750 0x5650e187c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1879b70 0x5650e187aa90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/4120025093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879250 0x5650e1881e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.673+0000 7fabd4f9f740 1 -- 172.21.15.45:0/4120025093 >> 172.21.15.45:0/4120025093 conn(0x5650e1873980 msgr2=0x5650e186ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- 172.21.15.45:0/4120025093 shutdown_connections 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- 172.21.15.45:0/4120025093 wait complete. 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- start start 2024-03-12T05:15:43.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 Processor -- start 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1879250 0x5650e1889860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 0x5650e1887c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1888720 con 0x5650e1879250 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1888890 con 0x5650e1882750 2024-03-12T05:15:43.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.674+0000 7fabd4f9f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650e1888a50 con 0x5650e1879b70 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd4f9f740 10 client.0 _ll_register_callbacks cb 0x7fabc40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1879250 0x5650e1889860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54308/0 (socket says 172.21.15.45:54308) 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 -- 172.21.15.45:0/1724524200 learned_addr learned my addr 172.21.15.45:0/1724524200 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 0x5650e1887c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 -- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 msgr2=0x5650e1887c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 0x5650e1887c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 -- 172.21.15.45:0/1724524200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1879250 msgr2=0x5650e1889860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1879250 0x5650e1889860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 -- 172.21.15.45:0/1724524200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650e17df6a0 con 0x5650e1882750 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabcb7fe640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 0x5650e1887c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd088e640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fabc001fa90 tx=0x7fabc0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebf46b14478ab98 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0017070 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fabc001eaa0 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.675+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650e187f560 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.676+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabc001ec60 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.676+0000 7fabd4f9f740 10 client.26264 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.676+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fabc0009e20 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.676+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5650e187f560 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.676+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fabc0029070 con 0x5650e1882750 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 10 client.26264 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 10 client.26264 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 20 client.26264 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18ea390 0x5650e18ec7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 4 client.26264 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fabbc02cc20 con 0x5650e18ea390 2024-03-12T05:15:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabcbfff640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18ea390 0x5650e18ec7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabcbfff640 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18ea390 0x5650e18ec7b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecf87efb28d2eac3 server_cookie=5116c7f718ea8c2c in_seq=0 out_seq=0 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.677+0000 7fabab7fe640 10 client.26264 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabab7fe640 1 -- 172.21.15.45:0/1724524200 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fabbc02cc20 con 0x5650e18ea390 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabab7fe640 10 client.26264 handle_command_reply: tid=0 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 client.26264 shutdown 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 msgr2=0x5650e1888140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fabc001fa90 tx=0x7fabc0004730 comp rx=0 tx=0).stop 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 shutdown_connections 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5650e18ea390 0x5650e18ec7b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabab7fe640 0 client.26264 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650e1882750 0x5650e1888140 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650e1879b70 0x5650e1887c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 --2- 172.21.15.45:0/1724524200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650e1879250 0x5650e1889860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 >> 172.21.15.45:0/1724524200 conn(0x5650e1873980 msgr2=0x5650e187aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 shutdown_connections 2024-03-12T05:15:43.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 1 -- 172.21.15.45:0/1724524200 wait complete. 2024-03-12T05:15:43.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.678+0000 7fabd4f9f740 20 client.26264 trim_cache size 0 max 16384 2024-03-12T05:15:43.678 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:43.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.679+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 msgr2=0x7fabb40a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 0x7fabb40a41b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009df0 tx=0x7fabc0005f90 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 msgr2=0x7fabcc1a6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 0x7fabcc1a6bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabc400eb00 tx=0x7fabc400c2d0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 shutdown_connections 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fabb40a1cf0 0x7fabb40a41b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabcc1aaf30 0x7fabcc1ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabcc11a960 0x7fabcc1a6bb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 --2- 172.21.15.45:0/3345016513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fabcc10d9d0 0x7fabcc1a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 >> 172.21.15.45:0/3345016513 conn(0x7fabcc06c260 msgr2=0x7fabcc119540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.680+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 shutdown_connections 2024-03-12T05:15:43.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.681+0000 7fabd22f1640 1 -- 172.21.15.45:0/3345016513 wait complete. 2024-03-12T05:15:43.694 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:15:43.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:15:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.861+0000 7f16fe242640 1 -- 172.21.15.45:0/3329725628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 msgr2=0x7f16f80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.861+0000 7f16fe242640 1 --2- 172.21.15.45:0/3329725628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f80689b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009930 tx=0x7f16ec02bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 -- 172.21.15.45:0/3329725628 shutdown_connections 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 --2- 172.21.15.45:0/3329725628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f8106430 0x7f16f810e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 --2- 172.21.15.45:0/3329725628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f8068f80 0x7f16f8105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 --2- 172.21.15.45:0/3329725628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f80689b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 -- 172.21.15.45:0/3329725628 >> 172.21.15.45:0/3329725628 conn(0x7f16f8077180 msgr2=0x7f16f8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.862+0000 7f16fe242640 1 -- 172.21.15.45:0/3329725628 shutdown_connections 2024-03-12T05:15:43.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.863+0000 7f16fe242640 1 -- 172.21.15.45:0/3329725628 wait complete. 2024-03-12T05:15:43.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.864+0000 7f16fe242640 1 Processor -- start 2024-03-12T05:15:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 -- start start 2024-03-12T05:15:43.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f810dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f8106430 0x7f16f810ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819fa90 con 0x7f16f80685d0 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819fc00 con 0x7f16f8106430 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16fe242640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819fdc0 con 0x7f16f810b370 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44202/0 (socket says 172.21.15.45:44202) 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.865+0000 7f16f7fff640 1 -- 172.21.15.45:0/3093857340 learned_addr learned my addr 172.21.15.45:0/3093857340 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.866+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f810dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 -- 172.21.15.45:0/3093857340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 msgr2=0x7f16f810b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 -- 172.21.15.45:0/3093857340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f8106430 msgr2=0x7f16f810ae30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f8106430 0x7f16f810ae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 -- 172.21.15.45:0/3093857340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec009560 con 0x7f16f80685d0 2024-03-12T05:15:43.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.867+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f810dd90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f16ec036040 tx=0x7f16ec005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.870+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.870+0000 7f16f4ff9640 1 -- 172.21.15.45:0/3093857340 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec004350 con 0x7f16f80685d0 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.870+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f81a0040 con 0x7f16f80685d0 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.870+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a0520 con 0x7f16f80685d0 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.871+0000 7f16f4ff9640 1 -- 172.21.15.45:0/3093857340 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16ec0044b0 con 0x7f16f80685d0 2024-03-12T05:15:43.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.871+0000 7f16f4ff9640 1 -- 172.21.15.45:0/3093857340 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec005320 con 0x7f16f80685d0 2024-03-12T05:15:43.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.875+0000 7f16f4ff9640 1 -- 172.21.15.45:0/3093857340 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f16ec005480 con 0x7f16f80685d0 2024-03-12T05:15:43.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.875+0000 7f1700e0b740 1 Processor -- start 2024-03-12T05:15:43.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 -- start start 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d52210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d54b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0d04960 con 0x55cee0d52750 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0cb8310 con 0x55cee0d49250 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f1700e0b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0cb5840 con 0x55cee0d49b70 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54342/0 (socket says 172.21.15.45:54342) 2024-03-12T05:15:43.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.876+0000 7f16f7fff640 1 -- 172.21.15.45:0/3119101591 learned_addr learned my addr 172.21.15.45:0/3119101591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d52210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 -- 172.21.15.45:0/3119101591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 msgr2=0x55cee0d52210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d52210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 -- 172.21.15.45:0/3119101591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 msgr2=0x55cee0d54b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d54b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 -- 172.21.15.45:0/3119101591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee0c73060 con 0x55cee0d49250 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16f000f390 tx=0x7f16f000f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d5d5ee0a4a1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.877+0000 7f16ddffb640 1 -- 172.21.15.45:0/3119101591 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0011700 con 0x55cee0d49250 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.878+0000 7f16ddffb640 1 -- 172.21.15.45:0/3119101591 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16f0011d10 con 0x55cee0d49250 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.878+0000 7f16ddffb640 1 -- 172.21.15.45:0/3119101591 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f001be40 con 0x55cee0d49250 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.878+0000 7f16f4ff9640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 0x7f16e00a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.878+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 0x7f16e00a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.878+0000 7f16f4ff9640 1 -- 172.21.15.45:0/3093857340 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f16ec02de40 con 0x7f16f80685d0 2024-03-12T05:15:43.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 -- 172.21.15.45:0/3119101591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 msgr2=0x55cee0d49630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f16f000f390 tx=0x7f16f000f850 comp rx=0 tx=0).stop 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f16f77fe640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 0x7f16e00a41b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0ed500 tx=0x7f16ec02bc30 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 -- 172.21.15.45:0/3119101591 shutdown_connections 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d54b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d52210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3119101591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d49630 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 -- 172.21.15.45:0/3119101591 >> 172.21.15.45:0/3119101591 conn(0x55cee0d43980 msgr2=0x55cee0d43d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.879+0000 7f1700e0b740 1 -- 172.21.15.45:0/3119101591 shutdown_connections 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.880+0000 7f1700e0b740 1 -- 172.21.15.45:0/3119101591 wait complete. 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.880+0000 7f1700e0b740 1 -- start start 2024-03-12T05:15:43.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.880+0000 7f1700e0b740 1 Processor -- start 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.881+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.881+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49b70 0x55cee0d4fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.881+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.881+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54348/0 (socket says 172.21.15.45:54348) 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.881+0000 7f16f7fff640 1 -- 172.21.15.45:0/2920104853 learned_addr learned my addr 172.21.15.45:0/2920104853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d52750 0x55cee0d50170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0cb1020 con 0x55cee0d49b70 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0984430 con 0x55cee0d49250 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0940e00 con 0x55cee0d52750 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49b70 0x55cee0d4fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f1700e0b740 10 client.0 _ll_register_callbacks cb 0x55cee0c58cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d52750 0x55cee0d50170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 -- 172.21.15.45:0/2920104853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d52750 msgr2=0x55cee0d50170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d52750 0x55cee0d50170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 -- 172.21.15.45:0/2920104853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49b70 msgr2=0x55cee0d4fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49b70 0x55cee0d4fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.882+0000 7f16f7fff640 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f0009ca0 con 0x55cee0d49250 2024-03-12T05:15:43.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.883+0000 7f16f7fff640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009370 tx=0x7f16f00092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b925516d9e3c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.883+0000 7f16d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:43.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.883+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f001f030 con 0x55cee0d49250 2024-03-12T05:15:43.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.884+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cee09b07e0 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.884+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f16f0020070 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.884+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f0004b00 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.884+0000 7f1700e0b740 10 client.26273 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.884+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cee0caf6a0 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16f0034080 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16f00051d0 con 0x55cee0d49250 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 10 client.26273 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 10 client.26273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 20 client.26273 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0d62ae0 0x55cee0d64f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 4 client.26273 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:43.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cee09ed720 con 0x55cee0d62ae0 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.885+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0d62ae0 0x55cee0d64f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.886+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0d62ae0 0x55cee0d64f00 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36901a4e031e3fd0 server_cookie=96f91055175c5c6 in_seq=0 out_seq=0 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.886+0000 7f16d6ffd640 10 client.26273 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.887+0000 7f16d6ffd640 1 -- 172.21.15.45:0/2920104853 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cee09ed720 con 0x55cee0d62ae0 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.887+0000 7f16d6ffd640 10 client.26273 handle_command_reply: tid=0 2024-03-12T05:15:43.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.887+0000 7f1700e0b740 1 client.26273 shutdown 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 msgr2=0x55cee0d4f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009370 tx=0x7f16f00092a0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 shutdown_connections 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0d62ae0 0x55cee0d64f00 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49b70 0x55cee0d4fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d52750 0x55cee0d50170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 --2- 172.21.15.45:0/2920104853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49250 0x55cee0d4f6f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f16d6ffd640 0 client.26273 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.888+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 >> 172.21.15.45:0/2920104853 conn(0x55cee0d43980 msgr2=0x55cee0d456d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.889+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 shutdown_connections 2024-03-12T05:15:43.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.889+0000 7f1700e0b740 1 -- 172.21.15.45:0/2920104853 wait complete. 2024-03-12T05:15:43.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.889+0000 7f1700e0b740 20 client.26273 trim_cache size 0 max 16384 2024-03-12T05:15:43.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 Processor -- start 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 -- start start 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49250 0x55cee0d51e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d52750 0x55cee0d4e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0c36130 con 0x55cee0d49250 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0d48eb0 con 0x55cee0d52750 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f1700e0b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0d4ea40 con 0x55cee0d49b70 2024-03-12T05:15:43.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44228/0 (socket says 172.21.15.45:44228) 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.910+0000 7f16f6ffd640 1 -- 172.21.15.45:0/3829252172 learned_addr learned my addr 172.21.15.45:0/3829252172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d52750 0x55cee0d4e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f7fff640 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49250 0x55cee0d51e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 -- 172.21.15.45:0/3829252172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d52750 msgr2=0x55cee0d4e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d52750 0x55cee0d4e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 -- 172.21.15.45:0/3829252172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49250 msgr2=0x55cee0d51e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49250 0x55cee0d51e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 -- 172.21.15.45:0/3829252172 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee0c4bc20 con 0x55cee0d49b70 2024-03-12T05:15:43.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f16f810c680 tx=0x7f16e8009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9c551d9ac069a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16dd7fa640 1 -- 172.21.15.45:0/3829252172 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8016040 con 0x55cee0d49b70 2024-03-12T05:15:43.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16dd7fa640 1 -- 172.21.15.45:0/3829252172 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16e80044d0 con 0x55cee0d49b70 2024-03-12T05:15:43.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.911+0000 7f16dd7fa640 1 -- 172.21.15.45:0/3829252172 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8010430 con 0x55cee0d49b70 2024-03-12T05:15:43.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 -- 172.21.15.45:0/3829252172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 msgr2=0x55cee0d4dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f16f810c680 tx=0x7f16e8009530 comp rx=0 tx=0).stop 2024-03-12T05:15:43.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 -- 172.21.15.45:0/3829252172 shutdown_connections 2024-03-12T05:15:43.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d52750 0x55cee0d4e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49b70 0x55cee0d4dfc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 --2- 172.21.15.45:0/3829252172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d49250 0x55cee0d51e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 -- 172.21.15.45:0/3829252172 >> 172.21.15.45:0/3829252172 conn(0x55cee0d43980 msgr2=0x55cee0d69840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 -- 172.21.15.45:0/3829252172 shutdown_connections 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.914+0000 7f1700e0b740 1 -- 172.21.15.45:0/3829252172 wait complete. 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.915+0000 7f1700e0b740 1 -- start start 2024-03-12T05:15:43.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.915+0000 7f1700e0b740 1 Processor -- start 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49250 0x55cee0d582f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d58db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0db9190 con 0x55cee0d52750 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0db9300 con 0x55cee0d49b70 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee0dca4d0 con 0x55cee0d49250 2024-03-12T05:15:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f1700e0b740 10 client.0 _ll_register_callbacks cb 0x7f16f0009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:43.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54368/0 (socket says 172.21.15.45:54368) 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f16f6ffd640 1 -- 172.21.15.45:0/360067520 learned_addr learned my addr 172.21.15.45:0/360067520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f16f7fff640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49250 0x55cee0d582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.916+0000 7f16f7fff640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d58db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 -- 172.21.15.45:0/360067520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49250 msgr2=0x55cee0d582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49250 0x55cee0d582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 -- 172.21.15.45:0/360067520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 msgr2=0x55cee0d58db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d58db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 -- 172.21.15.45:0/360067520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e8009880 con 0x55cee0d49b70 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16f6ffd640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009a10 tx=0x7f16e8003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8efb6db3fe04ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:43.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.917+0000 7f16d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8018c90 con 0x55cee0d49b70 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f16e8018560 con 0x55cee0d49b70 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16f0011d10 con 0x55cee0d49b70 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e8018df0 con 0x55cee0d49b70 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f1700e0b740 10 client.26279 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:43.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.918+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cee0c718c0 con 0x55cee0d49b70 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.919+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f16f0011d10 con 0x55cee0d49b70 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16e802c670 con 0x55cee0d49b70 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 10 client.26279 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 20 client.26279 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:43.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0dcbc30 0x55cee0dce050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 4 client.26279 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:15:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cee0a3c890 con 0x55cee0dcbc30 2024-03-12T05:15:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.920+0000 7f16f7fff640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0dcbc30 0x55cee0dce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:43.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.921+0000 7f16d6ffd640 10 client.26279 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:43.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.921+0000 7f16f7fff640 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0dcbc30 0x55cee0dce050 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a383462f72301 server_cookie=486702902417be7d in_seq=0 out_seq=0 2024-03-12T05:15:43.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.921+0000 7f16d6ffd640 1 -- 172.21.15.45:0/360067520 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cee0a3c890 con 0x55cee0dcbc30 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.921+0000 7f16d6ffd640 10 client.26279 handle_command_reply: tid=0 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.921+0000 7f1700e0b740 1 client.26279 shutdown 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 msgr2=0x55cee0d58830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009a10 tx=0x7f16e8003d50 comp rx=0 tx=0).stop 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 shutdown_connections 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55cee0dcbc30 0x55cee0dce050 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cee0d52750 0x55cee0d58db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cee0d49b70 0x55cee0d58830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 --2- 172.21.15.45:0/360067520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cee0d49250 0x55cee0d582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f16d6ffd640 0 client.26279 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 >> 172.21.15.45:0/360067520 conn(0x55cee0d43980 msgr2=0x55cee0d4db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 shutdown_connections 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 1 -- 172.21.15.45:0/360067520 wait complete. 2024-03-12T05:15:43.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.922+0000 7f1700e0b740 20 client.26279 trim_cache size 0 max 16384 2024-03-12T05:15:43.920 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 msgr2=0x7f16e00a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 0x7f16e00a41b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0ed500 tx=0x7f16ec02bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 msgr2=0x7f16f810dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f810dd90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f16ec036040 tx=0x7f16ec005dc0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 shutdown_connections 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f16e00a1cf0 0x7f16e00a41b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16f810b370 0x7f16f810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16f8106430 0x7f16f810ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 --2- 172.21.15.45:0/3093857340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16f80685d0 0x7f16f810dd90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.923+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 >> 172.21.15.45:0/3093857340 conn(0x7f16f8077180 msgr2=0x7f16f80fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.924+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 shutdown_connections 2024-03-12T05:15:43.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:43.924+0000 7f16fe242640 1 -- 172.21.15.45:0/3093857340 wait complete. 2024-03-12T05:15:44.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:43 smithi045 ceph-mon[32174]: pgmap v148: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:44.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:43 smithi130 ceph-mon[35398]: pgmap v148: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:44.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:43 smithi116 ceph-mon[35486]: pgmap v148: 97 pgs: 97 active+clean; 2.5 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:15:44.940 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:15:44.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:15:44.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:15:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 220.0 s, 1319.2 tps, lat 75.759 ms stddev 39.457 2024-03-12T05:15:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 -- 172.21.15.45:0/2740732437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4560105170 msgr2=0x7f456010cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 --2- 172.21.15.45:0/2740732437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4560105170 0x7f456010cd40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f455800aca0 tx=0x7f455802c2e0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 -- 172.21.15.45:0/2740732437 shutdown_connections 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 --2- 172.21.15.45:0/2740732437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4560105170 0x7f456010cd40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 --2- 172.21.15.45:0/2740732437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4560100a50 0x7f4560104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 --2- 172.21.15.45:0/2740732437 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f45601000a0 0x7f4560100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 -- 172.21.15.45:0/2740732437 >> 172.21.15.45:0/2740732437 conn(0x7f4560076c30 msgr2=0x7f45600fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 -- 172.21.15.45:0/2740732437 shutdown_connections 2024-03-12T05:15:45.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.119+0000 7f4567055640 1 -- 172.21.15.45:0/2740732437 wait complete. 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.120+0000 7f4567055640 1 Processor -- start 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 -- start start 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4560105170 0x7f4560199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 0x7f456019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456019e930 con 0x7f4560105170 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456019eaa0 con 0x7f45601000a0 2024-03-12T05:15:45.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4567055640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456019ec10 con 0x7f456019df40 2024-03-12T05:15:45.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4564df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4564df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54394/0 (socket says 172.21.15.45:54394) 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f4564df1640 1 -- 172.21.15.45:0/747561413 learned_addr learned my addr 172.21.15.45:0/747561413 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f45655f2640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 0x7f456019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.121+0000 7f455ffff640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4560105170 0x7f4560199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 -- 172.21.15.45:0/747561413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 msgr2=0x7f456019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 0x7f456019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 -- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4560105170 msgr2=0x7f4560199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4560105170 0x7f4560199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 -- 172.21.15.45:0/747561413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455800a920 con 0x7f45601000a0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f45655f2640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 0x7f456019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4564df1640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f455400e9f0 tx=0x7f455400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f455dffb640 1 -- 172.21.15.45:0/747561413 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554009700 con 0x7f45601000a0 2024-03-12T05:15:45.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f455dffb640 1 -- 172.21.15.45:0/747561413 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f455400cd80 con 0x7f45601000a0 2024-03-12T05:15:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f455dffb640 1 -- 172.21.15.45:0/747561413 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554017800 con 0x7f45601000a0 2024-03-12T05:15:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.122+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45601a61f0 con 0x7f45601000a0 2024-03-12T05:15:45.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.123+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45601a65f0 con 0x7f45601000a0 2024-03-12T05:15:45.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.125+0000 7f455dffb640 1 -- 172.21.15.45:0/747561413 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f45540179a0 con 0x7f45601000a0 2024-03-12T05:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.125+0000 7f455dffb640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 0x7f45500a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 Processor -- start 2024-03-12T05:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f455dffb640 1 -- 172.21.15.45:0/747561413 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f4554014070 con 0x7f45601000a0 2024-03-12T05:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 -- start start 2024-03-12T05:15:45.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f455ffff640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 0x7f45500a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8aee930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8af9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8a5cf90 con 0x5569d8aeef00 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8a5a4c0 con 0x5569d8aee550 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4569c22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8a55ca0 con 0x5569d8af7a80 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f455ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f4564df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8aee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f455ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33974/0 (socket says 172.21.15.45:33974) 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f455ffff640 1 -- 172.21.15.45:0/3624850358 learned_addr learned my addr 172.21.15.45:0/3624850358 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.126+0000 7f45655f2640 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8af9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 -- 172.21.15.45:0/3624850358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 msgr2=0x5569d8af9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8af9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 -- 172.21.15.45:0/3624850358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 msgr2=0x5569d8af74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 -- 172.21.15.45:0/3624850358 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569d8a180b0 con 0x5569d8aee550 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f45655f2640 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8af9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f4564df1640 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8aee930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45540cb3b0 tx=0x7f455409bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb515d00a81fe728 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f455ffff640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 0x7f45500a41f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f454c013910 tx=0x7f454c009aa0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.127+0000 7f454affd640 1 -- 172.21.15.45:0/3624850358 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455409f920 con 0x5569d8aee550 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.128+0000 7f454affd640 1 -- 172.21.15.45:0/3624850358 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45540a3f70 con 0x5569d8aee550 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.128+0000 7f454affd640 1 -- 172.21.15.45:0/3624850358 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45540cb790 con 0x5569d8aee550 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- 172.21.15.45:0/3624850358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 msgr2=0x5569d8aee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8aee930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45540cb3b0 tx=0x7f455409bed0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- 172.21.15.45:0/3624850358 shutdown_connections 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8af9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 --2- 172.21.15.45:0/3624850358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8aee930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- 172.21.15.45:0/3624850358 >> 172.21.15.45:0/3624850358 conn(0x5569d8ae8a10 msgr2=0x5569d8ae8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- 172.21.15.45:0/3624850358 shutdown_connections 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- 172.21.15.45:0/3624850358 wait complete. 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.129+0000 7f4569c22740 1 -- start start 2024-03-12T05:15:45.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.130+0000 7f4569c22740 1 Processor -- start 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 0x5569d8af4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 0x5569d8af4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8af7a80 0x5569d8af54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4564df1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 0x5569d8af4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4564df1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 0x5569d8af4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44286/0 (socket says 172.21.15.45:44286) 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4564df1640 1 -- 172.21.15.45:0/3383760754 learned_addr learned my addr 172.21.15.45:0/3383760754 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f455ffff640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 0x5569d8af4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d89faf40 con 0x5569d8af7a80 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d87294b0 con 0x5569d8aeef00 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d89db670 con 0x5569d8aee550 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.131+0000 7f4569c22740 10 client.0 _ll_register_callbacks cb 0x5569d8a48f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 -- 172.21.15.45:0/3383760754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 msgr2=0x5569d8af4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 0x5569d8af4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 -- 172.21.15.45:0/3383760754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8af7a80 msgr2=0x5569d8af54e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8af7a80 0x5569d8af54e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45601a65f0 con 0x5569d8aeef00 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f455ffff640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 0x5569d8af4fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x5569d8ae9c90 tx=0x7f454c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7867fa5483295dad server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569d8a2ff40 con 0x5569d8aeef00 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f453ffff640 10 client.26291 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:45.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.132+0000 7f453ffff640 1 -- 172.21.15.45:0/3383760754 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c01fac0 con 0x5569d8aeef00 2024-03-12T05:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.133+0000 7f453ffff640 1 -- 172.21.15.45:0/3383760754 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f454c025030 con 0x5569d8aeef00 2024-03-12T05:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.133+0000 7f453ffff640 1 -- 172.21.15.45:0/3383760754 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f454c01fc20 con 0x5569d8aeef00 2024-03-12T05:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.133+0000 7f4569c22740 10 client.26291 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.133+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5569d8a54320 con 0x5569d8aeef00 2024-03-12T05:15:45.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.133+0000 7f453ffff640 1 -- 172.21.15.45:0/3383760754 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f454c005020 con 0x5569d8aeef00 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 10 client.26291 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 10 client.26291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 20 client.26291 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b0a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 4 client.26291 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569d8a2ff40 con 0x5569d8b07fb0 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4564df1640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b0a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f453ffff640 10 client.26291 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.134+0000 7f4564df1640 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b0a3e0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3a4b243b4fc49 server_cookie=621e6cb7ee6f006e in_seq=0 out_seq=0 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.135+0000 7f453ffff640 1 -- 172.21.15.45:0/3383760754 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5569d8a2ff40 con 0x5569d8b07fb0 2024-03-12T05:15:45.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.135+0000 7f453ffff640 10 client.26291 handle_command_reply: tid=0 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.135+0000 7f4569c22740 1 client.26291 shutdown 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.135+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 msgr2=0x5569d8af4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.135+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 0x5569d8af4fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x5569d8ae9c90 tx=0x7f454c01d000 comp rx=0 tx=0).stop 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 shutdown_connections 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b0a3e0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f453ffff640 0 client.26291 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8af7a80 0x5569d8af54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aeef00 0x5569d8af4fa0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 --2- 172.21.15.45:0/3383760754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aee550 0x5569d8af4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 >> 172.21.15.45:0/3383760754 conn(0x5569d8ae8a10 msgr2=0x5569d8aea970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 shutdown_connections 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 1 -- 172.21.15.45:0/3383760754 wait complete. 2024-03-12T05:15:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.136+0000 7f4569c22740 20 client.26291 trim_cache size 0 max 16384 2024-03-12T05:15:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 Processor -- start 2024-03-12T05:15:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 -- start start 2024-03-12T05:15:45.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8aea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8aef950 con 0x5569d8aeef00 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8aeecd0 con 0x5569d8aee550 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.170+0000 7f4569c22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8af0680 con 0x5569d8af7a80 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54430/0 (socket says 172.21.15.45:54430) 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 -- 172.21.15.45:0/3104015871 learned_addr learned my addr 172.21.15.45:0/3104015871 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f45655f2640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8aea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f455ffff640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 -- 172.21.15.45:0/3104015871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 msgr2=0x5569d8aea050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8aea050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 -- 172.21.15.45:0/3104015871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 msgr2=0x5569d8af2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 -- 172.21.15.45:0/3104015871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569d89f0c00 con 0x5569d8aee550 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f455ffff640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f4564df1640 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4554097c30 tx=0x7f4554097470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8347e6d63dea8e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.171+0000 7f454a7fc640 1 -- 172.21.15.45:0/3104015871 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455409f920 con 0x5569d8aee550 2024-03-12T05:15:45.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.172+0000 7f454a7fc640 1 -- 172.21.15.45:0/3104015871 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45540b32d0 con 0x5569d8aee550 2024-03-12T05:15:45.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.172+0000 7f454a7fc640 1 -- 172.21.15.45:0/3104015871 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554098020 con 0x5569d8aee550 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 -- 172.21.15.45:0/3104015871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 msgr2=0x5569d8af1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4554097c30 tx=0x7f4554097470 comp rx=0 tx=0).stop 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 -- 172.21.15.45:0/3104015871 shutdown_connections 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aeef00 0x5569d8af2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8af7a80 0x5569d8aea050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 --2- 172.21.15.45:0/3104015871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8aee550 0x5569d8af1ce0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 -- 172.21.15.45:0/3104015871 >> 172.21.15.45:0/3104015871 conn(0x5569d8ae8a10 msgr2=0x5569d8af3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 -- 172.21.15.45:0/3104015871 shutdown_connections 2024-03-12T05:15:45.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.173+0000 7f4569c22740 1 -- 172.21.15.45:0/3104015871 wait complete. 2024-03-12T05:15:45.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.174+0000 7f4569c22740 1 -- start start 2024-03-12T05:15:45.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.174+0000 7f4569c22740 1 Processor -- start 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 0x5569d8b58900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aeef00 0x5569d8b58e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 0x5569d8afde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8afe410 con 0x5569d8aee550 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8afe580 con 0x5569d8af7a80 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569d8afe740 con 0x5569d8aeef00 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4569c22740 10 client.0 _ll_register_callbacks cb 0x7f4554097a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4564df1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 0x5569d8b58900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4564df1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 0x5569d8b58900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33996/0 (socket says 172.21.15.45:33996) 2024-03-12T05:15:45.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f4564df1640 1 -- 172.21.15.45:0/1603799998 learned_addr learned my addr 172.21.15.45:0/1603799998 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f455ffff640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aeef00 0x5569d8b58e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 0x5569d8afde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 -- 172.21.15.45:0/1603799998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aeef00 msgr2=0x5569d8b58e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aeef00 0x5569d8b58e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 -- 172.21.15.45:0/1603799998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 msgr2=0x5569d8b58900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 0x5569d8b58900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.175+0000 7f45655f2640 1 -- 172.21.15.45:0/1603799998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569d8a180b0 con 0x5569d8af7a80 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.176+0000 7f45655f2640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 0x5569d8afde50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f455802c8f0 tx=0x7f4558033b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f947893029e54b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.176+0000 7f453ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:45.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558046070 con 0x5569d8af7a80 2024-03-12T05:15:45.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f455803e650 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569d8a7b900 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f455803e830 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f4569c22740 10 client.26297 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5569d8a16910 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.177+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4558057080 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4558051030 con 0x5569d8af7a80 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 10 client.26297 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 10 client.26297 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 20 client.26297 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b62850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 4 client.26297 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:15:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.178+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5569d8a7b900 con 0x5569d8b07fb0 2024-03-12T05:15:45.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.180+0000 7f4564df1640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b62850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.180+0000 7f4564df1640 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b62850 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a99532d25e2b097 server_cookie=5ff0bf31ed6f4376 in_seq=0 out_seq=0 2024-03-12T05:15:45.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.180+0000 7f453ffff640 10 client.26297 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f453ffff640 1 -- 172.21.15.45:0/1603799998 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5569d8a7b900 con 0x5569d8b07fb0 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f453ffff640 10 client.26297 handle_command_reply: tid=0 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 client.26297 shutdown 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 msgr2=0x5569d8afde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 0x5569d8afde50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f455802c8f0 tx=0x7f4558033b10 comp rx=0 tx=0).stop 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 shutdown_connections 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5569d8b07fb0 0x5569d8b62850 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5569d8af7a80 0x5569d8afde50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5569d8aeef00 0x5569d8b58e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 --2- 172.21.15.45:0/1603799998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5569d8aee550 0x5569d8b58900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 >> 172.21.15.45:0/1603799998 conn(0x5569d8ae8a10 msgr2=0x5569d8af52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f453ffff640 0 client.26297 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 shutdown_connections 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.182+0000 7f4569c22740 1 -- 172.21.15.45:0/1603799998 wait complete. 2024-03-12T05:15:45.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.183+0000 7f4569c22740 20 client.26297 trim_cache size 0 max 16384 2024-03-12T05:15:45.180 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:45.180 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:15:45.180 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "1ef972c7-0d65-45e9-9a20-7785287335d4", 2024-03-12T05:15:45.180 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:15:45.180 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.184+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 msgr2=0x7f45500a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.184+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 0x7f45500a41f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f454c013910 tx=0x7f454c009aa0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.184+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 msgr2=0x7f4560199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.184+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f455400e9f0 tx=0x7f455400eeb0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 shutdown_connections 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45500a1d30 0x7f45500a41f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4560105170 0x7f4560199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f456019df40 0x7f456019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 --2- 172.21.15.45:0/747561413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45601000a0 0x7f4560199930 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 >> 172.21.15.45:0/747561413 conn(0x7f4560076c30 msgr2=0x7f45601021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 shutdown_connections 2024-03-12T05:15:45.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.185+0000 7f4567055640 1 -- 172.21.15.45:0/747561413 wait complete. 2024-03-12T05:15:45.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:15:45.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:15:45.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 -- 172.21.15.45:0/1876817821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100c20 msgr2=0x7f6634104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 --2- 172.21.15.45:0/1876817821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100c20 0x7f6634104c60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6624009930 tx=0x7f662402bc30 comp rx=0 tx=0).stop 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 -- 172.21.15.45:0/1876817821 shutdown_connections 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 --2- 172.21.15.45:0/1876817821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6634105340 0x7f663410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 --2- 172.21.15.45:0/1876817821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100c20 0x7f6634104c60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 --2- 172.21.15.45:0/1876817821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634100270 0x7f6634100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 -- 172.21.15.45:0/1876817821 >> 172.21.15.45:0/1876817821 conn(0x7f6634077230 msgr2=0x7f6634077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 -- 172.21.15.45:0/1876817821 shutdown_connections 2024-03-12T05:15:45.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.414+0000 7f6639081640 1 -- 172.21.15.45:0/1876817821 wait complete. 2024-03-12T05:15:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.416+0000 7f6639081640 1 Processor -- start 2024-03-12T05:15:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.416+0000 7f6639081640 1 -- start start 2024-03-12T05:15:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.416+0000 7f6639081640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.416+0000 7f6639081640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634105340 0x7f6634197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6639081640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f663419bf50 0x7f663419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6639081640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663419c940 con 0x7f663419bf50 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6639081640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663419cab0 con 0x7f6634100270 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6639081640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663419cc20 con 0x7f6634105340 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54452/0 (socket says 172.21.15.45:54452) 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 -- 172.21.15.45:0/104900165 learned_addr learned my addr 172.21.15.45:0/104900165 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f663259c640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634105340 0x7f6634197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f663359e640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f663419bf50 0x7f663419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 -- 172.21.15.45:0/104900165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634105340 msgr2=0x7f6634197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634105340 0x7f6634197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 -- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f663419bf50 msgr2=0x7f663419c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f663419bf50 0x7f663419c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.417+0000 7f6632d9d640 1 -- 172.21.15.45:0/104900165 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624009560 con 0x7f6634100270 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f6632d9d640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f662800e9f0 tx=0x7f662800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f661bfff640 1 -- 172.21.15.45:0/104900165 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628009700 con 0x7f6634100270 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f661bfff640 1 -- 172.21.15.45:0/104900165 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f662800cd80 con 0x7f6634100270 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f661bfff640 1 -- 172.21.15.45:0/104900165 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628018840 con 0x7f6634100270 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66341a4200 con 0x7f6634100270 2024-03-12T05:15:45.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.418+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66341a4570 con 0x7f6634100270 2024-03-12T05:15:45.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.421+0000 7f661bfff640 1 -- 172.21.15.45:0/104900165 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f66280189a0 con 0x7f6634100270 2024-03-12T05:15:45.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.421+0000 7f663bc4f740 1 Processor -- start 2024-03-12T05:15:45.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.421+0000 7f663bc4f740 1 -- start start 2024-03-12T05:15:45.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e6fe0 0x55d5307ef680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 0x55d5307f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d530751280 con 0x55d5307e66c0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d53074e7b0 con 0x55d5307efbc0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663bc4f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d530749f90 con 0x55d5307e6fe0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34032/0 (socket says 172.21.15.45:34032) 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 -- 172.21.15.45:0/2212149190 learned_addr learned my addr 172.21.15.45:0/2212149190 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663259c640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 0x55d5307f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f6632d9d640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e6fe0 0x55d5307ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 -- 172.21.15.45:0/2212149190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e6fe0 msgr2=0x55d5307ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e6fe0 0x55d5307ef680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 -- 172.21.15.45:0/2212149190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 msgr2=0x55d5307f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 0x55d5307f1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.422+0000 7f663359e640 1 -- 172.21.15.45:0/2212149190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d530708570 con 0x55d5307e66c0 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f663259c640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 0x55d5307f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f661bfff640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 0x7f661c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f663359e640 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f662c009f90 tx=0x7f662c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6312f6c5d7c41375 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f6632d9d640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 0x7f661c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f661bfff640 1 -- 172.21.15.45:0/104900165 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6628014070 con 0x7f6634100270 2024-03-12T05:15:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f66197fa640 1 -- 172.21.15.45:0/2212149190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c011700 con 0x55d5307e66c0 2024-03-12T05:15:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f66197fa640 1 -- 172.21.15.45:0/2212149190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f662c015070 con 0x55d5307e66c0 2024-03-12T05:15:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.423+0000 7f66197fa640 1 -- 172.21.15.45:0/2212149190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c019b50 con 0x55d5307e66c0 2024-03-12T05:15:45.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 -- 172.21.15.45:0/2212149190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 msgr2=0x55d5307e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f6632d9d640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 0x7f661c0a4160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55d5307e6190 tx=0x7f6628009310 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f662c009f90 tx=0x7f662c010860 comp rx=0 tx=0).stop 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 -- 172.21.15.45:0/2212149190 shutdown_connections 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307efbc0 0x55d5307f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e6fe0 0x55d5307ef680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2212149190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307e66c0 0x55d5307e6aa0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.424+0000 7f663bc4f740 1 -- 172.21.15.45:0/2212149190 >> 172.21.15.45:0/2212149190 conn(0x55d5307e0da0 msgr2=0x55d5307e1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.425+0000 7f663bc4f740 1 -- 172.21.15.45:0/2212149190 shutdown_connections 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.425+0000 7f663bc4f740 1 -- 172.21.15.45:0/2212149190 wait complete. 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.425+0000 7f663bc4f740 1 -- start start 2024-03-12T05:15:45.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.425+0000 7f663bc4f740 1 Processor -- start 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307eca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d53041d430 con 0x55d5307efbc0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5303d9e00 con 0x55d5307e6fe0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5306cefd0 con 0x55d5307e66c0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54474/0 (socket says 172.21.15.45:54474) 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663259c640 1 -- 172.21.15.45:0/352605060 learned_addr learned my addr 172.21.15.45:0/352605060 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663bc4f740 10 client.0 _ll_register_callbacks cb 0x55d530724fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.426+0000 7f663359e640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663359e640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 -- 172.21.15.45:0/352605060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 msgr2=0x55d5307eca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307eca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 -- 172.21.15.45:0/352605060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 msgr2=0x55d5307ed4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307ed4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 -- 172.21.15.45:0/352605060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c009bf0 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663359e640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663259c640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6624005ab0 tx=0x7f6624005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba73921a892e979a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f6616ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624005220 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f66240046c0 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.427+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5304497e0 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6624004880 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 10 client.26309 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d530748610 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d5304497e0 con 0x55d5307e6fe0 2024-03-12T05:15:45.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6624045020 con 0x55d5307e6fe0 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 10 client.26309 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 10 client.26309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 20 client.26309 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5307ffd10 0x55d530802170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 4 client.26309 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d530486720 con 0x55d5307ffd10 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.428+0000 7f663359e640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5307ffd10 0x55d530802170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.429+0000 7f6616ff5640 10 client.26309 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.429+0000 7f663359e640 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5307ffd10 0x55d530802170 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb58907019867172 server_cookie=ed006da9ba4134bc in_seq=0 out_seq=0 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f6616ff5640 1 -- 172.21.15.45:0/352605060 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d530486720 con 0x55d5307ffd10 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f6616ff5640 10 client.26309 handle_command_reply: tid=0 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 client.26309 shutdown 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 msgr2=0x55d5307ecf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6624005ab0 tx=0x7f6624005e20 comp rx=0 tx=0).stop 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 shutdown_connections 2024-03-12T05:15:45.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5307ffd10 0x55d530802170 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307ed4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ecf80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 --2- 172.21.15.45:0/352605060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 >> 172.21.15.45:0/352605060 conn(0x55d5307e0da0 msgr2=0x55d5307e2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f6616ff5640 0 client.26309 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 shutdown_connections 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.430+0000 7f663bc4f740 1 -- 172.21.15.45:0/352605060 wait complete. 2024-03-12T05:15:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.431+0000 7f663bc4f740 20 client.26309 trim_cache size 0 max 16384 2024-03-12T05:15:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.456+0000 7f663bc4f740 1 Processor -- start 2024-03-12T05:15:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.456+0000 7f663bc4f740 1 -- start start 2024-03-12T05:15:45.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307ef170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307e6360 con 0x55d5307efbc0 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307e2020 con 0x55d5307e6fe0 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663bc4f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307e1df0 con 0x55d5307e66c0 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34062/0 (socket says 172.21.15.45:34062) 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663359e640 1 -- 172.21.15.45:0/2461080342 learned_addr learned my addr 172.21.15.45:0/2461080342 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663359e640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307ef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 -- 172.21.15.45:0/2461080342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 msgr2=0x55d5307ef170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307ef170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 -- 172.21.15.45:0/2461080342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 msgr2=0x55d5307eba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663259c640 1 -- 172.21.15.45:0/2461080342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5306e4ac0 con 0x55d5307e6fe0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.457+0000 7f663359e640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.458+0000 7f663259c640 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ed2e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6624004ff0 tx=0x7f6624005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4264e58cd2dcc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.458+0000 7f6618ff9640 1 -- 172.21.15.45:0/2461080342 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624005220 con 0x55d5307e6fe0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.458+0000 7f6618ff9640 1 -- 172.21.15.45:0/2461080342 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6624009780 con 0x55d5307e6fe0 2024-03-12T05:15:45.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.458+0000 7f6618ff9640 1 -- 172.21.15.45:0/2461080342 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624003fe0 con 0x55d5307e6fe0 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 -- 172.21.15.45:0/2461080342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 msgr2=0x55d5307ed2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ed2e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6624004ff0 tx=0x7f6624005980 comp rx=0 tx=0).stop 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 -- 172.21.15.45:0/2461080342 shutdown_connections 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307ed2e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2461080342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307ef170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 -- 172.21.15.45:0/2461080342 >> 172.21.15.45:0/2461080342 conn(0x55d5307e0da0 msgr2=0x55d5307eb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 -- 172.21.15.45:0/2461080342 shutdown_connections 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.459+0000 7f663bc4f740 1 -- 172.21.15.45:0/2461080342 wait complete. 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.460+0000 7f663bc4f740 1 -- start start 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.460+0000 7f663bc4f740 1 Processor -- start 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307deff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307f7370 con 0x55d5307efbc0 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:54488/0 (socket says 172.21.15.45:54488) 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663259c640 1 -- 172.21.15.45:0/2547555680 learned_addr learned my addr 172.21.15.45:0/2547555680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307df530 con 0x55d5307e6fe0 2024-03-12T05:15:45.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663359e640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.461+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5307df6f0 con 0x55d5307e66c0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663359e640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307deff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663bc4f740 10 client.0 _ll_register_callbacks cb 0x7f662c009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 -- 172.21.15.45:0/2547555680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 msgr2=0x55d5307f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 -- 172.21.15.45:0/2547555680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 msgr2=0x55d5307deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307deff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c01e040 con 0x55d5307e6fe0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f662c015070 con 0x55d5307e6fe0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f663259c640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f662402f020 tx=0x7f6624005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d39504406c619 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f6616ff5640 10 client.26315 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f6616ff5640 1 -- 172.21.15.45:0/2547555680 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240048c0 con 0x55d5307e6fe0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.462+0000 7f6616ff5640 1 -- 172.21.15.45:0/2547555680 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f662402c6e0 con 0x55d5307e6fe0 2024-03-12T05:15:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f6616ff5640 1 -- 172.21.15.45:0/2547555680 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6624004a20 con 0x55d5307e6fe0 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 10 client.26315 fetch_fsmap learned FSMap version 33 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55d530748610 con 0x55d5307e6fe0 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f6616ff5640 1 -- 172.21.15.45:0/2547555680 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6624045040 con 0x55d5307e6fe0 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 10 client.26315 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 10 client.26315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 20 client.26315 populate_metadata read hostname 'smithi045' 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d53085b840 0x55d53085dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 4 client.26315 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d5304d5890 con 0x55d53085b840 2024-03-12T05:15:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.463+0000 7f663359e640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d53085b840 0x55d53085dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663359e640 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d53085b840 0x55d53085dc30 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a50092b3fd3f3a79 server_cookie=2163141459eaaf53 in_seq=0 out_seq=0 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f6616ff5640 10 client.26315 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f6616ff5640 1 -- 172.21.15.45:0/2547555680 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d5304d5890 con 0x55d53085b840 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f6616ff5640 10 client.26315 handle_command_reply: tid=0 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 client.26315 shutdown 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 msgr2=0x55d5307deab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f662402f020 tx=0x7f6624005e40 comp rx=0 tx=0).stop 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 shutdown_connections 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d53085b840 0x55d53085dc30 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f6616ff5640 0 client.26315 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:15:45.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5307efbc0 0x55d5307deff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5307e6fe0 0x55d5307deab0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 --2- 172.21.15.45:0/2547555680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5307e66c0 0x55d5307f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.464+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 >> 172.21.15.45:0/2547555680 conn(0x55d5307e0da0 msgr2=0x55d5307db1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.465+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 shutdown_connections 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.465+0000 7f663bc4f740 1 -- 172.21.15.45:0/2547555680 wait complete. 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.465+0000 7f663bc4f740 20 client.26315 trim_cache size 0 max 16384 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stdout: "1ef972c7-0d65-45e9-9a20-7785287335d4": { 2024-03-12T05:15:45.465 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:15:45.466 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "1ef972c7-0d65-45e9-9a20-7785287335d4", 2024-03-12T05:15:45.466 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:15:45.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:15:45.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:15:45.466 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 msgr2=0x7f661c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 0x7f661c0a4160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55d5307e6190 tx=0x7f6628009310 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 msgr2=0x7f6634197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f662800e9f0 tx=0x7f662800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 shutdown_connections 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f661c0a1ca0 0x7f661c0a4160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f663419bf50 0x7f663419c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6634105340 0x7f6634197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 --2- 172.21.15.45:0/104900165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6634100270 0x7f6634197690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 >> 172.21.15.45:0/104900165 conn(0x7f6634077230 msgr2=0x7f66340fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:15:45.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 shutdown_connections 2024-03-12T05:15:45.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:15:45.466+0000 7f6639081640 1 -- 172.21.15.45:0/104900165 wait complete. 2024-03-12T05:15:45.480 INFO:tasks.cephfs.filesystem:scrub status for tag:1ef972c7-0d65-45e9-9a20-7785287335d4 - {'path': '/', 'tag': '1ef972c7-0d65-45e9-9a20-7785287335d4', 'options': 'recursive,force'} 2024-03-12T05:15:46.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: pgmap v149: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 1.93k op/s 2024-03-12T05:15:46.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:45 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: pgmap v149: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 1.93k op/s 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:45 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: pgmap v149: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 1.93k op/s 2024-03-12T05:15:46.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:15:46.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:46.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:46.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:46.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:46.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:45 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:47.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:46 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:15:47.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:47.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:46 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:15:47.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:47.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:46 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:15:47.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:15:48.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:47 smithi045 ceph-mon[32174]: pgmap v150: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:15:48.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:47 smithi130 ceph-mon[35398]: pgmap v150: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:15:48.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:47 smithi116 ceph-mon[35486]: pgmap v150: 97 pgs: 97 active+clean; 2.5 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:15:49.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 225.0 s, 1450.4 tps, lat 68.968 ms stddev 67.940 2024-03-12T05:15:50.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:49 smithi045 ceph-mon[32174]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:50.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:49 smithi130 ceph-mon[35398]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:50.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:49 smithi116 ceph-mon[35486]: pgmap v151: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 1.82k op/s 2024-03-12T05:15:52.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:51 smithi045 ceph-mon[32174]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:52.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:51 smithi130 ceph-mon[35398]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:52.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:51 smithi116 ceph-mon[35486]: pgmap v152: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:53 smithi045 ceph-mon[32174]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:54.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:53 smithi130 ceph-mon[35398]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:54.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:53 smithi116 ceph-mon[35486]: pgmap v153: 97 pgs: 97 active+clean; 2.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.16k op/s 2024-03-12T05:15:54.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 230.0 s, 1662.8 tps, lat 60.160 ms stddev 16.665 2024-03-12T05:15:56.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:55 smithi045 ceph-mon[32174]: pgmap v154: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 17 MiB/s wr, 1.80k op/s 2024-03-12T05:15:56.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:55 smithi130 ceph-mon[35398]: pgmap v154: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 17 MiB/s wr, 1.80k op/s 2024-03-12T05:15:56.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:55 smithi116 ceph-mon[35486]: pgmap v154: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 17 MiB/s wr, 1.80k op/s 2024-03-12T05:15:58.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:57 smithi045 ceph-mon[32174]: pgmap v155: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:58.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:57 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:58.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:57 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:58.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:57 smithi130 ceph-mon[35398]: pgmap v155: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:58.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:57 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:58.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:57 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:58.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:57 smithi116 ceph-mon[35486]: pgmap v155: 97 pgs: 97 active+clean; 2.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 12 MiB/s wr, 1.20k op/s 2024-03-12T05:15:58.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:57 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:58.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:57 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:59.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:58 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:59.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:58 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:15:59.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:58 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:59.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:58 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:15:59.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:58 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:59.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:58 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:15:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 235.0 s, 1653.8 tps, lat 60.370 ms stddev 16.065 2024-03-12T05:16:00.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:15:59 smithi045 ceph-mon[32174]: pgmap v156: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.91k op/s 2024-03-12T05:16:00.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:15:59 smithi130 ceph-mon[35398]: pgmap v156: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.91k op/s 2024-03-12T05:16:00.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:15:59 smithi116 ceph-mon[35486]: pgmap v156: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.91k op/s 2024-03-12T05:16:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:01 smithi045 ceph-mon[32174]: pgmap v157: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:02.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:01 smithi130 ceph-mon[35398]: pgmap v157: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:02.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:01 smithi116 ceph-mon[35486]: pgmap v157: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:04.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:03 smithi045 ceph-mon[32174]: pgmap v158: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:04.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:03 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:16:04.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:03 smithi130 ceph-mon[35398]: pgmap v158: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:04.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:03 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:16:04.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:03 smithi116 ceph-mon[35486]: pgmap v158: 97 pgs: 97 active+clean; 2.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:16:04.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:03 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:16:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 240.0 s, 1432.2 tps, lat 69.797 ms stddev 32.548 2024-03-12T05:16:05.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:04 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:16:05.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:04 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:16:05.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:04 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:16:05.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:04 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:16:05.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:04 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:16:05.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:04 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:16:05.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:04 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:16:05.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:04 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:16:05.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:04 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:16:06.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:06 smithi045 ceph-mon[32174]: pgmap v159: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.96k op/s 2024-03-12T05:16:06.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:06 smithi130 ceph-mon[35398]: pgmap v159: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.96k op/s 2024-03-12T05:16:06.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:06 smithi116 ceph-mon[35486]: pgmap v159: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 MiB/s wr, 1.96k op/s 2024-03-12T05:16:08.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:08 smithi130 ceph-mon[35398]: pgmap v160: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:16:08.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:08 smithi116 ceph-mon[35486]: pgmap v160: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:16:08.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:08 smithi045 ceph-mon[32174]: pgmap v160: 97 pgs: 97 active+clean; 2.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:16:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 245.0 s, 1500.2 tps, lat 66.766 ms stddev 24.017 2024-03-12T05:16:10.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:10 smithi130 ceph-mon[35398]: pgmap v161: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 MiB/s wr, 1.95k op/s 2024-03-12T05:16:10.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:10 smithi116 ceph-mon[35486]: pgmap v161: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 MiB/s wr, 1.95k op/s 2024-03-12T05:16:10.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:10 smithi045 ceph-mon[32174]: pgmap v161: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 MiB/s wr, 1.95k op/s 2024-03-12T05:16:12.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:12 smithi130 ceph-mon[35398]: pgmap v162: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:12.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:12 smithi116 ceph-mon[35486]: pgmap v162: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:12.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:12 smithi045 ceph-mon[32174]: pgmap v162: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:14.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:14 smithi130 ceph-mon[35398]: pgmap v163: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:14.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:14 smithi116 ceph-mon[35486]: pgmap v163: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:14.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:14 smithi045 ceph-mon[32174]: pgmap v163: 97 pgs: 97 active+clean; 2.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.24k op/s 2024-03-12T05:16:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 250.0 s, 1378.0 tps, lat 72.429 ms stddev 77.591 2024-03-12T05:16:15.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:16:15.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 -- 172.21.15.45:0/2302702945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 msgr2=0x7eff2c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 --2- 172.21.15.45:0/2302702945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c108770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009890 tx=0x7eff1c02bbf0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 -- 172.21.15.45:0/2302702945 shutdown_connections 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 --2- 172.21.15.45:0/2302702945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 --2- 172.21.15.45:0/2302702945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 --2- 172.21.15.45:0/2302702945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c108770 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 -- 172.21.15.45:0/2302702945 >> 172.21.15.45:0/2302702945 conn(0x7eff2c06c270 msgr2=0x7eff2c06c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 -- 172.21.15.45:0/2302702945 shutdown_connections 2024-03-12T05:16:15.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.667+0000 7eff32b56640 1 -- 172.21.15.45:0/2302702945 wait complete. 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.668+0000 7eff32b56640 1 Processor -- start 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.668+0000 7eff32b56640 1 -- start start 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.668+0000 7eff32b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff32b56640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff32b56640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53102/0 (socket says 172.21.15.45:53102) 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff308f2640 1 -- 172.21.15.45:0/2882131469 learned_addr learned my addr 172.21.15.45:0/2882131469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1ab390 con 0x7eff2c1142d0 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1ab500 con 0x7eff2c108390 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1ab6c0 con 0x7eff2c108cb0 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.669+0000 7eff310f3640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff2bfff640 1 -- 172.21.15.45:0/2882131469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 msgr2=0x7eff2c113100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff2bfff640 1 -- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 msgr2=0x7eff2c1aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff2bfff640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c009490 con 0x7eff2c108cb0 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff308f2640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:15.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.670+0000 7eff310f3640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.671+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff2000e9f0 tx=0x7eff2000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.671+0000 7eff29ffb640 1 -- 172.21.15.45:0/2882131469 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2000ccc0 con 0x7eff2c108cb0 2024-03-12T05:16:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.671+0000 7eff29ffb640 1 -- 172.21.15.45:0/2882131469 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff2000ce20 con 0x7eff2c108cb0 2024-03-12T05:16:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.672+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2c1ab9a0 con 0x7eff2c108cb0 2024-03-12T05:16:15.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.672+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c1b3220 con 0x7eff2c108cb0 2024-03-12T05:16:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.673+0000 7eff29ffb640 1 -- 172.21.15.45:0/2882131469 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200187c0 con 0x7eff2c108cb0 2024-03-12T05:16:15.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.675+0000 7eff29ffb640 1 -- 172.21.15.45:0/2882131469 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7eff20010430 con 0x7eff2c108cb0 2024-03-12T05:16:15.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 Processor -- start 2024-03-12T05:16:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 -- start start 2024-03-12T05:16:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff29ffb640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 0x7eff140a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc3394270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3396ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc32fa280 con 0x55ebc33947b0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc32f77b0 con 0x55ebc338bbd0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff35809740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc32f2f90 con 0x55ebc338b2b0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.676+0000 7eff29ffb640 1 -- 172.21.15.45:0/2882131469 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7eff20014070 con 0x7eff2c108cb0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42246/0 (socket says 172.21.15.45:42246) 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 -- 172.21.15.45:0/202086034 learned_addr learned my addr 172.21.15.45:0/202086034 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff310f3640 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3396ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff2bfff640 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc3394270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 0x7eff140a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 -- 172.21.15.45:0/202086034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 msgr2=0x55ebc3394270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc3394270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 -- 172.21.15.45:0/202086034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 msgr2=0x55ebc3396ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3396ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 -- 172.21.15.45:0/202086034 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc32b1570 con 0x55ebc338b2b0 2024-03-12T05:16:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.677+0000 7eff308f2640 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff1c02cf50 tx=0x7eff1c004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426728187b576b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.678+0000 7eff12ffd640 1 -- 172.21.15.45:0/202086034 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c02c960 con 0x55ebc338b2b0 2024-03-12T05:16:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.678+0000 7eff12ffd640 1 -- 172.21.15.45:0/202086034 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff1c02cac0 con 0x55ebc338b2b0 2024-03-12T05:16:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.678+0000 7eff12ffd640 1 -- 172.21.15.45:0/202086034 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c03f660 con 0x55ebc338b2b0 2024-03-12T05:16:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- 172.21.15.45:0/202086034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 msgr2=0x55ebc338b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff1c02cf50 tx=0x7eff1c004580 comp rx=0 tx=0).stop 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff308f2640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 0x7eff140a4230 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff1c04c000 tx=0x7eff1c047000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- 172.21.15.45:0/202086034 shutdown_connections 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3396ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc3394270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 --2- 172.21.15.45:0/202086034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338b2b0 0x55ebc338b690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- 172.21.15.45:0/202086034 >> 172.21.15.45:0/202086034 conn(0x55ebc3385990 msgr2=0x55ebc3385d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- 172.21.15.45:0/202086034 shutdown_connections 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- 172.21.15.45:0/202086034 wait complete. 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 -- start start 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.679+0000 7eff35809740 1 Processor -- start 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33914d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3391a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc2fc6430 con 0x55ebc33947b0 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc2f82e00 con 0x55ebc338b2b0 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc3277fd0 con 0x55ebc338bbd0 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.680+0000 7eff35809740 10 client.0 _ll_register_callbacks cb 0x55ebc32cdfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53126/0 (socket says 172.21.15.45:53126) 2024-03-12T05:16:15.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 -- 172.21.15.45:0/2310866613 learned_addr learned my addr 172.21.15.45:0/2310866613 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff310f3640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3391a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33914d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 -- 172.21.15.45:0/2310866613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 msgr2=0x55ebc33914d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33914d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 -- 172.21.15.45:0/2310866613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 msgr2=0x55ebc3391a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3391a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.681+0000 7eff308f2640 1 -- 172.21.15.45:0/2310866613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c117010 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff308f2640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff1c037040 tx=0x7eff1c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5de704cc5c3d6f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c02c960 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebc2ff27e0 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7eff1c0035f0 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff1c02cc30 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff35809740 10 client.26327 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.682+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ebc32f1610 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ebc2ff27e0 con 0x55ebc338b2b0 2024-03-12T05:16:15.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff1c053910 con 0x55ebc338b2b0 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff35809740 10 client.26327 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff35809740 10 client.26327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff35809740 20 client.26327 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc33b5670 0x55ebc33b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.683+0000 7eff35809740 4 client.26327 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.684+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebc302f720 con 0x55ebc33b5670 2024-03-12T05:16:15.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.684+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc33b5670 0x55ebc33b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.685+0000 7eff2bfff640 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc33b5670 0x55ebc33b7a90 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b148018c979e1b26 server_cookie=8579d01beab6d08d in_seq=0 out_seq=0 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.685+0000 7eff0bfff640 10 client.26327 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.686+0000 7eff0bfff640 1 -- 172.21.15.45:0/2310866613 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ebc302f720 con 0x55ebc33b5670 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.686+0000 7eff0bfff640 10 client.26327 handle_command_reply: tid=0 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.686+0000 7eff35809740 1 client.26327 shutdown 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.686+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 msgr2=0x55ebc33940d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.686+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff1c037040 tx=0x7eff1c005c90 comp rx=0 tx=0).stop 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 shutdown_connections 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc33b5670 0x55ebc33b7a90 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc3391a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 --2- 172.21.15.45:0/2310866613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33940d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 >> 172.21.15.45:0/2310866613 conn(0x55ebc3385990 msgr2=0x55ebc3387730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.687+0000 7eff0bfff640 0 client.26327 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.688+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 shutdown_connections 2024-03-12T05:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.688+0000 7eff35809740 1 -- 172.21.15.45:0/2310866613 wait complete. 2024-03-12T05:16:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.688+0000 7eff35809740 20 client.26327 trim_cache size 0 max 16384 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.715+0000 7eff35809740 1 Processor -- start 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.715+0000 7eff35809740 1 -- start start 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.715+0000 7eff35809740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.715+0000 7eff35809740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc33bc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff35809740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc33947b0 0x55ebc338c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff35809740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc33a8490 con 0x55ebc338b2b0 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff35809740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc33869e0 con 0x55ebc338bbd0 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37798/0 (socket says 172.21.15.45:37798) 2024-03-12T05:16:15.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 -- 172.21.15.45:0/3383546804 learned_addr learned my addr 172.21.15.45:0/3383546804 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff310f3640 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc33947b0 0x55ebc338c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff2bfff640 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc33bc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc33810d0 con 0x55ebc33947b0 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 -- 172.21.15.45:0/3383546804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc33947b0 msgr2=0x55ebc338c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc33947b0 0x55ebc338c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 -- 172.21.15.45:0/3383546804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 msgr2=0x55ebc33bc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc33bc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.716+0000 7eff308f2640 1 -- 172.21.15.45:0/3383546804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc328dac0 con 0x55ebc338b2b0 2024-03-12T05:16:15.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.717+0000 7eff308f2640 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7eff1c041490 tx=0x7eff1c004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d9c53e00e37f0d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.718+0000 7eff127fc640 1 -- 172.21.15.45:0/3383546804 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c02c960 con 0x55ebc338b2b0 2024-03-12T05:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.718+0000 7eff127fc640 1 -- 172.21.15.45:0/3383546804 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff1c02cc30 con 0x55ebc338b2b0 2024-03-12T05:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.718+0000 7eff127fc640 1 -- 172.21.15.45:0/3383546804 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0536c0 con 0x55ebc338b2b0 2024-03-12T05:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.719+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 msgr2=0x55ebc3393c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.719+0000 7eff35809740 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7eff1c041490 tx=0x7eff1c004580 comp rx=0 tx=0).stop 2024-03-12T05:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.720+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 shutdown_connections 2024-03-12T05:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.720+0000 7eff35809740 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc33947b0 0x55ebc338c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.720+0000 7eff35809740 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338bbd0 0x55ebc33bc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.720+0000 7eff35809740 1 --2- 172.21.15.45:0/3383546804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc338b2b0 0x55ebc3393c00 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.720+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 >> 172.21.15.45:0/3383546804 conn(0x55ebc3385990 msgr2=0x55ebc3390a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.721+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 shutdown_connections 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.721+0000 7eff35809740 1 -- 172.21.15.45:0/3383546804 wait complete. 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.721+0000 7eff35809740 1 -- start start 2024-03-12T05:16:15.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.722+0000 7eff35809740 1 Processor -- start 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.722+0000 7eff35809740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.722+0000 7eff35809740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff35809740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc338d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff308f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33ac6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53142/0 (socket says 172.21.15.45:53142) 2024-03-12T05:16:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff2bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff2bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42274/0 (socket says 172.21.15.45:42274) 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff35809740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc338df30 con 0x55ebc33947b0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff35809740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc338e0a0 con 0x55ebc338b2b0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff35809740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc338e260 con 0x55ebc338bbd0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff308f2640 1 -- 172.21.15.45:0/1242455152 learned_addr learned my addr 172.21.15.45:0/1242455152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff35809740 10 client.0 _ll_register_callbacks cb 0x7eff24009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff310f3640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc338d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff2bfff640 1 -- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 msgr2=0x55ebc33ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.723+0000 7eff2bfff640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff2bfff640 1 -- 172.21.15.45:0/1242455152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 msgr2=0x55ebc338d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff2bfff640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc338d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff2bfff640 1 -- 172.21.15.45:0/1242455152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebc32f1610 con 0x55ebc338bbd0 2024-03-12T05:16:15.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff2bfff640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff200cb200 tx=0x7eff200cd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db8a1849207262e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200a5070 con 0x55ebc338bbd0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7eff200b31c0 con 0x55ebc338bbd0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.724+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff1c02cac0 con 0x55ebc338bbd0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.725+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff20010b00 con 0x55ebc338bbd0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.725+0000 7eff35809740 10 client.26335 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.725+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ebc32b36a0 con 0x55ebc338bbd0 2024-03-12T05:16:15.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff1c02cac0 con 0x55ebc338bbd0 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff200a8020 con 0x55ebc338bbd0 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 10 client.26335 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 20 client.26335 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc340ccd0 0x55ebc340f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 4 client.26335 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:16:15.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.726+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ebc307e890 con 0x55ebc340ccd0 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff308f2640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc340ccd0 0x55ebc340f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff308f2640 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc340ccd0 0x55ebc340f0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb6bfa280e3c28b6 server_cookie=3775f92fe69e5410 in_seq=0 out_seq=0 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff0bfff640 10 client.26335 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff0bfff640 1 -- 172.21.15.45:0/1242455152 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ebc307e890 con 0x55ebc340ccd0 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff0bfff640 10 client.26335 handle_command_reply: tid=0 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.727+0000 7eff35809740 1 client.26335 shutdown 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 msgr2=0x55ebc33acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff200cb200 tx=0x7eff200cd040 comp rx=0 tx=0).stop 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 shutdown_connections 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ebc340ccd0 0x55ebc340f0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ebc33947b0 0x55ebc338d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebc338bbd0 0x55ebc33acc20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 --2- 172.21.15.45:0/1242455152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc338b2b0 0x55ebc33ac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 >> 172.21.15.45:0/1242455152 conn(0x55ebc3385990 msgr2=0x55ebc3390a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff0bfff640 0 client.26335 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.728+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 shutdown_connections 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.729+0000 7eff35809740 1 -- 172.21.15.45:0/1242455152 wait complete. 2024-03-12T05:16:15.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.729+0000 7eff35809740 20 client.26335 trim_cache size 0 max 16384 2024-03-12T05:16:15.727 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:15.727 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:16:15.727 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:16:15.727 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 msgr2=0x7eff140a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 0x7eff140a4230 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff1c04c000 tx=0x7eff1c047000 comp rx=0 tx=0).stop 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 msgr2=0x7eff2c113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff2000e9f0 tx=0x7eff2000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 shutdown_connections 2024-03-12T05:16:15.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff140a1d70 0x7eff140a4230 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff2c1142d0 0x7eff2c1aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff2c108cb0 0x7eff2c113640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 --2- 172.21.15.45:0/2882131469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff2c108390 0x7eff2c113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 >> 172.21.15.45:0/2882131469 conn(0x7eff2c06c270 msgr2=0x7eff2c06fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 shutdown_connections 2024-03-12T05:16:15.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.732+0000 7eff32b56640 1 -- 172.21.15.45:0/2882131469 wait complete. 2024-03-12T05:16:15.746 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:16:15.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:16:15.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 -- 172.21.15.45:0/2169063438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301002f0 msgr2=0x7efc301006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 --2- 172.21.15.45:0/2169063438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301002f0 0x7efc301006d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efc20009890 tx=0x7efc2002bb70 comp rx=0 tx=0).stop 2024-03-12T05:16:15.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 -- 172.21.15.45:0/2169063438 shutdown_connections 2024-03-12T05:16:15.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 --2- 172.21.15.45:0/2169063438 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301053c0 0x7efc3010cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 --2- 172.21.15.45:0/2169063438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc30100ca0 0x7efc30104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 --2- 172.21.15.45:0/2169063438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301002f0 0x7efc301006d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 -- 172.21.15.45:0/2169063438 >> 172.21.15.45:0/2169063438 conn(0x7efc30077180 msgr2=0x7efc30077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:15.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 -- 172.21.15.45:0/2169063438 shutdown_connections 2024-03-12T05:16:15.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.922+0000 7efc362a3640 1 -- 172.21.15.45:0/2169063438 wait complete. 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.923+0000 7efc362a3640 1 Processor -- start 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.923+0000 7efc362a3640 1 -- start start 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301002f0 0x7efc3019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301975f0 0x7efc30197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30197fe0 con 0x7efc301002f0 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30198150 con 0x7efc301975f0 2024-03-12T05:16:15.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc362a3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30198310 con 0x7efc301053c0 2024-03-12T05:16:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42284/0 (socket says 172.21.15.45:42284) 2024-03-12T05:16:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301002f0 0x7efc3019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 -- 172.21.15.45:0/1379155293 learned_addr learned my addr 172.21.15.45:0/1379155293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc34840640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301975f0 0x7efc30197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 -- 172.21.15.45:0/1379155293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301975f0 msgr2=0x7efc30197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301975f0 0x7efc30197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 -- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301002f0 msgr2=0x7efc3019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301002f0 0x7efc3019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.924+0000 7efc2f7fe640 1 -- 172.21.15.45:0/1379155293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28009840 con 0x7efc301053c0 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.925+0000 7efc2f7fe640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efc24005c80 tx=0x7efc2400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.925+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc24009a30 con 0x7efc301053c0 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.925+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc20009490 con 0x7efc301053c0 2024-03-12T05:16:15.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.925+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc30102080 con 0x7efc301053c0 2024-03-12T05:16:15.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.925+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efc240044d0 con 0x7efc301053c0 2024-03-12T05:16:15.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.926+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc24003cc0 con 0x7efc301053c0 2024-03-12T05:16:15.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.926+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c005290 con 0x7efc301053c0 2024-03-12T05:16:15.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.929+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efc24009b90 con 0x7efc301053c0 2024-03-12T05:16:15.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.930+0000 7efc2d7fa640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 0x7efc180a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.930+0000 7efc2ffff640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 0x7efc180a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.930+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efc240ccee0 con 0x7efc301053c0 2024-03-12T05:16:15.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.931+0000 7efc2ffff640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 0x7efc180a3fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efc2002c180 tx=0x7efc200056d0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:15.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:15.935+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc24095dd0 con 0x7efc301053c0 2024-03-12T05:16:16.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.344+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc1c0058f0 con 0x7efc301053c0 2024-03-12T05:16:16.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.345+0000 7efc2d7fa640 1 -- 172.21.15.45:0/1379155293 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7efc1c0058f0 con 0x7efc301053c0 2024-03-12T05:16:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:16:16.341 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:16:16.341 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:16:16.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 msgr2=0x7efc180a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 0x7efc180a3fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efc2002c180 tx=0x7efc200056d0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 msgr2=0x7efc3019d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efc24005c80 tx=0x7efc2400dde0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 shutdown_connections 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efc180a1b00 0x7efc180a3fc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc301975f0 0x7efc30197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc301053c0 0x7efc3019d980 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 --2- 172.21.15.45:0/1379155293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc301002f0 0x7efc3019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 >> 172.21.15.45:0/1379155293 conn(0x7efc30077180 msgr2=0x7efc30104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 shutdown_connections 2024-03-12T05:16:16.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.348+0000 7efc362a3640 1 -- 172.21.15.45:0/1379155293 wait complete. 2024-03-12T05:16:16.362 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:16.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:16:16.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:16 smithi045 ceph-mon[32174]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 MiB/s wr, 1.80k op/s 2024-03-12T05:16:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:16 smithi130 ceph-mon[35398]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 MiB/s wr, 1.80k op/s 2024-03-12T05:16:16.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:16 smithi116 ceph-mon[35486]: pgmap v164: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 MiB/s wr, 1.80k op/s 2024-03-12T05:16:16.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2597976995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb410cf00 msgr2=0x7f8bb4114a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2597976995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb410cf00 0x7f8bb4114a80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009900 tx=0x7f8bac02bc40 comp rx=0 tx=0).stop 2024-03-12T05:16:16.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2597976995 shutdown_connections 2024-03-12T05:16:16.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2597976995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb410cf00 0x7f8bb4114a80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2597976995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb4108da0 0x7f8bb410c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2597976995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb41083f0 0x7f8bb41087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.539+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2597976995 >> 172.21.15.45:0/2597976995 conn(0x7f8bb406c6c0 msgr2=0x7f8bb406cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.540+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2597976995 shutdown_connections 2024-03-12T05:16:16.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.540+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2597976995 wait complete. 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.540+0000 7f8bb8c3b640 1 Processor -- start 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 -- start start 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 0x7f8bb4111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb41118c0 0x7f8bb41b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb41b2a30 con 0x7f8bb41083f0 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb41b2ba0 con 0x7f8bb410cf00 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb8c3b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb41b2d60 con 0x7f8bb41118c0 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37832/0 (socket says 172.21.15.45:37832) 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb259c640 1 -- 172.21.15.45:0/2066401473 learned_addr learned my addr 172.21.15.45:0/2066401473 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 0x7f8bb4111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb41118c0 0x7f8bb41b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2066401473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb41118c0 msgr2=0x7f8bb41b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb41118c0 0x7f8bb41b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 msgr2=0x7f8bb4110e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.541+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2066401473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac009560 con 0x7f8bb410cf00 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bb259c640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 0x7f8bb4111380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4073380 tx=0x7f8ba400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bab7fe640 1 -- 172.21.15.45:0/2066401473 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400cc40 con 0x7f8bb410cf00 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb41b2fe0 con 0x7f8bb410cf00 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bab7fe640 1 -- 172.21.15.45:0/2066401473 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ba400cda0 con 0x7f8bb410cf00 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bab7fe640 1 -- 172.21.15.45:0/2066401473 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400cc40 con 0x7f8bb410cf00 2024-03-12T05:16:16.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.542+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb41b34c0 con 0x7f8bb410cf00 2024-03-12T05:16:16.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 Processor -- start 2024-03-12T05:16:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 -- start start 2024-03-12T05:16:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bab7fe640 1 -- 172.21.15.45:0/2066401473 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8ba4010430 con 0x7f8bb410cf00 2024-03-12T05:16:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d38630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d41210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d43b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668cf3960 con 0x559668d38b70 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668ca7310 con 0x559668d41750 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.546+0000 7f8bbb805740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668ca4840 con 0x559668d38250 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d43b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d38630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d38630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42314/0 (socket says 172.21.15.45:42314) 2024-03-12T05:16:16.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/2648069614 learned_addr learned my addr 172.21.15.45:0/2648069614 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb259c640 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d41210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2648069614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 msgr2=0x559668d38630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d38630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2648069614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 msgr2=0x559668d41210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d41210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.547+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/2648069614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559668c62060 con 0x559668d41750 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d43b40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40cae40 tx=0x7f8ba40d3910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea94a339de819f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8bb259c640 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d41210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:16.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8bab7fe640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 0x7f8b940a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8ba8ff9640 1 -- 172.21.15.45:0/2648069614 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4096670 con 0x559668d41750 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8ba8ff9640 1 -- 172.21.15.45:0/2648069614 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ba4095f40 con 0x559668d41750 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8ba8ff9640 1 -- 172.21.15.45:0/2648069614 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40cdd10 con 0x559668d41750 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 0x7f8b940a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.548+0000 7f8bab7fe640 1 -- 172.21.15.45:0/2066401473 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8ba4014070 con 0x7f8bb410cf00 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 0x7f8b940a41b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac02c160 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 -- 172.21.15.45:0/2648069614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 msgr2=0x559668d43b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d43b40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40cae40 tx=0x7f8ba40d3910 comp rx=0 tx=0).stop 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 -- 172.21.15.45:0/2648069614 shutdown_connections 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d41210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d43b40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 --2- 172.21.15.45:0/2648069614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d38630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 -- 172.21.15.45:0/2648069614 >> 172.21.15.45:0/2648069614 conn(0x559668d32980 msgr2=0x559668d32d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.549+0000 7f8bbb805740 1 -- 172.21.15.45:0/2648069614 shutdown_connections 2024-03-12T05:16:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.550+0000 7f8bbb805740 1 -- 172.21.15.45:0/2648069614 wait complete. 2024-03-12T05:16:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.550+0000 7f8bbb805740 1 -- start start 2024-03-12T05:16:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.550+0000 7f8bbb805740 1 Processor -- start 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d3e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d3ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668ca0020 con 0x559668d38b70 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668973430 con 0x559668d41750 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55966892fe00 con 0x559668d38250 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bbb805740 10 client.0 _ll_register_callbacks cb 0x559668c47cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53200/0 (socket says 172.21.15.45:53200) 2024-03-12T05:16:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3419030237 learned_addr learned my addr 172.21.15.45:0/3419030237 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d3ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 msgr2=0x559668d3e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d3e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3419030237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 msgr2=0x559668d3ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d3ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3419030237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb41b34c0 con 0x559668d41750 2024-03-12T05:16:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.551+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d3ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.552+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02c1d0 tx=0x7f8bac009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06b0ac4457a86e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.552+0000 7f8ba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.552+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0047e0 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.552+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55966899f7e0 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.552+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8bac004440 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.553+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bac003b40 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.553+0000 7f8bbb805740 10 client.26351 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.553+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559668c9e6a0 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.553+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8bac061080 con 0x559668d41750 2024-03-12T05:16:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8bac05b030 con 0x559668d41750 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 10 client.26351 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 10 client.26351 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 20 client.26351 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668d51c00 0x559668d540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 4 client.26351 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55966899f7e0 con 0x559668d51c00 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.554+0000 7f8bb259c640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668d51c00 0x559668d540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.555+0000 7f8ba1ffb640 10 client.26351 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.555+0000 7f8bb259c640 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668d51c00 0x559668d540a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=719407eb14d947ad server_cookie=8e97b2a0584421b4 in_seq=0 out_seq=0 2024-03-12T05:16:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.556+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/3419030237 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55966899f7e0 con 0x559668d51c00 2024-03-12T05:16:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.556+0000 7f8ba1ffb640 10 client.26351 handle_command_reply: tid=0 2024-03-12T05:16:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.556+0000 7f8bbb805740 1 client.26351 shutdown 2024-03-12T05:16:16.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.557+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 msgr2=0x559668d3f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.557+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02c1d0 tx=0x7f8bac009260 comp rx=0 tx=0).stop 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.557+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 shutdown_connections 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668d51c00 0x559668d540a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668d3ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8ba1ffb640 0 client.26351 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3f240 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3419030237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668d3e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 >> 172.21.15.45:0/3419030237 conn(0x559668d32980 msgr2=0x559668d346d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 shutdown_connections 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 1 -- 172.21.15.45:0/3419030237 wait complete. 2024-03-12T05:16:16.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.558+0000 7f8bbb805740 20 client.26351 trim_cache size 0 max 16384 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.580+0000 7f8bbb805740 1 Processor -- start 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 -- start start 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 0x559668d40ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38b70 0x559668d39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668c25130 con 0x559668d38250 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53214/0 (socket says 172.21.15.45:53214) 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3197315356 learned_addr learned my addr 172.21.15.45:0/3197315356 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668d592f0 con 0x559668d41750 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.581+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668d3cd10 con 0x559668d38b70 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb259c640 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 0x559668d40ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3197315356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38b70 msgr2=0x559668d39af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38b70 0x559668d39af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3197315356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 msgr2=0x559668d40ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 0x559668d40ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/3197315356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559668c3ac20 con 0x559668d41750 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb259c640 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 0x559668d40ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.582+0000 7f8bb2d9d640 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02c720 tx=0x7f8bac02cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97cdc279ac010c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.583+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3197315356 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac064070 con 0x559668d41750 2024-03-12T05:16:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.584+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3197315356 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8bac050070 con 0x559668d41750 2024-03-12T05:16:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.584+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3197315356 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac055430 con 0x559668d41750 2024-03-12T05:16:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 msgr2=0x559668d3a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8bac02c720 tx=0x7f8bac02cc40 comp rx=0 tx=0).stop 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 shutdown_connections 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d3a030 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38b70 0x559668d39af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 --2- 172.21.15.45:0/3197315356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38250 0x559668d40ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 >> 172.21.15.45:0/3197315356 conn(0x559668d32980 msgr2=0x559668d2da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.585+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 shutdown_connections 2024-03-12T05:16:16.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 -- 172.21.15.45:0/3197315356 wait complete. 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 -- start start 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 Processor -- start 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668da1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668da1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d47b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668d48200 con 0x559668d38b70 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668d48370 con 0x559668d41750 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.586+0000 7f8bbb805740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559668d48530 con 0x559668d38250 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bbb805740 10 client.0 _ll_register_callbacks cb 0x7f8ba40d0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d47b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668da1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d47b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53218/0 (socket says 172.21.15.45:53218) 2024-03-12T05:16:16.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb2d9d640 1 -- 172.21.15.45:0/1096055839 learned_addr learned my addr 172.21.15.45:0/1096055839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 msgr2=0x559668da1680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.587+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668da1680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.588+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/1096055839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 msgr2=0x559668d47b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.588+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d47b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.588+0000 7f8bb1d9b640 1 -- 172.21.15.45:0/1096055839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559668c9e6a0 con 0x559668d38b70 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.588+0000 7f8ba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.588+0000 7f8bb1d9b640 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668da1bc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409a080 tx=0x7f8ba4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754452695bed8f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.589+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4099f10 con 0x559668d38b70 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.589+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559668d3e5f0 con 0x559668d38b70 2024-03-12T05:16:16.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.589+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f8ba409a3d0 con 0x559668d38b70 2024-03-12T05:16:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.590+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ba40a2ac0 con 0x559668d38b70 2024-03-12T05:16:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.590+0000 7f8bbb805740 10 client.16884 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.590+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559668c608c0 con 0x559668d38b70 2024-03-12T05:16:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.590+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ba40a2ac0 con 0x559668d38b70 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8ba40106e0 con 0x559668d38b70 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 10 client.16884 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 10 client.16884 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 20 client.16884 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668daa360 0x559668dac780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 4 client.16884 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f8ba4095f40 con 0x559668daa360 2024-03-12T05:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.591+0000 7f8bb259c640 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668daa360 0x559668dac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.592+0000 7f8bb259c640 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668daa360 0x559668dac780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d5cf0ecac26f937c server_cookie=2997006dcaded61c in_seq=0 out_seq=0 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.592+0000 7f8ba1ffb640 10 client.16884 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.592+0000 7f8ba1ffb640 1 -- 172.21.15.45:0/1096055839 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f8ba4095f40 con 0x559668daa360 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.592+0000 7f8ba1ffb640 10 client.16884 handle_command_reply: tid=0 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.592+0000 7f8bbb805740 1 client.16884 shutdown 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 msgr2=0x559668da1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668da1bc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409a080 tx=0x7f8ba4005f10 comp rx=0 tx=0).stop 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 shutdown_connections 2024-03-12T05:16:16.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x559668daa360 0x559668dac780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559668d38b70 0x559668da1bc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559668d41750 0x559668d47b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 --2- 172.21.15.45:0/1096055839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559668d38250 0x559668da1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 >> 172.21.15.45:0/1096055839 conn(0x559668d32980 msgr2=0x559668d39af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8ba1ffb640 0 client.16884 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:16.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 shutdown_connections 2024-03-12T05:16:16.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 1 -- 172.21.15.45:0/1096055839 wait complete. 2024-03-12T05:16:16.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.593+0000 7f8bbb805740 20 client.16884 trim_cache size 0 max 16384 2024-03-12T05:16:16.591 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.594+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 msgr2=0x7f8b940a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.594+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 0x7f8b940a41b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bac006010 tx=0x7f8bac02c160 comp rx=0 tx=0).stop 2024-03-12T05:16:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.594+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 msgr2=0x7f8bb4111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.594+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 0x7f8bb4111380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4073380 tx=0x7f8ba400ee50 comp rx=0 tx=0).stop 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 shutdown_connections 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1cf0 0x7f8b940a41b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8bb41118c0 0x7f8bb41b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb410cf00 0x7f8bb4111380 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 --2- 172.21.15.45:0/2066401473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bb41083f0 0x7f8bb4110e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 >> 172.21.15.45:0/2066401473 conn(0x7f8bb406c6c0 msgr2=0x7f8bb4070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 shutdown_connections 2024-03-12T05:16:16.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.595+0000 7f8bb8c3b640 1 -- 172.21.15.45:0/2066401473 wait complete. 2024-03-12T05:16:16.608 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:16.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:16:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.823+0000 7f9bf1e62640 1 -- 172.21.15.45:0/1533936983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 msgr2=0x7f9bec11cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.823+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/1533936983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec11cc10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009860 tx=0x7f9be402bb70 comp rx=0 tx=0).stop 2024-03-12T05:16:16.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.823+0000 7f9bf1e62640 1 -- 172.21.15.45:0/1533936983 shutdown_connections 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/1533936983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06d430 0x7f9bec11a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/1533936983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec11cc10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/1533936983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06cb10 0x7f9bec06cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 -- 172.21.15.45:0/1533936983 >> 172.21.15.45:0/1533936983 conn(0x7f9bec06c180 msgr2=0x7f9bec10a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 -- 172.21.15.45:0/1533936983 shutdown_connections 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 -- 172.21.15.45:0/1533936983 wait complete. 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.824+0000 7f9bf1e62640 1 Processor -- start 2024-03-12T05:16:16.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 -- start start 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06cb10 0x7f9bec1a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec1aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1ab2f0 con 0x7f9bec06cb10 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1ab460 con 0x7f9bec11a820 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9bf1e62640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1ab620 con 0x7f9bec06d430 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9beaffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9beaffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42344/0 (socket says 172.21.15.45:42344) 2024-03-12T05:16:16.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.825+0000 7f9beaffd640 1 -- 172.21.15.45:0/2098853785 learned_addr learned my addr 172.21.15.45:0/2098853785 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9bebfff640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec1aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06cb10 0x7f9bec1a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 -- 172.21.15.45:0/2098853785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 msgr2=0x7f9bec1aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec1aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 -- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06cb10 msgr2=0x7f9bec1a6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06cb10 0x7f9bec1a6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 -- 172.21.15.45:0/2098853785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009490 con 0x7f9bec06d430 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00eeb0 tx=0x7f9bdc00c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9be8ff9640 1 -- 172.21.15.45:0/2098853785 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc003b70 con 0x7f9bec06d430 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec1ab900 con 0x7f9bec06d430 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.826+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec1b3180 con 0x7f9bec06d430 2024-03-12T05:16:16.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.827+0000 7f9be8ff9640 1 -- 172.21.15.45:0/2098853785 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9bdc0099a0 con 0x7f9bec06d430 2024-03-12T05:16:16.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.827+0000 7f9be8ff9640 1 -- 172.21.15.45:0/2098853785 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc00f940 con 0x7f9bec06d430 2024-03-12T05:16:16.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.830+0000 7f9bf4a31740 1 Processor -- start 2024-03-12T05:16:16.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.830+0000 7f9bf4a31740 1 -- start start 2024-03-12T05:16:16.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.830+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df59423630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.830+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df5942eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df593de960 con 0x55df59423250 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df59392310 con 0x55df5942c750 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5938f840 con 0x55df59423b70 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42348/0 (socket says 172.21.15.45:42348) 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 -- 172.21.15.45:0/4082457410 learned_addr learned my addr 172.21.15.45:0/4082457410 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9bebfff640 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df59423630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 -- 172.21.15.45:0/4082457410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 msgr2=0x55df5942eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df5942eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 -- 172.21.15.45:0/4082457410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 msgr2=0x55df59423630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df59423630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.831+0000 7f9beb7fe640 1 -- 172.21.15.45:0/4082457410 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df5934d060 con 0x55df59423b70 2024-03-12T05:16:16.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.832+0000 7f9be8ff9640 1 -- 172.21.15.45:0/2098853785 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9bdc00a040 con 0x7f9bec06d430 2024-03-12T05:16:16.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.832+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009c80 tx=0x7f9be00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84c0cb5887cb64cc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.832+0000 7f9bd1ffb640 1 -- 172.21.15.45:0/4082457410 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0012450 con 0x55df59423b70 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.833+0000 7f9be8ff9640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 0x7f9bd40a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.833+0000 7f9bd1ffb640 1 -- 172.21.15.45:0/4082457410 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9be0012a60 con 0x55df59423b70 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.833+0000 7f9be8ff9640 1 -- 172.21.15.45:0/2098853785 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9bdc09a900 con 0x7f9bec06d430 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.833+0000 7f9bd1ffb640 1 -- 172.21.15.45:0/4082457410 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000fb10 con 0x55df59423b70 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.834+0000 7f9bebfff640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 0x7f9bd40a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.834+0000 7f9bf4a31740 1 -- 172.21.15.45:0/4082457410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 msgr2=0x55df5942c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.834+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009c80 tx=0x7f9be00117b0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 -- 172.21.15.45:0/4082457410 shutdown_connections 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df5942eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df5942c210 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/4082457410 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df59423630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 -- 172.21.15.45:0/4082457410 >> 172.21.15.45:0/4082457410 conn(0x55df5941d980 msgr2=0x55df5941dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 -- 172.21.15.45:0/4082457410 shutdown_connections 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 -- 172.21.15.45:0/4082457410 wait complete. 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.835+0000 7f9bf4a31740 1 -- start start 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bebfff640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 0x7f9bd40a41b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009830 tx=0x7f9be4005a90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 Processor -- start 2024-03-12T05:16:16.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 0x55df59429640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df5942a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5938b020 con 0x55df5942c750 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5905e430 con 0x55df59423b70 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5901ae00 con 0x55df59423250 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.836+0000 7f9bf4a31740 10 client.0 _ll_register_callbacks cb 0x55df59332cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 0x55df59429640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53248/0 (socket says 172.21.15.45:53248) 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beaffd640 1 -- 172.21.15.45:0/2784150351 learned_addr learned my addr 172.21.15.45:0/2784150351 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9bebfff640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df5942a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 -- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 msgr2=0x55df59429b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 -- 172.21.15.45:0/2784150351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 msgr2=0x55df5942a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df5942a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 -- 172.21.15.45:0/2784150351 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be00098b0 con 0x55df59423250 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.837+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 0x55df59429640 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be0009fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9db152e352d64b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00128e0 con 0x55df59423250 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9be0004040 con 0x55df59423250 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df5908a7e0 con 0x55df59423250 2024-03-12T05:16:16.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9be0004220 con 0x55df59423250 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bf4a31740 10 client.26362 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55df593896a0 con 0x55df59423250 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55df5908a7e0 con 0x55df59423250 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.838+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9be0022040 con 0x55df59423250 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 10 client.26362 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 10 client.26362 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 20 client.26362 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df5943cb40 0x55df5943ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 4 client.26362 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df590c7720 con 0x55df5943cb40 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.839+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df5943cb40 0x55df5943ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.840+0000 7f9beaffd640 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df5943cb40 0x55df5943ef60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dedbfcd17df9be25 server_cookie=42cf87260226ce1d in_seq=0 out_seq=0 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.840+0000 7f9bcaffd640 10 client.26362 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.840+0000 7f9bcaffd640 1 -- 172.21.15.45:0/2784150351 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55df590c7720 con 0x55df5943cb40 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.840+0000 7f9bcaffd640 10 client.26362 handle_command_reply: tid=0 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.840+0000 7f9bf4a31740 1 client.26362 shutdown 2024-03-12T05:16:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 msgr2=0x55df59429640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 0x55df59429640 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be0009fa0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 shutdown_connections 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df5943cb40 0x55df5943ef60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df5942a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423b70 0x55df59429b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/2784150351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423250 0x55df59429640 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 >> 172.21.15.45:0/2784150351 conn(0x55df5941d980 msgr2=0x55df5941f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bcaffd640 0 client.26362 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 shutdown_connections 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 1 -- 172.21.15.45:0/2784150351 wait complete. 2024-03-12T05:16:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.841+0000 7f9bf4a31740 20 client.26362 trim_cache size 0 max 16384 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 Processor -- start 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 -- start start 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 0x55df5942bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59429ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df59310130 con 0x55df5942c750 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df59422eb0 con 0x55df59423250 2024-03-12T05:16:16.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df594195c0 con 0x55df59423b70 2024-03-12T05:16:16.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37942/0 (socket says 172.21.15.45:37942) 2024-03-12T05:16:16.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9bebfff640 1 -- 172.21.15.45:0/3067240899 learned_addr learned my addr 172.21.15.45:0/3067240899 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 0x55df5942bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.863+0000 7f9beaffd640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59429ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3067240899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 msgr2=0x55df59429ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59429ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3067240899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 msgr2=0x55df594268a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3067240899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df59325c20 con 0x55df59423250 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beaffd640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59429ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9bebfff640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 0x55df5942bd00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9be000a010 tx=0x7f9be000f5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34057c26581e4f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9bd17fa640 1 -- 172.21.15.45:0/3067240899 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0009530 con 0x55df59423250 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9bd17fa640 1 -- 172.21.15.45:0/3067240899 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9be00035f0 con 0x55df59423250 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.864+0000 7f9bd17fa640 1 -- 172.21.15.45:0/3067240899 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000f900 con 0x55df59423250 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.865+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3067240899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 msgr2=0x55df5942bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.865+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 0x55df5942bd00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9be000a010 tx=0x7f9be000f5a0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3067240899 shutdown_connections 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df5942c750 0x55df594268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59429ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3067240899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df59423250 0x55df5942bd00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3067240899 >> 172.21.15.45:0/3067240899 conn(0x55df5941d980 msgr2=0x55df594438e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3067240899 shutdown_connections 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3067240899 wait complete. 2024-03-12T05:16:16.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.866+0000 7f9bf4a31740 1 -- start start 2024-03-12T05:16:16.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 Processor -- start 2024-03-12T05:16:16.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59431c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df594321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df594326f0 con 0x55df59423250 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df59432860 con 0x55df5942c750 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df594329d0 con 0x55df59423b70 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bf4a31740 10 client.0 _ll_register_callbacks cb 0x7f9be00096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9beb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9beb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37954/0 (socket says 172.21.15.45:37954) 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3016352403 learned_addr learned my addr 172.21.15.45:0/3016352403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.867+0000 7f9bebfff640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df594321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3016352403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 msgr2=0x55df59431c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59431c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 msgr2=0x55df594321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:16.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df594321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 -- 172.21.15.45:0/3016352403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec11d640 con 0x55df59423250 2024-03-12T05:16:16.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9bebfff640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df594321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.868+0000 7f9beb7fe640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009880 tx=0x7f9be0003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674d7fb284c27d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.869+0000 7f9bcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00054f0 con 0x55df59423250 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f9be000f6a0 con 0x55df59423250 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9be0012a60 con 0x55df59423250 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9be0009530 con 0x55df59423250 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bf4a31740 10 client.16902 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.870+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55df593896a0 con 0x55df59423250 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9be0012a60 con 0x55df59423250 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9be0022040 con 0x55df59423250 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 10 client.16902 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 10 client.16902 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 20 client.16902 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df59494150 0x55df59496570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 4 client.16902 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55df59116890 con 0x55df59494150 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.871+0000 7f9beaffd640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df59494150 0x55df59496570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.872+0000 7f9beaffd640 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df59494150 0x55df59496570 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c13b8e91552fc64 server_cookie=e0cbf52ad96d5681 in_seq=0 out_seq=0 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.872+0000 7f9bcaffd640 10 client.16902 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:16.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.872+0000 7f9bcaffd640 1 -- 172.21.15.45:0/3016352403 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55df59116890 con 0x55df59494150 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.872+0000 7f9bcaffd640 10 client.16902 handle_command_reply: tid=0 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.872+0000 7f9bf4a31740 1 client.16902 shutdown 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 msgr2=0x55df594338d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009880 tx=0x7f9be0003c20 comp rx=0 tx=0).stop 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 shutdown_connections 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55df59494150 0x55df59496570 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df5942c750 0x55df594321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55df59423b70 0x55df59431c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 --2- 172.21.15.45:0/3016352403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55df59423250 0x55df594338d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bcaffd640 0 client.16902 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:16.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.873+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 >> 172.21.15.45:0/3016352403 conn(0x55df5941d980 msgr2=0x55df5941c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.874+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 shutdown_connections 2024-03-12T05:16:16.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.874+0000 7f9bf4a31740 1 -- 172.21.15.45:0/3016352403 wait complete. 2024-03-12T05:16:16.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.875+0000 7f9bf4a31740 20 client.16902 trim_cache size 0 max 16384 2024-03-12T05:16:16.873 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 msgr2=0x7f9bd40a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 0x7f9bd40a41b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009830 tx=0x7f9be4005a90 comp rx=0 tx=0).stop 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 msgr2=0x7f9bec1a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00eeb0 tx=0x7f9bdc00c6e0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 shutdown_connections 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9bd40a1cf0 0x7f9bd40a41b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bec11a820 0x7f9bec1aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bec06d430 0x7f9bec1a6970 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 --2- 172.21.15.45:0/2098853785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bec06cb10 0x7f9bec1a6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 >> 172.21.15.45:0/2098853785 conn(0x7f9bec06c180 msgr2=0x7f9bec11bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.876+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 shutdown_connections 2024-03-12T05:16:16.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:16.877+0000 7f9bf1e62640 1 -- 172.21.15.45:0/2098853785 wait complete. 2024-03-12T05:16:16.891 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:16.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:16:17.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.071+0000 7f6e841e9640 1 -- 172.21.15.45:0/481884486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c10d9d0 msgr2=0x7f6e7c10fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.071+0000 7f6e841e9640 1 --2- 172.21.15.45:0/481884486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c10fdc0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009ca0 tx=0x7f6e7402c140 comp rx=0 tx=0).stop 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 -- 172.21.15.45:0/481884486 shutdown_connections 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 --2- 172.21.15.45:0/481884486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c10fdc0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 --2- 172.21.15.45:0/481884486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c069c30 0x7f6e7c06a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 --2- 172.21.15.45:0/481884486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c102040 0x7f6e7c0696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 -- 172.21.15.45:0/481884486 >> 172.21.15.45:0/481884486 conn(0x7f6e7c077060 msgr2=0x7f6e7c077470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.072+0000 7f6e841e9640 1 -- 172.21.15.45:0/481884486 shutdown_connections 2024-03-12T05:16:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.073+0000 7f6e841e9640 1 -- 172.21.15.45:0/481884486 wait complete. 2024-03-12T05:16:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.074+0000 7f6e841e9640 1 Processor -- start 2024-03-12T05:16:17.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.074+0000 7f6e841e9640 1 -- start start 2024-03-12T05:16:17.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.074+0000 7f6e841e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 0x7f6e7c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e841e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 0x7f6e7c103de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 0x7f6e7c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 0x7f6e7c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37968/0 (socket says 172.21.15.45:37968) 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e81f85640 1 -- 172.21.15.45:0/3755443725 learned_addr learned my addr 172.21.15.45:0/3755443725 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c1043b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c104b40 con 0x7f6e7c069c30 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e81784640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 0x7f6e7c103de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c104cb0 con 0x7f6e7c10d9d0 2024-03-12T05:16:17.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e82786640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c1043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.075+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7c1b2ac0 con 0x7f6e7c102040 2024-03-12T05:16:17.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 -- 172.21.15.45:0/3755443725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c10d9d0 msgr2=0x7f6e7c1043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c1043b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 -- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 msgr2=0x7f6e7c10d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 0x7f6e7c10d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74009900 con 0x7f6e7c102040 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.076+0000 7f6e81784640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 0x7f6e7c103de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e7800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.077+0000 7f6e72ffd640 1 -- 172.21.15.45:0/3755443725 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78011830 con 0x7f6e7c102040 2024-03-12T05:16:17.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.077+0000 7f6e72ffd640 1 -- 172.21.15.45:0/3755443725 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e78011e40 con 0x7f6e7c102040 2024-03-12T05:16:17.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.077+0000 7f6e72ffd640 1 -- 172.21.15.45:0/3755443725 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800f2e0 con 0x7f6e7c102040 2024-03-12T05:16:17.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.078+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7c1b2da0 con 0x7f6e7c102040 2024-03-12T05:16:17.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.078+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c1b31d0 con 0x7f6e7c102040 2024-03-12T05:16:17.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.081+0000 7f6e72ffd640 1 -- 172.21.15.45:0/3755443725 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6e78011990 con 0x7f6e7c102040 2024-03-12T05:16:17.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.081+0000 7f6e86dc3740 1 Processor -- start 2024-03-12T05:16:17.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 -- start start 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d6d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d76210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9d28960 con 0x55e5a9d6d250 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9cdc310 con 0x55e5a9d76750 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9cd9840 con 0x55e5a9d6db70 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e82786640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d6d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d76210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e81784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e81784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53290/0 (socket says 172.21.15.45:53290) 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e82786640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d6d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37976/0 (socket says 172.21.15.45:37976) 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.082+0000 7f6e81784640 1 -- 172.21.15.45:0/1736483326 learned_addr learned my addr 172.21.15.45:0/1736483326 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 -- 172.21.15.45:0/1736483326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 msgr2=0x55e5a9d76210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d76210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 -- 172.21.15.45:0/1736483326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 msgr2=0x55e5a9d6d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d6d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 -- 172.21.15.45:0/1736483326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a9c97060 con 0x55e5a9d76750 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e81784640 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e780ca5e0 tx=0x7f6e780d54b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88aa8966c5a096b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e67fff640 1 -- 172.21.15.45:0/1736483326 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780db400 con 0x55e5a9d76750 2024-03-12T05:16:17.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.083+0000 7f6e67fff640 1 -- 172.21.15.45:0/1736483326 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e780dba10 con 0x55e5a9d76750 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.084+0000 7f6e72ffd640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 0x7f6e680a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.084+0000 7f6e72ffd640 1 -- 172.21.15.45:0/3755443725 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e780cce10 con 0x7f6e7c102040 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.084+0000 7f6e82786640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 0x7f6e680a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.084+0000 7f6e86dc3740 1 -- 172.21.15.45:0/1736483326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 msgr2=0x55e5a9d78b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.084+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e780ca5e0 tx=0x7f6e780d54b0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 -- 172.21.15.45:0/1736483326 shutdown_connections 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d78b40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d76210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/1736483326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d6d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 -- 172.21.15.45:0/1736483326 >> 172.21.15.45:0/1736483326 conn(0x55e5a9d67980 msgr2=0x55e5a9d67d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e82786640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 0x7f6e680a40e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e7402c680 tx=0x7f6e74009800 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 -- 172.21.15.45:0/1736483326 shutdown_connections 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 -- 172.21.15.45:0/1736483326 wait complete. 2024-03-12T05:16:17.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 -- start start 2024-03-12T05:16:17.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.085+0000 7f6e86dc3740 1 Processor -- start 2024-03-12T05:16:17.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6d250 0x55e5a9d73660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9cd5020 con 0x55e5a9d6db70 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a99a8430 con 0x55e5a9d76750 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9964e00 con 0x55e5a9d6d250 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e86dc3740 10 client.0 _ll_register_callbacks cb 0x55e5a9c7ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e81784640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e81784640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37984/0 (socket says 172.21.15.45:37984) 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e81784640 1 -- 172.21.15.45:0/966493078 learned_addr learned my addr 172.21.15.45:0/966493078 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e82786640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.086+0000 7f6e81f85640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6d250 0x55e5a9d73660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 -- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6d250 msgr2=0x55e5a9d73660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6d250 0x55e5a9d73660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 -- 172.21.15.45:0/966493078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 msgr2=0x55e5a9d740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 -- 172.21.15.45:0/966493078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c1b31d0 con 0x55e5a9d6db70 2024-03-12T05:16:17.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e82786640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:17.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e81784640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6e78096520 tx=0x7f6e78010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94d15ed028368c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.087+0000 7f6e657fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:17.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780d0450 con 0x55e5a9d6db70 2024-03-12T05:16:17.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e78010610 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5a99d47e0 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e780091f0 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e86dc3740 10 client.16914 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.088+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e5a9cd36a0 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e78095df0 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6e780ca7e0 con 0x55e5a9d6db70 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 10 client.16914 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 10 client.16914 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 20 client.16914 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9d86710 0x55e5a9d88b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 4 client.16914 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:17.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5a99d47e0 con 0x55e5a9d86710 2024-03-12T05:16:17.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.089+0000 7f6e81f85640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9d86710 0x55e5a9d88b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.092+0000 7f6e657fa640 10 client.16914 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:17.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.092+0000 7f6e81f85640 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9d86710 0x55e5a9d88b30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d851d95ee9e7e0d7 server_cookie=7a7ff572e399da7 in_seq=0 out_seq=0 2024-03-12T05:16:17.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.093+0000 7f6e657fa640 1 -- 172.21.15.45:0/966493078 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e5a99d47e0 con 0x55e5a9d86710 2024-03-12T05:16:17.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.093+0000 7f6e657fa640 10 client.16914 handle_command_reply: tid=0 2024-03-12T05:16:17.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.093+0000 7f6e86dc3740 1 client.16914 shutdown 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.093+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 msgr2=0x55e5a9d73ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.093+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6e78096520 tx=0x7f6e78010800 comp rx=0 tx=0).stop 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 shutdown_connections 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9d86710 0x55e5a9d88b30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6db70 0x55e5a9d73ba0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/966493078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6d250 0x55e5a9d73660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e657fa640 0 client.16914 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:17.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 >> 172.21.15.45:0/966493078 conn(0x55e5a9d67980 msgr2=0x55e5a9d696d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 shutdown_connections 2024-03-12T05:16:17.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 1 -- 172.21.15.45:0/966493078 wait complete. 2024-03-12T05:16:17.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.094+0000 7f6e86dc3740 20 client.16914 trim_cache size 0 max 16384 2024-03-12T05:16:17.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.116+0000 7f6e86dc3740 1 Processor -- start 2024-03-12T05:16:17.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.116+0000 7f6e86dc3740 1 -- start start 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.116+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.116+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d72640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.116+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 0x55e5a9d708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53302/0 (socket says 172.21.15.45:53302) 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e81784640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d72640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e81784640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d72640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42396/0 (socket says 172.21.15.45:42396) 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e81f85640 1 -- 172.21.15.45:0/3535186964 learned_addr learned my addr 172.21.15.45:0/3535186964 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e82786640 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 0x55e5a9d708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9c5a130 con 0x55e5a9d76750 2024-03-12T05:16:17.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9d68af0 con 0x55e5a9d6d250 2024-03-12T05:16:17.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.117+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9d635c0 con 0x55e5a9d6db70 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e81f85640 1 -- 172.21.15.45:0/3535186964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 msgr2=0x55e5a9d72640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e81f85640 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d72640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e81f85640 1 -- 172.21.15.45:0/3535186964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 msgr2=0x55e5a9d708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e81f85640 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 0x55e5a9d708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e81f85640 1 -- 172.21.15.45:0/3535186964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5a9c6fc20 con 0x55e5a9d6d250 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.118+0000 7f6e82786640 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 0x55e5a9d708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.119+0000 7f6e81f85640 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009af0 tx=0x7f6e6c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2328469945c55a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.119+0000 7f6e677fe640 1 -- 172.21.15.45:0/3535186964 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c01cb00 con 0x55e5a9d6d250 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.119+0000 7f6e677fe640 1 -- 172.21.15.45:0/3535186964 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e6c01c040 con 0x55e5a9d6d250 2024-03-12T05:16:17.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.119+0000 7f6e677fe640 1 -- 172.21.15.45:0/3535186964 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c024780 con 0x55e5a9d6d250 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 msgr2=0x55e5a9d75d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009af0 tx=0x7f6e6c008040 comp rx=0 tx=0).stop 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 shutdown_connections 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d76750 0x55e5a9d708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d72640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/3535186964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d6d250 0x55e5a9d75d90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 >> 172.21.15.45:0/3535186964 conn(0x55e5a9d67980 msgr2=0x55e5a9d71f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 shutdown_connections 2024-03-12T05:16:17.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.121+0000 7f6e86dc3740 1 -- 172.21.15.45:0/3535186964 wait complete. 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.122+0000 7f6e86dc3740 1 -- start start 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.122+0000 7f6e86dc3740 1 Processor -- start 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d7c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d7c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9ddcc60 con 0x55e5a9d6d250 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9ddcda0 con 0x55e5a9d76750 2024-03-12T05:16:17.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5a9ddcee0 con 0x55e5a9d6db70 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e86dc3740 10 client.0 _ll_register_callbacks cb 0x7f6e780d07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e82786640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e81f85640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d7c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e82786640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53316/0 (socket says 172.21.15.45:53316) 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e82786640 1 -- 172.21.15.45:0/2951559121 learned_addr learned my addr 172.21.15.45:0/2951559121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.123+0000 7f6e81784640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d7c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 -- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 msgr2=0x55e5a9d7c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d7c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 -- 172.21.15.45:0/2951559121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 msgr2=0x55e5a9d7c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d7c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 -- 172.21.15.45:0/2951559121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c110a20 con 0x55e5a9d76750 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e82786640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c06a6d0 tx=0x7f6e740095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94bdb45c125b407 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e657fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7404f070 con 0x55e5a9d76750 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6e74032a00 con 0x55e5a9d76750 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e780dba10 con 0x55e5a9d76750 2024-03-12T05:16:17.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e74003970 con 0x55e5a9d76750 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e86dc3740 10 client.26387 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.124+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f6e6c009c80 con 0x55e5a9d76750 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6e780dba10 con 0x55e5a9d76750 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6e7405a070 con 0x55e5a9d76750 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 10 client.26387 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 10 client.26387 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 20 client.26387 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9dde730 0x55e5a9de0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 4 client.26387 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e81f85640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9dde730 0x55e5a9de0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.125+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e5a9a11720 con 0x55e5a9dde730 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.126+0000 7f6e81f85640 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9dde730 0x55e5a9de0b50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ae4fdb85408d933 server_cookie=fa4bb72b18a207f4 in_seq=0 out_seq=0 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.126+0000 7f6e657fa640 10 client.26387 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.126+0000 7f6e657fa640 1 -- 172.21.15.45:0/2951559121 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e5a9a11720 con 0x55e5a9dde730 2024-03-12T05:16:17.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.126+0000 7f6e657fa640 10 client.26387 handle_command_reply: tid=0 2024-03-12T05:16:17.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.126+0000 7f6e86dc3740 1 client.26387 shutdown 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 msgr2=0x55e5a9d7ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c06a6d0 tx=0x7f6e740095c0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 shutdown_connections 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55e5a9dde730 0x55e5a9de0b50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5a9d76750 0x55e5a9d7ccb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5a9d6db70 0x55e5a9d7c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 --2- 172.21.15.45:0/2951559121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5a9d6d250 0x55e5a9d7c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 >> 172.21.15.45:0/2951559121 conn(0x55e5a9d67980 msgr2=0x55e5a9d73600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e657fa640 0 client.26387 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 shutdown_connections 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.127+0000 7f6e86dc3740 1 -- 172.21.15.45:0/2951559121 wait complete. 2024-03-12T05:16:17.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.128+0000 7f6e86dc3740 20 client.26387 trim_cache size 0 max 16384 2024-03-12T05:16:17.126 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.130+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 msgr2=0x7f6e680a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.130+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 0x7f6e680a40e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6e7402c680 tx=0x7f6e74009800 comp rx=0 tx=0).stop 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.130+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 msgr2=0x7f6e7c103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.130+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 0x7f6e7c103de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e7800b870 comp rx=0 tx=0).stop 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 shutdown_connections 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6e680a1c20 0x7f6e680a40e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e7c10d9d0 0x7f6e7c1043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e7c102040 0x7f6e7c103de0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 --2- 172.21.15.45:0/3755443725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e7c069c30 0x7f6e7c10d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.131+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 >> 172.21.15.45:0/3755443725 conn(0x7f6e7c077060 msgr2=0x7f6e7c10e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.132+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 shutdown_connections 2024-03-12T05:16:17.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.132+0000 7f6e841e9640 1 -- 172.21.15.45:0/3755443725 wait complete. 2024-03-12T05:16:17.149 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:17.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:16:17.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 -- 172.21.15.45:0/3564600776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 msgr2=0x7ff1e4114400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3564600776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4114400 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff1d401a940 tx=0x7ff1d403cc60 comp rx=0 tx=0).stop 2024-03-12T05:16:17.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 -- 172.21.15.45:0/3564600776 shutdown_connections 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3564600776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4114400 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3564600776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3564600776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 0x7ff1e410da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 -- 172.21.15.45:0/3564600776 >> 172.21.15.45:0/3564600776 conn(0x7ff1e4076f50 msgr2=0x7ff1e4105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 -- 172.21.15.45:0/3564600776 shutdown_connections 2024-03-12T05:16:17.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.328+0000 7ff1e914f640 1 -- 172.21.15.45:0/3564600776 wait complete. 2024-03-12T05:16:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.330+0000 7ff1e914f640 1 Processor -- start 2024-03-12T05:16:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.330+0000 7ff1e914f640 1 -- start start 2024-03-12T05:16:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.330+0000 7ff1e914f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.330+0000 7ff1e914f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.330+0000 7ff1e914f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 0x7ff1e4199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e914f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e419a390 con 0x7ff1e410d640 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e914f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e419a500 con 0x7ff1e41076c0 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e914f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e419a6c0 con 0x7ff1e4108080 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42426/0 (socket says 172.21.15.45:42426) 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 -- 172.21.15.45:0/3868268498 learned_addr learned my addr 172.21.15.45:0/3868268498 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e359e640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 0x7ff1e4199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 -- 172.21.15.45:0/3868268498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 msgr2=0x7ff1e4110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4110130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 -- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 msgr2=0x7ff1e4199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 0x7ff1e4199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 -- 172.21.15.45:0/3868268498 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d401a570 con 0x7ff1e4108080 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.331+0000 7ff1e259c640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00eab0 tx=0x7ff1dc00ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.332+0000 7ff1cbfff640 1 -- 172.21.15.45:0/3868268498 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc00ce40 con 0x7ff1e4108080 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.332+0000 7ff1cbfff640 1 -- 172.21.15.45:0/3868268498 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff1dc011920 con 0x7ff1e4108080 2024-03-12T05:16:17.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.332+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e419a9a0 con 0x7ff1e4108080 2024-03-12T05:16:17.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.333+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e41a1fc0 con 0x7ff1e4108080 2024-03-12T05:16:17.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.333+0000 7ff1cbfff640 1 -- 172.21.15.45:0/3868268498 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0109e0 con 0x7ff1e4108080 2024-03-12T05:16:17.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.336+0000 7ff1cbfff640 1 -- 172.21.15.45:0/3868268498 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff1dc011a90 con 0x7ff1e4108080 2024-03-12T05:16:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 Processor -- start 2024-03-12T05:16:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 -- start start 2024-03-12T05:16:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa92840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa9b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baa4d960 con 0x55f9baa9b960 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baa01310 con 0x55f9baa92d80 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ba9fe840 con 0x55f9baa92460 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.337+0000 7ff1cbfff640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 0x7ff1cc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1cbfff640 1 -- 172.21.15.45:0/3868268498 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff1dc0cd6c0 con 0x7ff1e4108080 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e359e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa92840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38020/0 (socket says 172.21.15.45:38020) 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e359e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa92840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42440/0 (socket says 172.21.15.45:42440) 2024-03-12T05:16:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 -- 172.21.15.45:0/2304681541 learned_addr learned my addr 172.21.15.45:0/2304681541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e359e640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 0x7ff1cc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa9b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 -- 172.21.15.45:0/2304681541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 msgr2=0x55f9baa92840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa92840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 -- 172.21.15.45:0/2304681541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 msgr2=0x55f9baa9b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa9b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 -- 172.21.15.45:0/2304681541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9ba9bc060 con 0x55f9baa9b960 2024-03-12T05:16:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.338+0000 7ff1e259c640 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000f80 tx=0x7ff1dc09a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c49c98f4f7a06d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.339+0000 7ff1e359e640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 0x7ff1cc0a40e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4047040 tx=0x7ff1d4005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.339+0000 7ff1c97fa640 1 -- 172.21.15.45:0/2304681541 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc09a990 con 0x55f9baa9b960 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.339+0000 7ff1c97fa640 1 -- 172.21.15.45:0/2304681541 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff1dc09a290 con 0x55f9baa9b960 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.339+0000 7ff1c97fa640 1 -- 172.21.15.45:0/2304681541 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc096900 con 0x55f9baa9b960 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/2304681541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 msgr2=0x55f9baa9dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000f80 tx=0x7ff1dc09a120 comp rx=0 tx=0).stop 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/2304681541 shutdown_connections 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa9dd50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa9b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/2304681541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa92840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/2304681541 >> 172.21.15.45:0/2304681541 conn(0x55f9baa8cb90 msgr2=0x55f9baa8cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.340+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/2304681541 shutdown_connections 2024-03-12T05:16:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.341+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/2304681541 wait complete. 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.341+0000 7ff1ebdfe740 1 -- start start 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.341+0000 7ff1ebdfe740 1 Processor -- start 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.341+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.341+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa98f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa99450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:42456/0 (socket says 172.21.15.45:42456) 2024-03-12T05:16:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa98f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ba9fa020 con 0x55f9baa9b960 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ba6cd430 con 0x55f9baa92d80 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1ebdfe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ba689e00 con 0x55f9baa92460 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa98f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53340/0 (socket says 172.21.15.45:53340) 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/1391261324 learned_addr learned my addr 172.21.15.45:0/1391261324 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1ebdfe740 10 client.0 _ll_register_callbacks cb 0x55f9ba9a1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e359e640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa99450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/1391261324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 msgr2=0x55f9baa98f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa98f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/1391261324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 msgr2=0x55f9baa99450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa99450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.342+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/1391261324 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e41a1fc0 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.343+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80117b0 tx=0x7ff1d8011c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b5cfaca3d658a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.343+0000 7ff1c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8016070 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff1d8010ba0 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9ba6f97e0 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1d8010d60 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1ebdfe740 10 client.26392 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.344+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f9ba9f86a0 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f9ba6f97e0 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff1d8021050 con 0x55f9baa92460 2024-03-12T05:16:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 10 client.26392 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 10 client.26392 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 20 client.26392 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9baaaba60 0x55f9baaade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 4 client.26392 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9ba736720 con 0x55f9baaaba60 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.345+0000 7ff1e259c640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9baaaba60 0x55f9baaade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.346+0000 7ff1c6ff5640 10 client.26392 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.346+0000 7ff1e259c640 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9baaaba60 0x55f9baaade70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3689a99435bb008 server_cookie=1aef401d36f7fe4f in_seq=0 out_seq=0 2024-03-12T05:16:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.347+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/1391261324 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f9ba736720 con 0x55f9baaaba60 2024-03-12T05:16:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.347+0000 7ff1c6ff5640 10 client.26392 handle_command_reply: tid=0 2024-03-12T05:16:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.347+0000 7ff1ebdfe740 1 client.26392 shutdown 2024-03-12T05:16:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 msgr2=0x55f9baa989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80117b0 tx=0x7ff1d8011c70 comp rx=0 tx=0).stop 2024-03-12T05:16:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 shutdown_connections 2024-03-12T05:16:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa9b960 0x55f9baa99450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9baaaba60 0x55f9baaade70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baa98f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/1391261324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa92460 0x55f9baa989d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1c6ff5640 0 client.26392 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 >> 172.21.15.45:0/1391261324 conn(0x55f9baa8cb90 msgr2=0x55f9baa8e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 shutdown_connections 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.348+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/1391261324 wait complete. 2024-03-12T05:16:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.349+0000 7ff1ebdfe740 20 client.26392 trim_cache size 0 max 16384 2024-03-12T05:16:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.373+0000 7ff1ebdfe740 1 Processor -- start 2024-03-12T05:16:17.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.373+0000 7ff1ebdfe740 1 -- start start 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 0x55f9baa95c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baa96190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53356/0 (socket says 172.21.15.45:53356) 2024-03-12T05:16:17.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/3102848833 learned_addr learned my addr 172.21.15.45:0/3102848833 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ba97f130 con 0x55f9baa92d80 2024-03-12T05:16:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baa88540 con 0x55f9baa92460 2024-03-12T05:16:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.374+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baa8db90 con 0x55f9baa9b960 2024-03-12T05:16:17.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/3102848833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 msgr2=0x55f9baa96190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baa96190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/3102848833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 msgr2=0x55f9baa95c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e259c640 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 0x55f9baa95c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 0x55f9baa95c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/3102848833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9ba994c20 con 0x55f9baa92460 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e259c640 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 0x55f9baa95c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.375+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8004de0 tx=0x7ff1d8005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb469835c24f757e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.376+0000 7ff1c8ff9640 1 -- 172.21.15.45:0/3102848833 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8016040 con 0x55f9baa92460 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.376+0000 7ff1c8ff9640 1 -- 172.21.15.45:0/3102848833 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff1d8004b10 con 0x55f9baa92460 2024-03-12T05:16:17.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.376+0000 7ff1c8ff9640 1 -- 172.21.15.45:0/3102848833 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80054f0 con 0x55f9baa92460 2024-03-12T05:16:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 msgr2=0x55f9baa9b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8004de0 tx=0x7ff1d8005900 comp rx=0 tx=0).stop 2024-03-12T05:16:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 shutdown_connections 2024-03-12T05:16:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92d80 0x55f9baa95c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baa96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/3102848833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92460 0x55f9baa9b100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 >> 172.21.15.45:0/3102848833 conn(0x55f9baa8cb90 msgr2=0x55f9baab2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 shutdown_connections 2024-03-12T05:16:17.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.377+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/3102848833 wait complete. 2024-03-12T05:16:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.378+0000 7ff1ebdfe740 1 -- start start 2024-03-12T05:16:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.378+0000 7ff1ebdfe740 1 Processor -- start 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baaa22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1e2d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38040/0 (socket says 172.21.15.45:38040) 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baaa22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1e259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baaa22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:53368/0 (socket says 172.21.15.45:53368) 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1ebdfe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baafb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/341506939 learned_addr learned my addr 172.21.15.45:0/341506939 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.379+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baafb660 con 0x55f9baa92460 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e359e640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baafb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baafb7d0 con 0x55f9baa92d80 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9baafb990 con 0x55f9baa9b960 2024-03-12T05:16:17.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1ebdfe740 10 client.0 _ll_register_callbacks cb 0x7ff1dc0bb200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 msgr2=0x55f9baafb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baafb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/341506939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 msgr2=0x55f9baaa22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baaa22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.380+0000 7ff1e2d9d640 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ba9f86a0 con 0x55f9baa92460 2024-03-12T05:16:17.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.381+0000 7ff1e2d9d640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8011cb0 tx=0x7ff1d8004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37337f93cf2c58b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.381+0000 7ff1c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.381+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8016070 con 0x55f9baa92460 2024-03-12T05:16:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.382+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7ff1d8004f80 con 0x55f9baa92460 2024-03-12T05:16:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.382+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1dc09a290 con 0x55f9baa92460 2024-03-12T05:16:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.383+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1d8005680 con 0x55f9baa92460 2024-03-12T05:16:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.383+0000 7ff1ebdfe740 10 client.16935 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.383+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f9ba9ba8c0 con 0x55f9baa92460 2024-03-12T05:16:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff1d803e030 con 0x55f9baa92460 2024-03-12T05:16:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff1d8021040 con 0x55f9baa92460 2024-03-12T05:16:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 10 client.16935 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 10 client.16935 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:16:17.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 20 client.16935 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9bab03f30 0x55f9bab06350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 4 client.16935 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9ba785890 con 0x55f9bab03f30 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.384+0000 7ff1e259c640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9bab03f30 0x55f9bab06350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.386+0000 7ff1e259c640 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9bab03f30 0x55f9bab06350 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acd81eaa0e2c99a0 server_cookie=3dad97efa8ee7df1 in_seq=0 out_seq=0 2024-03-12T05:16:17.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.386+0000 7ff1c6ff5640 10 client.16935 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:17.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.386+0000 7ff1c6ff5640 1 -- 172.21.15.45:0/341506939 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9ba785890 con 0x55f9bab03f30 2024-03-12T05:16:17.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.386+0000 7ff1c6ff5640 10 client.16935 handle_command_reply: tid=0 2024-03-12T05:16:17.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.386+0000 7ff1ebdfe740 1 client.16935 shutdown 2024-03-12T05:16:17.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 msgr2=0x55f9baaa1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8011cb0 tx=0x7ff1d8004890 comp rx=0 tx=0).stop 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 shutdown_connections 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f9bab03f30 0x55f9bab06350 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9baa9b960 0x55f9baafb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1c6ff5640 0 client.16935 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9baa92d80 0x55f9baaa22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 --2- 172.21.15.45:0/341506939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f9baa92460 0x55f9baaa1db0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.387+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 >> 172.21.15.45:0/341506939 conn(0x55f9baa8cb90 msgr2=0x55f9baa8b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.388+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 shutdown_connections 2024-03-12T05:16:17.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.388+0000 7ff1ebdfe740 1 -- 172.21.15.45:0/341506939 wait complete. 2024-03-12T05:16:17.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.388+0000 7ff1ebdfe740 20 client.16935 trim_cache size 0 max 16384 2024-03-12T05:16:17.385 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:17.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.389+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 msgr2=0x7ff1cc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.389+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 0x7ff1cc0a40e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4047040 tx=0x7ff1d4005a20 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.389+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 msgr2=0x7ff1e4110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.389+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00eab0 tx=0x7ff1dc00ef70 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 shutdown_connections 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff1cc0a1c20 0x7ff1cc0a40e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1e410d640 0x7ff1e4199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1e4108080 0x7ff1e4110670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 --2- 172.21.15.45:0/3868268498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1e41076c0 0x7ff1e4110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 >> 172.21.15.45:0/3868268498 conn(0x7ff1e4076f50 msgr2=0x7ff1e40781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 shutdown_connections 2024-03-12T05:16:17.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.390+0000 7ff1e914f640 1 -- 172.21.15.45:0/3868268498 wait complete. 2024-03-12T05:16:17.405 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:17 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1379155293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:17.406 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:17.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:16:17.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:17 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1379155293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:17.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:17 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1379155293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 -- 172.21.15.45:0/1417550798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c1060d0 msgr2=0x7fa22c068380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 --2- 172.21.15.45:0/1417550798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c1060d0 0x7fa22c068380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009930 tx=0x7fa21c02bc20 comp rx=0 tx=0).stop 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 -- 172.21.15.45:0/1417550798 shutdown_connections 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 --2- 172.21.15.45:0/1417550798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 0x7fa22c1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 --2- 172.21.15.45:0/1417550798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c0688c0 0x7fa22c068d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 --2- 172.21.15.45:0/1417550798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c1060d0 0x7fa22c068380 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 -- 172.21.15.45:0/1417550798 >> 172.21.15.45:0/1417550798 conn(0x7fa22c077180 msgr2=0x7fa22c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 -- 172.21.15.45:0/1417550798 shutdown_connections 2024-03-12T05:16:17.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.580+0000 7fa23224e640 1 -- 172.21.15.45:0/1417550798 wait complete. 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.581+0000 7fa23224e640 1 Processor -- start 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.581+0000 7fa23224e640 1 -- start start 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 0x7fa22c1a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c1060d0 0x7fa22c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c19de60 con 0x7fa22c103c40 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c19dfd0 con 0x7fa22c0688c0 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa23224e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22c19e190 con 0x7fa22c1060d0 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58456/0 (socket says 172.21.15.45:58456) 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa22b7fe640 1 -- 172.21.15.45:0/250260517 learned_addr learned my addr 172.21.15.45:0/250260517 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa22bfff640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c1060d0 0x7fa22c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.582+0000 7fa222ffd640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 0x7fa22c1a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 -- 172.21.15.45:0/250260517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c1060d0 msgr2=0x7fa22c19d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c1060d0 0x7fa22c19d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 -- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 msgr2=0x7fa22c1a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 0x7fa22c1a3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 -- 172.21.15.45:0/250260517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c009560 con 0x7fa22c0688c0 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.583+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009850 tx=0x7fa21c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.584+0000 7fa2297fa640 1 -- 172.21.15.45:0/250260517 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0044a0 con 0x7fa22c0688c0 2024-03-12T05:16:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.584+0000 7fa2297fa640 1 -- 172.21.15.45:0/250260517 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa21c003cb0 con 0x7fa22c0688c0 2024-03-12T05:16:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.584+0000 7fa2297fa640 1 -- 172.21.15.45:0/250260517 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c005620 con 0x7fa22c0688c0 2024-03-12T05:16:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.584+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22c19e410 con 0x7fa22c0688c0 2024-03-12T05:16:17.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.584+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22c1aa4a0 con 0x7fa22c0688c0 2024-03-12T05:16:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.587+0000 7fa2297fa640 1 -- 172.21.15.45:0/250260517 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa21c003800 con 0x7fa22c0688c0 2024-03-12T05:16:17.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa2297fa640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 0x7fa2180a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa2297fa640 1 -- 172.21.15.45:0/250260517 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa21c0f03b0 con 0x7fa22c0688c0 2024-03-12T05:16:17.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa234e1e740 1 Processor -- start 2024-03-12T05:16:17.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa234e1e740 1 -- start start 2024-03-12T05:16:17.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa222ffd640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 0x7fa2180a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.588+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9b70 0x55dc03902210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 0x55dc03904b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa234e1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc038b4960 con 0x55dc038f9b70 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa234e1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03868310 con 0x55dc038f9250 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa234e1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03865840 con 0x55dc03902750 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58466/0 (socket says 172.21.15.45:58466) 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa222ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9b70 0x55dc03902210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 -- 172.21.15.45:0/655082856 learned_addr learned my addr 172.21.15.45:0/655082856 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22bfff640 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 0x55dc03904b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 -- 172.21.15.45:0/655082856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 msgr2=0x55dc03904b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 0x55dc03904b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 -- 172.21.15.45:0/655082856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9b70 msgr2=0x55dc03902210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9b70 0x55dc03902210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 -- 172.21.15.45:0/655082856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc03823060 con 0x55dc038f9250 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22bfff640 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 0x55dc03904b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.589+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0b9b80 tx=0x7fa21c0bcb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0798b4135684c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.590+0000 7fa221ffb640 1 -- 172.21.15.45:0/655082856 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0c10d0 con 0x55dc038f9250 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.590+0000 7fa221ffb640 1 -- 172.21.15.45:0/655082856 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa21c0bd1b0 con 0x55dc038f9250 2024-03-12T05:16:17.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.590+0000 7fa221ffb640 1 -- 172.21.15.45:0/655082856 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0edd20 con 0x55dc038f9250 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 -- 172.21.15.45:0/655082856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 msgr2=0x55dc038f9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0b9b80 tx=0x7fa21c0bcb10 comp rx=0 tx=0).stop 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa222ffd640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 0x7fa2180a41a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa214005e10 tx=0x7fa2140127c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 -- 172.21.15.45:0/655082856 shutdown_connections 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9b70 0x55dc03902210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03902750 0x55dc03904b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 --2- 172.21.15.45:0/655082856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038f9630 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 -- 172.21.15.45:0/655082856 >> 172.21.15.45:0/655082856 conn(0x55dc038f3980 msgr2=0x55dc038f3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 -- 172.21.15.45:0/655082856 shutdown_connections 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.591+0000 7fa234e1e740 1 -- 172.21.15.45:0/655082856 wait complete. 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.592+0000 7fa234e1e740 1 -- start start 2024-03-12T05:16:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.592+0000 7fa234e1e740 1 Processor -- start 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.593+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038ff740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.593+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc038ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.593+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc039001c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.593+0000 7fa222ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc038ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.593+0000 7fa222ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc038ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46684/0 (socket says 172.21.15.45:46684) 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa222ffd640 1 -- 172.21.15.45:0/3542174135 learned_addr learned my addr 172.21.15.45:0/3542174135 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03861020 con 0x55dc03902750 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03534430 con 0x55dc038f9250 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc034f0e00 con 0x55dc038f9b70 2024-03-12T05:16:17.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22bfff640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc039001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa234e1e740 10 client.0 _ll_register_callbacks cb 0x55dc03808cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 -- 172.21.15.45:0/3542174135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 msgr2=0x55dc038ffc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc038ffc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 -- 172.21.15.45:0/3542174135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 msgr2=0x55dc039001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc039001c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.594+0000 7fa22b7fe640 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22c1aa4a0 con 0x55dc038f9250 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc035607e0 con 0x55dc038f9250 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038ff740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0b9b80 tx=0x7fa21c0bcc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ddd4f40c1894734 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa20affd640 10 client.26417 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa20affd640 1 -- 172.21.15.45:0/3542174135 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0c1300 con 0x55dc038f9250 2024-03-12T05:16:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa20affd640 1 -- 172.21.15.45:0/3542174135 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa21c0c5950 con 0x55dc038f9250 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.595+0000 7fa20affd640 1 -- 172.21.15.45:0/3542174135 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa21c0c10d0 con 0x55dc038f9250 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 10 client.26417 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55dc0385f6a0 con 0x55dc038f9250 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa20affd640 1 -- 172.21.15.45:0/3542174135 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa21c0b7d20 con 0x55dc038f9250 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 10 client.26417 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 10 client.26417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 20 client.26417 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc039255e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 4 client.26417 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc0359d720 con 0x55dc039231f0 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.596+0000 7fa222ffd640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc039255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.597+0000 7fa20affd640 10 client.26417 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.597+0000 7fa222ffd640 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc039255e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a988f947192a1e9d server_cookie=725c36f93ecb4e2c in_seq=0 out_seq=0 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.597+0000 7fa20affd640 1 -- 172.21.15.45:0/3542174135 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dc0359d720 con 0x55dc039231f0 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.597+0000 7fa20affd640 10 client.26417 handle_command_reply: tid=0 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.597+0000 7fa234e1e740 1 client.26417 shutdown 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 msgr2=0x55dc038ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038ff740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0b9b80 tx=0x7fa21c0bcc40 comp rx=0 tx=0).stop 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 shutdown_connections 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc039255e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc039001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa20affd640 0 client.26417 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc038ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 --2- 172.21.15.45:0/3542174135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9250 0x55dc038ff740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 >> 172.21.15.45:0/3542174135 conn(0x55dc038f3980 msgr2=0x55dc038f56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 shutdown_connections 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 1 -- 172.21.15.45:0/3542174135 wait complete. 2024-03-12T05:16:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.598+0000 7fa234e1e740 20 client.26417 trim_cache size 0 max 16384 2024-03-12T05:16:17.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 Processor -- start 2024-03-12T05:16:17.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 -- start start 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9250 0x55dc03901e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 0x55dc038fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc038fc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc037e6130 con 0x55dc03902750 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc038f8620 con 0x55dc038f9b70 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa234e1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc038f8eb0 con 0x55dc038f9250 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa22bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc038fc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa222ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 0x55dc038fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa22bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc038fc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39148/0 (socket says 172.21.15.45:39148) 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa22bfff640 1 -- 172.21.15.45:0/1585891839 learned_addr learned my addr 172.21.15.45:0/1585891839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.620+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9250 0x55dc03901e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 -- 172.21.15.45:0/1585891839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9250 msgr2=0x55dc03901e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9250 0x55dc03901e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 -- 172.21.15.45:0/1585891839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 msgr2=0x55dc038fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc038fc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 -- 172.21.15.45:0/1585891839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc037fbc20 con 0x55dc038f9b70 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa222ffd640 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 0x55dc038fffe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55dc03925cb0 tx=0x7fa214018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebd95f2be88ccac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.621+0000 7fa2217fa640 1 -- 172.21.15.45:0/1585891839 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214024070 con 0x55dc038f9b70 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.622+0000 7fa2217fa640 1 -- 172.21.15.45:0/1585891839 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa21400ad50 con 0x55dc038f9b70 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.622+0000 7fa2217fa640 1 -- 172.21.15.45:0/1585891839 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214025790 con 0x55dc038f9b70 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 -- 172.21.15.45:0/1585891839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 msgr2=0x55dc038fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 0x55dc038fffe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55dc03925cb0 tx=0x7fa214018000 comp rx=0 tx=0).stop 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 -- 172.21.15.45:0/1585891839 shutdown_connections 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc03902750 0x55dc038fc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc038f9b70 0x55dc038fffe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 --2- 172.21.15.45:0/1585891839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9250 0x55dc03901e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 -- 172.21.15.45:0/1585891839 >> 172.21.15.45:0/1585891839 conn(0x55dc038f3980 msgr2=0x55dc038faa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 -- 172.21.15.45:0/1585891839 shutdown_connections 2024-03-12T05:16:17.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.623+0000 7fa234e1e740 1 -- 172.21.15.45:0/1585891839 wait complete. 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.624+0000 7fa234e1e740 1 -- start start 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.624+0000 7fa234e1e740 1 Processor -- start 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 0x55dc03919410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc03919950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 0x55dc039794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 0x55dc03919410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0391a050 con 0x55dc038f9250 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03979a30 con 0x55dc03902750 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa22b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 0x55dc03919410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39158/0 (socket says 172.21.15.45:39158) 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03979b70 con 0x55dc038f9b70 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa222ffd640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc03919950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa22b7fe640 1 -- 172.21.15.45:0/2616630324 learned_addr learned my addr 172.21.15.45:0/2616630324 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa234e1e740 10 client.0 _ll_register_callbacks cb 0x7fa21c004ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.625+0000 7fa22bfff640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 0x55dc039794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 -- 172.21.15.45:0/2616630324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 msgr2=0x55dc03919950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc03919950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 -- 172.21.15.45:0/2616630324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 msgr2=0x55dc03919410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 0x55dc03919410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 -- 172.21.15.45:0/2616630324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa214018070 con 0x55dc03902750 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa22bfff640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 0x55dc039794f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa224005f40 tx=0x7fa224003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c825ec3c42dc5b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa20affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224011d30 con 0x55dc03902750 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa224010460 con 0x55dc03902750 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa21c0bd1b0 con 0x55dc03902750 2024-03-12T05:16:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa224011e90 con 0x55dc03902750 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa234e1e740 10 client.26423 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.626+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dc038218c0 con 0x55dc03902750 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa224029080 con 0x55dc03902750 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa224023030 con 0x55dc03902750 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 10 client.26423 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 10 client.26423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 20 client.26423 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc0397d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 4 client.26423 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dc035ec890 con 0x55dc039231f0 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.627+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc0397d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.628+0000 7fa20affd640 10 client.26423 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.628+0000 7fa22b7fe640 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc0397d3f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f863728cb06114ae server_cookie=49a92f1b5e04029b in_seq=0 out_seq=0 2024-03-12T05:16:17.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.628+0000 7fa20affd640 1 -- 172.21.15.45:0/2616630324 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dc035ec890 con 0x55dc039231f0 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.628+0000 7fa20affd640 10 client.26423 handle_command_reply: tid=0 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.628+0000 7fa234e1e740 1 client.26423 shutdown 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 msgr2=0x55dc039794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 0x55dc039794f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa224005f40 tx=0x7fa224003f90 comp rx=0 tx=0).stop 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 shutdown_connections 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dc039231f0 0x55dc0397d3f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc03902750 0x55dc039794f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc038f9b70 0x55dc03919950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 --2- 172.21.15.45:0/2616630324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc038f9250 0x55dc03919410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa20affd640 0 client.26423 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 >> 172.21.15.45:0/2616630324 conn(0x55dc038f3980 msgr2=0x55dc038f4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 shutdown_connections 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.629+0000 7fa234e1e740 1 -- 172.21.15.45:0/2616630324 wait complete. 2024-03-12T05:16:17.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.630+0000 7fa234e1e740 20 client.26423 trim_cache size 0 max 16384 2024-03-12T05:16:17.628 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.630+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 msgr2=0x7fa2180a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.630+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 0x7fa2180a41a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa214005e10 tx=0x7fa2140127c0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 msgr2=0x7fa22c1a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa21c009850 tx=0x7fa21c004270 comp rx=0 tx=0).stop 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 shutdown_connections 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa2180a1ce0 0x7fa2180a41a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa22c103c40 0x7fa22c1a3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa22c1060d0 0x7fa22c19d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 --2- 172.21.15.45:0/250260517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa22c0688c0 0x7fa22c1a3650 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 >> 172.21.15.45:0/250260517 conn(0x7fa22c077180 msgr2=0x7fa22c0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 shutdown_connections 2024-03-12T05:16:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:17.631+0000 7fa23224e640 1 -- 172.21.15.45:0/250260517 wait complete. 2024-03-12T05:16:18.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:18 smithi130 ceph-mon[35398]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-03-12T05:16:18.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:18 smithi116 ceph-mon[35486]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-03-12T05:16:18.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:18 smithi045 ceph-mon[32174]: pgmap v165: 97 pgs: 97 active+clean; 2.8 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.18k op/s 2024-03-12T05:16:18.644 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:16:18.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:16:18.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 -- 172.21.15.45:0/3100344345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de8004730 msgr2=0x7f2de8004bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3100344345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de8004730 0x7f2de8004bb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009860 tx=0x7f2dec02bb70 comp rx=0 tx=0).stop 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 -- 172.21.15.45:0/3100344345 shutdown_connections 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3100344345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de80a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3100344345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de8004730 0x7f2de8004bb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3100344345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de809be00 0x7f2de809c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.816+0000 7f2df68d8640 1 -- 172.21.15.45:0/3100344345 >> 172.21.15.45:0/3100344345 conn(0x7f2de800a730 msgr2=0x7f2de800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.817+0000 7f2df68d8640 1 -- 172.21.15.45:0/3100344345 shutdown_connections 2024-03-12T05:16:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.817+0000 7f2df68d8640 1 -- 172.21.15.45:0/3100344345 wait complete. 2024-03-12T05:16:18.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.817+0000 7f2df68d8640 1 Processor -- start 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.817+0000 7f2df68d8640 1 -- start start 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 0x7f2de812e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de812e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8133290 con 0x7f2de80050f0 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8133400 con 0x7f2de809be00 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df68d8640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81335c0 con 0x7f2de8004730 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58516/0 (socket says 172.21.15.45:58516) 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 -- 172.21.15.45:0/3070981078 learned_addr learned my addr 172.21.15.45:0/3070981078 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:18.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df50d5640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de812e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df58d6640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 0x7f2de812e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 -- 172.21.15.45:0/3070981078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 msgr2=0x7f2de812e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 0x7f2de812e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.818+0000 7f2df60d7640 1 -- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 msgr2=0x7f2de812e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df60d7640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de812e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df60d7640 1 -- 172.21.15.45:0/3070981078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df004f750 con 0x7f2de809be00 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df58d6640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 0x7f2de812e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df50d5640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de812e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df60d7640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2df00507e0 tx=0x7f2df006a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2de6ffd640 1 -- 172.21.15.45:0/3070981078 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df007b050 con 0x7f2de809be00 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2de6ffd640 1 -- 172.21.15.45:0/3070981078 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2df007c040 con 0x7f2de809be00 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2de6ffd640 1 -- 172.21.15.45:0/3070981078 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df00816a0 con 0x7f2de809be00 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec009490 con 0x7f2de809be00 2024-03-12T05:16:18.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.819+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de813af40 con 0x7f2de809be00 2024-03-12T05:16:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.823+0000 7f2de6ffd640 1 -- 172.21.15.45:0/3070981078 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2df0074a70 con 0x7f2de809be00 2024-03-12T05:16:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.823+0000 7f2de6ffd640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 0x7f2ddc0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.824+0000 7f2de6ffd640 1 -- 172.21.15.45:0/3070981078 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f2df0138180 con 0x7f2de809be00 2024-03-12T05:16:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.824+0000 7f2df58d6640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 0x7f2ddc0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.824+0000 7f2dfa7e9740 1 Processor -- start 2024-03-12T05:16:18.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.825+0000 7f2dfa7e9740 1 -- start start 2024-03-12T05:16:18.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.825+0000 7f2df58d6640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 0x7f2ddc0a4160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2de000abe0 tx=0x7f2de000a5c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df4d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df443f90 con 0x5614df4d55d0 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4414c0 con 0x5614df4deb00 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df43cca0 con 0x5614df4d5f80 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46730/0 (socket says 172.21.15.45:46730) 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 -- 172.21.15.45:0/989963056 learned_addr learned my addr 172.21.15.45:0/989963056 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df60d7640 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 -- 172.21.15.45:0/989963056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 msgr2=0x5614df4e0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 -- 172.21.15.45:0/989963056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 msgr2=0x5614df4d59b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df4d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.826+0000 7f2df50d5640 1 -- 172.21.15.45:0/989963056 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614df3ff0b0 con 0x5614df4d5f80 2024-03-12T05:16:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.827+0000 7f2df50d5640 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2dec02c0b0 tx=0x7f2dec032360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7afb39aa9b22f2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.827+0000 7f2ddbfff640 1 -- 172.21.15.45:0/989963056 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec004380 con 0x5614df4d5f80 2024-03-12T05:16:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.827+0000 7f2ddbfff640 1 -- 172.21.15.45:0/989963056 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2dec02ce90 con 0x5614df4d5f80 2024-03-12T05:16:18.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.827+0000 7f2ddbfff640 1 -- 172.21.15.45:0/989963056 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0035f0 con 0x5614df4d5f80 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/989963056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 msgr2=0x5614df4de530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2dec02c0b0 tx=0x7f2dec032360 comp rx=0 tx=0).stop 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/989963056 shutdown_connections 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df4de530 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/989963056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df4d59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.828+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/989963056 >> 172.21.15.45:0/989963056 conn(0x5614df4cfa90 msgr2=0x5614df4cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.829+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/989963056 shutdown_connections 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.829+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/989963056 wait complete. 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.829+0000 7f2dfa7e9740 1 -- start start 2024-03-12T05:16:18.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.829+0000 7f2dfa7e9740 1 Processor -- start 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4dbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4dc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df3e1f40 con 0x5614df4d5f80 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df1104b0 con 0x5614df4deb00 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df3c2670 con 0x5614df4d55d0 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2dfa7e9740 10 client.0 _ll_register_callbacks cb 0x5614df42ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58544/0 (socket says 172.21.15.45:58544) 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 -- 172.21.15.45:0/2816607802 learned_addr learned my addr 172.21.15.45:0/2816607802 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df50d5640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4dbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 -- 172.21.15.45:0/2816607802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 msgr2=0x5614df4dbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4dbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 -- 172.21.15.45:0/2816607802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 msgr2=0x5614df4dc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4dc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.830+0000 7f2df60d7640 1 -- 172.21.15.45:0/2816607802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de80a3480 con 0x5614df4deb00 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.831+0000 7f2df60d7640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2df0101720 tx=0x7f2df0105510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67eb901094d1c91c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.831+0000 7f2dd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.831+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0135cf0 con 0x5614df4deb00 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.831+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f2df010d4f0 con 0x5614df4deb00 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.831+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614df416f40 con 0x5614df4deb00 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df01041a0 con 0x5614df4deb00 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 10 client.26435 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:18.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5614df43b320 con 0x5614df4deb00 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2df0123030 con 0x5614df4deb00 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2df0074650 con 0x5614df4deb00 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 10 client.26435 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 20 client.26435 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df4ef280 0x5614df4f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 4 client.26435 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.832+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5614df433310 con 0x5614df4ef280 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.833+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df4ef280 0x5614df4f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.834+0000 7f2dd97fa640 10 client.26435 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.834+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df4ef280 0x5614df4f16a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=291f3bffdebc057b server_cookie=d1cfd8ab01f78a22 in_seq=0 out_seq=0 2024-03-12T05:16:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.834+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2816607802 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5614df433310 con 0x5614df4ef280 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.834+0000 7f2dd97fa640 10 client.26435 handle_command_reply: tid=0 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.834+0000 7f2dfa7e9740 1 client.26435 shutdown 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 msgr2=0x5614df4dc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2df0101720 tx=0x7f2df0105510 comp rx=0 tx=0).stop 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 shutdown_connections 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df4ef280 0x5614df4f16a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4dc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4dc5b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2816607802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4dbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 >> 172.21.15.45:0/2816607802 conn(0x5614df4cfa90 msgr2=0x5614df4d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dd97fa640 0 client.26435 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 shutdown_connections 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2816607802 wait complete. 2024-03-12T05:16:18.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.835+0000 7f2dfa7e9740 20 client.26435 trim_cache size 0 max 16384 2024-03-12T05:16:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.863+0000 7f2dfa7e9740 1 Processor -- start 2024-03-12T05:16:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 -- start start 2024-03-12T05:16:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2df58d6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2df58d6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46736/0 (socket says 172.21.15.45:46736) 2024-03-12T05:16:18.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4d4f20 con 0x5614df4d5f80 2024-03-12T05:16:18.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4ebc50 con 0x5614df4deb00 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4cb6e0 con 0x5614df4d55d0 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2df50d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2df58d6640 1 -- 172.21.15.45:0/367651644 learned_addr learned my addr 172.21.15.45:0/367651644 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.864+0000 7f2df60d7640 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 -- 172.21.15.45:0/367651644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 msgr2=0x5614df4e5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 -- 172.21.15.45:0/367651644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 msgr2=0x5614df4ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 -- 172.21.15.45:0/367651644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614df3d7c00 con 0x5614df4d55d0 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df50d5640 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2df58d6640 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2de000a910 tx=0x7f2de0011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=686c3a85913cee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2ddb7fe640 1 -- 172.21.15.45:0/367651644 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0024d10 con 0x5614df4d55d0 2024-03-12T05:16:18.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2ddb7fe640 1 -- 172.21.15.45:0/367651644 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2de0024040 con 0x5614df4d55d0 2024-03-12T05:16:18.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.865+0000 7f2ddb7fe640 1 -- 172.21.15.45:0/367651644 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0010a10 con 0x5614df4d55d0 2024-03-12T05:16:18.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/367651644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 msgr2=0x5614df4de260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2de000a910 tx=0x7f2de0011000 comp rx=0 tx=0).stop 2024-03-12T05:16:18.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/367651644 shutdown_connections 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d5f80 0x5614df4ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/367651644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d55d0 0x5614df4de260 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.866+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/367651644 >> 172.21.15.45:0/367651644 conn(0x5614df4cfa90 msgr2=0x5614df4ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.867+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/367651644 shutdown_connections 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.867+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/367651644 wait complete. 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.867+0000 7f2dfa7e9740 1 -- start start 2024-03-12T05:16:18.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.867+0000 7f2dfa7e9740 1 Processor -- start 2024-03-12T05:16:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df548890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4e5a30 con 0x5614df4d55d0 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4e5ba0 con 0x5614df4deb00 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614df4e5d60 con 0x5614df4d5f80 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2dfa7e9740 10 client.0 _ll_register_callbacks cb 0x7f2df004fdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46740/0 (socket says 172.21.15.45:46740) 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df60d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58562/0 (socket says 172.21.15.45:58562) 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 -- 172.21.15.45:0/2674005830 learned_addr learned my addr 172.21.15.45:0/2674005830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df58d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df548890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 -- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 msgr2=0x5614df4e5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 -- 172.21.15.45:0/2674005830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 msgr2=0x5614df548890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df548890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df50d5640 1 -- 172.21.15.45:0/2674005830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5614df3ff0b0 con 0x5614df4d5f80 2024-03-12T05:16:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.868+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df548890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2df50d5640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2dec035fd0 tx=0x7f2dec0347b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d9b7afb1ee8d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dec0348e0 con 0x5614df4d5f80 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f2dec02c720 con 0x5614df4d5f80 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614df462900 con 0x5614df4d5f80 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dec02c8e0 con 0x5614df4d5f80 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dfa7e9740 10 client.26422 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:18.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.869+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5614df3fd910 con 0x5614df4d5f80 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2dec05f080 con 0x5614df4d5f80 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2dec03f480 con 0x5614df4d5f80 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 10 client.26422 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 20 client.26422 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df5503f0 0x5614df552810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 4 client.26422 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5614df13c860 con 0x5614df5503f0 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.870+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df5503f0 0x5614df552810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.871+0000 7f2dd97fa640 10 client.26422 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.871+0000 7f2df58d6640 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df5503f0 0x5614df552810 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2613000dba8596c1 server_cookie=831de22bb161f51c in_seq=0 out_seq=0 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dd97fa640 1 -- 172.21.15.45:0/2674005830 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5614df13c860 con 0x5614df5503f0 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dd97fa640 10 client.26422 handle_command_reply: tid=0 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 client.26422 shutdown 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 msgr2=0x5614df548dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2dec035fd0 tx=0x7f2dec0347b0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 shutdown_connections 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5614df5503f0 0x5614df552810 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614df4deb00 0x5614df4e5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614df4d5f80 0x5614df548dd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 --2- 172.21.15.45:0/2674005830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5614df4d55d0 0x5614df548890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dd97fa640 0 client.26422 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 >> 172.21.15.45:0/2674005830 conn(0x5614df4cfa90 msgr2=0x5614df4cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 shutdown_connections 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.872+0000 7f2dfa7e9740 1 -- 172.21.15.45:0/2674005830 wait complete. 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.873+0000 7f2dfa7e9740 20 client.26422 trim_cache size 0 max 16384 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "f757b256-1736-497c-9b96-d1e1be062b12", 2024-03-12T05:16:18.874 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:16:18.875 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:18.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.873+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 msgr2=0x7f2ddc0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.873+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 0x7f2ddc0a4160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2de000abe0 tx=0x7f2de000a5c0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.873+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 msgr2=0x7f2de8132c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.873+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2df00507e0 tx=0x7f2df006a410 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 shutdown_connections 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2ddc0a1ca0 0x7f2ddc0a4160 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2de80050f0 0x7f2de812e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de809be00 0x7f2de8132c90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 --2- 172.21.15.45:0/3070981078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2de8004730 0x7f2de812e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 >> 172.21.15.45:0/3070981078 conn(0x7f2de800a730 msgr2=0x7f2de80a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 shutdown_connections 2024-03-12T05:16:18.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:18.874+0000 7f2df68d8640 1 -- 172.21.15.45:0/3070981078 wait complete. 2024-03-12T05:16:18.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:16:18.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 -- 172.21.15.45:0/3424266851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c075fd0 msgr2=0x7f5d1c111a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 --2- 172.21.15.45:0/3424266851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c075fd0 0x7f5d1c111a80 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009900 tx=0x7f5d1002bc30 comp rx=0 tx=0).stop 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 -- 172.21.15.45:0/3424266851 shutdown_connections 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 --2- 172.21.15.45:0/3424266851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c075fd0 0x7f5d1c111a80 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 --2- 172.21.15.45:0/3424266851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c1143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 --2- 172.21.15.45:0/3424266851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.073+0000 7f5d226df640 1 -- 172.21.15.45:0/3424266851 >> 172.21.15.45:0/3424266851 conn(0x7f5d1c0fee70 msgr2=0x7f5d1c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.074+0000 7f5d226df640 1 -- 172.21.15.45:0/3424266851 shutdown_connections 2024-03-12T05:16:19.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.074+0000 7f5d226df640 1 -- 172.21.15.45:0/3424266851 wait complete. 2024-03-12T05:16:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 Processor -- start 2024-03-12T05:16:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 -- start start 2024-03-12T05:16:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 0x7f5d1c1a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1a3210 con 0x7f5d1c111fc0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1a3380 con 0x7f5d1c0756b0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d226df640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1a34f0 con 0x7f5d1c1a2820 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d20c7c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 0x7f5d1c1a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39230/0 (socket says 172.21.15.45:39230) 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.075+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/1975616866 learned_addr learned my addr 172.21.15.45:0/1975616866 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1975616866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 msgr2=0x7f5d1c1a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 0x7f5d1c1a2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 msgr2=0x7f5d1c19e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1975616866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10009560 con 0x7f5d1c0756b0 2024-03-12T05:16:19.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 0x7f5d1c1a2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c19df60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d14005fd0 tx=0x7f5d1400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d197fa640 1 -- 172.21.15.45:0/1975616866 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400cd80 con 0x7f5d1c0756b0 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d197fa640 1 -- 172.21.15.45:0/1975616866 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d14005020 con 0x7f5d1c0756b0 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d197fa640 1 -- 172.21.15.45:0/1975616866 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d140116d0 con 0x7f5d1c0756b0 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.076+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1c1aaad0 con 0x7f5d1c0756b0 2024-03-12T05:16:19.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.077+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c1aaf50 con 0x7f5d1c0756b0 2024-03-12T05:16:19.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.080+0000 7f5d197fa640 1 -- 172.21.15.45:0/1975616866 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5d14005190 con 0x7f5d1c0756b0 2024-03-12T05:16:19.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.080+0000 7f5d25394740 1 Processor -- start 2024-03-12T05:16:19.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.080+0000 7f5d25394740 1 -- start start 2024-03-12T05:16:19.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.080+0000 7f5d197fa640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 0x7f5d040a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d197fa640 1 -- 172.21.15.45:0/1975616866 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5d140cd8a0 con 0x7f5d1c0756b0 2024-03-12T05:16:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed99bd0 0x55ffeeda2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 0x7f5d040a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 0x55ffeeda4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed08280 con 0x55ffeeda27b0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed057b0 con 0x55ffeed992b0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d25394740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed00f90 con 0x55ffeed99bd0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58574/0 (socket says 172.21.15.45:58574) 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.081+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1807547315 learned_addr learned my addr 172.21.15.45:0/1807547315 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed99bd0 0x55ffeeda2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 0x55ffeeda4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1807547315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed99bd0 msgr2=0x55ffeeda2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed99bd0 0x55ffeeda2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1807547315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 msgr2=0x55ffeeda4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 0x55ffeeda4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1807547315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffeecbf570 con 0x55ffeed992b0 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 0x55ffeeda4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.082+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d1409a510 tx=0x7f5d1409a1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22f81cb112678555 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.083+0000 7f5d027fc640 1 -- 172.21.15.45:0/1807547315 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d140ba0d0 con 0x55ffeed992b0 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.083+0000 7f5d027fc640 1 -- 172.21.15.45:0/1807547315 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d140a2c70 con 0x55ffeed992b0 2024-03-12T05:16:19.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.083+0000 7f5d027fc640 1 -- 172.21.15.45:0/1807547315 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14096bf0 con 0x55ffeed992b0 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 0x7f5d040a4120 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55ffeed94b70 tx=0x7f5d0c009ba0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 -- 172.21.15.45:0/1807547315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 msgr2=0x55ffeed99690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d1409a510 tx=0x7f5d1409a1b0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 -- 172.21.15.45:0/1807547315 shutdown_connections 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda27b0 0x55ffeeda4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed99bd0 0x55ffeeda2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 --2- 172.21.15.45:0/1807547315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed992b0 0x55ffeed99690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 -- 172.21.15.45:0/1807547315 >> 172.21.15.45:0/1807547315 conn(0x55ffeed93990 msgr2=0x55ffeed93d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 -- 172.21.15.45:0/1807547315 shutdown_connections 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.084+0000 7f5d25394740 1 -- 172.21.15.45:0/1807547315 wait complete. 2024-03-12T05:16:19.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 -- start start 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 Processor -- start 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeed9f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeed9fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffee9d4430 con 0x55ffeed992b0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffee990e00 con 0x55ffeed99bd0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeec85fd0 con 0x55ffeeda27b0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.085+0000 7f5d25394740 10 client.0 _ll_register_callbacks cb 0x55ffeecdbfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeed9f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeed9f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58586/0 (socket says 172.21.15.45:58586) 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/1717364757 learned_addr learned my addr 172.21.15.45:0/1717364757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeed9fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 -- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 msgr2=0x55ffeed9f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeed9f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 -- 172.21.15.45:0/1717364757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 msgr2=0x55ffeeda2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 -- 172.21.15.45:0/1717364757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c1aaf50 con 0x55ffeeda27b0 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.086+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeed9fd30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c012040 tx=0x7f5d0c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bf84066deb91a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5cfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c00fcc0 con 0x55ffeeda27b0 2024-03-12T05:16:19.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ffeea007e0 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5d0c00fe20 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d0c004020 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5d25394740 10 client.26434 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.087+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ffeecff610 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ffeea007e0 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5d0c003c90 con 0x55ffeeda27b0 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 10 client.26434 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 20 client.26434 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeedb29d0 0x55ffeedb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 4 client.26434 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:19.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ffeea3d720 con 0x55ffeedb29d0 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.088+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeedb29d0 0x55ffeedb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.089+0000 7f5cfb7fe640 10 client.26434 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.089+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeedb29d0 0x55ffeedb4df0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba26007c413ea59f server_cookie=2af7dd41c1df28ce in_seq=0 out_seq=0 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.089+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1717364757 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ffeea3d720 con 0x55ffeedb29d0 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.089+0000 7f5cfb7fe640 10 client.26434 handle_command_reply: tid=0 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 client.26434 shutdown 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 msgr2=0x55ffeed9fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeed9fd30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c012040 tx=0x7f5d0c005f90 comp rx=0 tx=0).stop 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 shutdown_connections 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeedb29d0 0x55ffeedb4df0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeed9fd30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeed9f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 --2- 172.21.15.45:0/1717364757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 >> 172.21.15.45:0/1717364757 conn(0x55ffeed93990 msgr2=0x55ffeed95730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.090+0000 7f5cfb7fe640 0 client.26434 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.091+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 shutdown_connections 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.091+0000 7f5d25394740 1 -- 172.21.15.45:0/1717364757 wait complete. 2024-03-12T05:16:19.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.091+0000 7f5d25394740 20 client.26434 trim_cache size 0 max 16384 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 Processor -- start 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 -- start start 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeeda1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed949e0 con 0x55ffeed992b0 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed98cf0 con 0x55ffeed99bd0 2024-03-12T05:16:19.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.119+0000 7f5d25394740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeed98f10 con 0x55ffeeda27b0 2024-03-12T05:16:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d20c7c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeeda1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58598/0 (socket says 172.21.15.45:58598) 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/2470415428 learned_addr learned my addr 172.21.15.45:0/2470415428 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/2470415428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 msgr2=0x55ffeeda1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeeda1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/2470415428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 msgr2=0x55ffeeda96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/2470415428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ffeec9bac0 con 0x55ffeed99bd0 2024-03-12T05:16:19.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.120+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d1002c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74de1667edab722f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.121+0000 7f5d01ffb640 1 -- 172.21.15.45:0/2470415428 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10045070 con 0x55ffeed99bd0 2024-03-12T05:16:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.121+0000 7f5d01ffb640 1 -- 172.21.15.45:0/2470415428 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d100044f0 con 0x55ffeed99bd0 2024-03-12T05:16:19.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.121+0000 7f5d01ffb640 1 -- 172.21.15.45:0/2470415428 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10003bb0 con 0x55ffeed99bd0 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 -- 172.21.15.45:0/2470415428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 msgr2=0x55ffeeda9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d1002c8e0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 -- 172.21.15.45:0/2470415428 shutdown_connections 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeeda27b0 0x55ffeeda1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeed99bd0 0x55ffeeda9bf0 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d1002c8e0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 --2- 172.21.15.45:0/2470415428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeed992b0 0x55ffeeda96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.122+0000 7f5d25394740 1 -- 172.21.15.45:0/2470415428 >> 172.21.15.45:0/2470415428 conn(0x55ffeed93990 msgr2=0x55ffeed9a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.124+0000 7f5d25394740 1 -- 172.21.15.45:0/2470415428 shutdown_connections 2024-03-12T05:16:19.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.125+0000 7f5d25394740 1 -- 172.21.15.45:0/2470415428 wait complete. 2024-03-12T05:16:19.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.125+0000 7f5d25394740 1 -- start start 2024-03-12T05:16:19.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.125+0000 7f5d25394740 1 Processor -- start 2024-03-12T05:16:19.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d25394740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 0x55ffeee03450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d25394740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d25394740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda8a90 0x55ffeeda8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 0x55ffeee03450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 0x55ffeee03450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46792/0 (socket says 172.21.15.45:46792) 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.126+0000 7f5d1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:58602/0 (socket says 172.21.15.45:58602) 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d1bfff640 1 -- 172.21.15.45:0/1201456192 learned_addr learned my addr 172.21.15.45:0/1201456192 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeeda9450 con 0x55ffeeda8a90 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeeda95c0 con 0x55ffeeda27b0 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ffeeda9780 con 0x55ffeed992b0 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d25394740 10 client.0 _ll_register_callbacks cb 0x7f5d1400cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.127+0000 7f5d20c7c640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda8a90 0x55ffeeda8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/1201456192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 msgr2=0x55ffeee03450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 0x55ffeee03450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/1201456192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda8a90 msgr2=0x55ffeeda8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda8a90 0x55ffeeda8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c114e40 con 0x55ffeeda27b0 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5d1b7fe640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55ffeedae4a0 tx=0x7f5d1002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2ae0b3db613c4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:19.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.128+0000 7f5cfb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:19.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10045050 con 0x55ffeeda27b0 2024-03-12T05:16:19.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f5d100051d0 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d140a2c70 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d10005390 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5d25394740 10 client.26459 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ffeecc16a0 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.129+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d140a2c70 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5d10050030 con 0x55ffeeda27b0 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 10 client.26459 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:19.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 20 client.26459 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeee0b3f0 0x55ffeee0d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 4 client.26459 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ffeea8c890 con 0x55ffeee0b3f0 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeee0b3f0 0x55ffeee0d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5d1bfff640 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeee0b3f0 0x55ffeee0d810 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e31971d5abbb8a server_cookie=a12f8f186dfb5776 in_seq=0 out_seq=0 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.130+0000 7f5cfb7fe640 10 client.26459 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5cfb7fe640 1 -- 172.21.15.45:0/1201456192 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ffeea8c890 con 0x55ffeee0b3f0 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5cfb7fe640 10 client.26459 handle_command_reply: tid=0 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 client.26459 shutdown 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 msgr2=0x55ffeee03990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55ffeedae4a0 tx=0x7f5d1002bc30 comp rx=0 tx=0).stop 2024-03-12T05:16:19.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 shutdown_connections 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ffeee0b3f0 0x55ffeee0d810 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ffeeda8a90 0x55ffeeda8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffeeda27b0 0x55ffeee03990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 --2- 172.21.15.45:0/1201456192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ffeed992b0 0x55ffeee03450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 >> 172.21.15.45:0/1201456192 conn(0x55ffeed93990 msgr2=0x55ffeed9ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5cfb7fe640 0 client.26459 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 shutdown_connections 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.131+0000 7f5d25394740 1 -- 172.21.15.45:0/1201456192 wait complete. 2024-03-12T05:16:19.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.132+0000 7f5d25394740 20 client.26459 trim_cache size 0 max 16384 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "f757b256-1736-497c-9b96-d1e1be062b12": { 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "f757b256-1736-497c-9b96-d1e1be062b12", 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:16:19.133 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.132+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 msgr2=0x7f5d040a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.132+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 0x7f5d040a4120 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55ffeed94b70 tx=0x7f5d0c009ba0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.132+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 msgr2=0x7f5d1c19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.132+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c19df60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d14005fd0 tx=0x7f5d1400ee80 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 shutdown_connections 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d040a1c60 0x7f5d040a4120 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c111fc0 0x7f5d1c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d1c1a2820 0x7f5d1c1a2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 --2- 172.21.15.45:0/1975616866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d1c0756b0 0x7f5d1c19df60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 >> 172.21.15.45:0/1975616866 conn(0x7f5d1c0fee70 msgr2=0x7f5d1c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:19.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 shutdown_connections 2024-03-12T05:16:19.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:19.133+0000 7f5d226df640 1 -- 172.21.15.45:0/1975616866 wait complete. 2024-03-12T05:16:19.146 INFO:tasks.cephfs.filesystem:scrub status for tag:f757b256-1736-497c-9b96-d1e1be062b12 - {'path': '/', 'tag': 'f757b256-1736-497c-9b96-d1e1be062b12', 'options': 'recursive,force'} 2024-03-12T05:16:19.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:19 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:19.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:19 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:19.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:19 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:19.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:19 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:19.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:19 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:19.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:19 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:19.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 255.0 s, 1239.7 tps, lat 72.268 ms stddev 19.637 2024-03-12T05:16:20.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 16 MiB/s wr, 1.77k op/s 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 16 MiB/s wr, 1.77k op/s 2024-03-12T05:16:20.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:16:20.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:20.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:20.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:20.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.493 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: pgmap v166: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 16 MiB/s wr, 1.77k op/s 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:20.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:20.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:22.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:22 smithi130 ceph-mon[35398]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:22.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:22 smithi116 ceph-mon[35486]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:22.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:22 smithi045 ceph-mon[32174]: pgmap v167: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:24 smithi130 ceph-mon[35398]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:24 smithi116 ceph-mon[35486]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:24.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:24 smithi045 ceph-mon[32174]: pgmap v168: 97 pgs: 97 active+clean; 2.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-03-12T05:16:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 260.0 s, 1208.2 tps, lat 85.866 ms stddev 147.390 2024-03-12T05:16:26.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:26 smithi130 ceph-mon[35398]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 MiB/s wr, 1.87k op/s 2024-03-12T05:16:26.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:26 smithi116 ceph-mon[35486]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 MiB/s wr, 1.87k op/s 2024-03-12T05:16:26.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:26 smithi045 ceph-mon[32174]: pgmap v169: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 MiB/s wr, 1.87k op/s 2024-03-12T05:16:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:28 smithi130 ceph-mon[35398]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 19 MiB/s wr, 1.31k op/s 2024-03-12T05:16:28.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:28 smithi116 ceph-mon[35486]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 19 MiB/s wr, 1.31k op/s 2024-03-12T05:16:28.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:28 smithi045 ceph-mon[32174]: pgmap v170: 97 pgs: 97 active+clean; 2.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 19 MiB/s wr, 1.31k op/s 2024-03-12T05:16:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 265.0 s, 1243.0 tps, lat 85.057 ms stddev 142.190 2024-03-12T05:16:30.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:30 smithi130 ceph-mon[35398]: pgmap v171: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 32 MiB/s wr, 1.90k op/s 2024-03-12T05:16:30.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:30 smithi116 ceph-mon[35486]: pgmap v171: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 32 MiB/s wr, 1.90k op/s 2024-03-12T05:16:30.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:30 smithi045 ceph-mon[32174]: pgmap v171: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 32 MiB/s wr, 1.90k op/s 2024-03-12T05:16:32.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:32 smithi130 ceph-mon[35398]: pgmap v172: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:32.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:32 smithi116 ceph-mon[35486]: pgmap v172: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:32.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:32 smithi045 ceph-mon[32174]: pgmap v172: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:33.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:33 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:16:33.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:33 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:16:33.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:33 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:16:33.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:33 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:16:33.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:33 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:16:33.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:33 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:16:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:33 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:16:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:33 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:16:33.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:33 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:16:34.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:34 smithi130 ceph-mon[35398]: pgmap v173: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:34.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:34 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:16:34.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:34 smithi116 ceph-mon[35486]: pgmap v173: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:34.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:34 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:16:34.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:34 smithi045 ceph-mon[32174]: pgmap v173: 97 pgs: 97 active+clean; 3.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 26 MiB/s wr, 1.31k op/s 2024-03-12T05:16:34.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:34 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:16:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 270.0 s, 1108.6 tps, lat 91.158 ms stddev 146.442 2024-03-12T05:16:36.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:36 smithi130 ceph-mon[35398]: pgmap v174: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 40 MiB/s wr, 1.89k op/s 2024-03-12T05:16:36.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:36 smithi116 ceph-mon[35486]: pgmap v174: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 40 MiB/s wr, 1.89k op/s 2024-03-12T05:16:36.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:36 smithi045 ceph-mon[32174]: pgmap v174: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 40 MiB/s wr, 1.89k op/s 2024-03-12T05:16:38.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:38 smithi130 ceph-mon[35398]: pgmap v175: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-03-12T05:16:38.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:38 smithi116 ceph-mon[35486]: pgmap v175: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-03-12T05:16:38.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:38 smithi045 ceph-mon[32174]: pgmap v175: 97 pgs: 97 active+clean; 3.0 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-03-12T05:16:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 275.0 s, 1141.6 tps, lat 84.361 ms stddev 126.063 2024-03-12T05:16:40.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:40 smithi130 ceph-mon[35398]: pgmap v176: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 38 MiB/s wr, 1.76k op/s 2024-03-12T05:16:40.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:40 smithi116 ceph-mon[35486]: pgmap v176: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 38 MiB/s wr, 1.76k op/s 2024-03-12T05:16:40.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:40 smithi045 ceph-mon[32174]: pgmap v176: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 38 MiB/s wr, 1.76k op/s 2024-03-12T05:16:42.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:42 smithi130 ceph-mon[35398]: pgmap v177: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:42.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:42 smithi116 ceph-mon[35486]: pgmap v177: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:42.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:42 smithi045 ceph-mon[32174]: pgmap v177: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:44.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:44 smithi130 ceph-mon[35398]: pgmap v178: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:44.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:44 smithi116 ceph-mon[35486]: pgmap v178: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:44.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:44 smithi045 ceph-mon[32174]: pgmap v178: 97 pgs: 97 active+clean; 3.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 26 MiB/s wr, 1.16k op/s 2024-03-12T05:16:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 280.0 s, 1108.6 tps, lat 93.524 ms stddev 153.989 2024-03-12T05:16:46.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:46 smithi130 ceph-mon[35398]: pgmap v179: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 37 MiB/s wr, 1.75k op/s 2024-03-12T05:16:46.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:46 smithi116 ceph-mon[35486]: pgmap v179: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 37 MiB/s wr, 1.75k op/s 2024-03-12T05:16:46.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:46 smithi045 ceph-mon[32174]: pgmap v179: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 37 MiB/s wr, 1.75k op/s 2024-03-12T05:16:48.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:48 smithi130 ceph-mon[35398]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 24 MiB/s wr, 1.18k op/s 2024-03-12T05:16:48.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:48 smithi116 ceph-mon[35486]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 24 MiB/s wr, 1.18k op/s 2024-03-12T05:16:48.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:48 smithi045 ceph-mon[32174]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 24 MiB/s wr, 1.18k op/s 2024-03-12T05:16:49.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:16:49.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 -- 172.21.15.45:0/987002573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde8102d10 msgr2=0x7efde810b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 --2- 172.21.15.45:0/987002573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde8102d10 0x7efde810b1f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009900 tx=0x7efdd802bd70 comp rx=0 tx=0).stop 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 -- 172.21.15.45:0/987002573 shutdown_connections 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 --2- 172.21.15.45:0/987002573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde8102d10 0x7efde810b1f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 --2- 172.21.15.45:0/987002573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde810b730 0x7efde810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 --2- 172.21.15.45:0/987002573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde81023f0 0x7efde81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 -- 172.21.15.45:0/987002573 >> 172.21.15.45:0/987002573 conn(0x7efde8077190 msgr2=0x7efde80775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 -- 172.21.15.45:0/987002573 shutdown_connections 2024-03-12T05:16:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.309+0000 7efdee04e640 1 -- 172.21.15.45:0/987002573 wait complete. 2024-03-12T05:16:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.310+0000 7efdee04e640 1 Processor -- start 2024-03-12T05:16:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.310+0000 7efdee04e640 1 -- start start 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 0x7efde8100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 0x7efde8100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde8101f90 con 0x7efde810b730 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde81aa030 con 0x7efde8102d10 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efdee04e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde81aa170 con 0x7efde81023f0 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41160/0 (socket says 172.21.15.45:41160) 2024-03-12T05:16:49.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 -- 172.21.15.45:0/812916844 learned_addr learned my addr 172.21.15.45:0/812916844 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde6ffd640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 0x7efde8100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde77fe640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 0x7efde8100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 -- 172.21.15.45:0/812916844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 msgr2=0x7efde8100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 0x7efde8100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 -- 172.21.15.45:0/812916844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 msgr2=0x7efde8100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 0x7efde8100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 -- 172.21.15.45:0/812916844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009560 con 0x7efde810b730 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde6ffd640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 0x7efde8100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde77fe640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 0x7efde8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:49.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde7fff640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efde000eaa0 tx=0x7efde000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.311+0000 7efde4ff9640 1 -- 172.21.15.45:0/812916844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000cdb0 con 0x7efde810b730 2024-03-12T05:16:49.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.312+0000 7efde4ff9640 1 -- 172.21.15.45:0/812916844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efde0005020 con 0x7efde810b730 2024-03-12T05:16:49.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.312+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde81aa450 con 0x7efde810b730 2024-03-12T05:16:49.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.312+0000 7efde4ff9640 1 -- 172.21.15.45:0/812916844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00116b0 con 0x7efde810b730 2024-03-12T05:16:49.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.312+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde81aa930 con 0x7efde810b730 2024-03-12T05:16:49.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.314+0000 7efde4ff9640 1 -- 172.21.15.45:0/812916844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efde0005190 con 0x7efde810b730 2024-03-12T05:16:49.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 Processor -- start 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde4ff9640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 0x7efdd40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 -- start start 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde4ff9640 1 -- 172.21.15.45:0/812916844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efde00cd940 con 0x7efde810b730 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d7d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde6ffd640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 0x7efdd40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 0x5574a9d88ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9cec280 con 0x5574a9d7dbd0 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9ce97b0 con 0x5574a9d867b0 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9ce4f90 con 0x5574a9d7d2b0 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41174/0 (socket says 172.21.15.45:41174) 2024-03-12T05:16:49.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde6ffd640 1 -- 172.21.15.45:0/3773967666 learned_addr learned my addr 172.21.15.45:0/3773967666 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde7fff640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 0x5574a9d88ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.315+0000 7efde77fe640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d7d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 -- 172.21.15.45:0/3773967666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 msgr2=0x5574a9d7d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d7d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 -- 172.21.15.45:0/3773967666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 msgr2=0x5574a9d86270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 -- 172.21.15.45:0/3773967666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574a9ca3570 con 0x5574a9d867b0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde6ffd640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde6ffd640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 0x7efdd40a40e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009230 tx=0x7efdd8004bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efde7fff640 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 0x5574a9d88ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb090 tx=0x7efde0099ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58f6195da64bec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efdddffb640 1 -- 172.21.15.45:0/3773967666 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde009e6c0 con 0x5574a9d867b0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efdddffb640 1 -- 172.21.15.45:0/3773967666 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efde00a2d10 con 0x5574a9d867b0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.316+0000 7efdddffb640 1 -- 172.21.15.45:0/3773967666 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00cb310 con 0x5574a9d867b0 2024-03-12T05:16:49.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- 172.21.15.45:0/3773967666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 msgr2=0x5574a9d88ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 0x5574a9d88ba0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb090 tx=0x7efde0099ed0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- 172.21.15.45:0/3773967666 shutdown_connections 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d86270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d867b0 0x5574a9d88ba0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/3773967666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d7d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- 172.21.15.45:0/3773967666 >> 172.21.15.45:0/3773967666 conn(0x5574a9d77990 msgr2=0x5574a9d77d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- 172.21.15.45:0/3773967666 shutdown_connections 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- 172.21.15.45:0/3773967666 wait complete. 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.317+0000 7efdf0c1e740 1 -- start start 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 Processor -- start 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d83590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d83cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a99b8430 con 0x5574a9d7d2b0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9974e00 con 0x5574a9d7dbd0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9c69fd0 con 0x5574a9d867b0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efdf0c1e740 10 client.0 _ll_register_callbacks cb 0x5574a9cbffd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d83590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41188/0 (socket says 172.21.15.45:41188) 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde77fe640 1 -- 172.21.15.45:0/4197797061 learned_addr learned my addr 172.21.15.45:0/4197797061 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde7fff640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d83cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 -- 172.21.15.45:0/4197797061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 msgr2=0x5574a9d83cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d83cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 -- 172.21.15.45:0/4197797061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 msgr2=0x5574a9d86200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde6ffd640 1 -- 172.21.15.45:0/4197797061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde81aa930 con 0x5574a9d7dbd0 2024-03-12T05:16:49.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.318+0000 7efde77fe640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efde6ffd640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d83590 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdd802c380 tx=0x7efdd8048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c85bfb3ffc2bbe82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8048070 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574a99e47e0 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efdd8005250 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd8005410 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 10 client.26471 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5574a9ce3610 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efdd805b020 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdd8034440 con 0x5574a9d7dbd0 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 10 client.26471 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 10 client.26471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 20 client.26471 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:49.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9d968d0 0x5574a9d98cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 4 client.26471 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.319+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574a9a21720 con 0x5574a9d968d0 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.320+0000 7efde77fe640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9d968d0 0x5574a9d98cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.320+0000 7efde77fe640 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9d968d0 0x5574a9d98cf0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d084089ed6974156 server_cookie=d7cac842251b5890 in_seq=0 out_seq=0 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.320+0000 7efdc6ffd640 10 client.26471 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdc6ffd640 1 -- 172.21.15.45:0/4197797061 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5574a9a21720 con 0x5574a9d968d0 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdc6ffd640 10 client.26471 handle_command_reply: tid=0 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 client.26471 shutdown 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 msgr2=0x5574a9d83590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d83590 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdd802c380 tx=0x7efdd8048000 comp rx=0 tx=0).stop 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 shutdown_connections 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9d968d0 0x5574a9d98cf0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d83cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d83590 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/4197797061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7d2b0 0x5574a9d86200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 >> 172.21.15.45:0/4197797061 conn(0x5574a9d77990 msgr2=0x5574a9d79730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdc6ffd640 0 client.26471 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 shutdown_connections 2024-03-12T05:16:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 1 -- 172.21.15.45:0/4197797061 wait complete. 2024-03-12T05:16:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.321+0000 7efdf0c1e740 20 client.26471 trim_cache size 0 max 16384 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 Processor -- start 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 -- start start 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9d7ccc0 con 0x5574a9d867b0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9d78770 con 0x5574a9d7dbd0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.339+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9d78560 con 0x5574a9d7d2b0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41196/0 (socket says 172.21.15.45:41196) 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 -- 172.21.15.45:0/1544050813 learned_addr learned my addr 172.21.15.45:0/1544050813 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde6ffd640 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde77fe640 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 -- 172.21.15.45:0/1544050813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 msgr2=0x5574a9d8d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 -- 172.21.15.45:0/1544050813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 msgr2=0x5574a9d8dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 -- 172.21.15.45:0/1544050813 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574a9c7fac0 con 0x5574a9d867b0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efde7fff640 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb140 tx=0x7efde009a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc51cbd86af81827 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efddd7fa640 1 -- 172.21.15.45:0/1544050813 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde009e6c0 con 0x5574a9d867b0 2024-03-12T05:16:49.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efddd7fa640 1 -- 172.21.15.45:0/1544050813 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efde0005520 con 0x5574a9d867b0 2024-03-12T05:16:49.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.340+0000 7efddd7fa640 1 -- 172.21.15.45:0/1544050813 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0096be0 con 0x5574a9d867b0 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 -- 172.21.15.45:0/1544050813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 msgr2=0x5574a9d85cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb140 tx=0x7efde009a000 comp rx=0 tx=0).stop 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 -- 172.21.15.45:0/1544050813 shutdown_connections 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d867b0 0x5574a9d85cc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/1544050813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 -- 172.21.15.45:0/1544050813 >> 172.21.15.45:0/1544050813 conn(0x5574a9d77990 msgr2=0x5574a9d81e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 -- 172.21.15.45:0/1544050813 shutdown_connections 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.341+0000 7efdf0c1e740 1 -- 172.21.15.45:0/1544050813 wait complete. 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 -- start start 2024-03-12T05:16:49.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 Processor -- start 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d8cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9dece00 con 0x5574a9d7dbd0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9decf70 con 0x5574a9d7d2b0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a9ded130 con 0x5574a9d867b0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efdf0c1e740 10 client.0 _ll_register_callbacks cb 0x7efde000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efde6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41210/0 (socket says 172.21.15.45:41210) 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efde6ffd640 1 -- 172.21.15.45:0/2618853760 learned_addr learned my addr 172.21.15.45:0/2618853760 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efde77fe640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.342+0000 7efde7fff640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d8cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 -- 172.21.15.45:0/2618853760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 msgr2=0x5574a9d8cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d8cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 -- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7d2b0 msgr2=0x5574a9d8c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 -- 172.21.15.45:0/2618853760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574a9ca3570 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efde6ffd640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efdd80348a0 tx=0x7efdd8034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcde37bc4ec41f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8005d90 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7efdd8034e40 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efde00a2d10 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd8004380 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdf0c1e740 10 client.17001 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5574a9ca56a0 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efde00a2d10 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.343+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdd8003cd0 con 0x5574a9d7dbd0 2024-03-12T05:16:49.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 10 client.17001 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 20 client.17001 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9dee6e0 0x5574a9df0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 4 client.17001 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5574a9a70890 con 0x5574a9dee6e0 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efde77fe640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9dee6e0 0x5574a9df0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efde77fe640 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9dee6e0 0x5574a9df0b00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e802749452612ee6 server_cookie=479e0c2093e572f4 in_seq=0 out_seq=0 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdc6ffd640 10 client.17001 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdc6ffd640 1 -- 172.21.15.45:0/2618853760 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5574a9a70890 con 0x5574a9dee6e0 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdc6ffd640 10 client.17001 handle_command_reply: tid=0 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.344+0000 7efdf0c1e740 1 client.17001 shutdown 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 msgr2=0x5574a9d8c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efdd80348a0 tx=0x7efdd8034670 comp rx=0 tx=0).stop 2024-03-12T05:16:49.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 shutdown_connections 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5574a9dee6e0 0x5574a9df0b00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdc6ffd640 0 client.17001 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574a9d7dbd0 0x5574a9d8c6f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574a9d867b0 0x5574a9d8cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 --2- 172.21.15.45:0/2618853760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574a9d7d2b0 0x5574a9d8c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 >> 172.21.15.45:0/2618853760 conn(0x5574a9d77990 msgr2=0x5574a9d81da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 shutdown_connections 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 1 -- 172.21.15.45:0/2618853760 wait complete. 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdf0c1e740 20 client.17001 trim_cache size 0 max 16384 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:49.346 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:16:49.347 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:16:49.347 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 msgr2=0x7efdd40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 0x7efdd40a40e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009230 tx=0x7efdd8004bc0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 msgr2=0x7efde8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7efde000eaa0 tx=0x7efde000ef60 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 shutdown_connections 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efdd40a1c20 0x7efdd40a40e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efde810b730 0x7efde8100ac0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8102d10 0x7efde8100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 --2- 172.21.15.45:0/812916844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efde81023f0 0x7efde8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.345+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 >> 172.21.15.45:0/812916844 conn(0x7efde8077190 msgr2=0x7efde80fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.346+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 shutdown_connections 2024-03-12T05:16:49.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.346+0000 7efdee04e640 1 -- 172.21.15.45:0/812916844 wait complete. 2024-03-12T05:16:49.353 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:16:49.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:16:49.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 -- 172.21.15.45:0/699854299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 msgr2=0x7f454c073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 --2- 172.21.15.45:0/699854299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c073900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f453c009900 tx=0x7f453c02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 -- 172.21.15.45:0/699854299 shutdown_connections 2024-03-12T05:16:49.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 --2- 172.21.15.45:0/699854299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c112120 0x7f454c114510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 --2- 172.21.15.45:0/699854299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c073e40 0x7f454c111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 --2- 172.21.15.45:0/699854299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c073900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 -- 172.21.15.45:0/699854299 >> 172.21.15.45:0/699854299 conn(0x7f454c078580 msgr2=0x7f454c078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 -- 172.21.15.45:0/699854299 shutdown_connections 2024-03-12T05:16:49.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.513+0000 7f4550bc9640 1 -- 172.21.15.45:0/699854299 wait complete. 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 Processor -- start 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 -- start start 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 0x7f454c19e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c112120 0x7f454c1a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c1a2be0 con 0x7f454c073e40 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c1a2d50 con 0x7f454c112120 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4550bc9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c1a2f10 con 0x7f454c073520 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f454a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f4549d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 0x7f454c19e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f454a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45686/0 (socket says 172.21.15.45:45686) 2024-03-12T05:16:49.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.514+0000 7f454a59c640 1 -- 172.21.15.45:0/2759572801 learned_addr learned my addr 172.21.15.45:0/2759572801 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454ad9d640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c112120 0x7f454c1a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 -- 172.21.15.45:0/2759572801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c112120 msgr2=0x7f454c1a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c112120 0x7f454c1a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 -- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 msgr2=0x7f454c19e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 0x7f454c19e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 -- 172.21.15.45:0/2759572801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c009560 con 0x7f454c073520 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f4549d9b640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 0x7f454c19e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.515+0000 7f454a59c640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f453c000c00 tx=0x7f453c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.516+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c02dc70 con 0x7f454c073520 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.516+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f453c02ddd0 con 0x7f454c073520 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.516+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0052b0 con 0x7f454c073520 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.516+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c1a3190 con 0x7f454c073520 2024-03-12T05:16:49.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.516+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c1aaa70 con 0x7f454c073520 2024-03-12T05:16:49.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.518+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c0676e0 con 0x7f454c073520 2024-03-12T05:16:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.520+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f453c003cc0 con 0x7f454c073520 2024-03-12T05:16:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.521+0000 7f453b7fe640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 0x7f45340a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.521+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f453c0efc70 con 0x7f454c073520 2024-03-12T05:16:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.521+0000 7f4549d9b640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 0x7f45340a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.522+0000 7f4549d9b640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 0x7f45340a40e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f454c10e330 tx=0x7f454000b390 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.526+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f453c0b8c50 con 0x7f454c073520 2024-03-12T05:16:49.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.910+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f454c19f070 con 0x7f454c073520 2024-03-12T05:16:49.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.911+0000 7f453b7fe640 1 -- 172.21.15.45:0/2759572801 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f453c043090 con 0x7f454c073520 2024-03-12T05:16:49.907 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:16:49.907 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:16:49.907 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 msgr2=0x7f45340a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 0x7f45340a40e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f454c10e330 tx=0x7f454000b390 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 msgr2=0x7f454c19dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f453c000c00 tx=0x7f453c0045e0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 shutdown_connections 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f45340a1c20 0x7f45340a40e0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f454c10e330 tx=0x7f454000b390 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f454c073e40 0x7f454c19e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f454c112120 0x7f454c1a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 --2- 172.21.15.45:0/2759572801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c073520 0x7f454c19dd20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 >> 172.21.15.45:0/2759572801 conn(0x7f454c078580 msgr2=0x7f454c1110d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 shutdown_connections 2024-03-12T05:16:49.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:49.914+0000 7f4550bc9640 1 -- 172.21.15.45:0/2759572801 wait complete. 2024-03-12T05:16:49.926 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:49.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:16:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 285.0 s, 1148.6 tps, lat 84.262 ms stddev 173.188 2024-03-12T05:16:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 -- 172.21.15.45:0/298951692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec1080d0 msgr2=0x7f23ec1145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 --2- 172.21.15.45:0/298951692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec1080d0 0x7f23ec1145e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009be0 tx=0x7f23e402bef0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 -- 172.21.15.45:0/298951692 shutdown_connections 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 --2- 172.21.15.45:0/298951692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec1080d0 0x7f23ec1145e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 --2- 172.21.15.45:0/298951692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 --2- 172.21.15.45:0/298951692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec10d720 0x7f23ec10db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 -- 172.21.15.45:0/298951692 >> 172.21.15.45:0/298951692 conn(0x7f23ec075690 msgr2=0x7f23ec077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 -- 172.21.15.45:0/298951692 shutdown_connections 2024-03-12T05:16:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.098+0000 7f23f30d4640 1 -- 172.21.15.45:0/298951692 wait complete. 2024-03-12T05:16:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.099+0000 7f23f30d4640 1 Processor -- start 2024-03-12T05:16:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f30d4640 1 -- start start 2024-03-12T05:16:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f30d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f30d4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec1080d0 0x7f23ec1a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f30d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec10d720 0x7f23ec19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f30d4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec19e0d0 con 0x7f23ec1080d0 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45704/0 (socket says 172.21.15.45:45704) 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f0e70640 1 -- 172.21.15.45:0/3452707828 learned_addr learned my addr 172.21.15.45:0/3452707828 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23e3fff640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec1080d0 0x7f23ec1a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.100+0000 7f23f1671640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec10d720 0x7f23ec19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec19e240 con 0x7f23ec10d720 2024-03-12T05:16:50.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec19e400 con 0x7f23ec107710 2024-03-12T05:16:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 -- 172.21.15.45:0/3452707828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec10d720 msgr2=0x7f23ec19da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec10d720 0x7f23ec19da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 -- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec1080d0 msgr2=0x7f23ec1a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec1080d0 0x7f23ec1a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 -- 172.21.15.45:0/3452707828 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8009560 con 0x7f23ec107710 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.101+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23e800e9c0 tx=0x7f23e800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.102+0000 7f23e1ffb640 1 -- 172.21.15.45:0/3452707828 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e800cde0 con 0x7f23ec107710 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.102+0000 7f23e1ffb640 1 -- 172.21.15.45:0/3452707828 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23e80044d0 con 0x7f23ec107710 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.102+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e4009840 con 0x7f23ec107710 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.102+0000 7f23e1ffb640 1 -- 172.21.15.45:0/3452707828 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8010430 con 0x7f23ec107710 2024-03-12T05:16:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.102+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec19e8d0 con 0x7f23ec107710 2024-03-12T05:16:50.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.106+0000 7f23e1ffb640 1 -- 172.21.15.45:0/3452707828 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f23e8010590 con 0x7f23ec107710 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.106+0000 7f23f5d7d740 1 Processor -- start 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 -- start start 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133d8750 0x55e0133dab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01338a960 con 0x55e0133d8750 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01333e310 con 0x55e0133cfb70 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01333b840 con 0x55e0133cf250 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f1671640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f1671640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45716/0 (socket says 172.21.15.45:45716) 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23f1671640 1 -- 172.21.15.45:0/881445225 learned_addr learned my addr 172.21.15.45:0/881445225 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.107+0000 7f23e3fff640 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133d8750 0x55e0133dab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 -- 172.21.15.45:0/881445225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 msgr2=0x55e0133d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 -- 172.21.15.45:0/881445225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133d8750 msgr2=0x55e0133dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133d8750 0x55e0133dab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 -- 172.21.15.45:0/881445225 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0132f9060 con 0x55e0133cf250 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f0e70640 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23e1ffb640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 0x7f23d40a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.108+0000 7f23f1671640 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23e4035000 tx=0x7f23e402c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e2f139b30c7f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.109+0000 7f23e1ffb640 1 -- 172.21.15.45:0/3452707828 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f23e8014070 con 0x7f23ec107710 2024-03-12T05:16:50.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.109+0000 7f23f1671640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 0x7f23d40a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.109+0000 7f23d2ffd640 1 -- 172.21.15.45:0/881445225 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e40335e0 con 0x55e0133cf250 2024-03-12T05:16:50.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.109+0000 7f23d2ffd640 1 -- 172.21.15.45:0/881445225 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23e4033ea0 con 0x55e0133cf250 2024-03-12T05:16:50.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.109+0000 7f23d2ffd640 1 -- 172.21.15.45:0/881445225 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e40043b0 con 0x55e0133cf250 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 -- 172.21.15.45:0/881445225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 msgr2=0x55e0133cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23e4035000 tx=0x7f23e402c890 comp rx=0 tx=0).stop 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f1671640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 0x7f23d40a4120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f23e40045a0 tx=0x7f23e4004510 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 -- 172.21.15.45:0/881445225 shutdown_connections 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133d8750 0x55e0133dab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/881445225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133cf630 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 -- 172.21.15.45:0/881445225 >> 172.21.15.45:0/881445225 conn(0x55e0133c9980 msgr2=0x55e0133c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 -- 172.21.15.45:0/881445225 shutdown_connections 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.110+0000 7f23f5d7d740 1 -- 172.21.15.45:0/881445225 wait complete. 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.111+0000 7f23f5d7d740 1 -- start start 2024-03-12T05:16:50.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.111+0000 7f23f5d7d740 1 Processor -- start 2024-03-12T05:16:50.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133d5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133d8750 0x55e0133d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e013337020 con 0x55e0133cf250 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01300a430 con 0x55e0133cfb70 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e012fc6e00 con 0x55e0133d8750 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f5d7d740 10 client.0 _ll_register_callbacks cb 0x55e0132decb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133d5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133d5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41270/0 (socket says 172.21.15.45:41270) 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f0e70640 1 -- 172.21.15.45:0/1979421610 learned_addr learned my addr 172.21.15.45:0/1979421610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23f1671640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133d8750 0x55e0133d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 -- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133d8750 msgr2=0x55e0133d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133d8750 0x55e0133d61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 -- 172.21.15.45:0/1979421610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 msgr2=0x55e0133d5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133d5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.112+0000 7f23e3fff640 1 -- 172.21.15.45:0/1979421610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec114ee0 con 0x55e0133cfb70 2024-03-12T05:16:50.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.113+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d5c60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f23ec19f270 tx=0x7f23dc012aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e8e5738abdc04c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.113+0000 7f23cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.113+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc017070 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.113+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f23dc00f560 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.113+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e013073720 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23dc00f720 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 10 client.26495 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e0133356a0 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e013073720 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f23dc0269d0 con 0x55e0133cfb70 2024-03-12T05:16:50.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 10 client.26495 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 10 client.26495 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 20 client.26495 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e0133e8b80 0x55e0133eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 4 client.26495 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.114+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0130c2890 con 0x55e0133e8b80 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.115+0000 7f23f0e70640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e0133e8b80 0x55e0133eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.115+0000 7f23f0e70640 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e0133e8b80 0x55e0133eafc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b267d88135d15c7a server_cookie=d3b40a02d32dd783 in_seq=0 out_seq=0 2024-03-12T05:16:50.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.115+0000 7f23cbfff640 10 client.26495 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23cbfff640 1 -- 172.21.15.45:0/1979421610 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e0130c2890 con 0x55e0133e8b80 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23cbfff640 10 client.26495 handle_command_reply: tid=0 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 client.26495 shutdown 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 msgr2=0x55e0133d5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d5c60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f23ec19f270 tx=0x7f23dc012aa0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 shutdown_connections 2024-03-12T05:16:50.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e0133e8b80 0x55e0133eafc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133d8750 0x55e0133d61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133cfb70 0x55e0133d5c60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1979421610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133d5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 >> 172.21.15.45:0/1979421610 conn(0x55e0133c9980 msgr2=0x55e0133cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.116+0000 7f23cbfff640 0 client.26495 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.117+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 shutdown_connections 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.117+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1979421610 wait complete. 2024-03-12T05:16:50.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.117+0000 7f23f5d7d740 20 client.26495 trim_cache size 0 max 16384 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 Processor -- start 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 -- start start 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 0x55e0133d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45748/0 (socket says 172.21.15.45:45748) 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e0133d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0132bc130 con 0x55e0133cfb70 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e013073720 con 0x55e0133d8750 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0133e5630 con 0x55e0133cf250 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 0x55e0133d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.138+0000 7f23f0e70640 1 -- 172.21.15.45:0/3973337548 learned_addr learned my addr 172.21.15.45:0/3973337548 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f1671640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e0133d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 -- 172.21.15.45:0/3973337548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 msgr2=0x55e0133d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e0133d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 -- 172.21.15.45:0/3973337548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 msgr2=0x55e0133d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 0x55e0133d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 -- 172.21.15.45:0/3973337548 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0132d1c20 con 0x55e0133cf250 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f1671640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e0133d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23e3fff640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 0x55e0133d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.139+0000 7f23f0e70640 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23e809a3e0 tx=0x7f23e8086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fbb82255fcd9403 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.140+0000 7f23d27fc640 1 -- 172.21.15.45:0/3973337548 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8096fb0 con 0x55e0133cf250 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.140+0000 7f23d27fc640 1 -- 172.21.15.45:0/3973337548 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f23e809e830 con 0x55e0133cf250 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.140+0000 7f23d27fc640 1 -- 172.21.15.45:0/3973337548 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80ba310 con 0x55e0133cf250 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 -- 172.21.15.45:0/3973337548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 msgr2=0x55e0133d7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23e809a3e0 tx=0x7f23e8086040 comp rx=0 tx=0).stop 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 -- 172.21.15.45:0/3973337548 shutdown_connections 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cfb70 0x55e0133d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e0133d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/3973337548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cf250 0x55e0133d7e50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 -- 172.21.15.45:0/3973337548 >> 172.21.15.45:0/3973337548 conn(0x55e0133c9980 msgr2=0x55e0133ef960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.141+0000 7f23f5d7d740 1 -- 172.21.15.45:0/3973337548 shutdown_connections 2024-03-12T05:16:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.142+0000 7f23f5d7d740 1 -- 172.21.15.45:0/3973337548 wait complete. 2024-03-12T05:16:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.142+0000 7f23f5d7d740 1 -- start start 2024-03-12T05:16:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.142+0000 7f23f5d7d740 1 Processor -- start 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133de870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 0x55e0133dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e01343f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01343f630 con 0x55e0133cf250 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01343f770 con 0x55e0133d8750 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01343f8b0 con 0x55e0133cfb70 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f5d7d740 10 client.0 _ll_register_callbacks cb 0x7f23e8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133de870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f1671640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e01343f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 0x55e0133dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f1671640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e01343f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37368/0 (socket says 172.21.15.45:37368) 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f0e70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133de870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41284/0 (socket says 172.21.15.45:41284) 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23f1671640 1 -- 172.21.15.45:0/1743935970 learned_addr learned my addr 172.21.15.45:0/1743935970 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 -- 172.21.15.45:0/1743935970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 msgr2=0x55e01343f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e01343f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 -- 172.21.15.45:0/1743935970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 msgr2=0x55e0133de870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133de870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.143+0000 7f23e3fff640 1 -- 172.21.15.45:0/1743935970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0133356a0 con 0x55e0133cfb70 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.144+0000 7f23e3fff640 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 0x55e0133dedb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f23dc001eb0 tx=0x7f23dc0055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3d205d2b3b411c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.144+0000 7f23cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.144+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc017070 con 0x55e0133cfb70 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.144+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f23dc000e40 con 0x55e0133cfb70 2024-03-12T05:16:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.144+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0130367e0 con 0x55e0133cfb70 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23dc009850 con 0x55e0133cfb70 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 10 client.26479 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f23ec19e8d0 con 0x55e0133cfb70 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e0130367e0 con 0x55e0133cfb70 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f23dc00f560 con 0x55e0133cfb70 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 10 client.26479 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 10 client.26479 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 20 client.26479 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e013440fe0 0x55e013443400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 4 client.26479 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.145+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e012fe2700 con 0x55e013440fe0 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.146+0000 7f23f0e70640 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e013440fe0 0x55e013443400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.146+0000 7f23f0e70640 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e013440fe0 0x55e013443400 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ac947b52de09dd35 server_cookie=7608b19311634ecf in_seq=0 out_seq=0 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.146+0000 7f23cbfff640 10 client.26479 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23cbfff640 1 -- 172.21.15.45:0/1743935970 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e012fe2700 con 0x55e013440fe0 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23cbfff640 10 client.26479 handle_command_reply: tid=0 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 client.26479 shutdown 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 msgr2=0x55e0133dedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 0x55e0133dedb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f23dc001eb0 tx=0x7f23dc0055f0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 shutdown_connections 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55e013440fe0 0x55e013443400 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e0133d8750 0x55e01343f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e0133cfb70 0x55e0133dedb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 --2- 172.21.15.45:0/1743935970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e0133cf250 0x55e0133de870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 >> 172.21.15.45:0/1743935970 conn(0x55e0133c9980 msgr2=0x55e0133c82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23cbfff640 0 client.26479 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 shutdown_connections 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.147+0000 7f23f5d7d740 1 -- 172.21.15.45:0/1743935970 wait complete. 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.148+0000 7f23f5d7d740 20 client.26479 trim_cache size 0 max 16384 2024-03-12T05:16:50.145 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 msgr2=0x7f23d40a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 0x7f23d40a4120 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f23e40045a0 tx=0x7f23e4004510 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 msgr2=0x7f23ec1a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23e800e9c0 tx=0x7f23e800ee80 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 shutdown_connections 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f23d40a1c60 0x7f23d40a4120 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23ec1080d0 0x7f23ec1a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23ec10d720 0x7f23ec19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 --2- 172.21.15.45:0/3452707828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23ec107710 0x7f23ec1a38d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 >> 172.21.15.45:0/3452707828 conn(0x7f23ec075690 msgr2=0x7f23ec075e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 shutdown_connections 2024-03-12T05:16:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.149+0000 7f23f30d4640 1 -- 172.21.15.45:0/3452707828 wait complete. 2024-03-12T05:16:50.162 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:50.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:16:50.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 -- 172.21.15.45:0/371998952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d40ffb30 msgr2=0x7fb5d40fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 --2- 172.21.15.45:0/371998952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d40ffb30 0x7fb5d40fff30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009cd0 tx=0x7fb5cc02c0e0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 -- 172.21.15.45:0/371998952 shutdown_connections 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 --2- 172.21.15.45:0/371998952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d4109010 0x7fb5d4109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 --2- 172.21.15.45:0/371998952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d41099d0 0x7fb5d41100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 --2- 172.21.15.45:0/371998952 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d40ffb30 0x7fb5d40fff30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 -- 172.21.15.45:0/371998952 >> 172.21.15.45:0/371998952 conn(0x7fb5d4075170 msgr2=0x7fb5d4075580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.373+0000 7fb5da38c640 1 -- 172.21.15.45:0/371998952 shutdown_connections 2024-03-12T05:16:50.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.374+0000 7fb5da38c640 1 -- 172.21.15.45:0/371998952 wait complete. 2024-03-12T05:16:50.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.374+0000 7fb5da38c640 1 Processor -- start 2024-03-12T05:16:50.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.374+0000 7fb5da38c640 1 -- start start 2024-03-12T05:16:50.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 0x7fb5d41996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d4109010 0x7fb5d4199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d419e530 con 0x7fb5d40ffb30 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d419e6a0 con 0x7fb5d4109010 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5da38c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d419e860 con 0x7fb5d41099d0 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5d8929640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5d8929640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45778/0 (socket says 172.21.15.45:45778) 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5d8929640 1 -- 172.21.15.45:0/486554036 learned_addr learned my addr 172.21.15.45:0/486554036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.375+0000 7fb5d37fe640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d4109010 0x7fb5d4199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 0x7fb5d41996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 -- 172.21.15.45:0/486554036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 msgr2=0x7fb5d419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 -- 172.21.15.45:0/486554036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d4109010 msgr2=0x7fb5d4199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d4109010 0x7fb5d4199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 -- 172.21.15.45:0/486554036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc009900 con 0x7fb5d40ffb30 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d8929640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.376+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 0x7fb5d41996c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc037040 tx=0x7fb5cc035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.377+0000 7fb5d17fa640 1 -- 172.21.15.45:0/486554036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc0329e0 con 0x7fb5d40ffb30 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.377+0000 7fb5d17fa640 1 -- 172.21.15.45:0/486554036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5cc035e90 con 0x7fb5d40ffb30 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.377+0000 7fb5d17fa640 1 -- 172.21.15.45:0/486554036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc0047d0 con 0x7fb5d40ffb30 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.377+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d419eae0 con 0x7fb5d40ffb30 2024-03-12T05:16:50.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.377+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d41a61e0 con 0x7fb5d40ffb30 2024-03-12T05:16:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 Processor -- start 2024-03-12T05:16:50.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 -- start start 2024-03-12T05:16:50.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5d17fa640 1 -- 172.21.15.45:0/486554036 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb5cc03b040 con 0x7fb5d40ffb30 2024-03-12T05:16:50.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12746d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127481560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127428960 con 0x55f12746d270 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1273dc310 con 0x55f12746dc20 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1273d9840 con 0x55f12747f170 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127481560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127481560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45788/0 (socket says 172.21.15.45:45788) 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.381+0000 7fb5d37fe640 1 -- 172.21.15.45:0/2320853189 learned_addr learned my addr 172.21.15.45:0/2320853189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d8929640 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12746d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 -- 172.21.15.45:0/2320853189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 msgr2=0x55f127481560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127481560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 -- 172.21.15.45:0/2320853189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 msgr2=0x55f12746d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12746d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 -- 172.21.15.45:0/2320853189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f127397060 con 0x55f12746dc20 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.382+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ec30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0ed800 tx=0x7fb5cc0f81a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87cd97437614a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5d17fa640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 0x7fb5bc0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5ba7fc640 1 -- 172.21.15.45:0/2320853189 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc0f8750 con 0x55f12746dc20 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5ba7fc640 1 -- 172.21.15.45:0/2320853189 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5cc0f22e0 con 0x55f12746dc20 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5ba7fc640 1 -- 172.21.15.45:0/2320853189 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc0b8de0 con 0x55f12746dc20 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5d8929640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 0x7fb5bc0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.383+0000 7fb5d17fa640 1 -- 172.21.15.45:0/486554036 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb5cc0effd0 con 0x7fb5d40ffb30 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- 172.21.15.45:0/2320853189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 msgr2=0x55f12747ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ec30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc0ed800 tx=0x7fb5cc0f81a0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- 172.21.15.45:0/2320853189 shutdown_connections 2024-03-12T05:16:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127481560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ec30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/2320853189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12746d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- 172.21.15.45:0/2320853189 >> 172.21.15.45:0/2320853189 conn(0x55f127467b90 msgr2=0x55f127467f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- 172.21.15.45:0/2320853189 shutdown_connections 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5d8929640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 0x7fb5bc0a41b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4005fd0 tx=0x7fb5c400b3a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- 172.21.15.45:0/2320853189 wait complete. 2024-03-12T05:16:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.384+0000 7fb5dcf63740 1 -- start start 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.385+0000 7fb5dcf63740 1 Processor -- start 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.385+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 0x55f12747c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.385+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 0x55f12747c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.385+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 0x55f12747cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.385+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1273d5020 con 0x55f12746dc20 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 0x55f12747c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 0x55f12747c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41330/0 (socket says 172.21.15.45:41330) 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5d37fe640 1 -- 172.21.15.45:0/232107374 learned_addr learned my addr 172.21.15.45:0/232107374 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1270a8430 con 0x55f12747f170 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5d8929640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 0x55f12747cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127064e00 con 0x55f12746d270 2024-03-12T05:16:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5dcf63740 10 client.0 _ll_register_callbacks cb 0x55f12737ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.386+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 0x55f12747c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 -- 172.21.15.45:0/232107374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 msgr2=0x55f12747c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 0x55f12747c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 -- 172.21.15.45:0/232107374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 msgr2=0x55f12747c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 0x55f12747c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d41a61e0 con 0x55f12747f170 2024-03-12T05:16:50.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 0x55f12747c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5d8929640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 0x55f12747cc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400b680 tx=0x7fb5c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e24878e07ed8bad server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.387+0000 7fb5b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.388+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c400fa60 con 0x55f12747f170 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.388+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb5c4018070 con 0x55f12747f170 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.388+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f127111720 con 0x55f12747f170 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.388+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5c400fbc0 con 0x55f12747f170 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.389+0000 7fb5dcf63740 10 client.26510 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.389+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f1273d36a0 con 0x55f12747f170 2024-03-12T05:16:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f127111720 con 0x55f12747f170 2024-03-12T05:16:50.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5c4025020 con 0x55f12747f170 2024-03-12T05:16:50.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 10 client.26510 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 10 client.26510 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 20 client.26510 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f12748f500 0x55f127491920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 4 client.26510 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f127160890 con 0x55f12748f500 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.390+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f12748f500 0x55f127491920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.391+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f12748f500 0x55f127491920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=35ea85919824cb29 server_cookie=c4caa0b692d3329a in_seq=0 out_seq=0 2024-03-12T05:16:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.391+0000 7fb5b37fe640 10 client.26510 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:50.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.393+0000 7fb5b37fe640 1 -- 172.21.15.45:0/232107374 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f127160890 con 0x55f12748f500 2024-03-12T05:16:50.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.393+0000 7fb5b37fe640 10 client.26510 handle_command_reply: tid=0 2024-03-12T05:16:50.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.393+0000 7fb5dcf63740 1 client.26510 shutdown 2024-03-12T05:16:50.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.393+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 msgr2=0x55f12747cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.393+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 0x55f12747cc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400b680 tx=0x7fb5c40046c0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 shutdown_connections 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f12748f500 0x55f127491920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746dc20 0x55f12747c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12747f170 0x55f12747cc30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/232107374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12746d270 0x55f12747c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 >> 172.21.15.45:0/232107374 conn(0x55f127467b90 msgr2=0x55f1274698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5b37fe640 0 client.26510 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:50.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.394+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 shutdown_connections 2024-03-12T05:16:50.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.395+0000 7fb5dcf63740 1 -- 172.21.15.45:0/232107374 wait complete. 2024-03-12T05:16:50.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.397+0000 7fb5dcf63740 20 client.26510 trim_cache size 0 max 16384 2024-03-12T05:16:50.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 Processor -- start 2024-03-12T05:16:50.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 -- start start 2024-03-12T05:16:50.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12747e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127496280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127463340 con 0x55f12746d270 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12747e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12747e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41340/0 (socket says 172.21.15.45:41340) 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5d3fff640 1 -- 172.21.15.45:0/3299906199 learned_addr learned my addr 172.21.15.45:0/3299906199 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f12735a130 con 0x55f12746dc20 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.423+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127481cf0 con 0x55f12747f170 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d8929640 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127496280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 -- 172.21.15.45:0/3299906199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 msgr2=0x55f127496280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127496280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 -- 172.21.15.45:0/3299906199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 msgr2=0x55f12747e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12747e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 -- 172.21.15.45:0/3299906199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f12736fc20 con 0x55f12746dc20 2024-03-12T05:16:50.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.424+0000 7fb5d37fe640 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ca50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55f12747d780 tx=0x7fb5c8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d0a9f6034b4c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.425+0000 7fb5b9ffb640 1 -- 172.21.15.45:0/3299906199 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8017050 con 0x55f12746dc20 2024-03-12T05:16:50.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.425+0000 7fb5b9ffb640 1 -- 172.21.15.45:0/3299906199 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5c8018040 con 0x55f12746dc20 2024-03-12T05:16:50.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.425+0000 7fb5b9ffb640 1 -- 172.21.15.45:0/3299906199 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8003d10 con 0x55f12746dc20 2024-03-12T05:16:50.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.426+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 msgr2=0x55f12747ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.426+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ca50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55f12747d780 tx=0x7fb5c8005f90 comp rx=0 tx=0).stop 2024-03-12T05:16:50.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 shutdown_connections 2024-03-12T05:16:50.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127496280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f12747ca50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/3299906199 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f12747e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 >> 172.21.15.45:0/3299906199 conn(0x55f127467b90 msgr2=0x55f127477550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 shutdown_connections 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.427+0000 7fb5dcf63740 1 -- 172.21.15.45:0/3299906199 wait complete. 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.428+0000 7fb5dcf63740 1 -- start start 2024-03-12T05:16:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.428+0000 7fb5dcf63740 1 Processor -- start 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f127484550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f127484a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127485980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127485f40 con 0x55f12746d270 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1274860b0 con 0x55f12746dc20 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f127486270 con 0x55f12747f170 2024-03-12T05:16:50.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5dcf63740 10 client.0 _ll_register_callbacks cb 0x7fb5cc0f39b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f127484550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f127484550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41350/0 (socket says 172.21.15.45:41350) 2024-03-12T05:16:50.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d3fff640 1 -- 172.21.15.45:0/1473949348 learned_addr learned my addr 172.21.15.45:0/1473949348 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127485980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 -- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 msgr2=0x55f127484a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:16:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f127484a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 -- 172.21.15.45:0/1473949348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 msgr2=0x55f127484550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f127484550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.429+0000 7fb5d8929640 1 -- 172.21.15.45:0/1473949348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c8009b70 con 0x55f12747f170 2024-03-12T05:16:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.430+0000 7fb5d8929640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127485980 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40059f0 tx=0x7fb5c4011e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ccfdafb112d061 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.430+0000 7fb5b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.430+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c400bb00 con 0x55f12747f170 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.430+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fb5c4018040 con 0x55f12747f170 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.430+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1270d47e0 con 0x55f12747f170 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.431+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5c400f860 con 0x55f12747f170 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.431+0000 7fb5dcf63740 10 client.26497 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.431+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f1273958c0 con 0x55f12747f170 2024-03-12T05:16:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.431+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f1270d47e0 con 0x55f12747f170 2024-03-12T05:16:50.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.432+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5c4042070 con 0x55f12747f170 2024-03-12T05:16:50.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.432+0000 7fb5dcf63740 10 client.26497 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5dcf63740 10 client.26497 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5dcf63740 20 client.26497 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f1274f7b50 0x55f1274f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5dcf63740 4 client.26497 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f127080700 con 0x55f1274f7b50 2024-03-12T05:16:50.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.433+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f1274f7b50 0x55f1274f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.434+0000 7fb5d3fff640 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f1274f7b50 0x55f1274f9f70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5ab43e9e745cb4d server_cookie=be7b14e70b4951ff in_seq=0 out_seq=0 2024-03-12T05:16:50.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.434+0000 7fb5b37fe640 10 client.26497 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:50.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.435+0000 7fb5b37fe640 1 -- 172.21.15.45:0/1473949348 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f127080700 con 0x55f1274f7b50 2024-03-12T05:16:50.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.435+0000 7fb5b37fe640 10 client.26497 handle_command_reply: tid=0 2024-03-12T05:16:50.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.435+0000 7fb5dcf63740 1 client.26497 shutdown 2024-03-12T05:16:50.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 msgr2=0x55f127485980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127485980 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40059f0 tx=0x7fb5c4011e10 comp rx=0 tx=0).stop 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 shutdown_connections 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f1274f7b50 0x55f1274f9f70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5b37fe640 0 client.26497 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.436+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f12747f170 0x55f127485980 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f12746dc20 0x55f127484a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 1 --2- 172.21.15.45:0/1473949348 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f12746d270 0x55f127484550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 >> 172.21.15.45:0/1473949348 conn(0x55f127467b90 msgr2=0x55f127496280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 shutdown_connections 2024-03-12T05:16:50.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 1 -- 172.21.15.45:0/1473949348 wait complete. 2024-03-12T05:16:50.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.437+0000 7fb5dcf63740 20 client.26497 trim_cache size 0 max 16384 2024-03-12T05:16:50.433 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:50.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 msgr2=0x7fb5bc0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 0x7fb5bc0a41b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4005fd0 tx=0x7fb5c400b3a0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 msgr2=0x7fb5d41996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 0x7fb5d41996c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc037040 tx=0x7fb5cc035ca0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 shutdown_connections 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5bc0a1cf0 0x7fb5bc0a41b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5d41099d0 0x7fb5d419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5d4109010 0x7fb5d4199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 --2- 172.21.15.45:0/486554036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5d40ffb30 0x7fb5d41996c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.438+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 >> 172.21.15.45:0/486554036 conn(0x7fb5d4075170 msgr2=0x7fb5d4100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.439+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 shutdown_connections 2024-03-12T05:16:50.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.439+0000 7fb5da38c640 1 -- 172.21.15.45:0/486554036 wait complete. 2024-03-12T05:16:50.452 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:50.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:16:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:50 smithi045 ceph-mon[32174]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-03-12T05:16:50.495 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:50 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2759572801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 -- 172.21.15.45:0/1983054695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10f9a0 msgr2=0x7f7b8c10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 --2- 172.21.15.45:0/1983054695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c10d490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b8802bc40 comp rx=0 tx=0).stop 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 -- 172.21.15.45:0/1983054695 shutdown_connections 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 --2- 172.21.15.45:0/1983054695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c11a960 0x7f7b8c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 --2- 172.21.15.45:0/1983054695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 --2- 172.21.15.45:0/1983054695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c10d490 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 -- 172.21.15.45:0/1983054695 >> 172.21.15.45:0/1983054695 conn(0x7f7b8c06c260 msgr2=0x7f7b8c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 -- 172.21.15.45:0/1983054695 shutdown_connections 2024-03-12T05:16:50.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.635+0000 7f7b93a62640 1 -- 172.21.15.45:0/1983054695 wait complete. 2024-03-12T05:16:50.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 Processor -- start 2024-03-12T05:16:50.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 -- start start 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c1a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c11a960 0x7f7b8c1aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1ab2b0 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1ab420 con 0x7f7b8c10d9d0 2024-03-12T05:16:50.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b93a62640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1ab5e0 con 0x7f7b8c11a960 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41382/0 (socket says 172.21.15.45:41382) 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b90ffd640 1 -- 172.21.15.45:0/3176197862 learned_addr learned my addr 172.21.15.45:0/3176197862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.637+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c1a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b91fff640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c11a960 0x7f7b8c1aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 -- 172.21.15.45:0/3176197862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c11a960 msgr2=0x7f7b8c1aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c11a960 0x7f7b8c1aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 -- 172.21.15.45:0/3176197862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 msgr2=0x7f7b8c1a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c1a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 -- 172.21.15.45:0/3176197862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88009560 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b917fe640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c1a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c920 tx=0x7f7b7c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b827fc640 1 -- 172.21.15.45:0/3176197862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c007c70 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b827fc640 1 -- 172.21.15.45:0/3176197862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b7c007dd0 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c1ab8c0 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.638+0000 7f7b827fc640 1 -- 172.21.15.45:0/3176197862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c0187c0 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.639+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c1b3140 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.641+0000 7f7b827fc640 1 -- 172.21.15.45:0/3176197862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7b7c00f450 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.642+0000 7f7b827fc640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 0x7f7b780a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.642+0000 7f7b827fc640 1 -- 172.21.15.45:0/3176197862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7b7c0cd910 con 0x7f7b8c10f9a0 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.642+0000 7f7b917fe640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 0x7f7b780a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 Processor -- start 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 -- start start 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 0x55a346736630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a346741b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3466f1960 con 0x55a346736250 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3466a5310 con 0x55a34673f750 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b96630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3466a2840 con 0x55a346736b70 2024-03-12T05:16:50.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b917fe640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 0x7f7b780a4160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000c00 tx=0x7f7b8802c6c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45828/0 (socket says 172.21.15.45:45828) 2024-03-12T05:16:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 0x55a346736630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b91fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a346741b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.643+0000 7f7b90ffd640 1 -- 172.21.15.45:0/940170787 learned_addr learned my addr 172.21.15.45:0/940170787 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.644+0000 7f7b90ffd640 1 -- 172.21.15.45:0/940170787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 msgr2=0x55a346741b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.644+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a346741b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.644+0000 7f7b90ffd640 1 -- 172.21.15.45:0/940170787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 msgr2=0x55a346736630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.644+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 0x55a346736630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.644+0000 7f7b90ffd640 1 -- 172.21.15.45:0/940170787 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a346660060 con 0x55a346736b70 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.645+0000 7f7b917fe640 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 0x55a346736630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.645+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c096eb0 tx=0x7f7b7c099e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d31bd1324226435 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.646+0000 7f7b777fe640 1 -- 172.21.15.45:0/940170787 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c09abe0 con 0x55a346736b70 2024-03-12T05:16:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.646+0000 7f7b777fe640 1 -- 172.21.15.45:0/940170787 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b7c096780 con 0x55a346736b70 2024-03-12T05:16:50.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.646+0000 7f7b777fe640 1 -- 172.21.15.45:0/940170787 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c096b50 con 0x55a346736b70 2024-03-12T05:16:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.647+0000 7f7b96630740 1 -- 172.21.15.45:0/940170787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 msgr2=0x55a34673f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.647+0000 7f7b96630740 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c096eb0 tx=0x7f7b7c099e40 comp rx=0 tx=0).stop 2024-03-12T05:16:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 -- 172.21.15.45:0/940170787 shutdown_connections 2024-03-12T05:16:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a346741b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673f210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 --2- 172.21.15.45:0/940170787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736250 0x55a346736630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 -- 172.21.15.45:0/940170787 >> 172.21.15.45:0/940170787 conn(0x55a346730980 msgr2=0x55a346730d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 -- 172.21.15.45:0/940170787 shutdown_connections 2024-03-12T05:16:50.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.648+0000 7f7b96630740 1 -- 172.21.15.45:0/940170787 wait complete. 2024-03-12T05:16:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.649+0000 7f7b96630740 1 -- start start 2024-03-12T05:16:50.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.649+0000 7f7b96630740 1 Processor -- start 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a34673d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37466/0 (socket says 172.21.15.45:37466) 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b917fe640 1 -- 172.21.15.45:0/2352381776 learned_addr learned my addr 172.21.15.45:0/2352381776 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34669e020 con 0x55a34673f750 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b91fff640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a34673d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a346371430 con 0x55a346736250 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34632de00 con 0x55a346736b70 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.650+0000 7f7b96630740 10 client.0 _ll_register_callbacks cb 0x55a346645cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 -- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 msgr2=0x55a34673cd00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673cd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 -- 172.21.15.45:0/2352381776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 msgr2=0x55a34673d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a34673d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c1b3140 con 0x55a346736250 2024-03-12T05:16:50.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b91fff640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a34673d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b88009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d5c0945d57677d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88003cc0 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7b88047040 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.651+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a34639d7e0 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b88003e20 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 10 client.26525 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a34669c6a0 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a34639d7e0 con 0x55a346736250 2024-03-12T05:16:50.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b8803f670 con 0x55a346736250 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 10 client.26525 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 10 client.26525 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 20 client.26525 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a34674fbb0 0x55a346751ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 4 client.26525 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.652+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3463da720 con 0x55a34674fbb0 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.653+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a34674fbb0 0x55a346751ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.653+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a34674fbb0 0x55a346751ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=64f25d1eb3b7361b server_cookie=9a6a2828753ebf24 in_seq=0 out_seq=0 2024-03-12T05:16:50.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.653+0000 7f7b74ff9640 10 client.26525 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2352381776 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a3463da720 con 0x55a34674fbb0 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b74ff9640 10 client.26525 handle_command_reply: tid=0 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 client.26525 shutdown 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 msgr2=0x55a34673c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b88009260 comp rx=0 tx=0).stop 2024-03-12T05:16:50.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 shutdown_connections 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a34674fbb0 0x55a346751ff0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b74ff9640 0 client.26525 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a34673d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 --2- 172.21.15.45:0/2352381776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673c7c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.654+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 >> 172.21.15.45:0/2352381776 conn(0x55a346730980 msgr2=0x55a3467326d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.655+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 shutdown_connections 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.655+0000 7f7b96630740 1 -- 172.21.15.45:0/2352381776 wait complete. 2024-03-12T05:16:50.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.655+0000 7f7b96630740 20 client.26525 trim_cache size 0 max 16384 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.677+0000 7f7b96630740 1 Processor -- start 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.677+0000 7f7b96630740 1 -- start start 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a346737b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b917fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37472/0 (socket says 172.21.15.45:37472) 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b90ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45852/0 (socket says 172.21.15.45:45852) 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b917fe640 1 -- 172.21.15.45:0/529510526 learned_addr learned my addr 172.21.15.45:0/529510526 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a346623130 con 0x55a34673f750 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b91fff640 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a346737b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34674f930 con 0x55a346736250 2024-03-12T05:16:50.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.678+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a346757270 con 0x55a346736b70 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 -- 172.21.15.45:0/529510526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 msgr2=0x55a34673eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 -- 172.21.15.45:0/529510526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 msgr2=0x55a346737b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a346737b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 -- 172.21.15.45:0/529510526 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a346638c20 con 0x55a346736b70 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b91fff640 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a346737b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.679+0000 7f7b90ffd640 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55a346735d20 tx=0x7f7b7c085040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6187480b6afa2076 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.680+0000 7f7b76ffd640 1 -- 172.21.15.45:0/529510526 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c09a4e0 con 0x55a346736b70 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.680+0000 7f7b76ffd640 1 -- 172.21.15.45:0/529510526 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b7c09e630 con 0x55a346736b70 2024-03-12T05:16:50.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.680+0000 7f7b76ffd640 1 -- 172.21.15.45:0/529510526 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00f700 con 0x55a346736b70 2024-03-12T05:16:50.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 msgr2=0x55a34673d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55a346735d20 tx=0x7f7b7c085040 comp rx=0 tx=0).stop 2024-03-12T05:16:50.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 shutdown_connections 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a34673f750 0x55a346737b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736b70 0x55a34673d060 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 --2- 172.21.15.45:0/529510526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a346736250 0x55a34673eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 >> 172.21.15.45:0/529510526 conn(0x55a346730980 msgr2=0x55a34672ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 shutdown_connections 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- 172.21.15.45:0/529510526 wait complete. 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.681+0000 7f7b96630740 1 -- start start 2024-03-12T05:16:50.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.682+0000 7f7b96630740 1 Processor -- start 2024-03-12T05:16:50.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.683+0000 7f7b96630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736250 0x55a346746a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.683+0000 7f7b96630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736b70 0x55a346744e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b96630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b96630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a346745b70 con 0x55a346736b70 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b96630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a346745ce0 con 0x55a34673f750 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b96630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3467a61a0 con 0x55a346736250 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b91fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b91fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37488/0 (socket says 172.21.15.45:37488) 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b91fff640 1 -- 172.21.15.45:0/2775049188 learned_addr learned my addr 172.21.15.45:0/2775049188 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.684+0000 7f7b96630740 10 client.0 _ll_register_callbacks cb 0x7f7b7c00cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736250 0x55a346746a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b91fff640 1 -- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736250 msgr2=0x55a346746a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b91fff640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736250 0x55a346746a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b91fff640 1 -- 172.21.15.45:0/2775049188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736b70 msgr2=0x55a346744e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b91fff640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736b70 0x55a346744e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.685+0000 7f7b91fff640 1 -- 172.21.15.45:0/2775049188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b84007e50 con 0x55a34673f750 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b91fff640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400f670 tx=0x7f7b8400fb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73eaa4c61426b261 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8400e940 con 0x55a34673f750 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f7b8400eaa0 con 0x55a34673f750 2024-03-12T05:16:50.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b7c096780 con 0x55a34673f750 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b8400ec60 con 0x55a34673f750 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b96630740 10 client.26531 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a346660060 con 0x55a34673f750 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.686+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7b84017040 con 0x55a34673f750 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7b84023070 con 0x55a34673f750 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 10 client.26531 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 10 client.26531 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 20 client.26531 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a3467a79a0 0x55a3467a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 4 client.26531 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f7b7c096780 con 0x55a3467a79a0 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.687+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a3467a79a0 0x55a3467a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.688+0000 7f7b917fe640 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a3467a79a0 0x55a3467a9dc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=819047a94862824b server_cookie=631acfb4de0b1593 in_seq=0 out_seq=0 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.688+0000 7f7b74ff9640 10 client.26531 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:50.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b74ff9640 1 -- 172.21.15.45:0/2775049188 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f7b7c096780 con 0x55a3467a79a0 2024-03-12T05:16:50.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b74ff9640 10 client.26531 handle_command_reply: tid=0 2024-03-12T05:16:50.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b96630740 1 client.26531 shutdown 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 msgr2=0x55a3467454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400f670 tx=0x7f7b8400fb30 comp rx=0 tx=0).stop 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 shutdown_connections 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.689+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a3467a79a0 0x55a3467a9dc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b74ff9640 0 client.26531 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a346736b70 0x55a346744e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a34673f750 0x55a3467454a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 --2- 172.21.15.45:0/2775049188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a346736250 0x55a346746a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 >> 172.21.15.45:0/2775049188 conn(0x55a346730980 msgr2=0x55a346737b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 shutdown_connections 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 1 -- 172.21.15.45:0/2775049188 wait complete. 2024-03-12T05:16:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.690+0000 7f7b96630740 20 client.26531 trim_cache size 0 max 16384 2024-03-12T05:16:50.687 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 msgr2=0x7f7b780a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 0x7f7b780a4160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000c00 tx=0x7f7b8802c6c0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 msgr2=0x7f7b8c1a6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c920 tx=0x7f7b7c00cde0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 shutdown_connections 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7b780a1ca0 0x7f7b780a4160 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b8c10f9a0 0x7f7b8c1a6930 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c11a960 0x7f7b8c1aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 --2- 172.21.15.45:0/3176197862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b8c10d9d0 0x7f7b8c1a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 >> 172.21.15.45:0/3176197862 conn(0x7f7b8c06c260 msgr2=0x7f7b8c11b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 shutdown_connections 2024-03-12T05:16:50.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.691+0000 7f7b93a62640 1 -- 172.21.15.45:0/3176197862 wait complete. 2024-03-12T05:16:50.704 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:50.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:16:50.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:50 smithi130 ceph-mon[35398]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-03-12T05:16:50.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:50 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2759572801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:50 smithi116 ceph-mon[35486]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 36 MiB/s wr, 1.82k op/s 2024-03-12T05:16:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:50 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2759572801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:16:50.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 -- 172.21.15.45:0/3745369563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8105330 msgr2=0x7f15a810cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 --2- 172.21.15.45:0/3745369563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8105330 0x7f15a810cf00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009be0 tx=0x7f15a002bef0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 -- 172.21.15.45:0/3745369563 shutdown_connections 2024-03-12T05:16:50.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 --2- 172.21.15.45:0/3745369563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100c10 0x7f15a8104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 --2- 172.21.15.45:0/3745369563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8105330 0x7f15a810cf00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 --2- 172.21.15.45:0/3745369563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8100260 0x7f15a8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 -- 172.21.15.45:0/3745369563 >> 172.21.15.45:0/3745369563 conn(0x7f15a8077180 msgr2=0x7f15a8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 -- 172.21.15.45:0/3745369563 shutdown_connections 2024-03-12T05:16:50.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.919+0000 7f15af7bc640 1 -- 172.21.15.45:0/3745369563 wait complete. 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.920+0000 7f15af7bc640 1 Processor -- start 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.920+0000 7f15af7bc640 1 -- start start 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100260 0x7f15a8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 0x7f15a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 0x7f15a81a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a81a3bd0 con 0x7f15a8100260 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15acd57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 0x7f15a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15acd57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 0x7f15a8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37496/0 (socket says 172.21.15.45:37496) 2024-03-12T05:16:50.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a81a3d40 con 0x7f15a8105330 2024-03-12T05:16:50.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15acd57640 1 -- 172.21.15.45:0/264719962 learned_addr learned my addr 172.21.15.45:0/264719962 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a81a3f00 con 0x7f15a8198f80 2024-03-12T05:16:50.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15ad558640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100260 0x7f15a8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.921+0000 7f15add59640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 0x7f15a81a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 -- 172.21.15.45:0/264719962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 msgr2=0x7f15a8198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 0x7f15a8198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 -- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100260 msgr2=0x7f15a8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100260 0x7f15a8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 -- 172.21.15.45:0/264719962 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4009560 con 0x7f15a8198f80 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f15add59640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 0x7f15a81a3580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f15a002c430 tx=0x7f15a0033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f159e7fc640 1 -- 172.21.15.45:0/264719962 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0003970 con 0x7f15a8198f80 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f159e7fc640 1 -- 172.21.15.45:0/264719962 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f15a0042070 con 0x7f15a8198f80 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.922+0000 7f159e7fc640 1 -- 172.21.15.45:0/264719962 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00042a0 con 0x7f15a8198f80 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.923+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a0009840 con 0x7f15a8198f80 2024-03-12T05:16:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.923+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a8196620 con 0x7f15a8198f80 2024-03-12T05:16:50.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.926+0000 7f159e7fc640 1 -- 172.21.15.45:0/264719962 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f15a0004400 con 0x7f15a8198f80 2024-03-12T05:16:50.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 Processor -- start 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 -- start start 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9a5d960 con 0x55e5d9ab3f60 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9a11310 con 0x55e5d9aaaa60 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15b238f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9a0e840 con 0x55e5d9aab380 2024-03-12T05:16:50.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15acd57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15ad558640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15acd57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37506/0 (socket says 172.21.15.45:37506) 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15acd57640 1 -- 172.21.15.45:0/2823669739 learned_addr learned my addr 172.21.15.45:0/2823669739 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.927+0000 7f15acd57640 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 -- 172.21.15.45:0/2823669739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 msgr2=0x55e5d9ab3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 -- 172.21.15.45:0/2823669739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9ab3f60 msgr2=0x55e5d9ab6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 -- 172.21.15.45:0/2823669739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5d99cc060 con 0x55e5d9aaaa60 2024-03-12T05:16:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f159e7fc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 0x7f15940a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15acd57640 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1598011810 tx=0x7f1598009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00b6ec1dcfe5de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f159e7fc640 1 -- 172.21.15.45:0/264719962 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f15a0044070 con 0x7f15a8198f80 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.928+0000 7f15937fe640 1 -- 172.21.15.45:0/2823669739 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598004030 con 0x55e5d9aaaa60 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.929+0000 7f15937fe640 1 -- 172.21.15.45:0/2823669739 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f159800f7c0 con 0x55e5d9aaaa60 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.929+0000 7f15937fe640 1 -- 172.21.15.45:0/2823669739 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15980038e0 con 0x55e5d9aaaa60 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.929+0000 7f15ad558640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 0x7f15940a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.929+0000 7f15b238f740 1 -- 172.21.15.45:0/2823669739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 msgr2=0x55e5d9aaae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.929+0000 7f15b238f740 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1598011810 tx=0x7f1598009530 comp rx=0 tx=0).stop 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15ad558640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 0x7f15940a40e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55e5d9aaa530 tx=0x7f15a4009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 -- 172.21.15.45:0/2823669739 shutdown_connections 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 --2- 172.21.15.45:0/2823669739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9aaae40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 -- 172.21.15.45:0/2823669739 >> 172.21.15.45:0/2823669739 conn(0x55e5d9aa5190 msgr2=0x55e5d9aa5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 -- 172.21.15.45:0/2823669739 shutdown_connections 2024-03-12T05:16:50.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.930+0000 7f15b238f740 1 -- 172.21.15.45:0/2823669739 wait complete. 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.931+0000 7f15b238f740 1 -- start start 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.931+0000 7f15b238f740 1 Processor -- start 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15b238f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15b238f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15b238f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15b238f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9a0a020 con 0x55e5d9aaaa60 2024-03-12T05:16:50.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15ad558640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15acd57640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15add59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15add59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37520/0 (socket says 172.21.15.45:37520) 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15add59640 1 -- 172.21.15.45:0/3383103974 learned_addr learned my addr 172.21.15.45:0/3383103974 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.932+0000 7f15ad558640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41452/0 (socket says 172.21.15.45:41452) 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.933+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d96dd430 con 0x55e5d9ab3f60 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.933+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9699e00 con 0x55e5d9aab380 2024-03-12T05:16:50.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.933+0000 7f15b238f740 10 client.0 _ll_register_callbacks cb 0x55e5d99b1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 -- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 msgr2=0x55e5d9ab1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 -- 172.21.15.45:0/3383103974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 msgr2=0x55e5d9ab0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1598019040 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15acd57640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15add59640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15a00c9a80 tx=0x7f15a00c1410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a86524fc6f2ca069 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f1590ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f15ad558640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00f16e0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.934+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f15a00c5960 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5d97097e0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15a00c9fb0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 10 client.26543 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e5d9a086a0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f15a00da1f0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f15a0004b40 con 0x55e5d9ab3f60 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 10 client.26543 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 10 client.26543 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 20 client.26543 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.935+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9ad74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.936+0000 7f15b238f740 4 client.26543 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.936+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5d97097e0 con 0x55e5d9ad50c0 2024-03-12T05:16:50.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.936+0000 7f15ad558640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9ad74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.936+0000 7f1590ff9640 10 client.26543 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.936+0000 7f15ad558640 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9ad74e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3945aed536c968c server_cookie=161d6ab6721c9497 in_seq=0 out_seq=0 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.937+0000 7f1590ff9640 1 -- 172.21.15.45:0/3383103974 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e5d97097e0 con 0x55e5d9ad50c0 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.937+0000 7f1590ff9640 10 client.26543 handle_command_reply: tid=0 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.937+0000 7f15b238f740 1 client.26543 shutdown 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 msgr2=0x55e5d9ab1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15a00c9a80 tx=0x7f15a00c1410 comp rx=0 tx=0).stop 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 shutdown_connections 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9ad74e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9ab1a70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f1590ff9640 0 client.26543 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 --2- 172.21.15.45:0/3383103974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9ab0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.938+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 >> 172.21.15.45:0/3383103974 conn(0x55e5d9aa5190 msgr2=0x55e5d9aa6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.939+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 shutdown_connections 2024-03-12T05:16:50.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.939+0000 7f15b238f740 1 -- 172.21.15.45:0/3383103974 wait complete. 2024-03-12T05:16:50.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.939+0000 7f15b238f740 20 client.26543 trim_cache size 0 max 16384 2024-03-12T05:16:50.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 Processor -- start 2024-03-12T05:16:50.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 -- start start 2024-03-12T05:16:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9a9ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d998f130 con 0x55e5d9aab380 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9ac7c40 con 0x55e5d9aaaa60 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15b238f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9a985f0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15acd57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15acd57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41458/0 (socket says 172.21.15.45:41458) 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15acd57640 1 -- 172.21.15.45:0/485982959 learned_addr learned my addr 172.21.15.45:0/485982959 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15ad558640 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9a9ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.961+0000 7f15add59640 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 -- 172.21.15.45:0/485982959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9ab3f60 msgr2=0x55e5d9acbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 -- 172.21.15.45:0/485982959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 msgr2=0x55e5d9ab1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 -- 172.21.15.45:0/485982959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5d99a4c20 con 0x55e5d9aaaa60 2024-03-12T05:16:50.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.962+0000 7f15ad558640 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9a9ba00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55e5d9ad7dc0 tx=0x7f15a401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d453c56fbe42cca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.963+0000 7f1592ffd640 1 -- 172.21.15.45:0/485982959 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4017070 con 0x55e5d9aaaa60 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.963+0000 7f1592ffd640 1 -- 172.21.15.45:0/485982959 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f15a4024040 con 0x55e5d9aaaa60 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.963+0000 7f1592ffd640 1 -- 172.21.15.45:0/485982959 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40047a0 con 0x55e5d9aaaa60 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.963+0000 7f15b238f740 1 -- 172.21.15.45:0/485982959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 msgr2=0x55e5d9a9ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.963+0000 7f15b238f740 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9a9ba00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55e5d9ad7dc0 tx=0x7f15a401b000 comp rx=0 tx=0).stop 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 -- 172.21.15.45:0/485982959 shutdown_connections 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aab380 0x55e5d9ab1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 --2- 172.21.15.45:0/485982959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9aaaa60 0x55e5d9a9ba00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 -- 172.21.15.45:0/485982959 >> 172.21.15.45:0/485982959 conn(0x55e5d9aa5190 msgr2=0x55e5d9a9b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 -- 172.21.15.45:0/485982959 shutdown_connections 2024-03-12T05:16:50.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 -- 172.21.15.45:0/485982959 wait complete. 2024-03-12T05:16:50.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.964+0000 7f15b238f740 1 -- start start 2024-03-12T05:16:50.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.965+0000 7f15b238f740 1 Processor -- start 2024-03-12T05:16:50.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.965+0000 7f15b238f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9acaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15ad558640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9b2b5d0 con 0x55e5d9aaaa60 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15acd57640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9acaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15ad558640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41474/0 (socket says 172.21.15.45:41474) 2024-03-12T05:16:50.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15ad558640 1 -- 172.21.15.45:0/3828659897 learned_addr learned my addr 172.21.15.45:0/3828659897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:50.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15add59640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9b2b740 con 0x55e5d9ab3f60 2024-03-12T05:16:50.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5d9b2b900 con 0x55e5d9aab380 2024-03-12T05:16:50.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.966+0000 7f15b238f740 10 client.0 _ll_register_callbacks cb 0x7f1598009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15add59640 1 -- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 msgr2=0x55e5d9acaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15add59640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9acaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15add59640 1 -- 172.21.15.45:0/3828659897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 msgr2=0x55e5d9acaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15add59640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15add59640 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5d99ca8c0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.967+0000 7f15ad558640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f15add59640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acb480 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15a00c1750 tx=0x7f15a00c1480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2606f9cd8fd90835 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:50.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f1590ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a00c9d20 con 0x55e5d9ab3f60 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f15a00c0c60 con 0x55e5d9ab3f60 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f159800f7c0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15a00bdab0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.968+0000 7f15b238f740 10 client.26549 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e5d99b1050 con 0x55e5d9ab3f60 2024-03-12T05:16:50.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f159800f7c0 con 0x55e5d9ab3f60 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f15a00c2c50 con 0x55e5d9ab3f60 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 10 client.26549 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 10 client.26549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 20 client.26549 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9b2ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 4 client.26549 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e5d9746720 con 0x55e5d9ad50c0 2024-03-12T05:16:50.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.969+0000 7f15ad558640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9b2ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:50.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.970+0000 7f1590ff9640 10 client.26549 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.970+0000 7f15ad558640 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9b2ebe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12d39770b7a8eea1 server_cookie=bc29350e7a128fe3 in_seq=0 out_seq=0 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f1590ff9640 1 -- 172.21.15.45:0/3828659897 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e5d9746720 con 0x55e5d9ad50c0 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f1590ff9640 10 client.26549 handle_command_reply: tid=0 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 client.26549 shutdown 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 msgr2=0x55e5d9acb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acb480 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15a00c1750 tx=0x7f15a00c1480 comp rx=0 tx=0).stop 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 shutdown_connections 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e5d9ad50c0 0x55e5d9b2ebe0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e5d9ab3f60 0x55e5d9acb480 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f1590ff9640 0 client.26549 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e5d9aab380 0x55e5d9acaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 --2- 172.21.15.45:0/3828659897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e5d9aaaa60 0x55e5d9acaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 >> 172.21.15.45:0/3828659897 conn(0x55e5d9aa5190 msgr2=0x55e5d9aa6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 shutdown_connections 2024-03-12T05:16:50.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.971+0000 7f15b238f740 1 -- 172.21.15.45:0/3828659897 wait complete. 2024-03-12T05:16:50.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.972+0000 7f15b238f740 20 client.26549 trim_cache size 0 max 16384 2024-03-12T05:16:50.969 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.973+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 msgr2=0x7f15940a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.973+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 0x7f15940a40e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55e5d9aaa530 tx=0x7f15a4009260 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.973+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 msgr2=0x7f15a81a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.973+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 0x7f15a81a3580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f15a002c430 tx=0x7f15a0033c20 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 shutdown_connections 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f15940a1c20 0x7f15940a40e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a8198f80 0x7f15a81a3580 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15a8105330 0x7f15a8198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 --2- 172.21.15.45:0/264719962 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15a8100260 0x7f15a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 >> 172.21.15.45:0/264719962 conn(0x7f15a8077180 msgr2=0x7f15a80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:50.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 shutdown_connections 2024-03-12T05:16:50.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:50.974+0000 7f15af7bc640 1 -- 172.21.15.45:0/264719962 wait complete. 2024-03-12T05:16:50.989 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:16:50.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.162+0000 7f6682c34640 1 -- 172.21.15.45:0/1653206786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66740969c0 msgr2=0x7f66740a2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.162+0000 7f6682c34640 1 --2- 172.21.15.45:0/1653206786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66740969c0 0x7f66740a2bb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f667c066e50 tx=0x7f667c096e60 comp rx=0 tx=0).stop 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 -- 172.21.15.45:0/1653206786 shutdown_connections 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 --2- 172.21.15.45:0/1653206786 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66740969c0 0x7f66740a2bb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 --2- 172.21.15.45:0/1653206786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6674096000 0x7f6674096480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 --2- 172.21.15.45:0/1653206786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f667409ad20 0x7f667409b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 -- 172.21.15.45:0/1653206786 >> 172.21.15.45:0/1653206786 conn(0x7f667400b830 msgr2=0x7f667400bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 -- 172.21.15.45:0/1653206786 shutdown_connections 2024-03-12T05:16:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.163+0000 7f6682c34640 1 -- 172.21.15.45:0/1653206786 wait complete. 2024-03-12T05:16:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.164+0000 7f6682c34640 1 Processor -- start 2024-03-12T05:16:51.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.164+0000 7f6682c34640 1 -- start start 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.164+0000 7f6682c34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 0x7f667412c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 0x7f667412c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682c34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682c34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674131070 con 0x7f667409ad20 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682433640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682433640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41496/0 (socket says 172.21.15.45:41496) 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682433640 1 -- 172.21.15.45:0/2258439430 learned_addr learned my addr 172.21.15.45:0/2258439430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681c32640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 0x7f667412c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741311e0 con 0x7f66740969c0 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741313a0 con 0x7f6674096000 2024-03-12T05:16:51.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681431640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 0x7f667412c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681c32640 1 -- 172.21.15.45:0/2258439430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 msgr2=0x7f667412c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681c32640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 0x7f667412c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681c32640 1 -- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 msgr2=0x7f6674130a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.165+0000 7f6681c32640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6681c32640 1 -- 172.21.15.45:0/2258439430 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c009490 con 0x7f6674096000 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6681431640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 0x7f667412c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6682433640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6681c32640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 0x7f667412c200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f667800e9f0 tx=0x7f667800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6672ffd640 1 -- 172.21.15.45:0/2258439430 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667800cd00 con 0x7f6674096000 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6672ffd640 1 -- 172.21.15.45:0/2258439430 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66780044d0 con 0x7f6674096000 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6678009490 con 0x7f6674096000 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.166+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c04f750 con 0x7f6674096000 2024-03-12T05:16:51.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.167+0000 7f6672ffd640 1 -- 172.21.15.45:0/2258439430 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6678010430 con 0x7f6674096000 2024-03-12T05:16:51.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.170+0000 7f6672ffd640 1 -- 172.21.15.45:0/2258439430 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6678010590 con 0x7f6674096000 2024-03-12T05:16:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.171+0000 7f6686a61740 1 Processor -- start 2024-03-12T05:16:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.171+0000 7f6686a61740 1 -- start start 2024-03-12T05:16:51.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.171+0000 7f6672ffd640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 0x7f66680a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 0x7f66680a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f3d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f48d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867ef8960 con 0x559867f3d460 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867eac310 con 0x559867f46960 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6686a61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867ea9840 con 0x559867f3dd80 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45944/0 (socket says 172.21.15.45:45944) 2024-03-12T05:16:51.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6681431640 1 -- 172.21.15.45:0/4235335819 learned_addr learned my addr 172.21.15.45:0/4235335819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f48d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6681c32640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f3d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 0x7f66680a40e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f667c069790 tx=0x7f667c067020 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6672ffd640 1 -- 172.21.15.45:0/2258439430 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f6678014070 con 0x7f6674096000 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 -- 172.21.15.45:0/4235335819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 msgr2=0x559867f46420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.172+0000 7f6682433640 1 -- 172.21.15.45:0/4235335819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 msgr2=0x559867f3d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6682433640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f3d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6682433640 1 -- 172.21.15.45:0/4235335819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559867e67060 con 0x559867f46960 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6681431640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6681c32640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6682433640 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f48d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f667c000b60 tx=0x7f667c0b0000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a2c9491a1245bc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6667fff640 1 -- 172.21.15.45:0/4235335819 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0762e0 con 0x559867f46960 2024-03-12T05:16:51.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6667fff640 1 -- 172.21.15.45:0/4235335819 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f667c0a6040 con 0x559867f46960 2024-03-12T05:16:51.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.173+0000 7f6667fff640 1 -- 172.21.15.45:0/4235335819 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0a06c0 con 0x559867f46960 2024-03-12T05:16:51.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 -- 172.21.15.45:0/4235335819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 msgr2=0x559867f48d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f48d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f667c000b60 tx=0x7f667c0b0000 comp rx=0 tx=0).stop 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 -- 172.21.15.45:0/4235335819 shutdown_connections 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f48d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f46420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 --2- 172.21.15.45:0/4235335819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.174+0000 7f6686a61740 1 -- 172.21.15.45:0/4235335819 >> 172.21.15.45:0/4235335819 conn(0x559867f37b90 msgr2=0x559867f37f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.175+0000 7f6686a61740 1 -- 172.21.15.45:0/4235335819 shutdown_connections 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.175+0000 7f6686a61740 1 -- 172.21.15.45:0/4235335819 wait complete. 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.175+0000 7f6686a61740 1 -- start start 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.175+0000 7f6686a61740 1 Processor -- start 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3d460 0x559867f438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 0x559867f43e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f44360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6681431640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 0x559867f43e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6681c32640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3d460 0x559867f438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6681431640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 0x559867f43e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41506/0 (socket says 172.21.15.45:41506) 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6681431640 1 -- 172.21.15.45:0/160669057 learned_addr learned my addr 172.21.15.45:0/160669057 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867ea5020 con 0x559867f3dd80 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867b78430 con 0x559867f46960 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867b34e00 con 0x559867f3d460 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.176+0000 7f6682433640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f44360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6686a61740 10 client.0 _ll_register_callbacks cb 0x559867e4ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 -- 172.21.15.45:0/160669057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3d460 msgr2=0x559867f438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3d460 0x559867f438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 -- 172.21.15.45:0/160669057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 msgr2=0x559867f43e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 0x559867f43e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66740a3420 con 0x559867f46960 2024-03-12T05:16:51.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6682433640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f44360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x559867f493a0 tx=0x7f667c099fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e158b09f0608243f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f66657fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0c3e30 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f667c074530 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.177+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559867ba47e0 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f667c0746f0 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 10 client.26561 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559867ea36a0 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f667c074d00 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f667c0a9070 con 0x559867f46960 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 10 client.26561 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:51.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 20 client.26561 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867f569e0 0x559867f58dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 4 client.26561 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.178+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559867ba47e0 con 0x559867f569e0 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.179+0000 7f6681c32640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867f569e0 0x559867f58dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.180+0000 7f66657fa640 10 client.26561 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.179+0000 7f6681c32640 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867f569e0 0x559867f58dd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25659bad4e0930aa server_cookie=77e645007de0f6e0 in_seq=0 out_seq=0 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.180+0000 7f66657fa640 1 -- 172.21.15.45:0/160669057 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559867ba47e0 con 0x559867f569e0 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.180+0000 7f66657fa640 10 client.26561 handle_command_reply: tid=0 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.180+0000 7f6686a61740 1 client.26561 shutdown 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 msgr2=0x559867f44360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f44360 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x559867f493a0 tx=0x7f667c099fa0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 shutdown_connections 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867f569e0 0x559867f58dd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3dd80 0x559867f43e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f44360 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 --2- 172.21.15.45:0/160669057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3d460 0x559867f438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 >> 172.21.15.45:0/160669057 conn(0x559867f37b90 msgr2=0x559867f398e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f66657fa640 0 client.26561 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 shutdown_connections 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 1 -- 172.21.15.45:0/160669057 wait complete. 2024-03-12T05:16:51.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.181+0000 7f6686a61740 20 client.26561 trim_cache size 0 max 16384 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.203+0000 7f6686a61740 1 Processor -- start 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.203+0000 7f6686a61740 1 -- start start 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.203+0000 7f6686a61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f46010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.203+0000 7f6686a61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f44180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6686a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f421b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6686a61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867e2a130 con 0x559867f3d460 2024-03-12T05:16:51.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6686a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867f56830 con 0x559867f46960 2024-03-12T05:16:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6686a61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867f59060 con 0x559867f3dd80 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f44180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f44180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45962/0 (socket says 172.21.15.45:45962) 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6681431640 1 -- 172.21.15.45:0/2527087946 learned_addr learned my addr 172.21.15.45:0/2527087946 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6681c32640 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f46010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 -- 172.21.15.45:0/2527087946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 msgr2=0x559867f44180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f44180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 -- 172.21.15.45:0/2527087946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 msgr2=0x559867f46010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f46010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 -- 172.21.15.45:0/2527087946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559867e3fc20 con 0x559867f46960 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.204+0000 7f6682433640 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f421b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f667c074fd0 tx=0x7f667c067c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e88d78bf9f4022 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.205+0000 7f66677fe640 1 -- 172.21.15.45:0/2527087946 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0b5d20 con 0x559867f46960 2024-03-12T05:16:51.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.205+0000 7f66677fe640 1 -- 172.21.15.45:0/2527087946 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f667c0762e0 con 0x559867f46960 2024-03-12T05:16:51.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.205+0000 7f66677fe640 1 -- 172.21.15.45:0/2527087946 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0d59d0 con 0x559867f46960 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 -- 172.21.15.45:0/2527087946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 msgr2=0x559867f421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f421b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f667c074fd0 tx=0x7f667c067c10 comp rx=0 tx=0).stop 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 -- 172.21.15.45:0/2527087946 shutdown_connections 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f46960 0x559867f421b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f44180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 --2- 172.21.15.45:0/2527087946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f3d460 0x559867f46010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 -- 172.21.15.45:0/2527087946 >> 172.21.15.45:0/2527087946 conn(0x559867f37b90 msgr2=0x559867f5d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 -- 172.21.15.45:0/2527087946 shutdown_connections 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.206+0000 7f6686a61740 1 -- 172.21.15.45:0/2527087946 wait complete. 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 -- start start 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 Processor -- start 2024-03-12T05:16:51.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f3d460 0x559867f4ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 0x559867fabef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867fac430 con 0x559867f46960 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867fac570 con 0x559867f3d460 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6686a61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559867fac6b0 con 0x559867f3dd80 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.207+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f3d460 0x559867f4ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45968/0 (socket says 172.21.15.45:45968) 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 -- 172.21.15.45:0/2975765449 learned_addr learned my addr 172.21.15.45:0/2975765449 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6682433640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 0x559867fabef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6686a61740 10 client.0 _ll_register_callbacks cb 0x7f6678005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 -- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f3d460 msgr2=0x559867f4ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f3d460 0x559867f4ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 -- 172.21.15.45:0/2975765449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 msgr2=0x559867fabef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 0x559867fabef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 -- 172.21.15.45:0/2975765449 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559867ea36a0 con 0x559867f3dd80 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6682433640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 0x559867fabef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559867be1720 con 0x559867f3dd80 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f6681431640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098d0 tx=0x7f666c009d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f076a002362f0f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:51.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f66657fa640 10 client.26545 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.208+0000 7f66657fa640 1 -- 172.21.15.45:0/2975765449 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c015040 con 0x559867f3dd80 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f66657fa640 1 -- 172.21.15.45:0/2975765449 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f666c003e20 con 0x559867f3dd80 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f66657fa640 1 -- 172.21.15.45:0/2975765449 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f666c01b620 con 0x559867f3dd80 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 10 client.26545 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x559867e658c0 con 0x559867f3dd80 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f66657fa640 1 -- 172.21.15.45:0/2975765449 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f666c021020 con 0x559867f3dd80 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 10 client.26545 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 10 client.26545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 20 client.26545 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867fadd20 0x559867fb0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 4 client.26545 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559867c30890 con 0x559867fadd20 2024-03-12T05:16:51.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.209+0000 7f6681c32640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867fadd20 0x559867fb0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.210+0000 7f6681c32640 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867fadd20 0x559867fb0140 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d52c3ef1430f3ef server_cookie=ffb817339f1a2979 in_seq=0 out_seq=0 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.210+0000 7f66657fa640 10 client.26545 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.210+0000 7f66657fa640 1 -- 172.21.15.45:0/2975765449 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559867c30890 con 0x559867fadd20 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.210+0000 7f66657fa640 10 client.26545 handle_command_reply: tid=0 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.210+0000 7f6686a61740 1 client.26545 shutdown 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 msgr2=0x559867f4bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098d0 tx=0x7f666c009d90 comp rx=0 tx=0).stop 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 shutdown_connections 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559867fadd20 0x559867fb0140 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559867f46960 0x559867fabef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559867f3dd80 0x559867f4bfc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f66657fa640 0 client.26545 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 --2- 172.21.15.45:0/2975765449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559867f3d460 0x559867f4ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 >> 172.21.15.45:0/2975765449 conn(0x559867f37b90 msgr2=0x559867f420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 shutdown_connections 2024-03-12T05:16:51.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 1 -- 172.21.15.45:0/2975765449 wait complete. 2024-03-12T05:16:51.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.211+0000 7f6686a61740 20 client.26545 trim_cache size 0 max 16384 2024-03-12T05:16:51.210 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:16:51.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.212+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 msgr2=0x7f66680a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.212+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 0x7f66680a40e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f667c069790 tx=0x7f667c067020 comp rx=0 tx=0).stop 2024-03-12T05:16:51.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.212+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 msgr2=0x7f667412c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:51.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.212+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 0x7f667412c200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f667800e9f0 tx=0x7f667800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 shutdown_connections 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66680a1c20 0x7f66680a40e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f667409ad20 0x7f6674130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66740969c0 0x7f667412c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 --2- 172.21.15.45:0/2258439430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6674096000 0x7f667412c200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 >> 172.21.15.45:0/2258439430 conn(0x7f667400b830 msgr2=0x7f66740940f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 shutdown_connections 2024-03-12T05:16:51.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:51.213+0000 7f6682c34640 1 -- 172.21.15.45:0/2258439430 wait complete. 2024-03-12T05:16:52.227 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:16:52.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:16:52.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:16:52.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 -- 172.21.15.45:0/2209804237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc106080 msgr2=0x7f95bc106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/2209804237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc106080 0x7f95bc106460 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009860 tx=0x7f95b802bc10 comp rx=0 tx=0).stop 2024-03-12T05:16:52.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 -- 172.21.15.45:0/2209804237 shutdown_connections 2024-03-12T05:16:52.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/2209804237 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0fffc0 0x7f95bc104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/2209804237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc0ff620 0x7f95bc0ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/2209804237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc106080 0x7f95bc106460 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 -- 172.21.15.45:0/2209804237 >> 172.21.15.45:0/2209804237 conn(0x7f95bc0765a0 msgr2=0x7f95bc0fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 -- 172.21.15.45:0/2209804237 shutdown_connections 2024-03-12T05:16:52.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.395+0000 7f95c3ba7640 1 -- 172.21.15.45:0/2209804237 wait complete. 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 Processor -- start 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 -- start start 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 0x7f95bc1a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1a2bf0 con 0x7f95bc0ff620 2024-03-12T05:16:52.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1a2d60 con 0x7f95bc106080 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c3ba7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1a2f20 con 0x7f95bc0fffc0 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c1943640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c1142640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c1943640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41560/0 (socket says 172.21.15.45:41560) 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c1142640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45984/0 (socket says 172.21.15.45:45984) 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.396+0000 7f95c1943640 1 -- 172.21.15.45:0/1547989282 learned_addr learned my addr 172.21.15.45:0/1547989282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c2144640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 0x7f95bc1a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 -- 172.21.15.45:0/1547989282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 msgr2=0x7f95bc19e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 -- 172.21.15.45:0/1547989282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 msgr2=0x7f95bc1a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 0x7f95bc1a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 -- 172.21.15.45:0/1547989282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b4009900 con 0x7f95bc0ff620 2024-03-12T05:16:52.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1142640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c2144640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 0x7f95bc1a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.397+0000 7f95c1943640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f95b80057c0 tx=0x7f95b80346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.398+0000 7f95b2ffd640 1 -- 172.21.15.45:0/1547989282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b802c7b0 con 0x7f95bc0ff620 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.398+0000 7f95b2ffd640 1 -- 172.21.15.45:0/1547989282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95b80035f0 con 0x7f95bc0ff620 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.398+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b8009490 con 0x7f95bc0ff620 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.398+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc1aa8a0 con 0x7f95bc0ff620 2024-03-12T05:16:52.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.398+0000 7f95b2ffd640 1 -- 172.21.15.45:0/1547989282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b803d630 con 0x7f95bc0ff620 2024-03-12T05:16:52.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.401+0000 7f95c6777740 1 Processor -- start 2024-03-12T05:16:52.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 -- start start 2024-03-12T05:16:52.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95b2ffd640 1 -- 172.21.15.45:0/1547989282 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f95b803b040 con 0x7f95bc0ff620 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cd1380 0x555de7cd1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cda2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7c3ff90 con 0x555de7cda8b0 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7c3d4c0 con 0x555de7cd1d30 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c6777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7c38ca0 con 0x555de7cd1380 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c1142640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c1142640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41572/0 (socket says 172.21.15.45:41572) 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c1142640 1 -- 172.21.15.45:0/1180551821 learned_addr learned my addr 172.21.15.45:0/1180551821 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c1943640 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cda2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.402+0000 7f95c2144640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cd1380 0x555de7cd1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 -- 172.21.15.45:0/1180551821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cd1380 msgr2=0x555de7cd1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cd1380 0x555de7cd1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 -- 172.21.15.45:0/1180551821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 msgr2=0x555de7cda2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cda2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 -- 172.21.15.45:0/1180551821 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555de7bfb0b0 con 0x555de7cda8b0 2024-03-12T05:16:52.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95c1142640 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f95ac010450 tx=0x7f95ac010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f493920c380cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.403+0000 7f95b2ffd640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 0x7f95a80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.404+0000 7f95a7fff640 1 -- 172.21.15.45:0/1180551821 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac011760 con 0x555de7cda8b0 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.404+0000 7f95c2144640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 0x7f95a80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.404+0000 7f95a7fff640 1 -- 172.21.15.45:0/1180551821 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95ac015070 con 0x555de7cda8b0 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.404+0000 7f95a7fff640 1 -- 172.21.15.45:0/1180551821 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac01cd30 con 0x555de7cda8b0 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.404+0000 7f95b2ffd640 1 -- 172.21.15.45:0/1547989282 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95b80ef0a0 con 0x7f95bc0ff620 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c2144640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 0x7f95a80a4230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009fd0 tx=0x7f95b40095c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- 172.21.15.45:0/1180551821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 msgr2=0x555de7cdcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f95ac010450 tx=0x7f95ac010910 comp rx=0 tx=0).stop 2024-03-12T05:16:52.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- 172.21.15.45:0/1180551821 shutdown_connections 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cda8b0 0x555de7cdcd00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cda2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 --2- 172.21.15.45:0/1180551821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cd1380 0x555de7cd1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- 172.21.15.45:0/1180551821 >> 172.21.15.45:0/1180551821 conn(0x555de7ccb840 msgr2=0x555de7ccbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- 172.21.15.45:0/1180551821 shutdown_connections 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- 172.21.15.45:0/1180551821 wait complete. 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.405+0000 7f95c6777740 1 -- start start 2024-03-12T05:16:52.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 Processor -- start 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 0x555de7cd7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cd7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7bddf40 con 0x555de7cd1380 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de790c4b0 con 0x555de7cd1d30 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7bbe670 con 0x555de7cda8b0 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c6777740 10 client.0 _ll_register_callbacks cb 0x555de7c2bf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c2144640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c2144640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46006/0 (socket says 172.21.15.45:46006) 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.406+0000 7f95c2144640 1 -- 172.21.15.45:0/3261136400 learned_addr learned my addr 172.21.15.45:0/3261136400 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c1943640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 0x555de7cd7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c1142640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cd7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c2144640 1 -- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 msgr2=0x555de7cd7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c2144640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cd7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c2144640 1 -- 172.21.15.45:0/3261136400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 msgr2=0x555de7cd7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c2144640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 0x555de7cd7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c2144640 1 -- 172.21.15.45:0/3261136400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac00bcd0 con 0x555de7cda8b0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c1142640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cd7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.407+0000 7f95c1943640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 0x555de7cd7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95c2144640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95b400f2b0 tx=0x7f95b4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca592f3020f093ec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b401b420 con 0x555de7cda8b0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555de7c12f40 con 0x555de7cda8b0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95b401b580 con 0x555de7cda8b0 2024-03-12T05:16:52.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95b401b740 con 0x555de7cda8b0 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95c6777740 10 client.26557 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.408+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555de7c37320 con 0x555de7cda8b0 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95b4033020 con 0x555de7cda8b0 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f95b4026910 con 0x555de7cda8b0 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 10 client.26557 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 10 client.26557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 20 client.26557 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7ceae80 0x555de7ced2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 4 client.26557 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555de7c2f310 con 0x555de7ceae80 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.409+0000 7f95c1943640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7ceae80 0x555de7ced2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.410+0000 7f95c1943640 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7ceae80 0x555de7ced2a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd082dcc73eb328 server_cookie=5df9d5a9828ee67d in_seq=0 out_seq=0 2024-03-12T05:16:52.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.410+0000 7f95a57fa640 10 client.26557 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.410+0000 7f95a57fa640 1 -- 172.21.15.45:0/3261136400 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555de7c2f310 con 0x555de7ceae80 2024-03-12T05:16:52.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.410+0000 7f95a57fa640 10 client.26557 handle_command_reply: tid=0 2024-03-12T05:16:52.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 client.26557 shutdown 2024-03-12T05:16:52.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 msgr2=0x555de7cd81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95b400f2b0 tx=0x7f95b4004ab0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 shutdown_connections 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7ceae80 0x555de7ced2a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd81c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1d30 0x555de7cd7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95a57fa640 0 client.26557 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 --2- 172.21.15.45:0/3261136400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1380 0x555de7cd7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 >> 172.21.15.45:0/3261136400 conn(0x555de7ccb840 msgr2=0x555de7ccd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 shutdown_connections 2024-03-12T05:16:52.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 1 -- 172.21.15.45:0/3261136400 wait complete. 2024-03-12T05:16:52.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.411+0000 7f95c6777740 20 client.26557 trim_cache size 0 max 16384 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.437+0000 7f95c6777740 1 Processor -- start 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 -- start start 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7cd7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7ccca50 con 0x555de7cd1d30 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7c2f310 con 0x555de7cd1380 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c6777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7cdccf0 con 0x555de7cda8b0 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c1142640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7cd7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c1943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c1943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37644/0 (socket says 172.21.15.45:37644) 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c1943640 1 -- 172.21.15.45:0/2050231645 learned_addr learned my addr 172.21.15.45:0/2050231645 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.438+0000 7f95c1142640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7cd7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41600/0 (socket says 172.21.15.45:41600) 2024-03-12T05:16:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c2144640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 -- 172.21.15.45:0/2050231645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 msgr2=0x555de7cd6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 -- 172.21.15.45:0/2050231645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 msgr2=0x555de7cd7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7cd7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 -- 172.21.15.45:0/2050231645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555de7bd3c00 con 0x555de7cd1380 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c2144640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95c1943640 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f95b80bbdc0 tx=0x7f95b80f1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0e9dd53cfdeb8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95a77fe640 1 -- 172.21.15.45:0/2050231645 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b80bb790 con 0x555de7cd1380 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95a77fe640 1 -- 172.21.15.45:0/2050231645 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95b80bbe30 con 0x555de7cd1380 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.439+0000 7f95a77fe640 1 -- 172.21.15.45:0/2050231645 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b80f10b0 con 0x555de7cd1380 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- 172.21.15.45:0/2050231645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 msgr2=0x555de7cd9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f95b80bbdc0 tx=0x7f95b80f1040 comp rx=0 tx=0).stop 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- 172.21.15.45:0/2050231645 shutdown_connections 2024-03-12T05:16:52.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7cd7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7cd6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 --2- 172.21.15.45:0/2050231645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7cd9e70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- 172.21.15.45:0/2050231645 >> 172.21.15.45:0/2050231645 conn(0x555de7ccb840 msgr2=0x555de7cd6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- 172.21.15.45:0/2050231645 shutdown_connections 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- 172.21.15.45:0/2050231645 wait complete. 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.441+0000 7f95c6777740 1 -- start start 2024-03-12T05:16:52.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 Processor -- start 2024-03-12T05:16:52.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7ce0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7ce0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7d413b0 con 0x555de7cd1d30 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7d41520 con 0x555de7cd1380 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555de7d416e0 con 0x555de7cda8b0 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.442+0000 7f95c6777740 10 client.0 _ll_register_callbacks cb 0x7f95ac00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1142640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7ce0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37656/0 (socket says 172.21.15.45:37656) 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 -- 172.21.15.45:0/2397028720 learned_addr learned my addr 172.21.15.45:0/2397028720 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c2144640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7ce0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 -- 172.21.15.45:0/2397028720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 msgr2=0x555de7ce0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7ce0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 -- 172.21.15.45:0/2397028720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 msgr2=0x555de7ce0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7ce0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 -- 172.21.15.45:0/2397028720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc1aa8a0 con 0x555de7cd1380 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1142640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7ce0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c2144640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7ce0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95c1943640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f95b80b8800 tx=0x7f95b80bbfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93fe7e80024e7459 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.443+0000 7f95a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b80c4520 con 0x555de7cd1380 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7f95b80f2a10 con 0x555de7cd1380 2024-03-12T05:16:52.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555de7c5e900 con 0x555de7cd1380 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95b80bfed0 con 0x555de7cd1380 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 10 client.26585 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555de7bf9910 con 0x555de7cd1380 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555de7c5e900 con 0x555de7cd1380 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f95b80c4090 con 0x555de7cd1380 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 10 client.26585 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 10 client.26585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 20 client.26585 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7d42ad0 0x555de7d44ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 4 client.26585 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.444+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555de7938860 con 0x555de7d42ad0 2024-03-12T05:16:52.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.445+0000 7f95c1142640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7d42ad0 0x555de7d44ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.445+0000 7f95a57fa640 10 client.26585 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.445+0000 7f95c1142640 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7d42ad0 0x555de7d44ec0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1044f9467f7f7438 server_cookie=1516b45936da45a8 in_seq=0 out_seq=0 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.446+0000 7f95a57fa640 1 -- 172.21.15.45:0/2397028720 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555de7938860 con 0x555de7d42ad0 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.446+0000 7f95a57fa640 10 client.26585 handle_command_reply: tid=0 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.446+0000 7f95c6777740 1 client.26585 shutdown 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 msgr2=0x555de7ce0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f95b80b8800 tx=0x7f95b80bbfa0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 shutdown_connections 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555de7d42ad0 0x555de7d44ec0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555de7cd1d30 0x555de7ce0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555de7cda8b0 0x555de7ce0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 --2- 172.21.15.45:0/2397028720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555de7cd1380 0x555de7ce0400 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 >> 172.21.15.45:0/2397028720 conn(0x555de7ccb840 msgr2=0x555de7cca050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95a57fa640 0 client.26585 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 shutdown_connections 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 1 -- 172.21.15.45:0/2397028720 wait complete. 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.447+0000 7f95c6777740 20 client.26585 trim_cache size 0 max 16384 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:16:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "4900eaf8-2406-41a6-b144-daa8ff968620", 2024-03-12T05:16:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:16:52.449 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:52.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 msgr2=0x7f95a80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 0x7f95a80a4230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009fd0 tx=0x7f95b40095c0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 msgr2=0x7f95bc19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f95b80057c0 tx=0x7f95b80346f0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 shutdown_connections 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95a80a1d70 0x7f95a80a4230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95bc106080 0x7f95bc1a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95bc0fffc0 0x7f95bc19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 --2- 172.21.15.45:0/1547989282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95bc0ff620 0x7f95bc19dd30 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.448+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 >> 172.21.15.45:0/1547989282 conn(0x7f95bc0765a0 msgr2=0x7f95bc0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.449+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 shutdown_connections 2024-03-12T05:16:52.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.449+0000 7f95c3ba7640 1 -- 172.21.15.45:0/1547989282 wait complete. 2024-03-12T05:16:52.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:16:52.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:16:52.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:52 smithi045 ceph-mon[32174]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:52.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 -- 172.21.15.45:0/405411040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa270106450 msgr2=0x7fa27010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 --2- 172.21.15.45:0/405411040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa270106450 0x7fa27010e020 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa268009ca0 tx=0x7fa26802c100 comp rx=0 tx=0).stop 2024-03-12T05:16:52.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 -- 172.21.15.45:0/405411040 shutdown_connections 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 --2- 172.21.15.45:0/405411040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa270106450 0x7fa27010e020 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 --2- 172.21.15.45:0/405411040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270101d30 0x7fa270105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 --2- 172.21.15.45:0/405411040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa270101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 -- 172.21.15.45:0/405411040 >> 172.21.15.45:0/405411040 conn(0x7fa2700fcbe0 msgr2=0x7fa2700ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 -- 172.21.15.45:0/405411040 shutdown_connections 2024-03-12T05:16:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.642+0000 7fa277be2640 1 -- 172.21.15.45:0/405411040 wait complete. 2024-03-12T05:16:52.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.643+0000 7fa277be2640 1 Processor -- start 2024-03-12T05:16:52.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.643+0000 7fa277be2640 1 -- start start 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.643+0000 7fa277be2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa27010acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.643+0000 7fa277be2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 0x7fa27010b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.643+0000 7fa277be2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2701a2520 0x7fa2701a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa277be2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a2f10 con 0x7fa270101380 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa277be2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a3080 con 0x7fa270106450 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa277be2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701a31f0 con 0x7fa2701a2520 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa27010acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa27010acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41630/0 (socket says 172.21.15.45:41630) 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27597e640 1 -- 172.21.15.45:0/2553282814 learned_addr learned my addr 172.21.15.45:0/2553282814 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 0x7fa27010b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27617f640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2701a2520 0x7fa2701a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 -- 172.21.15.45:0/2553282814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2701a2520 msgr2=0x7fa2701a29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2701a2520 0x7fa2701a29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 -- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 msgr2=0x7fa27010acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa27010acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.644+0000 7fa27517d640 1 -- 172.21.15.45:0/2553282814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268009900 con 0x7fa270106450 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa27517d640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 0x7fa27010b1f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e9c0 tx=0x7fa26000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa266ffd640 1 -- 172.21.15.45:0/2553282814 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26000ccc0 con 0x7fa270106450 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa266ffd640 1 -- 172.21.15.45:0/2553282814 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa26000ce20 con 0x7fa270106450 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa266ffd640 1 -- 172.21.15.45:0/2553282814 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260018710 con 0x7fa270106450 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2701aa7d0 con 0x7fa270106450 2024-03-12T05:16:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.645+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2701aac30 con 0x7fa270106450 2024-03-12T05:16:52.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.648+0000 7fa266ffd640 1 -- 172.21.15.45:0/2553282814 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa260010430 con 0x7fa270106450 2024-03-12T05:16:52.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.649+0000 7fa27a7b5740 1 Processor -- start 2024-03-12T05:16:52.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 -- start start 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 0x55f8f18f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f1859280 con 0x55f8f18f37b0 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18567b0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f1851f90 con 0x55f8f18eabd0 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37674/0 (socket says 172.21.15.45:37674) 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 -- 172.21.15.45:0/1584625747 learned_addr learned my addr 172.21.15.45:0/1584625747 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27517d640 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 0x55f8f18f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 -- 172.21.15.45:0/1584625747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18eabd0 msgr2=0x55f8f18f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 -- 172.21.15.45:0/1584625747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 msgr2=0x55f8f18f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 0x55f8f18f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27617f640 1 -- 172.21.15.45:0/1584625747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8f1810570 con 0x55f8f18ea2b0 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.650+0000 7fa27517d640 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 0x55f8f18f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa27617f640 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa268009ca0 tx=0x7fa268032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9feab62758bd69 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa266ffd640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 0x7fa25c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa25bfff640 1 -- 172.21.15.45:0/1584625747 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268032e90 con 0x55f8f18ea2b0 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa25bfff640 1 -- 172.21.15.45:0/1584625747 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa268034bc0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa266ffd640 1 -- 172.21.15.45:0/2553282814 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa260014070 con 0x7fa270106450 2024-03-12T05:16:52.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.651+0000 7fa27597e640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 0x7fa25c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa25bfff640 1 -- 172.21.15.45:0/1584625747 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268003bd0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1584625747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 msgr2=0x55f8f18ea690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa268009ca0 tx=0x7fa268032340 comp rx=0 tx=0).stop 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27597e640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 0x7fa25c0a40e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55f8f18e9d80 tx=0x7fa26c004140 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1584625747 shutdown_connections 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18f37b0 0x55f8f18f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.652+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1584625747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18ea690 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.653+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1584625747 >> 172.21.15.45:0/1584625747 conn(0x55f8f18e4990 msgr2=0x55f8f18e4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.653+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1584625747 shutdown_connections 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.653+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1584625747 wait complete. 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.653+0000 7fa27a7b5740 1 -- start start 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.653+0000 7fa27a7b5740 1 Processor -- start 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f1525430 con 0x55f8f18ea2b0 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f14e1e00 con 0x55f8f18eabd0 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f17d6fd0 con 0x55f8f18f37b0 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27a7b5740 10 client.0 _ll_register_callbacks cb 0x55f8f182cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41644/0 (socket says 172.21.15.45:41644) 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 -- 172.21.15.45:0/783562862 learned_addr learned my addr 172.21.15.45:0/783562862 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27517d640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27617f640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 -- 172.21.15.45:0/783562862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 msgr2=0x55f8f18f0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 -- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 msgr2=0x55f8f18f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.654+0000 7fa27597e640 1 -- 172.21.15.45:0/783562862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27010e830 con 0x55f8f18ea2b0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa27597e640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa26c003df0 tx=0x7fa26c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1576bd07f70e13fb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa2597fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c003ee0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa26c017070 con 0x55f8f18ea2b0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8f15517e0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c01e620 con 0x55f8f18ea2b0 2024-03-12T05:16:52.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa27a7b5740 10 client.17118 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:52.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.655+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f8f1850610 con 0x55f8f18ea2b0 2024-03-12T05:16:52.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f8f15517e0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa26c024020 con 0x55f8f18ea2b0 2024-03-12T05:16:52.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 10 client.17118 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 20 client.17118 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f1903900 0x55f8f1905d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 4 client.17118 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8f158e720 con 0x55f8f1903900 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.656+0000 7fa27517d640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f1903900 0x55f8f1905d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.657+0000 7fa2597fa640 10 client.17118 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.657+0000 7fa27517d640 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f1903900 0x55f8f1905d10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb16d485435c8c30 server_cookie=9db0a278f8354f45 in_seq=0 out_seq=0 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.658+0000 7fa2597fa640 1 -- 172.21.15.45:0/783562862 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f8f158e720 con 0x55f8f1903900 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.658+0000 7fa2597fa640 10 client.17118 handle_command_reply: tid=0 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.658+0000 7fa27a7b5740 1 client.17118 shutdown 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 msgr2=0x55f8f18f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa26c003df0 tx=0x7fa26c00b4a0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 shutdown_connections 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f1903900 0x55f8f1905d10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/783562862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f3250 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 >> 172.21.15.45:0/783562862 conn(0x55f8f18e4990 msgr2=0x55f8f18e6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa2597fa640 0 client.17118 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 shutdown_connections 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.659+0000 7fa27a7b5740 1 -- 172.21.15.45:0/783562862 wait complete. 2024-03-12T05:16:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.660+0000 7fa27a7b5740 20 client.17118 trim_cache size 0 max 16384 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 Processor -- start 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 -- start start 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18e0320 con 0x55f8f18ea2b0 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18e59e0 con 0x55f8f18eabd0 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18e5790 con 0x55f8f18f37b0 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27517d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27597e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41660/0 (socket says 172.21.15.45:41660) 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27517d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37702/0 (socket says 172.21.15.45:37702) 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27597e640 1 -- 172.21.15.45:0/2101749061 learned_addr learned my addr 172.21.15.45:0/2101749061 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.686+0000 7fa27617f640 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 -- 172.21.15.45:0/2101749061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 msgr2=0x55f8f18fa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18fa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 -- 172.21.15.45:0/2101749061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 msgr2=0x55f8f18f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 -- 172.21.15.45:0/2101749061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8f17ecac0 con 0x55f8f18eabd0 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27597e640 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:16:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa27517d640 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cb1c0 tx=0x7fa260085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90325bd373bfe4f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.687+0000 7fa25b7fe640 1 -- 172.21.15.45:0/2101749061 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260098a50 con 0x55f8f18eabd0 2024-03-12T05:16:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.688+0000 7fa25b7fe640 1 -- 172.21.15.45:0/2101749061 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa260010a70 con 0x55f8f18eabd0 2024-03-12T05:16:52.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.688+0000 7fa25b7fe640 1 -- 172.21.15.45:0/2101749061 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2600cb440 con 0x55f8f18eabd0 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- 172.21.15.45:0/2101749061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 msgr2=0x55f8f18f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cb1c0 tx=0x7fa260085040 comp rx=0 tx=0).stop 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- 172.21.15.45:0/2101749061 shutdown_connections 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18eabd0 0x55f8f18f3250 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/2101749061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18ea2b0 0x55f8f18f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- 172.21.15.45:0/2101749061 >> 172.21.15.45:0/2101749061 conn(0x55f8f18e4990 msgr2=0x55f8f190a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- 172.21.15.45:0/2101749061 shutdown_connections 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- 172.21.15.45:0/2101749061 wait complete. 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.689+0000 7fa27a7b5740 1 -- start start 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 Processor -- start 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18eabd0 0x55f8f18f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18f9bd0 con 0x55f8f18eabd0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f18f9d40 con 0x55f8f18ea2b0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f1959e40 con 0x55f8f18f37b0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27597e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27a7b5740 10 client.0 _ll_register_callbacks cb 0x7fa260005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27597e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37718/0 (socket says 172.21.15.45:37718) 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27597e640 1 -- 172.21.15.45:0/1833865838 learned_addr learned my addr 172.21.15.45:0/1833865838 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.690+0000 7fa27517d640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18eabd0 0x55f8f18f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27617f640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 -- 172.21.15.45:0/1833865838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 msgr2=0x55f8f18f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 -- 172.21.15.45:0/1833865838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18eabd0 msgr2=0x55f8f18f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18eabd0 0x55f8f18f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 -- 172.21.15.45:0/1833865838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8f1850610 con 0x55f8f18ea2b0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27597e640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa26c02f710 tx=0x7fa26c02f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6338760e92f834e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa2597fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c003ee0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8520+0+0 (secure 0 0 0) 0x7fa26c017040 con 0x55f8f18ea2b0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.691+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa268034bc0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c02f8e0 con 0x55f8f18ea2b0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 10 client.26603 fetch_fsmap learned FSMap version 33 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fa2701aac30 con 0x55f8f18ea2b0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa26c045020 con 0x55f8f18ea2b0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa26c024040 con 0x55f8f18ea2b0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 10 client.26603 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 10 client.26603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 20 client.26603 populate_metadata read hostname 'smithi045' 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f195b560 0x55f8f195d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 4 client.26603 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fa268034bc0 con 0x55f8f195b560 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.692+0000 7fa27517d640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f195b560 0x55f8f195d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.693+0000 7fa2597fa640 10 client.26603 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.693+0000 7fa27517d640 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f195b560 0x55f8f195d980 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8731527d51ab204 server_cookie=e60ba2d873b59a30 in_seq=0 out_seq=0 2024-03-12T05:16:52.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.693+0000 7fa2597fa640 1 -- 172.21.15.45:0/1833865838 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fa268034bc0 con 0x55f8f195b560 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.693+0000 7fa2597fa640 10 client.26603 handle_command_reply: tid=0 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.693+0000 7fa27a7b5740 1 client.26603 shutdown 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 msgr2=0x55f8f18faae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa26c02f710 tx=0x7fa26c02f740 comp rx=0 tx=0).stop 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 shutdown_connections 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f8f195b560 0x55f8f195d980 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8f18eabd0 0x55f8f18f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa2597fa640 0 client.26603 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8f18f37b0 0x55f8f18f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 --2- 172.21.15.45:0/1833865838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8f18ea2b0 0x55f8f18faae0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 >> 172.21.15.45:0/1833865838 conn(0x55f8f18e4990 msgr2=0x55f8f18eb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 shutdown_connections 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 1 -- 172.21.15.45:0/1833865838 wait complete. 2024-03-12T05:16:52.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.694+0000 7fa27a7b5740 20 client.26603 trim_cache size 0 max 16384 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "4900eaf8-2406-41a6-b144-daa8ff968620": { 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "4900eaf8-2406-41a6-b144-daa8ff968620", 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:16:52.694 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:16:52.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.695+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 msgr2=0x7fa25c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.695+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 0x7fa25c0a40e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55f8f18e9d80 tx=0x7fa26c004140 comp rx=0 tx=0).stop 2024-03-12T05:16:52.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.695+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 msgr2=0x7fa27010b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.695+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 0x7fa27010b1f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e9c0 tx=0x7fa26000ee80 comp rx=0 tx=0).stop 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 shutdown_connections 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa25c0a1c20 0x7fa25c0a40e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2701a2520 0x7fa2701a29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa270106450 0x7fa27010b1f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 --2- 172.21.15.45:0/2553282814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa270101380 0x7fa27010acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 >> 172.21.15.45:0/2553282814 conn(0x7fa2700fcbe0 msgr2=0x7fa2700fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 shutdown_connections 2024-03-12T05:16:52.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:16:52.696+0000 7fa277be2640 1 -- 172.21.15.45:0/2553282814 wait complete. 2024-03-12T05:16:52.710 INFO:tasks.cephfs.filesystem:scrub status for tag:4900eaf8-2406-41a6-b144-daa8ff968620 - {'path': '/', 'tag': '4900eaf8-2406-41a6-b144-daa8ff968620', 'options': 'recursive,force'} 2024-03-12T05:16:52.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:52 smithi130 ceph-mon[35398]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:52.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:52 smithi116 ceph-mon[35486]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:53.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:16:53.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:53.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:16:53.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:53.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:53.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:16:53.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:54.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:54 smithi130 ceph-mon[35398]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:54.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:54 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:54.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:54 smithi116 ceph-mon[35486]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:54.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:54 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:54.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:54 smithi045 ceph-mon[32174]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 23 MiB/s wr, 1.24k op/s 2024-03-12T05:16:54.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:54 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:16:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 290.0 s, 1105.6 tps, lat 93.404 ms stddev 128.469 2024-03-12T05:16:56.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:56 smithi130 ceph-mon[35398]: pgmap v184: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 34 MiB/s wr, 1.82k op/s 2024-03-12T05:16:56.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:56 smithi116 ceph-mon[35486]: pgmap v184: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 34 MiB/s wr, 1.82k op/s 2024-03-12T05:16:56.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:56 smithi045 ceph-mon[32174]: pgmap v184: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 34 MiB/s wr, 1.82k op/s 2024-03-12T05:16:58.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:16:58 smithi130 ceph-mon[35398]: pgmap v185: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 22 MiB/s wr, 1.23k op/s 2024-03-12T05:16:58.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:16:58 smithi116 ceph-mon[35486]: pgmap v185: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 22 MiB/s wr, 1.23k op/s 2024-03-12T05:16:58.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:16:58 smithi045 ceph-mon[32174]: pgmap v185: 97 pgs: 97 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 22 MiB/s wr, 1.23k op/s 2024-03-12T05:16:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 295.0 s, 1211.6 tps, lat 82.237 ms stddev 108.697 2024-03-12T05:17:00.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:00 smithi130 ceph-mon[35398]: pgmap v186: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 33 MiB/s wr, 1.90k op/s 2024-03-12T05:17:00.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:00 smithi116 ceph-mon[35486]: pgmap v186: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 33 MiB/s wr, 1.90k op/s 2024-03-12T05:17:00.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:00 smithi045 ceph-mon[32174]: pgmap v186: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 33 MiB/s wr, 1.90k op/s 2024-03-12T05:17:02.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:02 smithi130 ceph-mon[35398]: pgmap v187: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 21 MiB/s wr, 1.25k op/s 2024-03-12T05:17:02.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:02 smithi116 ceph-mon[35486]: pgmap v187: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 21 MiB/s wr, 1.25k op/s 2024-03-12T05:17:02.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:02 smithi045 ceph-mon[32174]: pgmap v187: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 21 MiB/s wr, 1.25k op/s 2024-03-12T05:17:03.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:03.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:03.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:03.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:03.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:03.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:03.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:03.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:03.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:04.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:04 smithi130 ceph-mon[35398]: pgmap v188: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 22 MiB/s wr, 1.25k op/s 2024-03-12T05:17:04.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:04 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:04.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:04 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:17:04.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:04 smithi116 ceph-mon[35486]: pgmap v188: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 22 MiB/s wr, 1.25k op/s 2024-03-12T05:17:04.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:04 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:04.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:04 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:17:04.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:04 smithi045 ceph-mon[32174]: pgmap v188: 97 pgs: 97 active+clean; 3.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 22 MiB/s wr, 1.25k op/s 2024-03-12T05:17:04.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:04 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:04.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:04 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:17:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 300.0 s, 1053.4 tps, lat 94.960 ms stddev 132.786 2024-03-12T05:17:05.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:05 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:17:05.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:05 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:17:05.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:05 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:17:05.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:05 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:17:05.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:05 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:17:05.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:05 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:17:05.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:05 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:17:05.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:05 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:17:05.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:05 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:17:06.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:06 smithi130 ceph-mon[35398]: pgmap v189: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-03-12T05:17:06.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:06 smithi116 ceph-mon[35486]: pgmap v189: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-03-12T05:17:06.767 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:06 smithi045 ceph-mon[32174]: pgmap v189: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-03-12T05:17:08.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:08 smithi130 ceph-mon[35398]: pgmap v190: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-03-12T05:17:08.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:08 smithi116 ceph-mon[35486]: pgmap v190: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-03-12T05:17:08.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:08 smithi045 ceph-mon[32174]: pgmap v190: 97 pgs: 97 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 20 MiB/s wr, 1.31k op/s 2024-03-12T05:17:09.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:09 smithi116 ceph-mon[35486]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:17:09.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:09 smithi045 ceph-mon[32174]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:17:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 305.0 s, 1132.6 tps, lat 88.194 ms stddev 294.834 2024-03-12T05:17:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:09 smithi130 ceph-mon[35398]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:17:10.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:10 smithi116 ceph-mon[35486]: pgmap v192: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:10.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:10 smithi045 ceph-mon[32174]: pgmap v192: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:10.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:10 smithi130 ceph-mon[35398]: pgmap v192: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:12.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:12 smithi116 ceph-mon[35486]: pgmap v193: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:12.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:12 smithi045 ceph-mon[32174]: pgmap v193: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:12.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:12 smithi130 ceph-mon[35398]: pgmap v193: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:13.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:13 smithi116 ceph-mon[35486]: pgmap v194: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:13.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:13 smithi045 ceph-mon[32174]: pgmap v194: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:13.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:13 smithi130 ceph-mon[35398]: pgmap v194: 97 pgs: 21 active+clean+snaptrim, 39 active+clean, 37 active+clean+snaptrim_wait; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 22 MiB/s wr, 1.57k op/s 2024-03-12T05:17:14.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 310.0 s, 1386.1 tps, lat 66.753 ms stddev 21.186 2024-03-12T05:17:15.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:15 smithi045 ceph-mon[32174]: pgmap v195: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:15.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:15 smithi130 ceph-mon[35398]: pgmap v195: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:15.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:15 smithi116 ceph-mon[35486]: pgmap v195: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:17.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:17 smithi045 ceph-mon[32174]: pgmap v196: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:17.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:17 smithi130 ceph-mon[35398]: pgmap v196: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:17.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:17 smithi116 ceph-mon[35486]: pgmap v196: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-03-12T05:17:19.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 315.0 s, 1194.1 tps, lat 90.012 ms stddev 133.181 2024-03-12T05:17:20.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:20 smithi130 ceph-mon[35398]: pgmap v197: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 29 MiB/s wr, 2.42k op/s 2024-03-12T05:17:20.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:20 smithi130 ceph-mon[35398]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T05:17:20.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:20 smithi116 ceph-mon[35486]: pgmap v197: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 29 MiB/s wr, 2.42k op/s 2024-03-12T05:17:20.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:20 smithi116 ceph-mon[35486]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T05:17:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:20 smithi045 ceph-mon[32174]: pgmap v197: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 29 MiB/s wr, 2.42k op/s 2024-03-12T05:17:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:20 smithi045 ceph-mon[32174]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T05:17:22.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:17:22.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:17:22.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:22 smithi130 ceph-mon[35398]: pgmap v199: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:22.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:22 smithi116 ceph-mon[35486]: pgmap v199: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:22.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:22 smithi045 ceph-mon[32174]: pgmap v199: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:22.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 -- 172.21.15.45:0/3751988241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2641002d0 msgr2=0x7fd264100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 --2- 172.21.15.45:0/3751988241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2641002d0 0x7fd264100730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd254009930 tx=0x7fd25402bc80 comp rx=0 tx=0).stop 2024-03-12T05:17:22.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 -- 172.21.15.45:0/3751988241 shutdown_connections 2024-03-12T05:17:22.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 --2- 172.21.15.45:0/3751988241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd264100c70 0x7fd26410ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 --2- 172.21.15.45:0/3751988241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2641002d0 0x7fd264100730 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 --2- 172.21.15.45:0/3751988241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264105010 0x7fd2641053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 -- 172.21.15.45:0/3751988241 >> 172.21.15.45:0/3751988241 conn(0x7fd264076ee0 msgr2=0x7fd2640772f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 -- 172.21.15.45:0/3751988241 shutdown_connections 2024-03-12T05:17:22.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.891+0000 7fd268a0e640 1 -- 172.21.15.45:0/3751988241 wait complete. 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.892+0000 7fd268a0e640 1 Processor -- start 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.892+0000 7fd268a0e640 1 -- start start 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264100c70 0x7fd26419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd264105010 0x7fd264199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264199ef0 con 0x7fd2641002d0 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26419a060 con 0x7fd264100c70 2024-03-12T05:17:22.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd268a0e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26419a220 con 0x7fd264105010 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50174/0 (socket says 172.21.15.45:50174) 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 -- 172.21.15.45:0/2643860599 learned_addr learned my addr 172.21.15.45:0/2643860599 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd261d9b640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264100c70 0x7fd26419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd262d9d640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd264105010 0x7fd264199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 -- 172.21.15.45:0/2643860599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd264105010 msgr2=0x7fd264199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd264105010 0x7fd264199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 -- 172.21.15.45:0/2643860599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264100c70 msgr2=0x7fd26419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264100c70 0x7fd26419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.893+0000 7fd26259c640 1 -- 172.21.15.45:0/2643860599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009840 con 0x7fd2641002d0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd26259c640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00d950 tx=0x7fd24c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd25b7fe640 1 -- 172.21.15.45:0/2643860599 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c009c30 con 0x7fd2641002d0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd25b7fe640 1 -- 172.21.15.45:0/2643860599 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd24c00de80 con 0x7fd2641002d0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd25b7fe640 1 -- 172.21.15.45:0/2643860599 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c00f6b0 con 0x7fd2641002d0 2024-03-12T05:17:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd254009560 con 0x7fd2641002d0 2024-03-12T05:17:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.894+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2641a64e0 con 0x7fd2641002d0 2024-03-12T05:17:22.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.897+0000 7fd25b7fe640 1 -- 172.21.15.45:0/2643860599 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd24c010460 con 0x7fd2641002d0 2024-03-12T05:17:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.898+0000 7fd26b5e6740 1 Processor -- start 2024-03-12T05:17:22.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.898+0000 7fd25b7fe640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 0x7fd2500a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.898+0000 7fd262d9d640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 0x7fd2500a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.898+0000 7fd25b7fe640 1 -- 172.21.15.45:0/2643860599 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd24c0cd2f0 con 0x7fd2641002d0 2024-03-12T05:17:22.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 -- start start 2024-03-12T05:17:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd262d9d640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 0x7fd2500a4230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd25c005fd0 tx=0x7fd25c009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c079690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c084ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bfe8280 con 0x56160c0827b0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bfe57b0 con 0x56160c079bd0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bfe0f90 con 0x56160c0792b0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd261d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd262d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c084ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd261d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44426/0 (socket says 172.21.15.45:44426) 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd262d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c084ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50178/0 (socket says 172.21.15.45:50178) 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd261d9b640 1 -- 172.21.15.45:0/2638167022 learned_addr learned my addr 172.21.15.45:0/2638167022 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.899+0000 7fd26259c640 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c079690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 -- 172.21.15.45:0/2638167022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 msgr2=0x56160c079690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c079690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 -- 172.21.15.45:0/2638167022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 msgr2=0x56160c084ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c084ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 -- 172.21.15.45:0/2638167022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56160bf9f570 con 0x56160c079bd0 2024-03-12T05:17:22.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd261d9b640 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd254005ab0 tx=0x7fd2540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43d43e5637415308 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd258ff9640 1 -- 172.21.15.45:0/2638167022 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254004970 con 0x56160c079bd0 2024-03-12T05:17:22.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd258ff9640 1 -- 172.21.15.45:0/2638167022 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd25402cc10 con 0x56160c079bd0 2024-03-12T05:17:22.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.900+0000 7fd258ff9640 1 -- 172.21.15.45:0/2638167022 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254003970 con 0x56160c079bd0 2024-03-12T05:17:22.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 -- 172.21.15.45:0/2638167022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 msgr2=0x56160c082270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd254005ab0 tx=0x7fd2540046c0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 -- 172.21.15.45:0/2638167022 shutdown_connections 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c084ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c082270 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/2638167022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c079690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 -- 172.21.15.45:0/2638167022 >> 172.21.15.45:0/2638167022 conn(0x56160c073990 msgr2=0x56160c073d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.902+0000 7fd26b5e6740 1 -- 172.21.15.45:0/2638167022 shutdown_connections 2024-03-12T05:17:22.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.903+0000 7fd26b5e6740 1 -- 172.21.15.45:0/2638167022 wait complete. 2024-03-12T05:17:22.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.903+0000 7fd26b5e6740 1 -- start start 2024-03-12T05:17:22.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.903+0000 7fd26b5e6740 1 Processor -- start 2024-03-12T05:17:22.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c07f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c07fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bcb4430 con 0x56160c0827b0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bc70e00 con 0x56160c079bd0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160bf65fd0 con 0x56160c0792b0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38090/0 (socket says 172.21.15.45:38090) 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 -- 172.21.15.45:0/180703881 learned_addr learned my addr 172.21.15.45:0/180703881 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd262d9d640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c07fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26b5e6740 10 client.0 _ll_register_callbacks cb 0x56160bfbbfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd261d9b640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c07f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 -- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 msgr2=0x56160c07f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c07f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 -- 172.21.15.45:0/180703881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 msgr2=0x56160c07fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.904+0000 7fd26259c640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c07fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd26259c640 1 -- 172.21.15.45:0/180703881 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26410d830 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd261d9b640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c07f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd26259c640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd24c010fd0 tx=0x7fd24c09dfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee4b85b207428205 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c098250 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56160bce07e0 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd24c096500 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.905+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd24c0966c0 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 10 client.26590 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56160bfdf610 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd24c0a6080 con 0x56160c0792b0 2024-03-12T05:17:22.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd24c095be0 con 0x56160c0792b0 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 10 client.26590 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 10 client.26590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 20 client.26590 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.906+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c09add0 0x56160c09d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.907+0000 7fd26b5e6740 4 client.26590 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.907+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56160bd1d720 con 0x56160c09add0 2024-03-12T05:17:22.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.907+0000 7fd261d9b640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c09add0 0x56160c09d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.907+0000 7fd261d9b640 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c09add0 0x56160c09d1c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cf47c60fa52fcfc server_cookie=6bcf0b9b1ccb62a2 in_seq=0 out_seq=0 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.907+0000 7fd249ffb640 10 client.26590 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.908+0000 7fd249ffb640 1 -- 172.21.15.45:0/180703881 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56160bd1d720 con 0x56160c09add0 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.908+0000 7fd249ffb640 10 client.26590 handle_command_reply: tid=0 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.908+0000 7fd26b5e6740 1 client.26590 shutdown 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 msgr2=0x56160c082260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd24c010fd0 tx=0x7fd24c09dfe0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 shutdown_connections 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c09add0 0x56160c09d1c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd249ffb640 0 client.26590 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c0827b0 0x56160c07fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c079bd0 0x56160c07f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/180703881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c082260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 >> 172.21.15.45:0/180703881 conn(0x56160c073990 msgr2=0x56160c075730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.909+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 shutdown_connections 2024-03-12T05:17:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.910+0000 7fd26b5e6740 1 -- 172.21.15.45:0/180703881 wait complete. 2024-03-12T05:17:22.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.910+0000 7fd26b5e6740 20 client.26590 trim_cache size 0 max 16384 2024-03-12T05:17:22.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 Processor -- start 2024-03-12T05:17:22.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 -- start start 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c082260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0827b0 0x56160c07a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c078f10 con 0x56160c079bd0 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c07f3e0 con 0x56160c0792b0 2024-03-12T05:17:22.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c084f20 con 0x56160c0827b0 2024-03-12T05:17:22.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd26259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44436/0 (socket says 172.21.15.45:44436) 2024-03-12T05:17:22.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.937+0000 7fd261d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c082260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 -- 172.21.15.45:0/4160234476 learned_addr learned my addr 172.21.15.45:0/4160234476 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:22.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd262d9d640 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0827b0 0x56160c07a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 -- 172.21.15.45:0/4160234476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0827b0 msgr2=0x56160c07a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0827b0 0x56160c07a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 -- 172.21.15.45:0/4160234476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 msgr2=0x56160c082260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c082260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 -- 172.21.15.45:0/4160234476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56160bf7bac0 con 0x56160c0792b0 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd26259c640 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e010 tx=0x7fd24c099b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63590270c3fbf42b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.938+0000 7fd24bfff640 1 -- 172.21.15.45:0/4160234476 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0a9070 con 0x56160c0792b0 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.939+0000 7fd24bfff640 1 -- 172.21.15.45:0/4160234476 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56160bce07e0 con 0x56160c0792b0 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.939+0000 7fd24bfff640 1 -- 172.21.15.45:0/4160234476 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0b8170 con 0x56160c0792b0 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.939+0000 7fd26b5e6740 1 -- 172.21.15.45:0/4160234476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 msgr2=0x56160c081d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.939+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e010 tx=0x7fd24c099b00 comp rx=0 tx=0).stop 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 -- 172.21.15.45:0/4160234476 shutdown_connections 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c082260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0827b0 0x56160c07a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/4160234476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0792b0 0x56160c081d20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 -- 172.21.15.45:0/4160234476 >> 172.21.15.45:0/4160234476 conn(0x56160c073990 msgr2=0x56160c0a1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 -- 172.21.15.45:0/4160234476 shutdown_connections 2024-03-12T05:17:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 -- 172.21.15.45:0/4160234476 wait complete. 2024-03-12T05:17:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 -- start start 2024-03-12T05:17:22.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.940+0000 7fd26b5e6740 1 Processor -- start 2024-03-12T05:17:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 0x56160c08eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c08f180 con 0x56160c079bd0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd261d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd261d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50224/0 (socket says 172.21.15.45:50224) 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd261d9b640 1 -- 172.21.15.45:0/715555909 learned_addr learned my addr 172.21.15.45:0/715555909 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c08f2f0 con 0x56160c0827b0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160c0f12e0 con 0x56160c0792b0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38102/0 (socket says 172.21.15.45:38102) 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd262d9d640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 0x56160c08eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.941+0000 7fd26b5e6740 10 client.0 _ll_register_callbacks cb 0x7fd254009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 -- 172.21.15.45:0/715555909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 msgr2=0x56160c0921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 -- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 msgr2=0x56160c08eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 0x56160c08eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 -- 172.21.15.45:0/715555909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56160bfdf610 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd262d9d640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 0x56160c08eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd26259c640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.942+0000 7fd261d9b640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd25402c290 tx=0x7fd25402cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f098effdd057d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25404f070 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd2540048b0 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd25402cc10 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd254050d10 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd26b5e6740 10 client.17142 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.943+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56160bfa16a0 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.944+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd25402cc10 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.944+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd25403c050 con 0x56160c079bd0 2024-03-12T05:17:22.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 10 client.17142 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 20 client.17142 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c0f2650 0x56160c0f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 4 client.17142 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56160bd6c890 con 0x56160c0f2650 2024-03-12T05:17:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26259c640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c0f2650 0x56160c0f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd249ffb640 10 client.17142 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd26259c640 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c0f2650 0x56160c0f4a70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce4532e1093f7f4 server_cookie=2d5b26cec1359503 in_seq=0 out_seq=0 2024-03-12T05:17:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd249ffb640 1 -- 172.21.15.45:0/715555909 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56160bd6c890 con 0x56160c0f2650 2024-03-12T05:17:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.945+0000 7fd249ffb640 10 client.17142 handle_command_reply: tid=0 2024-03-12T05:17:22.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 client.17142 shutdown 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 msgr2=0x56160c08e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd25402c290 tx=0x7fd25402cb90 comp rx=0 tx=0).stop 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 shutdown_connections 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56160c0f2650 0x56160c0f4a70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56160c079bd0 0x56160c08e560 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56160c0827b0 0x56160c08eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 --2- 172.21.15.45:0/715555909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56160c0792b0 0x56160c0921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 >> 172.21.15.45:0/715555909 conn(0x56160c073990 msgr2=0x56160c07a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd249ffb640 0 client.17142 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 shutdown_connections 2024-03-12T05:17:22.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.946+0000 7fd26b5e6740 1 -- 172.21.15.45:0/715555909 wait complete. 2024-03-12T05:17:22.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.947+0000 7fd26b5e6740 20 client.17142 trim_cache size 0 max 16384 2024-03-12T05:17:22.946 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:17:22.946 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:17:22.946 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:17:22.946 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:17:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.947+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 msgr2=0x7fd2500a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.947+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 0x7fd2500a4230 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd25c005fd0 tx=0x7fd25c009500 comp rx=0 tx=0).stop 2024-03-12T05:17:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.947+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 msgr2=0x7fd26419f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.947+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00d950 tx=0x7fd24c00dd50 comp rx=0 tx=0).stop 2024-03-12T05:17:22.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 shutdown_connections 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd2500a1d70 0x7fd2500a4230 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd264105010 0x7fd264199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd264100c70 0x7fd26419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 --2- 172.21.15.45:0/2643860599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2641002d0 0x7fd26419f690 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 >> 172.21.15.45:0/2643860599 conn(0x7fd264076ee0 msgr2=0x7fd26410b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 shutdown_connections 2024-03-12T05:17:22.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:22.948+0000 7fd268a0e640 1 -- 172.21.15.45:0/2643860599 wait complete. 2024-03-12T05:17:22.961 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:17:22.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:17:23.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3609333990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc107730 msgr2=0x7ff6cc107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3609333990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc107730 0x7ff6cc107bb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009930 tx=0x7ff6c002bc20 comp rx=0 tx=0).stop 2024-03-12T05:17:23.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3609333990 shutdown_connections 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3609333990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc107730 0x7ff6cc107bb0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3609333990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1080f0 0x7ff6cc114600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3609333990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc10d740 0x7ff6cc10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3609333990 >> 172.21.15.45:0/3609333990 conn(0x7ff6cc0756d0 msgr2=0x7ff6cc077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3609333990 shutdown_connections 2024-03-12T05:17:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.176+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3609333990 wait complete. 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.177+0000 7ff6d0cc3640 1 Processor -- start 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 -- start start 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc1080f0 0x7ff6cc19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 0x7ff6cc19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc1a31a0 con 0x7ff6cc10d740 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc1a3310 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6d0cc3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc1a3480 con 0x7ff6cc1080f0 2024-03-12T05:17:23.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6cad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6c9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 0x7ff6cc19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6cad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44466/0 (socket says 172.21.15.45:44466) 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6cad9d640 1 -- 172.21.15.45:0/3776636579 learned_addr learned my addr 172.21.15.45:0/3776636579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.178+0000 7ff6ca59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc1080f0 0x7ff6cc19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 -- 172.21.15.45:0/3776636579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc1080f0 msgr2=0x7ff6cc19def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc1080f0 0x7ff6cc19def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 -- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 msgr2=0x7ff6cc19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 0x7ff6cc19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 -- 172.21.15.45:0/3776636579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c0009560 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6c9d9b640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 0x7ff6cc19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6cad9d640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400eb10 tx=0x7ff6c400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.179+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4004b20 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.180+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff6c4004c80 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.180+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4010690 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.180+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc1aaa60 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.180+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc1aae80 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.181+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc0683c0 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.182+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff6c4005020 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.183+0000 7ff6bb7fe640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 0x7ff6b40a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.183+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff6c4014070 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.183+0000 7ff6ca59c640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 0x7ff6b40a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.184+0000 7ff6ca59c640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 0x7ff6b40a4120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009b50 tx=0x7ff6bc005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.188+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff6c4096c50 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.613+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6cc10fcf0 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.614+0000 7ff6bb7fe640 1 -- 172.21.15.45:0/3776636579 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7ff6c409a310 con 0x7ff6cc1a27b0 2024-03-12T05:17:23.610 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:17:23.611 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:17:23.611 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 msgr2=0x7ff6b40a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 0x7ff6b40a4120 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009b50 tx=0x7ff6bc005f90 comp rx=0 tx=0).stop 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 msgr2=0x7ff6cc1a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400eb10 tx=0x7ff6c400c2d0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 shutdown_connections 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff6b40a1c60 0x7ff6b40a4120 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6cc10d740 0x7ff6cc19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6cc1a27b0 0x7ff6cc1a2c60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 --2- 172.21.15.45:0/3776636579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc1080f0 0x7ff6cc19def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 >> 172.21.15.45:0/3776636579 conn(0x7ff6cc0756d0 msgr2=0x7ff6cc076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 shutdown_connections 2024-03-12T05:17:23.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.617+0000 7ff6d0cc3640 1 -- 172.21.15.45:0/3776636579 wait complete. 2024-03-12T05:17:23.630 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:23.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 -- 172.21.15.45:0/1679185506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 msgr2=0x7f5e2c0ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 --2- 172.21.15.45:0/1679185506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c0ffcd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009930 tx=0x7f5e1c02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 -- 172.21.15.45:0/1679185506 shutdown_connections 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 --2- 172.21.15.45:0/1679185506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c111d80 0x7f5e2c114170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 --2- 172.21.15.45:0/1679185506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c100210 0x7f5e2c111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 --2- 172.21.15.45:0/1679185506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c0ffcd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 -- 172.21.15.45:0/1679185506 >> 172.21.15.45:0/1679185506 conn(0x7f5e2c0751c0 msgr2=0x7f5e2c0755d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 -- 172.21.15.45:0/1679185506 shutdown_connections 2024-03-12T05:17:23.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.805+0000 7f5e3332a640 1 -- 172.21.15.45:0/1679185506 wait complete. 2024-03-12T05:17:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 Processor -- start 2024-03-12T05:17:23.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 -- start start 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 0x7f5e2c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 0x7f5e2c06b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c06c160 con 0x7f5e2c100210 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c06c2d0 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.806+0000 7f5e3332a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c06c490 con 0x7f5e2c111d80 2024-03-12T05:17:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.807+0000 7f5e32b29640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 0x7f5e2c06b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.807+0000 7f5e32b29640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 0x7f5e2c06b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38136/0 (socket says 172.21.15.45:38136) 2024-03-12T05:17:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.807+0000 7f5e32b29640 1 -- 172.21.15.45:0/2045005883 learned_addr learned my addr 172.21.15.45:0/2045005883 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.807+0000 7f5e31b27640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 0x7f5e2c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.807+0000 7f5e32328640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e32328640 1 -- 172.21.15.45:0/2045005883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 msgr2=0x7f5e2c06b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e32328640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 0x7f5e2c06b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e32328640 1 -- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 msgr2=0x7f5e2c072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e32328640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 0x7f5e2c072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e32328640 1 -- 172.21.15.45:0/2045005883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c009560 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.808+0000 7f5e31b27640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 0x7f5e2c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e32328640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c072830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009230 tx=0x7f5e1c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e237fe640 1 -- 172.21.15.45:0/2045005883 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c004350 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e237fe640 1 -- 172.21.15.45:0/2045005883 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e1c0044b0 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e237fe640 1 -- 172.21.15.45:0/2045005883 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0053c0 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c06c710 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.809+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c070650 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.813+0000 7f5e237fe640 1 -- 172.21.15.45:0/2045005883 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5e1c005520 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.813+0000 7f5e36dfd740 1 Processor -- start 2024-03-12T05:17:23.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.813+0000 7f5e36dfd740 1 -- start start 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8fef820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea8ff8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8faa960 con 0x55fea8fefd60 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8f5e310 con 0x55fea8ff8940 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8f5b840 con 0x55fea8fef440 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44492/0 (socket says 172.21.15.45:44492) 2024-03-12T05:17:23.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 -- 172.21.15.45:0/2011713985 learned_addr learned my addr 172.21.15.45:0/2011713985 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e32b29640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8fef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e32328640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea8ff8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 -- 172.21.15.45:0/2011713985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 msgr2=0x55fea8fef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8fef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 -- 172.21.15.45:0/2011713985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 msgr2=0x55fea8ff8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea8ff8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e31b27640 1 -- 172.21.15.45:0/2011713985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea8f19060 con 0x55fea8ff8940 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.814+0000 7f5e32328640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea8ff8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.815+0000 7f5e31b27640 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e280114f0 tx=0x7f5e280119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84ffa302c251ba30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.815+0000 7f5e237fe640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 0x7f5e180a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.815+0000 7f5e237fe640 1 -- 172.21.15.45:0/2045005883 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5e1c0efd20 con 0x7f5e2c0ff8f0 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.815+0000 7f5e32b29640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 0x7f5e180a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.815+0000 7f5e20bfe640 1 -- 172.21.15.45:0/2011713985 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28005020 con 0x55fea8ff8940 2024-03-12T05:17:23.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.816+0000 7f5e20bfe640 1 -- 172.21.15.45:0/2011713985 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e28005630 con 0x55fea8ff8940 2024-03-12T05:17:23.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.816+0000 7f5e20bfe640 1 -- 172.21.15.45:0/2011713985 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800f880 con 0x55fea8ff8940 2024-03-12T05:17:23.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- 172.21.15.45:0/2011713985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 msgr2=0x55fea8ffad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e32b29640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 0x7f5e180a40e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009fd0 tx=0x7f5e24009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e280114f0 tx=0x7f5e280119b0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- 172.21.15.45:0/2011713985 shutdown_connections 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea8ff8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea8ffad30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/2011713985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8fef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- 172.21.15.45:0/2011713985 >> 172.21.15.45:0/2011713985 conn(0x55fea8fe9b70 msgr2=0x55fea8fe9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- 172.21.15.45:0/2011713985 shutdown_connections 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- 172.21.15.45:0/2011713985 wait complete. 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.817+0000 7f5e36dfd740 1 -- start start 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 Processor -- start 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea8ff5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 0x55fea8ff64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8f57020 con 0x55fea8ff8940 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8c2a430 con 0x55fea8fefd60 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8be6e00 con 0x55fea8fef440 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e32328640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e32328640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38168/0 (socket says 172.21.15.45:38168) 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e32328640 1 -- 172.21.15.45:0/1145388919 learned_addr learned my addr 172.21.15.45:0/1145388919 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e36dfd740 10 client.0 _ll_register_callbacks cb 0x55fea8efecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e31b27640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea8ff5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.818+0000 7f5e32b29640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 0x55fea8ff64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 -- 172.21.15.45:0/1145388919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 msgr2=0x55fea8ff5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea8ff5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 -- 172.21.15.45:0/1145388919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 msgr2=0x55fea8ff64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 0x55fea8ff64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 -- 172.21.15.45:0/1145388919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e28009c80 con 0x55fea8fef440 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32b29640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 0x55fea8ff64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e32328640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c006010 tx=0x7f5e1c0b8cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9fd3f66baa98947 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.819+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0f3340 con 0x55fea8fef440 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.820+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fea8c567e0 con 0x55fea8fef440 2024-03-12T05:17:23.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.820+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5e1c0f1070 con 0x55fea8fef440 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.820+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e1c0bc390 con 0x55fea8fef440 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.820+0000 7f5e36dfd740 10 client.26614 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fea8f556a0 con 0x55fea8fef440 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e1c0d4070 con 0x55fea8fef440 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5e1c03c070 con 0x55fea8fef440 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e36dfd740 10 client.26614 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e36dfd740 10 client.26614 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.821+0000 7f5e36dfd740 20 client.26614 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.822+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea9008950 0x55fea900ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.822+0000 7f5e36dfd740 4 client.26614 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:23.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.822+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fea8c93720 con 0x55fea9008950 2024-03-12T05:17:23.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.822+0000 7f5e31b27640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea9008950 0x55fea900ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.823+0000 7f5e15ffb640 10 client.26614 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.823+0000 7f5e31b27640 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea9008950 0x55fea900ad40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afcd83a2fbc737a3 server_cookie=ffbd5eb454379623 in_seq=0 out_seq=0 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e15ffb640 1 -- 172.21.15.45:0/1145388919 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fea8c93720 con 0x55fea9008950 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e15ffb640 10 client.26614 handle_command_reply: tid=0 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e36dfd740 1 client.26614 shutdown 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 msgr2=0x55fea8ff5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c006010 tx=0x7f5e1c0b8cd0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 shutdown_connections 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.824+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea9008950 0x55fea900ad40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8ff8940 0x55fea8ff64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea8ff5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1145388919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff5a20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 >> 172.21.15.45:0/1145388919 conn(0x55fea8fe9b70 msgr2=0x55fea8feb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e15ffb640 0 client.26614 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 shutdown_connections 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1145388919 wait complete. 2024-03-12T05:17:23.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.825+0000 7f5e36dfd740 20 client.26614 trim_cache size 0 max 16384 2024-03-12T05:17:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.845+0000 7f5e36dfd740 1 Processor -- start 2024-03-12T05:17:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.845+0000 7f5e36dfd740 1 -- start start 2024-03-12T05:17:23.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea900f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8edc130 con 0x55fea8fefd60 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8fe5520 con 0x55fea8ff8940 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e36dfd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8feadd0 con 0x55fea8fef440 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e31b27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38174/0 (socket says 172.21.15.45:38174) 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e31b27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50290/0 (socket says 172.21.15.45:50290) 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 -- 172.21.15.45:0/1339090487 learned_addr learned my addr 172.21.15.45:0/1339090487 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32b29640 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea900f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 -- 172.21.15.45:0/1339090487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 msgr2=0x55fea900f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea900f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 -- 172.21.15.45:0/1339090487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 msgr2=0x55fea900f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e32328640 1 -- 172.21.15.45:0/1339090487 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea8ef1c20 con 0x55fea8fef440 2024-03-12T05:17:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.846+0000 7f5e31b27640 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.847+0000 7f5e32328640 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0b9400 tx=0x7f5e1c0bc900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f59f720897751e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.847+0000 7f5e17fff640 1 -- 172.21.15.45:0/1339090487 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0f2020 con 0x55fea8fef440 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.847+0000 7f5e17fff640 1 -- 172.21.15.45:0/1339090487 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5e1c0d4070 con 0x55fea8fef440 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.847+0000 7f5e17fff640 1 -- 172.21.15.45:0/1339090487 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0c4de0 con 0x55fea8fef440 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1339090487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 msgr2=0x55fea8ff8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0b9400 tx=0x7f5e1c0bc900 comp rx=0 tx=0).stop 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1339090487 shutdown_connections 2024-03-12T05:17:23.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fefd60 0x55fea900f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8ff8940 0x55fea900f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/1339090487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8fef440 0x55fea8ff8150 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1339090487 >> 172.21.15.45:0/1339090487 conn(0x55fea8fe9b70 msgr2=0x55fea8ff42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1339090487 shutdown_connections 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- 172.21.15.45:0/1339090487 wait complete. 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.849+0000 7f5e36dfd740 1 -- start start 2024-03-12T05:17:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e36dfd740 1 Processor -- start 2024-03-12T05:17:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e36dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea900e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e32328640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e32328640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50302/0 (socket says 172.21.15.45:50302) 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.850+0000 7f5e32328640 1 -- 172.21.15.45:0/3315412221 learned_addr learned my addr 172.21.15.45:0/3315412221 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 0x55fea900efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea900f570 con 0x55fea8fef440 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e31b27640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea900e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e32b29640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 0x55fea900efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea900f6e0 con 0x55fea8fefd60 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea900f8a0 con 0x55fea8ff8940 2024-03-12T05:17:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.851+0000 7f5e36dfd740 10 client.0 _ll_register_callbacks cb 0x7f5e28009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:23.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 -- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 msgr2=0x55fea900efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 0x55fea900efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 -- 172.21.15.45:0/3315412221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 msgr2=0x55fea900e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea900e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fea8f19060 con 0x55fea8fef440 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32b29640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 0x55fea900efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e32328640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0c5390 tx=0x7f5e1c0c0bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cd0f2f3c0c0a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.852+0000 7f5e15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0f2030 con 0x55fea8fef440 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5e1c0bd130 con 0x55fea8fef440 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fea8ff0910 con 0x55fea8fef440 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e1c003dc0 con 0x55fea8fef440 2024-03-12T05:17:23.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e36dfd740 10 client.17166 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fea8f556a0 con 0x55fea8fef440 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.853+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fea8ff0910 con 0x55fea8fef440 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5e1c03c050 con 0x55fea8fef440 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 10 client.17166 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 10 client.17166 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 20 client.17166 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea905fa80 0x55fea9061e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 4 client.17166 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fea8fe52f0 con 0x55fea905fa80 2024-03-12T05:17:23.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.854+0000 7f5e31b27640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea905fa80 0x55fea9061e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e15ffb640 10 client.17166 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e31b27640 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea905fa80 0x55fea9061e70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f2f7086ea4bcc6cc server_cookie=bb5d44c6e32a83e7 in_seq=0 out_seq=0 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e15ffb640 1 -- 172.21.15.45:0/3315412221 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fea8fe52f0 con 0x55fea905fa80 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e15ffb640 10 client.17166 handle_command_reply: tid=0 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 client.17166 shutdown 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 msgr2=0x55fea900db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0c5390 tx=0x7f5e1c0c0bf0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 shutdown_connections 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55fea905fa80 0x55fea9061e70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea8ff8940 0x55fea900efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e15ffb640 0 client.17166 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fea8fefd60 0x55fea900e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 --2- 172.21.15.45:0/3315412221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fea8fef440 0x55fea900db40 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 >> 172.21.15.45:0/3315412221 conn(0x55fea8fe9b70 msgr2=0x55fea8fe8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 shutdown_connections 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.855+0000 7f5e36dfd740 1 -- 172.21.15.45:0/3315412221 wait complete. 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e36dfd740 20 client.17166 trim_cache size 0 max 16384 2024-03-12T05:17:23.854 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 msgr2=0x7f5e180a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 0x7f5e180a40e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009fd0 tx=0x7f5e24009500 comp rx=0 tx=0).stop 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 msgr2=0x7f5e2c072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c072830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009230 tx=0x7f5e1c005dc0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 shutdown_connections 2024-03-12T05:17:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5e180a1c20 0x7f5e180a40e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.856+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c100210 0x7f5e2c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.857+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5e2c111d80 0x7f5e2c06b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.857+0000 7f5e3332a640 1 --2- 172.21.15.45:0/2045005883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e2c0ff8f0 0x7f5e2c072830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.857+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 >> 172.21.15.45:0/2045005883 conn(0x7f5e2c0751c0 msgr2=0x7f5e2c10f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.857+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 shutdown_connections 2024-03-12T05:17:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:23.857+0000 7f5e3332a640 1 -- 172.21.15.45:0/2045005883 wait complete. 2024-03-12T05:17:23.870 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:23.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 -- 172.21.15.45:0/880598571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1060103590 msgr2=0x7f1060103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 --2- 172.21.15.45:0/880598571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1060103590 0x7f1060103970 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1050009cd0 tx=0x7f105002c0e0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 -- 172.21.15.45:0/880598571 shutdown_connections 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 --2- 172.21.15.45:0/880598571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060108660 0x7f1060110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 --2- 172.21.15.45:0/880598571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060103f40 0x7f1060107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 --2- 172.21.15.45:0/880598571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1060103590 0x7f1060103970 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 -- 172.21.15.45:0/880598571 >> 172.21.15.45:0/880598571 conn(0x7f10600fedf0 msgr2=0x7f1060101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 -- 172.21.15.45:0/880598571 shutdown_connections 2024-03-12T05:17:24.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.052+0000 7f106614f640 1 -- 172.21.15.45:0/880598571 wait complete. 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.053+0000 7f106614f640 1 Processor -- start 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.053+0000 7f106614f640 1 -- start start 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f106614f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f106614f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060108660 0x7f106019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50320/0 (socket says 172.21.15.45:50320) 2024-03-12T05:17:24.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f105f7fe640 1 -- 172.21.15.45:0/2601468762 learned_addr learned my addr 172.21.15.45:0/2601468762 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 0x7f1060199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106019a280 con 0x7f1060103590 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.054+0000 7f1057fff640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060108660 0x7f106019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105ffff640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 0x7f1060199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106019a3f0 con 0x7f10601997d0 2024-03-12T05:17:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106019a560 con 0x7f1060108660 2024-03-12T05:17:24.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105f7fe640 1 -- 172.21.15.45:0/2601468762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060108660 msgr2=0x7f106019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105f7fe640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060108660 0x7f106019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105f7fe640 1 -- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 msgr2=0x7f1060199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105f7fe640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 0x7f1060199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105f7fe640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050009900 con 0x7f1060103590 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.055+0000 7f105ffff640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 0x7f1060199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f105f7fe640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1050000f80 tx=0x7f1050035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f105d7fa640 1 -- 172.21.15.45:0/2601468762 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10500329e0 con 0x7f1060103590 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10601a64b0 con 0x7f1060103590 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10601a6960 con 0x7f1060103590 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f105d7fa640 1 -- 172.21.15.45:0/2601468762 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1050035e90 con 0x7f1060103590 2024-03-12T05:17:24.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.056+0000 7f105d7fa640 1 -- 172.21.15.45:0/2601468762 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10500046f0 con 0x7f1060103590 2024-03-12T05:17:24.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.059+0000 7f105d7fa640 1 -- 172.21.15.45:0/2601468762 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f105003b040 con 0x7f1060103590 2024-03-12T05:17:24.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.060+0000 7f1068e03740 1 Processor -- start 2024-03-12T05:17:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.060+0000 7f105d7fa640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 0x7f104c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.060+0000 7f105d7fa640 1 -- 172.21.15.45:0/2601468762 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f10500f0090 con 0x7f1060103590 2024-03-12T05:17:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 -- start start 2024-03-12T05:17:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834d4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278348f960 con 0x5627834d4a10 2024-03-12T05:17:24.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562783443310 con 0x5627834d4060 2024-03-12T05:17:24.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.061+0000 7f1068e03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562783440840 con 0x5627834e5f60 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38214/0 (socket says 172.21.15.45:38214) 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 -- 172.21.15.45:0/1452412692 learned_addr learned my addr 172.21.15.45:0/1452412692 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f1057fff640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105f7fe640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834d4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 0x7f104c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 -- 172.21.15.45:0/1452412692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 msgr2=0x5627834d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834d4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 -- 172.21.15.45:0/1452412692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 msgr2=0x5627834e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105ffff640 1 -- 172.21.15.45:0/1452412692 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627833fe060 con 0x5627834e5f60 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f105f7fe640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834d4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.062+0000 7f1057fff640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.063+0000 7f105ffff640 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1058012dd0 tx=0x7f1058010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859b099774bba0e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.063+0000 7f105ffff640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 0x7f104c0a40e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1058010c40 tx=0x7f1058017000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.063+0000 7f1055ffb640 1 -- 172.21.15.45:0/1452412692 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058024070 con 0x5627834e5f60 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.063+0000 7f1055ffb640 1 -- 172.21.15.45:0/1452412692 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f105800bd90 con 0x5627834e5f60 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.063+0000 7f1055ffb640 1 -- 172.21.15.45:0/1452412692 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105801f930 con 0x5627834e5f60 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.064+0000 7f1068e03740 1 -- 172.21.15.45:0/1452412692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 msgr2=0x5627834e8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.064+0000 7f1068e03740 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1058012dd0 tx=0x7f1058010740 comp rx=0 tx=0).stop 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 -- 172.21.15.45:0/1452412692 shutdown_connections 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834e5f60 0x5627834e8350 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 --2- 172.21.15.45:0/1452412692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834d4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 -- 172.21.15.45:0/1452412692 >> 172.21.15.45:0/1452412692 conn(0x5627834ce980 msgr2=0x5627834ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 -- 172.21.15.45:0/1452412692 shutdown_connections 2024-03-12T05:17:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.065+0000 7f1068e03740 1 -- 172.21.15.45:0/1452412692 wait complete. 2024-03-12T05:17:24.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.066+0000 7f1068e03740 1 -- start start 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.066+0000 7f1068e03740 1 Processor -- start 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.068+0000 7f1068e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.068+0000 7f1068e03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4a10 0x5627834dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.068+0000 7f1068e03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 0x5627834e56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f1068e03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278343c020 con 0x5627834e5f60 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44556/0 (socket says 172.21.15.45:44556) 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 -- 172.21.15.45:0/1213722602 learned_addr learned my addr 172.21.15.45:0/1213722602 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278310f430 con 0x5627834d4060 2024-03-12T05:17:24.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627830cbe00 con 0x5627834d4a10 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f1057fff640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4a10 0x5627834dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f1068e03740 10 client.0 _ll_register_callbacks cb 0x5627833e3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105ffff640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 0x5627834e56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 -- 172.21.15.45:0/1213722602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4a10 msgr2=0x5627834dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4a10 0x5627834dfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 -- 172.21.15.45:0/1213722602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 msgr2=0x5627834e56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 0x5627834e56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105801a040 con 0x5627834d4060 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105ffff640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 0x5627834e56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:24.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.069+0000 7f105f7fe640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f10500ed8c0 tx=0x7f10500bd440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=661ca53f2cd8b232 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f103effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10500edea0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56278313b7e0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f10500c54e0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10500bcfa0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.070+0000 7f1068e03740 10 client.26657 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56278343a6a0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56278313b7e0 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f10500c5070 con 0x5627834d4060 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 10 client.26657 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 10 client.26657 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:17:24.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 20 client.26657 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627834fe9b0 0x562783500da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 4 client.26657 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.071+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562783178720 con 0x5627834fe9b0 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.072+0000 7f1057fff640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627834fe9b0 0x562783500da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.073+0000 7f1057fff640 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627834fe9b0 0x562783500da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30baa9b11acb5572 server_cookie=3ca91a7f9171ea86 in_seq=0 out_seq=0 2024-03-12T05:17:24.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.073+0000 7f103effd640 10 client.26657 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.074+0000 7f103effd640 1 -- 172.21.15.45:0/1213722602 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562783178720 con 0x5627834fe9b0 2024-03-12T05:17:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.074+0000 7f103effd640 10 client.26657 handle_command_reply: tid=0 2024-03-12T05:17:24.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.075+0000 7f1068e03740 1 client.26657 shutdown 2024-03-12T05:17:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.075+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 msgr2=0x5627834df980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.075+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f10500ed8c0 tx=0x7f10500bd440 comp rx=0 tx=0).stop 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 shutdown_connections 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627834fe9b0 0x562783500da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834e5f60 0x5627834e56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4a10 0x5627834dfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 --2- 172.21.15.45:0/1213722602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834d4060 0x5627834df980 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 >> 172.21.15.45:0/1213722602 conn(0x5627834ce980 msgr2=0x5627834d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f103effd640 0 client.26657 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 shutdown_connections 2024-03-12T05:17:24.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.077+0000 7f1068e03740 1 -- 172.21.15.45:0/1213722602 wait complete. 2024-03-12T05:17:24.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.078+0000 7f1068e03740 20 client.26657 trim_cache size 0 max 16384 2024-03-12T05:17:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.103+0000 7f1068e03740 1 Processor -- start 2024-03-12T05:17:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.103+0000 7f1068e03740 1 -- start start 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627834d6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1057fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f105f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38222/0 (socket says 172.21.15.45:38222) 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1057fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50356/0 (socket says 172.21.15.45:50356) 2024-03-12T05:17:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f105f7fe640 1 -- 172.21.15.45:0/2016822541 learned_addr learned my addr 172.21.15.45:0/2016822541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f105ffff640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627834d6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627833c1130 con 0x5627834d4a10 2024-03-12T05:17:24.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627834d3340 con 0x5627834e5f60 2024-03-12T05:17:24.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.104+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627834cf550 con 0x5627834d4060 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 -- 172.21.15.45:0/2016822541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 msgr2=0x5627834e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 -- 172.21.15.45:0/2016822541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 msgr2=0x5627834d6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627834d6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 -- 172.21.15.45:0/2016822541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627833d6c20 con 0x5627834d4a10 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f105f7fe640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f105ffff640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627834d6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.105+0000 7f1057fff640 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x5627834da5e0 tx=0x7f1048005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d9838c0132ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.106+0000 7f10557fa640 1 -- 172.21.15.45:0/2016822541 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048017040 con 0x5627834d4a10 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.106+0000 7f10557fa640 1 -- 172.21.15.45:0/2016822541 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1048018040 con 0x5627834d4a10 2024-03-12T05:17:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.106+0000 7f10557fa640 1 -- 172.21.15.45:0/2016822541 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104801f110 con 0x5627834d4a10 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 msgr2=0x5627834cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x5627834da5e0 tx=0x7f1048005f90 comp rx=0 tx=0).stop 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 shutdown_connections 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x5627834cda00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627834d6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 --2- 172.21.15.45:0/2016822541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x5627834e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.107+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 >> 172.21.15.45:0/2016822541 conn(0x5627834ce980 msgr2=0x5627834cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.108+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 shutdown_connections 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.108+0000 7f1068e03740 1 -- 172.21.15.45:0/2016822541 wait complete. 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.108+0000 7f1068e03740 1 -- start start 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.109+0000 7f1068e03740 1 Processor -- start 2024-03-12T05:17:24.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.109+0000 7f1068e03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x56278354f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.109+0000 7f1068e03740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x56278354f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.109+0000 7f1068e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.109+0000 7f1068e03740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562783549b90 con 0x5627834d4a10 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f1068e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562783549d00 con 0x5627834e5f60 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f1068e03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562783549ec0 con 0x5627834d4060 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44570/0 (socket says 172.21.15.45:44570) 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 -- 172.21.15.45:0/840310293 learned_addr learned my addr 172.21.15.45:0/840310293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f1068e03740 10 client.0 _ll_register_callbacks cb 0x7f10580123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f1057fff640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x56278354f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105f7fe640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x56278354f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 -- 172.21.15.45:0/840310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 msgr2=0x56278354f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x56278354f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 -- 172.21.15.45:0/840310293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 msgr2=0x56278354f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x56278354f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105ffff640 1 -- 172.21.15.45:0/840310293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56278343a6a0 con 0x5627834e5f60 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.110+0000 7f105f7fe640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x56278354f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f105ffff640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f105800bf40 tx=0x7f1058012660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5350ab9f5d2596aa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f103effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105802e040 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1058005850 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f105800bd90 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1058024040 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f1068e03740 10 client.26663 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.111+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5627833fe060 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f105803f020 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f10580038b0 con 0x5627834e5f60 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 10 client.26663 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 10 client.26663 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 20 client.26663 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627835570f0 0x562783559510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 4 client.26663 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5627831c7890 con 0x5627835570f0 2024-03-12T05:17:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.112+0000 7f105f7fe640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627835570f0 0x562783559510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.113+0000 7f105f7fe640 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627835570f0 0x562783559510 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4aff9e2ffbd79d67 server_cookie=c93302e1c33b988d in_seq=0 out_seq=0 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.113+0000 7f103effd640 10 client.26663 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.113+0000 7f103effd640 1 -- 172.21.15.45:0/840310293 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5627831c7890 con 0x5627835570f0 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.113+0000 7f103effd640 10 client.26663 handle_command_reply: tid=0 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.113+0000 7f1068e03740 1 client.26663 shutdown 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 msgr2=0x5627835495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f105800bf40 tx=0x7f1058012660 comp rx=0 tx=0).stop 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 shutdown_connections 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5627835570f0 0x562783559510 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5627834d4a10 0x56278354f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5627834e5f60 0x5627835495d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 --2- 172.21.15.45:0/840310293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5627834d4060 0x56278354f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f103effd640 0 client.26663 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:24.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 >> 172.21.15.45:0/840310293 conn(0x5627834ce980 msgr2=0x5627834d9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 shutdown_connections 2024-03-12T05:17:24.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 1 -- 172.21.15.45:0/840310293 wait complete. 2024-03-12T05:17:24.112 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.114+0000 7f1068e03740 20 client.26663 trim_cache size 0 max 16384 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 msgr2=0x7f104c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 0x7f104c0a40e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1058010c40 tx=0x7f1058017000 comp rx=0 tx=0).stop 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 msgr2=0x7f106019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1050000f80 tx=0x7f1050035ca0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 shutdown_connections 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f104c0a1c20 0x7f104c0a40e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10601997d0 0x7f1060199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1060108660 0x7f106019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 --2- 172.21.15.45:0/2601468762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1060103590 0x7f106019f660 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.116+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 >> 172.21.15.45:0/2601468762 conn(0x7f10600fedf0 msgr2=0x7f1060100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.117+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 shutdown_connections 2024-03-12T05:17:24.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.117+0000 7f106614f640 1 -- 172.21.15.45:0/2601468762 wait complete. 2024-03-12T05:17:24.131 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:24.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:17:24.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 -- 172.21.15.45:0/592818161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa38100c10 msgr2=0x7ffa38104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/592818161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa38100c10 0x7ffa38104c50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009930 tx=0x7ffa2802bc20 comp rx=0 tx=0).stop 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 -- 172.21.15.45:0/592818161 shutdown_connections 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/592818161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa38100c10 0x7ffa38104c50 secure :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ffa28009930 tx=0x7ffa2802bc20 comp rx=0 tx=0).stop 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/592818161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38105330 0x7ffa3810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/592818161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38100260 0x7ffa38100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 -- 172.21.15.45:0/592818161 >> 172.21.15.45:0/592818161 conn(0x7ffa38077180 msgr2=0x7ffa38077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 -- 172.21.15.45:0/592818161 shutdown_connections 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.310+0000 7ffa3f19a640 1 -- 172.21.15.45:0/592818161 wait complete. 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.311+0000 7ffa3f19a640 1 Processor -- start 2024-03-12T05:17:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.311+0000 7ffa3f19a640 1 -- start start 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.311+0000 7ffa3f19a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.311+0000 7ffa3f19a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38105330 0x7ffa3810b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3f19a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 0x7ffa3810c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44584/0 (socket says 172.21.15.45:44584) 2024-03-12T05:17:24.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3cf36640 1 -- 172.21.15.45:0/1599927261 learned_addr learned my addr 172.21.15.45:0/1599927261 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa37fff640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38105330 0x7ffa3810b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3d737640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 0x7ffa3810c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3819b6d0 con 0x7ffa3810bcf0 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3819b810 con 0x7ffa38100260 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.312+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3819b9d0 con 0x7ffa38105330 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 -- 172.21.15.45:0/1599927261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38105330 msgr2=0x7ffa3810b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38105330 0x7ffa3810b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 -- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 msgr2=0x7ffa3810c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 0x7ffa3810c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa30009840 con 0x7ffa38100260 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3d737640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 0x7ffa3810c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009d50 tx=0x7ffa3000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa35ffb640 1 -- 172.21.15.45:0/1599927261 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30009b40 con 0x7ffa38100260 2024-03-12T05:17:24.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20009490 con 0x7ffa38100260 2024-03-12T05:17:24.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa35ffb640 1 -- 172.21.15.45:0/1599927261 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa30003b70 con 0x7ffa38100260 2024-03-12T05:17:24.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa35ffb640 1 -- 172.21.15.45:0/1599927261 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30010720 con 0x7ffa38100260 2024-03-12T05:17:24.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.313+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa28009560 con 0x7ffa38100260 2024-03-12T05:17:24.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 Processor -- start 2024-03-12T05:17:24.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa35ffb640 1 -- 172.21.15.45:0/1599927261 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ffa30003f90 con 0x7ffa38100260 2024-03-12T05:17:24.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 -- start start 2024-03-12T05:17:24.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64bab5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64ba70960 con 0x55a64bab5b70 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64ba24310 con 0x55a64bab5250 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa41d72740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64ba21840 con 0x55a64babe750 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa37fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50386/0 (socket says 172.21.15.45:50386) 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa37fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38244/0 (socket says 172.21.15.45:38244) 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.317+0000 7ffa3cf36640 1 -- 172.21.15.45:0/2462752455 learned_addr learned my addr 172.21.15.45:0/2462752455 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa3d737640 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64bab5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 -- 172.21.15.45:0/2462752455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 msgr2=0x55a64bab5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64bab5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 -- 172.21.15.45:0/2462752455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 msgr2=0x55a64babe210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 -- 172.21.15.45:0/2462752455 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a64b9df060 con 0x55a64babe750 2024-03-12T05:17:24.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffa38101160 tx=0x7ffa28005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=912ec805e7e34433 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.318+0000 7ffa2effd640 1 -- 172.21.15.45:0/2462752455 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa280047c0 con 0x55a64babe750 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.319+0000 7ffa2effd640 1 -- 172.21.15.45:0/2462752455 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa2802cc10 con 0x55a64babe750 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.319+0000 7ffa2effd640 1 -- 172.21.15.45:0/2462752455 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28003850 con 0x55a64babe750 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.319+0000 7ffa35ffb640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 0x7ffa240a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.319+0000 7ffa35ffb640 1 -- 172.21.15.45:0/1599927261 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffa30014070 con 0x7ffa38100260 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.319+0000 7ffa3d737640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 0x7ffa240a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 -- 172.21.15.45:0/2462752455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 msgr2=0x55a64bac0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ffa38101160 tx=0x7ffa28005dd0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 -- 172.21.15.45:0/2462752455 shutdown_connections 2024-03-12T05:17:24.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64babe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac0b40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2462752455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64bab5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.320+0000 7ffa41d72740 1 -- 172.21.15.45:0/2462752455 >> 172.21.15.45:0/2462752455 conn(0x55a64baaf980 msgr2=0x55a64baafd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.321+0000 7ffa3d737640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 0x7ffa240a40e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa200042d0 tx=0x7ffa20005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.321+0000 7ffa41d72740 1 -- 172.21.15.45:0/2462752455 shutdown_connections 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.321+0000 7ffa41d72740 1 -- 172.21.15.45:0/2462752455 wait complete. 2024-03-12T05:17:24.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.322+0000 7ffa41d72740 1 -- start start 2024-03-12T05:17:24.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.322+0000 7ffa41d72740 1 Processor -- start 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64babbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 0x55a64babc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64ba1d020 con 0x55a64babe750 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64b6f0430 con 0x55a64bab5b70 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64b6ace00 con 0x55a64bab5250 2024-03-12T05:17:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38258/0 (socket says 172.21.15.45:38258) 2024-03-12T05:17:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa3cf36640 1 -- 172.21.15.45:0/482030164 learned_addr learned my addr 172.21.15.45:0/482030164 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa41d72740 10 client.0 _ll_register_callbacks cb 0x55a64b9c4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.324+0000 7ffa37fff640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64babbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3d737640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 0x55a64babc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3cf36640 1 -- 172.21.15.45:0/482030164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 msgr2=0x55a64babbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64babbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3cf36640 1 -- 172.21.15.45:0/482030164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 msgr2=0x55a64babc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 0x55a64babc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3cf36640 1 -- 172.21.15.45:0/482030164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3810d710 con 0x55a64bab5250 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.325+0000 7ffa3d737640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 0x55a64babc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa300a2df0 tx=0x7ffa30096f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac4dad130e077e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa17fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3009b280 con 0x55a64bab5250 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffa300cf070 con 0x55a64bab5250 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a64b71c7e0 con 0x55a64bab5250 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.326+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa3009ab80 con 0x55a64bab5250 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.327+0000 7ffa41d72740 10 client.26647 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.327+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a64ba1b6a0 con 0x55a64bab5250 2024-03-12T05:17:24.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.327+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a64b71c7e0 con 0x55a64bab5250 2024-03-12T05:17:24.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.327+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffa300971f0 con 0x55a64bab5250 2024-03-12T05:17:24.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 10 client.26647 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 10 client.26647 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 20 client.26647 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bad2f90 0x55a64bad5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 4 client.26647 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a64b759720 con 0x55a64bad2f90 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.328+0000 7ffa37fff640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bad2f90 0x55a64bad5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.329+0000 7ffa17fff640 10 client.26647 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:24.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.329+0000 7ffa37fff640 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bad2f90 0x55a64bad5380 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb8bdcfcc8872c4b server_cookie=10ed089fafee6f9e in_seq=0 out_seq=0 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.329+0000 7ffa17fff640 1 -- 172.21.15.45:0/482030164 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a64b759720 con 0x55a64bad2f90 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.329+0000 7ffa17fff640 10 client.26647 handle_command_reply: tid=0 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.329+0000 7ffa41d72740 1 client.26647 shutdown 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 msgr2=0x55a64babb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffa300a2df0 tx=0x7ffa30096f10 comp rx=0 tx=0).stop 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 shutdown_connections 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bad2f90 0x55a64bad5380 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64babe750 0x55a64babc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64babbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 --2- 172.21.15.45:0/482030164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64bab5250 0x55a64babb770 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 >> 172.21.15.45:0/482030164 conn(0x55a64baaf980 msgr2=0x55a64bab16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.330+0000 7ffa17fff640 0 client.26647 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:24.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.331+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 shutdown_connections 2024-03-12T05:17:24.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.331+0000 7ffa41d72740 1 -- 172.21.15.45:0/482030164 wait complete. 2024-03-12T05:17:24.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.331+0000 7ffa41d72740 20 client.26647 trim_cache size 0 max 16384 2024-03-12T05:17:24.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.352+0000 7ffa41d72740 1 Processor -- start 2024-03-12T05:17:24.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 -- start start 2024-03-12T05:17:24.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64babdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64b9a2130 con 0x55a64bab5b70 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64bab65c0 con 0x55a64bab5250 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.353+0000 7ffa41d72740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64bab0300 con 0x55a64babe750 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50408/0 (socket says 172.21.15.45:50408) 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 -- 172.21.15.45:0/2931043612 learned_addr learned my addr 172.21.15.45:0/2931043612 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa3d737640 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64babdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 -- 172.21.15.45:0/2931043612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 msgr2=0x55a64bac99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 -- 172.21.15.45:0/2931043612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 msgr2=0x55a64babdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64babdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.354+0000 7ffa37fff640 1 -- 172.21.15.45:0/2931043612 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a64b9b7c20 con 0x55a64bab5b70 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.355+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006010 tx=0x7ffa28042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bddedf73efd484 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.355+0000 7ffa2e7fc640 1 -- 172.21.15.45:0/2931043612 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa28052d70 con 0x55a64bab5b70 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.355+0000 7ffa2e7fc640 1 -- 172.21.15.45:0/2931043612 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffa28004040 con 0x55a64bab5b70 2024-03-12T05:17:24.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.355+0000 7ffa2e7fc640 1 -- 172.21.15.45:0/2931043612 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa280525d0 con 0x55a64bab5b70 2024-03-12T05:17:24.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.356+0000 7ffa41d72740 1 -- 172.21.15.45:0/2931043612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 msgr2=0x55a64bab6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.356+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffa28006010 tx=0x7ffa28042040 comp rx=0 tx=0).stop 2024-03-12T05:17:24.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.356+0000 7ffa41d72740 1 -- 172.21.15.45:0/2931043612 shutdown_connections 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5b70 0x55a64bab6ad0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2931043612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5250 0x55a64babdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 -- 172.21.15.45:0/2931043612 >> 172.21.15.45:0/2931043612 conn(0x55a64baaf980 msgr2=0x55a64bad9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 -- 172.21.15.45:0/2931043612 shutdown_connections 2024-03-12T05:17:24.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 -- 172.21.15.45:0/2931043612 wait complete. 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 -- start start 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.357+0000 7ffa41d72740 1 Processor -- start 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64bac99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa3cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50420/0 (socket says 172.21.15.45:50420) 2024-03-12T05:17:24.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa3cf36640 1 -- 172.21.15.45:0/2795472496 learned_addr learned my addr 172.21.15.45:0/2795472496 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64bac99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64baca2c0 con 0x55a64bab5250 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64bac3a70 con 0x55a64bab5b70 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.358+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a64bac3c30 con 0x55a64babe750 2024-03-12T05:17:24.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3d737640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa41d72740 10 client.0 _ll_register_callbacks cb 0x7ffa300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3cf36640 1 -- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 msgr2=0x55a64bac3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3cf36640 1 -- 172.21.15.45:0/2795472496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 msgr2=0x55a64bac99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64bac99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.359+0000 7ffa3cf36640 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a64b9df060 con 0x55a64bab5250 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa3cf36640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ffa30097550 tx=0x7ffa3009a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5907f56cd1b9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa2802cc10 con 0x55a64bab5250 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa17fff640 10 client.17202 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:24.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64bac99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa17fff640 1 -- 172.21.15.45:0/2795472496 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa300cbb20 con 0x55a64bab5250 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa17fff640 1 -- 172.21.15.45:0/2795472496 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffa300d0030 con 0x55a64bab5250 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa17fff640 1 -- 172.21.15.45:0/2795472496 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa30096c80 con 0x55a64bab5250 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa41d72740 10 client.17202 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.360+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55a64b9dd8c0 con 0x55a64bab5250 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa17fff640 1 -- 172.21.15.45:0/2795472496 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffa300aebd0 con 0x55a64bab5250 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 10 client.17202 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 10 client.17202 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 20 client.17202 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bb2e4c0 0x55a64bb308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 4 client.17202 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a64b7a8890 con 0x55a64bb2e4c0 2024-03-12T05:17:24.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.361+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bb2e4c0 0x55a64bb308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa17fff640 10 client.17202 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa37fff640 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bb2e4c0 0x55a64bb308b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f6457666bc03a0 server_cookie=91e1dfd0c0d50699 in_seq=0 out_seq=0 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa17fff640 1 -- 172.21.15.45:0/2795472496 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a64b7a8890 con 0x55a64bb2e4c0 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa17fff640 10 client.17202 handle_command_reply: tid=0 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa41d72740 1 client.17202 shutdown 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 msgr2=0x55a64bac5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.362+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ffa30097550 tx=0x7ffa3009a7b0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 shutdown_connections 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a64bb2e4c0 0x55a64bb308b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa17fff640 0 client.17202 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a64babe750 0x55a64bac3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a64bab5b70 0x55a64bac99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 --2- 172.21.15.45:0/2795472496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a64bab5250 0x55a64bac5070 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 >> 172.21.15.45:0/2795472496 conn(0x55a64baaf980 msgr2=0x55a64bab6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 shutdown_connections 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 1 -- 172.21.15.45:0/2795472496 wait complete. 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.363+0000 7ffa41d72740 20 client.17202 trim_cache size 0 max 16384 2024-03-12T05:17:24.361 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:24.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 msgr2=0x7ffa240a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 0x7ffa240a40e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa200042d0 tx=0x7ffa20005f90 comp rx=0 tx=0).stop 2024-03-12T05:17:24.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 msgr2=0x7ffa3810cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009d50 tx=0x7ffa3000c6a0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 shutdown_connections 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffa240a1c20 0x7ffa240a40e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa3810bcf0 0x7ffa3810c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa38105330 0x7ffa3810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 --2- 172.21.15.45:0/1599927261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa38100260 0x7ffa3810cc40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 >> 172.21.15.45:0/1599927261 conn(0x7ffa38077180 msgr2=0x7ffa38102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 shutdown_connections 2024-03-12T05:17:24.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.364+0000 7ffa3f19a640 1 -- 172.21.15.45:0/1599927261 wait complete. 2024-03-12T05:17:24.387 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:24.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 -- 172.21.15.45:0/1287114833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486810a440 msgr2=0x7f4868112b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 --2- 172.21.15.45:0/1287114833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486810a440 0x7f4868112b90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4858009900 tx=0x7f485802bc30 comp rx=0 tx=0).stop 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 -- 172.21.15.45:0/1287114833 shutdown_connections 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 --2- 172.21.15.45:0/1287114833 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48681130d0 0x7f48681154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 --2- 172.21.15.45:0/1287114833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486810a440 0x7f4868112b90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 --2- 172.21.15.45:0/1287114833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4868109b20 0x7f4868109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 -- 172.21.15.45:0/1287114833 >> 172.21.15.45:0/1287114833 conn(0x7f486806c260 msgr2=0x7f486806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 -- 172.21.15.45:0/1287114833 shutdown_connections 2024-03-12T05:17:24.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.578+0000 7f486fa40640 1 -- 172.21.15.45:0/1287114833 wait complete. 2024-03-12T05:17:24.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.579+0000 7f486fa40640 1 Processor -- start 2024-03-12T05:17:24.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.579+0000 7f486fa40640 1 -- start start 2024-03-12T05:17:24.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4868109b20 0x7f48681a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 0x7f48681a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 0x7f48681aad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681ab350 con 0x7f486810a440 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681ab4c0 con 0x7f48681130d0 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486fa40640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681ab680 con 0x7f4868109b20 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 0x7f48681a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 0x7f48681a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50438/0 (socket says 172.21.15.45:50438) 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486cfdb640 1 -- 172.21.15.45:0/1721710944 learned_addr learned my addr 172.21.15.45:0/1721710944 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4868109b20 0x7f48681a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.580+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 0x7f48681aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 -- 172.21.15.45:0/1721710944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4868109b20 msgr2=0x7f48681a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4868109b20 0x7f48681a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 -- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 msgr2=0x7f48681a69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 0x7f48681a69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 -- 172.21.15.45:0/1721710944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4858009560 con 0x7f48681130d0 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 0x7f48681aad50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4860005fd0 tx=0x7f48600071c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f485e7fc640 1 -- 172.21.15.45:0/1721710944 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860010070 con 0x7f48681130d0 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f485e7fc640 1 -- 172.21.15.45:0/1721710944 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4860007bd0 con 0x7f48681130d0 2024-03-12T05:17:24.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f485e7fc640 1 -- 172.21.15.45:0/1721710944 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860005940 con 0x7f48681130d0 2024-03-12T05:17:24.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48681ab960 con 0x7f48681130d0 2024-03-12T05:17:24.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.581+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48681b31e0 con 0x7f48681130d0 2024-03-12T05:17:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.585+0000 7f487260d740 1 Processor -- start 2024-03-12T05:17:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.585+0000 7f485e7fc640 1 -- 172.21.15.45:0/1721710944 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4860007d40 con 0x7f48681130d0 2024-03-12T05:17:24.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f487260d740 1 -- start start 2024-03-12T05:17:24.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f487260d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f487260d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55af210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50454/0 (socket says 172.21.15.45:50454) 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f486d7dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.586+0000 7f486d7dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38324/0 (socket says 172.21.15.45:38324) 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f487260d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f487260d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c5561960 con 0x5610c55a6250 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f487260d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c5515310 con 0x5610c55af750 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f487260d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c5512840 con 0x5610c55a6b70 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 -- 172.21.15.45:0/1167304667 learned_addr learned my addr 172.21.15.45:0/1167304667 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 -- 172.21.15.45:0/1167304667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 msgr2=0x5610c55af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 -- 172.21.15.45:0/1167304667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 msgr2=0x5610c55b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.587+0000 7f486cfdb640 1 -- 172.21.15.45:0/1167304667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610c54d0060 con 0x5610c55a6250 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.588+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4858009230 tx=0x7f4858005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3087567b4bd4da server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.588+0000 7f48537fe640 1 -- 172.21.15.45:0/1167304667 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858005220 con 0x5610c55a6250 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.588+0000 7f48537fe640 1 -- 172.21.15.45:0/1167304667 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4858005500 con 0x5610c55a6250 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.588+0000 7f48537fe640 1 -- 172.21.15.45:0/1167304667 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858004100 con 0x5610c55a6250 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.588+0000 7f485e7fc640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 0x7f48540a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f485e7fc640 1 -- 172.21.15.45:0/1721710944 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48600ccdb0 con 0x7f48681130d0 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 0x7f48540a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 -- 172.21.15.45:0/1167304667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 msgr2=0x5610c55a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4858009230 tx=0x7f4858005df0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 -- 172.21.15.45:0/1167304667 shutdown_connections 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 --2- 172.21.15.45:0/1167304667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55a6630 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 -- 172.21.15.45:0/1167304667 >> 172.21.15.45:0/1167304667 conn(0x5610c55a0980 msgr2=0x5610c55a0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.589+0000 7f487260d740 1 -- 172.21.15.45:0/1167304667 shutdown_connections 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.590+0000 7f487260d740 1 -- 172.21.15.45:0/1167304667 wait complete. 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.590+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 0x7f48540a40e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x5610c55a5d20 tx=0x7f486400b410 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.590+0000 7f487260d740 1 -- start start 2024-03-12T05:17:24.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.590+0000 7f487260d740 1 Processor -- start 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 0x5610c55ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c550e020 con 0x5610c55a6b70 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c51e1430 con 0x5610c55af750 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f487260d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c519de00 con 0x5610c55a6250 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50468/0 (socket says 172.21.15.45:50468) 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.591+0000 7f486cfdb640 1 -- 172.21.15.45:0/1544177580 learned_addr learned my addr 172.21.15.45:0/1544177580 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 0x5610c55ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f487260d740 10 client.0 _ll_register_callbacks cb 0x5610c54b5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 -- 172.21.15.45:0/1544177580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 msgr2=0x5610c55ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 -- 172.21.15.45:0/1544177580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 msgr2=0x5610c55acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 -- 172.21.15.45:0/1544177580 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4868115ef0 con 0x5610c55a6250 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 0x5610c55ac7f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4864005d00 tx=0x7f4864004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33d40763fb13075e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.592+0000 7f4850ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.593+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864016070 con 0x5610c55a6250 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.593+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48640047a0 con 0x5610c55a6250 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.593+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610c520d7e0 con 0x5610c55a6250 2024-03-12T05:17:24.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.593+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4864004960 con 0x5610c55a6250 2024-03-12T05:17:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.595+0000 7f487260d740 10 client.26665 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.595+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5610c550c6a0 con 0x5610c55a6250 2024-03-12T05:17:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.595+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4864033020 con 0x5610c55a6250 2024-03-12T05:17:24.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f486401c660 con 0x5610c55a6250 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 10 client.26665 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 10 client.26665 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 20 client.26665 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c55bfb80 0x5610c55c2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 4 client.26665 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610c520d7e0 con 0x5610c55bfb80 2024-03-12T05:17:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.596+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c55bfb80 0x5610c55c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.597+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c55bfb80 0x5610c55c2040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f78912b1b2344d80 server_cookie=5cfcd37ac14e7889 in_seq=0 out_seq=0 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.597+0000 7f4850ff9640 10 client.26665 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.597+0000 7f4850ff9640 1 -- 172.21.15.45:0/1544177580 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5610c520d7e0 con 0x5610c55bfb80 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f4850ff9640 10 client.26665 handle_command_reply: tid=0 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 client.26665 shutdown 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 msgr2=0x5610c55ac7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 0x5610c55ac7f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4864005d00 tx=0x7f4864004730 comp rx=0 tx=0).stop 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 shutdown_connections 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6b70 0x5610c55acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c55bfb80 0x5610c55c2040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c55ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 --2- 172.21.15.45:0/1544177580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6250 0x5610c55ac7f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f4850ff9640 0 client.26665 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.598+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 >> 172.21.15.45:0/1544177580 conn(0x5610c55a0980 msgr2=0x5610c55a26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.599+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 shutdown_connections 2024-03-12T05:17:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.599+0000 7f487260d740 1 -- 172.21.15.45:0/1544177580 wait complete. 2024-03-12T05:17:24.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.599+0000 7f487260d740 20 client.26665 trim_cache size 0 max 16384 2024-03-12T05:17:24.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.624+0000 7f487260d740 1 Processor -- start 2024-03-12T05:17:24.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.624+0000 7f487260d740 1 -- start start 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.624+0000 7f487260d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55aef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f487260d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f487260d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55af750 0x5610c55a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f487260d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c5493130 con 0x5610c55a6250 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f486cfdb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44660/0 (socket says 172.21.15.45:44660) 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f487260d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c55a5eb0 con 0x5610c55a6b70 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f486cfdb640 1 -- 172.21.15.45:0/1182379068 learned_addr learned my addr 172.21.15.45:0/1182379068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c55ad900 con 0x5610c55af750 2024-03-12T05:17:24.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f486d7dc640 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.625+0000 7f486dfdd640 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55af750 0x5610c55a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 -- 172.21.15.45:0/1182379068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55af750 msgr2=0x5610c55a9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55af750 0x5610c55a9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 -- 172.21.15.45:0/1182379068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 msgr2=0x5610c55aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 -- 172.21.15.45:0/1182379068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610c54a8c20 con 0x5610c55a6b70 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f486cfdb640 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4858006010 tx=0x7f4858004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15871ff8de6c00b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f4852ffd640 1 -- 172.21.15.45:0/1182379068 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858005670 con 0x5610c55a6b70 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f4852ffd640 1 -- 172.21.15.45:0/1182379068 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4858004040 con 0x5610c55a6b70 2024-03-12T05:17:24.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.626+0000 7f4852ffd640 1 -- 172.21.15.45:0/1182379068 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858053350 con 0x5610c55a6b70 2024-03-12T05:17:24.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.627+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 msgr2=0x5610c55a9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.627+0000 7f487260d740 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4858006010 tx=0x7f4858004b00 comp rx=0 tx=0).stop 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 shutdown_connections 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55af750 0x5610c55a9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55a6b70 0x5610c55a9a90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 --2- 172.21.15.45:0/1182379068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 >> 172.21.15.45:0/1182379068 conn(0x5610c55a0980 msgr2=0x5610c55ab0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 shutdown_connections 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 -- 172.21.15.45:0/1182379068 wait complete. 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.628+0000 7f487260d740 1 -- start start 2024-03-12T05:17:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.629+0000 7f487260d740 1 Processor -- start 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.629+0000 7f487260d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55b5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c5616130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c55b6150 con 0x5610c55a6250 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c5616670 con 0x5610c55af750 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610c56167b0 con 0x5610c55a6b70 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f487260d740 10 client.0 _ll_register_callbacks cb 0x7f4858005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486d7dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486d7dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50490/0 (socket says 172.21.15.45:50490) 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486d7dc640 1 -- 172.21.15.45:0/116669773 learned_addr learned my addr 172.21.15.45:0/116669773 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486cfdb640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c5616130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 -- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 msgr2=0x5610c55b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 -- 172.21.15.45:0/116669773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 msgr2=0x5610c55b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486dfdd640 1 -- 172.21.15.45:0/116669773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610c54d0060 con 0x5610c55af750 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.630+0000 7f486cfdb640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f486dfdd640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c5616130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f48600ca470 tx=0x7f4860099a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4caa787e64f0e54e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f4850ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48600ca8f0 con 0x5610c55af750 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48600b2280 con 0x5610c55af750 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4858005500 con 0x5610c55af750 2024-03-12T05:17:24.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.631+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48600b2460 con 0x5610c55af750 2024-03-12T05:17:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.632+0000 7f487260d740 10 client.26696 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.632+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5610c54ce8c0 con 0x5610c55af750 2024-03-12T05:17:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4860099bb0 con 0x5610c55af750 2024-03-12T05:17:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48600cabc0 con 0x5610c55af750 2024-03-12T05:17:24.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 10 client.26696 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 10 client.26696 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 20 client.26696 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c5617ea0 0x5610c561a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 4 client.26696 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4858005500 con 0x5610c5617ea0 2024-03-12T05:17:24.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.633+0000 7f486d7dc640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c5617ea0 0x5610c561a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.634+0000 7f486d7dc640 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c5617ea0 0x5610c561a2c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35d6ad05ea98d042 server_cookie=94092ee8fdab2dbe in_seq=0 out_seq=0 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.634+0000 7f4850ff9640 10 client.26696 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.634+0000 7f4850ff9640 1 -- 172.21.15.45:0/116669773 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4858005500 con 0x5610c5617ea0 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.634+0000 7f4850ff9640 10 client.26696 handle_command_reply: tid=0 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.635+0000 7f487260d740 1 client.26696 shutdown 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.635+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 msgr2=0x5610c5616130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.635+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c5616130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f48600ca470 tx=0x7f4860099a10 comp rx=0 tx=0).stop 2024-03-12T05:17:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 shutdown_connections 2024-03-12T05:17:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5610c5617ea0 0x5610c561a2c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610c55af750 0x5610c5616130 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c55a6b70 0x5610c55b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 --2- 172.21.15.45:0/116669773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610c55a6250 0x5610c55b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 >> 172.21.15.45:0/116669773 conn(0x5610c55a0980 msgr2=0x5610c559f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f4850ff9640 0 client.26696 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 shutdown_connections 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 1 -- 172.21.15.45:0/116669773 wait complete. 2024-03-12T05:17:24.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.636+0000 7f487260d740 20 client.26696 trim_cache size 0 max 16384 2024-03-12T05:17:24.634 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.637+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 msgr2=0x7f48540a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.637+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 0x7f48540a40e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x5610c55a5d20 tx=0x7f486400b410 comp rx=0 tx=0).stop 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.637+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 msgr2=0x7f48681aad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.637+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 0x7f48681aad50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4860005fd0 tx=0x7f48600071c0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 shutdown_connections 2024-03-12T05:17:24.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48540a1c20 0x7f48540a40e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f486810a440 0x7f48681a69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48681130d0 0x7f48681aad50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 --2- 172.21.15.45:0/1721710944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4868109b20 0x7f48681a6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 >> 172.21.15.45:0/1721710944 conn(0x7f486806c260 msgr2=0x7f486806fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 shutdown_connections 2024-03-12T05:17:24.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.638+0000 7f486fa40640 1 -- 172.21.15.45:0/1721710944 wait complete. 2024-03-12T05:17:24.656 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:24.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:17:24.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:24 smithi045 ceph-mon[32174]: pgmap v200: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:24.695 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:24 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3776636579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:24.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:24 smithi130 ceph-mon[35398]: pgmap v200: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:24.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:24 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3776636579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:24.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:24 smithi116 ceph-mon[35486]: pgmap v200: 97 pgs: 97 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 19 MiB/s wr, 1.65k op/s 2024-03-12T05:17:24.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:24 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3776636579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 -- 172.21.15.45:0/3409762155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 msgr2=0x7f0efc0a8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 --2- 172.21.15.45:0/3409762155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc0a8f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009890 tx=0x7f0ef802bb70 comp rx=0 tx=0).stop 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 -- 172.21.15.45:0/3409762155 shutdown_connections 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 --2- 172.21.15.45:0/3409762155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc0a8f60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 --2- 172.21.15.45:0/3409762155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009970 0x7f0efc0a6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 --2- 172.21.15.45:0/3409762155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009050 0x7f0efc009430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 -- 172.21.15.45:0/3409762155 >> 172.21.15.45:0/3409762155 conn(0x7f0efc0939e0 msgr2=0x7f0efc095e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 -- 172.21.15.45:0/3409762155 shutdown_connections 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.841+0000 7f0f09975640 1 -- 172.21.15.45:0/3409762155 wait complete. 2024-03-12T05:17:24.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.842+0000 7f0f09975640 1 Processor -- start 2024-03-12T05:17:24.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 -- start start 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009970 0x7f0efc0a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc09f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc09fa80 con 0x7f0efc009970 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc09fbf0 con 0x7f0efc009050 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.843+0000 7f0f09975640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc09fdb0 con 0x7f0efc0a6b70 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44686/0 (socket says 172.21.15.45:44686) 2024-03-12T05:17:24.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 -- 172.21.15.45:0/3634322405 learned_addr learned my addr 172.21.15.45:0/3634322405 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f09174640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc09f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009970 0x7f0efc0a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 -- 172.21.15.45:0/3634322405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 msgr2=0x7f0efc09f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc09f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 -- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009970 msgr2=0x7f0efc0a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009970 0x7f0efc0a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.844+0000 7f0f08973640 1 -- 172.21.15.45:0/3634322405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0404f750 con 0x7f0efc009050 2024-03-12T05:17:24.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f08973640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000b60 tx=0x7f0f0407b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f01ffb640 1 -- 172.21.15.45:0/3634322405 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04076e10 con 0x7f0efc009050 2024-03-12T05:17:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f01ffb640 1 -- 172.21.15.45:0/3634322405 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0f0407c040 con 0x7f0efc009050 2024-03-12T05:17:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f01ffb640 1 -- 172.21.15.45:0/3634322405 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040713e0 con 0x7f0efc009050 2024-03-12T05:17:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef8009490 con 0x7f0efc009050 2024-03-12T05:17:24.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.845+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc0a3dc0 con 0x7f0efc009050 2024-03-12T05:17:24.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.848+0000 7f0f01ffb640 1 -- 172.21.15.45:0/3634322405 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0f04074530 con 0x7f0efc009050 2024-03-12T05:17:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.850+0000 7f0f01ffb640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 0x7f0ef40a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.850+0000 7f0f0d7ac740 1 Processor -- start 2024-03-12T05:17:24.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.850+0000 7f0f01ffb640 1 -- 172.21.15.45:0/3634322405 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0f04130730 con 0x7f0efc009050 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 -- start start 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a69b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a6a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a656960 con 0x56529a6a4960 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a60a310 con 0x56529a69b460 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a607840 con 0x56529a69bd80 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f09174640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f09174640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50498/0 (socket says 172.21.15.45:50498) 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f09174640 1 -- 172.21.15.45:0/2485710189 learned_addr learned my addr 172.21.15.45:0/2485710189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f08973640 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a69b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.851+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 0x7f0ef40a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f03fff640 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a6a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 -- 172.21.15.45:0/2485710189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 msgr2=0x56529a6a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a6a4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 -- 172.21.15.45:0/2485710189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 msgr2=0x56529a69b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a69b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 -- 172.21.15.45:0/2485710189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529a5c5060 con 0x56529a6a4960 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f09174640 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005370 tx=0x7f0ef8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174532c4a2fbd578 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.852+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 0x7f0ef40a4230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0a0900 tx=0x7f0ef0009f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.853+0000 7f0eeeffd640 1 -- 172.21.15.45:0/2485710189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8034cb0 con 0x56529a6a4960 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.853+0000 7f0eeeffd640 1 -- 172.21.15.45:0/2485710189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ef802ce90 con 0x56529a6a4960 2024-03-12T05:17:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.853+0000 7f0eeeffd640 1 -- 172.21.15.45:0/2485710189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80052a0 con 0x56529a6a4960 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/2485710189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 msgr2=0x56529a6a6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005370 tx=0x7f0ef8005c90 comp rx=0 tx=0).stop 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/2485710189 shutdown_connections 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6a6d50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a6a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/2485710189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a69b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.854+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/2485710189 >> 172.21.15.45:0/2485710189 conn(0x56529a695b90 msgr2=0x56529a695f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.855+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/2485710189 shutdown_connections 2024-03-12T05:17:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.855+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/2485710189 wait complete. 2024-03-12T05:17:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.855+0000 7f0f0d7ac740 1 -- start start 2024-03-12T05:17:24.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.856+0000 7f0f0d7ac740 1 Processor -- start 2024-03-12T05:17:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.856+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a6a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a6a4960 0x56529a6a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50512/0 (socket says 172.21.15.45:50512) 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f03fff640 1 -- 172.21.15.45:0/4195291175 learned_addr learned my addr 172.21.15.45:0/4195291175 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a603020 con 0x56529a69bd80 2024-03-12T05:17:24.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a2d6430 con 0x56529a69b460 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a292e00 con 0x56529a6a4960 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f08973640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a6a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.857+0000 7f0f09174640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a6a4960 0x56529a6a23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f0d7ac740 10 client.0 _ll_register_callbacks cb 0x56529a5aacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 -- 172.21.15.45:0/4195291175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a6a4960 msgr2=0x56529a6a23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a6a4960 0x56529a6a23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 -- 172.21.15.45:0/4195291175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 msgr2=0x56529a6a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc0a99f0 con 0x56529a69b460 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f03fff640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.858+0000 7f0f08973640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a6a1960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f0412ddf0 tx=0x7f0f040fb7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe3791a9eba352b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0412e270 con 0x56529a69b460 2024-03-12T05:17:24.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0f040f7e40 con 0x56529a69b460 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56529a3027e0 con 0x56529a69b460 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f040747e0 con 0x56529a69b460 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0f0d7ac740 10 client.26708 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.859+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56529a6016a0 con 0x56529a69b460 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.860+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0f04074940 con 0x56529a69b460 2024-03-12T05:17:24.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0f040f9220 con 0x56529a69b460 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 10 client.26708 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 10 client.26708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 20 client.26708 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a6b4e80 0x56529a6b7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 4 client.26708 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56529a3027e0 con 0x56529a6b4e80 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.861+0000 7f0f03fff640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a6b4e80 0x56529a6b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.862+0000 7f0f03fff640 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a6b4e80 0x56529a6b7340 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8293228889362670 server_cookie=20f1428f7acd0f6c in_seq=0 out_seq=0 2024-03-12T05:17:24.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.862+0000 7f0ee3fff640 10 client.26708 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.862+0000 7f0ee3fff640 1 -- 172.21.15.45:0/4195291175 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56529a3027e0 con 0x56529a6b4e80 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.862+0000 7f0ee3fff640 10 client.26708 handle_command_reply: tid=0 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 client.26708 shutdown 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 msgr2=0x56529a6a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a6a1960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f0412ddf0 tx=0x7f0f040fb7e0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 shutdown_connections 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a6b4e80 0x56529a6b7340 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69bd80 0x56529a6a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a6a4960 0x56529a6a23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/4195291175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69b460 0x56529a6a1960 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.863+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 >> 172.21.15.45:0/4195291175 conn(0x56529a695b90 msgr2=0x56529a6978e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.864+0000 7f0ee3fff640 0 client.26708 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.864+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 shutdown_connections 2024-03-12T05:17:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.864+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/4195291175 wait complete. 2024-03-12T05:17:24.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.864+0000 7f0f0d7ac740 20 client.26708 trim_cache size 0 max 16384 2024-03-12T05:17:24.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.887+0000 7f0f0d7ac740 1 Processor -- start 2024-03-12T05:17:24.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.887+0000 7f0f0d7ac740 1 -- start start 2024-03-12T05:17:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69b460 0x56529a6a4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a69cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a588130 con 0x56529a69b460 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f09174640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f09174640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44718/0 (socket says 172.21.15.45:44718) 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f09174640 1 -- 172.21.15.45:0/1928804052 learned_addr learned my addr 172.21.15.45:0/1928804052 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a6a4670 con 0x56529a6a4960 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.888+0000 7f0f08973640 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69b460 0x56529a6a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a6956c0 con 0x56529a69bd80 2024-03-12T05:17:24.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f03fff640 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a69cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f09174640 1 -- 172.21.15.45:0/1928804052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 msgr2=0x56529a69cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f09174640 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a69cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f09174640 1 -- 172.21.15.45:0/1928804052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69b460 msgr2=0x56529a6a4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f09174640 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69b460 0x56529a6a4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.889+0000 7f0f09174640 1 -- 172.21.15.45:0/1928804052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529a59dc20 con 0x56529a6a4960 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.890+0000 7f0f09174640 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005370 tx=0x7f0ef80347a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65b5c04067ec0312 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.890+0000 7f0eee7fc640 1 -- 172.21.15.45:0/1928804052 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80058b0 con 0x56529a6a4960 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.890+0000 7f0eee7fc640 1 -- 172.21.15.45:0/1928804052 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ef8034440 con 0x56529a6a4960 2024-03-12T05:17:24.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.890+0000 7f0eee7fc640 1 -- 172.21.15.45:0/1928804052 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef802c510 con 0x56529a6a4960 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 msgr2=0x56529a69ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0efc005370 tx=0x7f0ef80347a0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 shutdown_connections 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a6a4960 0x56529a69ebc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69bd80 0x56529a69cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/1928804052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a69b460 0x56529a6a4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 >> 172.21.15.45:0/1928804052 conn(0x56529a695b90 msgr2=0x56529a6bbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 shutdown_connections 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.891+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/1928804052 wait complete. 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.892+0000 7f0f0d7ac740 1 -- start start 2024-03-12T05:17:24.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.892+0000 7f0f0d7ac740 1 Processor -- start 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69b460 0x56529a6abc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6aa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a6aace0 con 0x56529a6a4960 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a6aae50 con 0x56529a69bd80 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529a70b430 con 0x56529a69b460 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f0d7ac740 10 client.0 _ll_register_callbacks cb 0x7f0ef8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44730/0 (socket says 172.21.15.45:44730) 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 -- 172.21.15.45:0/3401766823 learned_addr learned my addr 172.21.15.45:0/3401766823 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f09174640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f08973640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69b460 0x56529a6abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 -- 172.21.15.45:0/3401766823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69b460 msgr2=0x56529a6abc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69b460 0x56529a6abc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 -- 172.21.15.45:0/3401766823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 msgr2=0x56529a6aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:24.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6aa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.893+0000 7f0f03fff640 1 -- 172.21.15.45:0/3401766823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56529a5c5060 con 0x56529a69bd80 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0f03fff640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x56529a6b7c50 tx=0x7f0ef002c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19fb034cf53fb78f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef003e070 con 0x56529a69bd80 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0ef003abc0 con 0x56529a69bd80 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ef802ce90 con 0x56529a69bd80 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef0037040 con 0x56529a69bd80 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0f0d7ac740 10 client.26714 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:24.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.894+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56529a5c38c0 con 0x56529a69bd80 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0ef802ce90 con 0x56529a69bd80 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0ef0041030 con 0x56529a69bd80 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 10 client.26714 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 10 client.26714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 20 client.26714 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a70cce0 0x56529a70f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 4 client.26714 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.895+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56529a33f720 con 0x56529a70cce0 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.896+0000 7f0f08973640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a70cce0 0x56529a70f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.897+0000 7f0f08973640 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a70cce0 0x56529a70f100 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be78801f3867f82c server_cookie=730e63621f059e97 in_seq=0 out_seq=0 2024-03-12T05:17:24.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.897+0000 7f0ee3fff640 10 client.26714 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:24.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.897+0000 7f0ee3fff640 1 -- 172.21.15.45:0/3401766823 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56529a33f720 con 0x56529a70cce0 2024-03-12T05:17:24.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.897+0000 7f0ee3fff640 10 client.26714 handle_command_reply: tid=0 2024-03-12T05:17:24.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.897+0000 7f0f0d7ac740 1 client.26714 shutdown 2024-03-12T05:17:24.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 msgr2=0x56529a6aa000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x56529a6b7c50 tx=0x7f0ef002c000 comp rx=0 tx=0).stop 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 shutdown_connections 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56529a70cce0 0x56529a70f100 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0ee3fff640 0 client.26714 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56529a6a4960 0x56529a6aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.898+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56529a69bd80 0x56529a6aa000 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.899+0000 7f0f0d7ac740 1 --2- 172.21.15.45:0/3401766823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56529a69b460 0x56529a6abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.899+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 >> 172.21.15.45:0/3401766823 conn(0x56529a695b90 msgr2=0x56529a69cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.899+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 shutdown_connections 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.899+0000 7f0f0d7ac740 1 -- 172.21.15.45:0/3401766823 wait complete. 2024-03-12T05:17:24.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.899+0000 7f0f0d7ac740 20 client.26714 trim_cache size 0 max 16384 2024-03-12T05:17:24.897 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:24.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 msgr2=0x7f0ef40a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 0x7f0ef40a4230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0a0900 tx=0x7f0ef0009f90 comp rx=0 tx=0).stop 2024-03-12T05:17:24.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 msgr2=0x7f0efc0a60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:24.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0f04000b60 tx=0x7f0f0407b510 comp rx=0 tx=0).stop 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 shutdown_connections 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0ef40a1d70 0x7f0ef40a4230 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0a0900 tx=0x7f0ef0009f90 comp rx=0 tx=0).stop 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0efc009970 0x7f0efc0a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0efc0a6b70 0x7f0efc09f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 --2- 172.21.15.45:0/3634322405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0efc009050 0x7f0efc0a60c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 >> 172.21.15.45:0/3634322405 conn(0x7f0efc0939e0 msgr2=0x7f0efc0a8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 shutdown_connections 2024-03-12T05:17:24.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:24.900+0000 7f0f09975640 1 -- 172.21.15.45:0/3634322405 wait complete. 2024-03-12T05:17:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 320.0 s, 1293.4 tps, lat 77.220 ms stddev 23.565 2024-03-12T05:17:25.916 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:17:25.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:17:25.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/3052524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6410ded0 msgr2=0x7f6f6410e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/3052524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6410ded0 0x7f6f6410e5a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009900 tx=0x7f6f6002bc40 comp rx=0 tx=0).stop 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/3052524200 shutdown_connections 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/3052524200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f64108cd0 0x7f6f6410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/3052524200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f6410ded0 0x7f6f6410e5a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/3052524200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6410d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/3052524200 >> 172.21.15.45:0/3052524200 conn(0x7f6f6406c260 msgr2=0x7f6f6406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/3052524200 shutdown_connections 2024-03-12T05:17:26.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.101+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/3052524200 wait complete. 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.102+0000 7f6f6bdd0640 1 Processor -- start 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.102+0000 7f6f6bdd0640 1 -- start start 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f64108cd0 0x7f6f641a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 0x7f6f641a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641a4c70 con 0x7f6f6410ded0 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641a4de0 con 0x7f6f641088f0 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f69b6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6bdd0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641a4fa0 con 0x7f6f64108cd0 2024-03-12T05:17:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f69b6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44742/0 (socket says 172.21.15.45:44742) 2024-03-12T05:17:26.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f69b6c640 1 -- 172.21.15.45:0/2920360417 learned_addr learned my addr 172.21.15.45:0/2920360417 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6936b640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f64108cd0 0x7f6f641a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.103+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 0x7f6f641a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 -- 172.21.15.45:0/2920360417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f64108cd0 msgr2=0x7f6f641a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f64108cd0 0x7f6f641a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 -- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 msgr2=0x7f6f641a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 0x7f6f641a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 -- 172.21.15.45:0/2920360417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60009560 con 0x7f6f641088f0 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 0x7f6f641a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f69b6c640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00ebc0 tx=0x7f6f5c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f5affd640 1 -- 172.21.15.45:0/2920360417 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c010430 con 0x7f6f641088f0 2024-03-12T05:17:26.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f5affd640 1 -- 172.21.15.45:0/2920360417 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f5c010a40 con 0x7f6f641088f0 2024-03-12T05:17:26.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.104+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f641a5280 con 0x7f6f641088f0 2024-03-12T05:17:26.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.105+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f641acb00 con 0x7f6f641088f0 2024-03-12T05:17:26.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.105+0000 7f6f5affd640 1 -- 172.21.15.45:0/2920360417 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c018a20 con 0x7f6f641088f0 2024-03-12T05:17:26.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 Processor -- start 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 -- start start 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8abb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8abdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8a20f90 con 0x55dca8ab2380 2024-03-12T05:17:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8a1e4c0 con 0x55dca8abb8b0 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.108+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8a19ca0 con 0x55dca8ab2d30 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50560/0 (socket says 172.21.15.45:50560) 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 -- 172.21.15.45:0/2914454855 learned_addr learned my addr 172.21.15.45:0/2914454855 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6936b640 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8abdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f5affd640 1 -- 172.21.15.45:0/2920360417 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6f5c018b80 con 0x7f6f641088f0 2024-03-12T05:17:26.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 -- 172.21.15.45:0/2914454855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 msgr2=0x55dca8abb2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:26.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8abb2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 -- 172.21.15.45:0/2914454855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 msgr2=0x55dca8abdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8abdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.109+0000 7f6f6a36d640 1 -- 172.21.15.45:0/2914454855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dca89dc0b0 con 0x55dca8ab2380 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.110+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009900 tx=0x7f6f600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b59c4c5d7cbce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.110+0000 7f6f4ffff640 1 -- 172.21.15.45:0/2914454855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6002dea0 con 0x55dca8ab2380 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.111+0000 7f6f4ffff640 1 -- 172.21.15.45:0/2914454855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f60004ab0 con 0x55dca8ab2380 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.111+0000 7f6f4ffff640 1 -- 172.21.15.45:0/2914454855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f600039b0 con 0x55dca8ab2380 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.111+0000 7f6f5affd640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 0x7f6f500a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.112+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/2914454855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 msgr2=0x55dca8ab2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.112+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009900 tx=0x7f6f600046c0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.112+0000 7f6f69b6c640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 0x7f6f500a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f69b6c640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 0x7f6f500a40e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55dca8ab1e50 tx=0x7f6f5c096e60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f5affd640 1 -- 172.21.15.45:0/2920360417 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6f5c014070 con 0x7f6f641088f0 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/2914454855 shutdown_connections 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8abdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8abb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.113+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/2914454855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab2760 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.114+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/2914454855 >> 172.21.15.45:0/2914454855 conn(0x55dca8aac840 msgr2=0x55dca8aacc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.114+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/2914454855 shutdown_connections 2024-03-12T05:17:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.114+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/2914454855 wait complete. 2024-03-12T05:17:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.115+0000 7f6f6e9a1740 1 -- start start 2024-03-12T05:17:26.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.115+0000 7f6f6e9a1740 1 Processor -- start 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.115+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.115+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.115+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8ab51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca89bef40 con 0x55dca8ab2380 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca86ed4b0 con 0x55dca8abb8b0 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca899f670 con 0x55dca8ab2d30 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50570/0 (socket says 172.21.15.45:50570) 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3584849570 learned_addr learned my addr 172.21.15.45:0/3584849570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6e9a1740 10 client.0 _ll_register_callbacks cb 0x55dca8a0cf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:26.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.116+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8ab51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3584849570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 msgr2=0x55dca8ab4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 msgr2=0x55dca8ab51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8ab51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3584849570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6410efd0 con 0x55dca8ab2380 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6f6002bc80 tx=0x7f6f60003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80098f70e72c56e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.117+0000 7f6f4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60005480 con 0x55dca8ab2380 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dca89f3f40 con 0x55dca8ab2380 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6f600055e0 con 0x55dca8ab2380 2024-03-12T05:17:26.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f60034440 con 0x55dca8ab2380 2024-03-12T05:17:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f6e9a1740 10 client.17247 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dca8a18320 con 0x55dca8ab2380 2024-03-12T05:17:26.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.118+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dca89f3f40 con 0x55dca8ab2380 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f6003c040 con 0x55dca8ab2380 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 10 client.17247 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 10 client.17247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 20 client.17247 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8ad41e0 0x55dca8ad65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 4 client.17247 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dca8a10310 con 0x55dca8ad41e0 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.119+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8ad41e0 0x55dca8ad65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.120+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8ad41e0 0x55dca8ad65d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479d90d154a81f78 server_cookie=5c12aac82ace2eb in_seq=0 out_seq=0 2024-03-12T05:17:26.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.120+0000 7f6f4d7fa640 10 client.17247 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.121+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3584849570 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dca8a10310 con 0x55dca8ad41e0 2024-03-12T05:17:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.121+0000 7f6f4d7fa640 10 client.17247 handle_command_reply: tid=0 2024-03-12T05:17:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.121+0000 7f6f6e9a1740 1 client.17247 shutdown 2024-03-12T05:17:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.121+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 msgr2=0x55dca8ab4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.121+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6f6002bc80 tx=0x7f6f60003690 comp rx=0 tx=0).stop 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 shutdown_connections 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8ad41e0 0x55dca8ad65d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8abb8b0 0x55dca8ab51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3584849570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2380 0x55dca8ab4740 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f4d7fa640 0 client.17247 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.122+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 >> 172.21.15.45:0/3584849570 conn(0x55dca8aac840 msgr2=0x55dca8aae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.123+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 shutdown_connections 2024-03-12T05:17:26.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.123+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3584849570 wait complete. 2024-03-12T05:17:26.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.124+0000 7f6f6e9a1740 20 client.17247 trim_cache size 0 max 16384 2024-03-12T05:17:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 Processor -- start 2024-03-12T05:17:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 -- start start 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 0x55dca8abae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8abb8b0 0x55dca8aab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8aac370 con 0x55dca8ab2d30 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8ac2790 con 0x55dca8ab2380 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6e9a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8abf880 con 0x55dca8abb8b0 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6936b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6936b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50580/0 (socket says 172.21.15.45:50580) 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6936b640 1 -- 172.21.15.45:0/1048748172 learned_addr learned my addr 172.21.15.45:0/1048748172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.164+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 0x55dca8abae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8abb8b0 0x55dca8aab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 -- 172.21.15.45:0/1048748172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8abb8b0 msgr2=0x55dca8aab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8abb8b0 0x55dca8aab760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 -- 172.21.15.45:0/1048748172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 msgr2=0x55dca8ab4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 -- 172.21.15.45:0/1048748172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dca89b4c00 con 0x55dca8ab2380 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6936b640 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:26.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.165+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 0x55dca8abae90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6f60005ec0 tx=0x7f6f600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dec8529ee83bef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.166+0000 7f6f4f7fe640 1 -- 172.21.15.45:0/1048748172 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6003e930 con 0x55dca8ab2380 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.166+0000 7f6f4f7fe640 1 -- 172.21.15.45:0/1048748172 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6f6002ce90 con 0x55dca8ab2380 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.166+0000 7f6f4f7fe640 1 -- 172.21.15.45:0/1048748172 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60034680 con 0x55dca8ab2380 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/1048748172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 msgr2=0x55dca8abae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 0x55dca8abae90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6f60005ec0 tx=0x7f6f600046c0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/1048748172 shutdown_connections 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8ab2d30 0x55dca8ab4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8abb8b0 0x55dca8aab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/1048748172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2380 0x55dca8abae90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.167+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/1048748172 >> 172.21.15.45:0/1048748172 conn(0x55dca8aac840 msgr2=0x55dca8aab050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.168+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/1048748172 shutdown_connections 2024-03-12T05:17:26.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.169+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/1048748172 wait complete. 2024-03-12T05:17:26.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.169+0000 7f6f6e9a1740 1 -- start start 2024-03-12T05:17:26.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.170+0000 7f6f6e9a1740 1 Processor -- start 2024-03-12T05:17:26.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.171+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.171+0000 7f6f6e9a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2d30 0x55dca8ace010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.171+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6a36d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:38468/0 (socket says 172.21.15.45:38468) 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3971682765 learned_addr learned my addr 172.21.15.45:0/3971682765 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8abb8b0 0x55dca8b2a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8ace7b0 con 0x55dca8abb8b0 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8b2ac40 con 0x55dca8ab2d30 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dca8b2ae00 con 0x55dca8ab2380 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6e9a1740 10 client.0 _ll_register_callbacks cb 0x7f6f5c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:26.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.172+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2d30 0x55dca8ace010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2d30 msgr2=0x55dca8ace010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2d30 0x55dca8ace010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3971682765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8abb8b0 msgr2=0x55dca8b2a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8abb8b0 0x55dca8b2a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5400a9f0 con 0x55dca8ab2380 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6a36d640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f6003e800 tx=0x7f6f6003e640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6abfb5a3f55f2326 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60005480 con 0x55dca8ab2380 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6f600055e0 con 0x55dca8ab2380 2024-03-12T05:17:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.173+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dca8a3f900 con 0x55dca8ab2380 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.174+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f60060440 con 0x55dca8ab2380 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.174+0000 7f6f6e9a1740 10 client.26704 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.174+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dca8a18320 con 0x55dca8ab2380 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.174+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dca8a3f900 con 0x55dca8ab2380 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f60034440 con 0x55dca8ab2380 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 10 client.26704 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 10 client.26704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:26.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 20 client.26704 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8b2c180 0x55dca8b2e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 4 client.26704 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55dca8719860 con 0x55dca8b2c180 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.175+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8b2c180 0x55dca8b2e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.176+0000 7f6f6936b640 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8b2c180 0x55dca8b2e570 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84243b29bcc1a995 server_cookie=cf6d8a9083956487 in_seq=0 out_seq=0 2024-03-12T05:17:26.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.176+0000 7f6f4d7fa640 10 client.26704 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f4d7fa640 1 -- 172.21.15.45:0/3971682765 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55dca8719860 con 0x55dca8b2c180 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f4d7fa640 10 client.26704 handle_command_reply: tid=0 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 client.26704 shutdown 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 msgr2=0x55dca8acdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6f6003e800 tx=0x7f6f6003e640 comp rx=0 tx=0).stop 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 shutdown_connections 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dca8b2c180 0x55dca8b2e570 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dca8abb8b0 0x55dca8b2a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.177+0000 7f6f4d7fa640 0 client.26704 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dca8ab2d30 0x55dca8ace010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 1 --2- 172.21.15.45:0/3971682765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dca8ab2380 0x55dca8acdad0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 >> 172.21.15.45:0/3971682765 conn(0x55dca8aac840 msgr2=0x55dca8aae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 shutdown_connections 2024-03-12T05:17:26.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 1 -- 172.21.15.45:0/3971682765 wait complete. 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.178+0000 7f6f6e9a1740 20 client.26704 trim_cache size 0 max 16384 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "f21ff1ac-60e2-46de-afd0-687ea39da1cb", 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:17:26.175 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 msgr2=0x7f6f500a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 0x7f6f500a40e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55dca8ab1e50 tx=0x7f6f5c096e60 comp rx=0 tx=0).stop 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 msgr2=0x7f6f6419fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00ebc0 tx=0x7f6f5c00c2d0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 shutdown_connections 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6f500a1c20 0x7f6f500a40e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f6410ded0 0x7f6f641a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f64108cd0 0x7f6f641a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 --2- 172.21.15.45:0/2920360417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f641088f0 0x7f6f6419fdb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.179+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 >> 172.21.15.45:0/2920360417 conn(0x7f6f6406c260 msgr2=0x7f6f64112e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.180+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 shutdown_connections 2024-03-12T05:17:26.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.180+0000 7f6f6bdd0640 1 -- 172.21.15.45:0/2920360417 wait complete. 2024-03-12T05:17:26.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:17:26.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 -- 172.21.15.45:0/380355857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 msgr2=0x7f66e010fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 --2- 172.21.15.45:0/380355857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e010fec0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009ca0 tx=0x7f66d802c100 comp rx=0 tx=0).stop 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 -- 172.21.15.45:0/380355857 shutdown_connections 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 --2- 172.21.15.45:0/380355857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e010fec0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 --2- 172.21.15.45:0/380355857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e010d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 --2- 172.21.15.45:0/380355857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.376+0000 7f66e75a1640 1 -- 172.21.15.45:0/380355857 >> 172.21.15.45:0/380355857 conn(0x7f66e0077230 msgr2=0x7f66e0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.377+0000 7f66e75a1640 1 -- 172.21.15.45:0/380355857 shutdown_connections 2024-03-12T05:17:26.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.377+0000 7f66e75a1640 1 -- 172.21.15.45:0/380355857 wait complete. 2024-03-12T05:17:26.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.378+0000 7f66e75a1640 1 Processor -- start 2024-03-12T05:17:26.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.379+0000 7f66e75a1640 1 -- start start 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e0102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e01066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e0106cb0 con 0x7f66e00fff20 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e0106e20 con 0x7f66e0100840 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e75a1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e0106fe0 con 0x7f66e010dad0 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.380+0000 7f66e533d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50598/0 (socket says 172.21.15.45:50598) 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 -- 172.21.15.45:0/2789105549 learned_addr learned my addr 172.21.15.45:0/2789105549 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e01066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e0102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 -- 172.21.15.45:0/2789105549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 msgr2=0x7f66e01066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e01066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 -- 172.21.15.45:0/2789105549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 msgr2=0x7f66e0102850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e0102850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 -- 172.21.15.45:0/2789105549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8009900 con 0x7f66e00fff20 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e01066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:26.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.381+0000 7f66e533d640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00e9f0 tx=0x7f66dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.382+0000 7f66d67fc640 1 -- 172.21.15.45:0/2789105549 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc009720 con 0x7f66e00fff20 2024-03-12T05:17:26.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.382+0000 7f66d67fc640 1 -- 172.21.15.45:0/2789105549 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66dc00cd80 con 0x7f66e00fff20 2024-03-12T05:17:26.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.382+0000 7f66d67fc640 1 -- 172.21.15.45:0/2789105549 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc010430 con 0x7f66e00fff20 2024-03-12T05:17:26.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.382+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e01072c0 con 0x7f66e00fff20 2024-03-12T05:17:26.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.382+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e010cb70 con 0x7f66e00fff20 2024-03-12T05:17:26.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66d67fc640 1 -- 172.21.15.45:0/2789105549 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f66dc0105f0 con 0x7f66e00fff20 2024-03-12T05:17:26.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66ea179740 1 Processor -- start 2024-03-12T05:17:26.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66ea179740 1 -- start start 2024-03-12T05:17:26.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66ea179740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66d67fc640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 0x7f66cc0a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.386+0000 7f66ea179740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66ea179740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff5eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66ea179740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2fec2280 con 0x55ba2ff5c7b0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66ea179740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2febf7b0 con 0x55ba2ff532b0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66ea179740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2febaf90 con 0x55ba2ff53bd0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44794/0 (socket says 172.21.15.45:44794) 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 -- 172.21.15.45:0/3059791304 learned_addr learned my addr 172.21.15.45:0/3059791304 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66d67fc640 1 -- 172.21.15.45:0/2789105549 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f66dc014070 con 0x7f66e00fff20 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e533d640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 0x7f66cc0a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff5eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 -- 172.21.15.45:0/3059791304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 msgr2=0x55ba2ff5c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 -- 172.21.15.45:0/3059791304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 msgr2=0x55ba2ff5eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff5eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.387+0000 7f66e5b3e640 1 -- 172.21.15.45:0/3059791304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba2fe79570 con 0x55ba2ff532b0 2024-03-12T05:17:26.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.388+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66d8005fe0 tx=0x7f66d8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4e2c95cf6c634b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.388+0000 7f66e533d640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 0x7f66cc0a41a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f66dc096c40 tx=0x7f66dc0049c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.388+0000 7f66cb7fe640 1 -- 172.21.15.45:0/3059791304 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8034640 con 0x55ba2ff532b0 2024-03-12T05:17:26.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.388+0000 7f66cb7fe640 1 -- 172.21.15.45:0/3059791304 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d8032aa0 con 0x55ba2ff532b0 2024-03-12T05:17:26.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.388+0000 7f66cb7fe640 1 -- 172.21.15.45:0/3059791304 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8004290 con 0x55ba2ff532b0 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 -- 172.21.15.45:0/3059791304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 msgr2=0x55ba2ff53690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66d8005fe0 tx=0x7f66d8005ef0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 -- 172.21.15.45:0/3059791304 shutdown_connections 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff5eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 --2- 172.21.15.45:0/3059791304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff53690 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 -- 172.21.15.45:0/3059791304 >> 172.21.15.45:0/3059791304 conn(0x55ba2ff4d990 msgr2=0x55ba2ff4dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.389+0000 7f66ea179740 1 -- 172.21.15.45:0/3059791304 shutdown_connections 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.390+0000 7f66ea179740 1 -- 172.21.15.45:0/3059791304 wait complete. 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.390+0000 7f66ea179740 1 -- start start 2024-03-12T05:17:26.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.390+0000 7f66ea179740 1 Processor -- start 2024-03-12T05:17:26.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff59cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44810/0 (socket says 172.21.15.45:44810) 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2fb8e430 con 0x55ba2ff53bd0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66e5b3e640 1 -- 172.21.15.45:0/2921626127 learned_addr learned my addr 172.21.15.45:0/2921626127 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2fb4ae00 con 0x55ba2ff532b0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2fe3ffd0 con 0x55ba2ff5c7b0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66ea179740 10 client.0 _ll_register_callbacks cb 0x55ba2fe95fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.391+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff59cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 -- 172.21.15.45:0/2921626127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 msgr2=0x55ba2ff59cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff59cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 -- 172.21.15.45:0/2921626127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff53bd0 msgr2=0x55ba2ff59570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e0110a20 con 0x55ba2ff532b0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f66d8003d60 tx=0x7f66d8003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39dfd44c44574247 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.392+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003a80 con 0x55ba2ff532b0 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f66d8034650 con 0x55ba2ff532b0 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba2fbba7e0 con 0x55ba2ff532b0 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d8034810 con 0x55ba2ff532b0 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66ea179740 10 client.26738 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:26.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ba2feb9610 con 0x55ba2ff532b0 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d8060080 con 0x55ba2ff532b0 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66d803e040 con 0x55ba2ff532b0 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66ea179740 10 client.26738 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.393+0000 7f66ea179740 10 client.26738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.394+0000 7f66ea179740 20 client.26738 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.394+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ff6c540 0x55ba2ff6e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.394+0000 7f66ea179740 4 client.26738 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.394+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba2fbf7720 con 0x55ba2ff6c540 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.394+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ff6c540 0x55ba2ff6e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.395+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ff6c540 0x55ba2ff6e930 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ca7497e02f8701e server_cookie=aa91395c9e57a663 in_seq=0 out_seq=0 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.395+0000 7f66c8ff9640 10 client.26738 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66c8ff9640 1 -- 172.21.15.45:0/2921626127 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba2fbf7720 con 0x55ba2ff6c540 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66c8ff9640 10 client.26738 handle_command_reply: tid=0 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 client.26738 shutdown 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 msgr2=0x55ba2ff5c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f66d8003d60 tx=0x7f66d8003c60 comp rx=0 tx=0).stop 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 shutdown_connections 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ff6c540 0x55ba2ff6e930 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff59570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff59cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 --2- 172.21.15.45:0/2921626127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5c1e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 >> 172.21.15.45:0/2921626127 conn(0x55ba2ff4d990 msgr2=0x55ba2ff4f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66c8ff9640 0 client.26738 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 shutdown_connections 2024-03-12T05:17:26.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 1 -- 172.21.15.45:0/2921626127 wait complete. 2024-03-12T05:17:26.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.396+0000 7f66ea179740 20 client.26738 trim_cache size 0 max 16384 2024-03-12T05:17:26.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.423+0000 7f66ea179740 1 Processor -- start 2024-03-12T05:17:26.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.423+0000 7f66ea179740 1 -- start start 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.423+0000 7f66ea179740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66ea179740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66ea179740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff57e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66ea179740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ff4e7e0 con 0x55ba2ff5c7b0 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66ea179740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ff52f10 con 0x55ba2ff532b0 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66ea179740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ff492b0 con 0x55ba2ff53bd0 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:44812/0 (socket says 172.21.15.45:44812) 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 -- 172.21.15.45:0/789137947 learned_addr learned my addr 172.21.15.45:0/789137947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff57e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 -- 172.21.15.45:0/789137947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 msgr2=0x55ba2ff5c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 -- 172.21.15.45:0/789137947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 msgr2=0x55ba2ff57e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff57e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.424+0000 7f66e5b3e640 1 -- 172.21.15.45:0/789137947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba2fe55ac0 con 0x55ba2ff532b0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.425+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66d8032d30 tx=0x7f66d802c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4803440c7cf2421 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.425+0000 7f66caffd640 1 -- 172.21.15.45:0/789137947 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003a80 con 0x55ba2ff532b0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.425+0000 7f66caffd640 1 -- 172.21.15.45:0/789137947 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ba2fbba7e0 con 0x55ba2ff532b0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.425+0000 7f66caffd640 1 -- 172.21.15.45:0/789137947 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003d90 con 0x55ba2ff532b0 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.426+0000 7f66ea179740 1 -- 172.21.15.45:0/789137947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 msgr2=0x55ba2ff5bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.426+0000 7f66ea179740 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66d8032d30 tx=0x7f66d802c100 comp rx=0 tx=0).stop 2024-03-12T05:17:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 -- 172.21.15.45:0/789137947 shutdown_connections 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff57e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff5c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 --2- 172.21.15.45:0/789137947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff532b0 0x55ba2ff5bca0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 -- 172.21.15.45:0/789137947 >> 172.21.15.45:0/789137947 conn(0x55ba2ff4d990 msgr2=0x55ba2ff54810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 -- 172.21.15.45:0/789137947 shutdown_connections 2024-03-12T05:17:26.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.427+0000 7f66ea179740 1 -- 172.21.15.45:0/789137947 wait complete. 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.428+0000 7f66ea179740 1 -- start start 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.428+0000 7f66ea179740 1 Processor -- start 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.429+0000 7f66ea179740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.429+0000 7f66ea179740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff55fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.429+0000 7f66ea179740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff56550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.429+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.429+0000 7f66e5b3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50634/0 (socket says 172.21.15.45:50634) 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff55fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e5b3e640 1 -- 172.21.15.45:0/3506042875 learned_addr learned my addr 172.21.15.45:0/3506042875 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ff56c50 con 0x55ba2ff532b0 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ff56dc0 con 0x55ba2ff53bd0 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba2ffc2a10 con 0x55ba2ff5c7b0 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66ea179740 10 client.0 _ll_register_callbacks cb 0x7f66dc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 -- 172.21.15.45:0/3506042875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 msgr2=0x55ba2ff56550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff56550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 -- 172.21.15.45:0/3506042875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 msgr2=0x55ba2ff63ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e4b3c640 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba2feb9610 con 0x55ba2ff53bd0 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.430+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:26.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66e4b3c640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff55fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55ba2ff6f210 tx=0x7f66d0005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2e28b6d02dd6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d000f830 con 0x55ba2ff53bd0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f66d0015040 con 0x55ba2ff53bd0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66d8032aa0 con 0x55ba2ff53bd0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d000f990 con 0x55ba2ff53bd0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66ea179740 10 client.26744 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ba2fe7b6a0 con 0x55ba2ff53bd0 2024-03-12T05:17:26.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.431+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f66d002d020 con 0x55ba2ff53bd0 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66d0028bb0 con 0x55ba2ff53bd0 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 10 client.26744 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 20 client.26744 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ffc41c0 0x55ba2ffc65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 4 client.26744 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:17:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.432+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ba2fc46890 con 0x55ba2ffc41c0 2024-03-12T05:17:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.433+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ffc41c0 0x55ba2ffc65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:26.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.434+0000 7f66e5b3e640 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ffc41c0 0x55ba2ffc65b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dcf4ddd32c53ac2 server_cookie=5c83a5c71be467aa in_seq=0 out_seq=0 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.434+0000 7f66c8ff9640 10 client.26744 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.434+0000 7f66c8ff9640 1 -- 172.21.15.45:0/3506042875 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ba2fc46890 con 0x55ba2ffc41c0 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.434+0000 7f66c8ff9640 10 client.26744 handle_command_reply: tid=0 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.434+0000 7f66ea179740 1 client.26744 shutdown 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 msgr2=0x55ba2ff55fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff55fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55ba2ff6f210 tx=0x7f66d0005d10 comp rx=0 tx=0).stop 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 shutdown_connections 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ba2ffc41c0 0x55ba2ffc65b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba2ff5c7b0 0x55ba2ff56550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba2ff53bd0 0x55ba2ff55fb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.435+0000 7f66ea179740 1 --2- 172.21.15.45:0/3506042875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ba2ff532b0 0x55ba2ff63ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.436+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 >> 172.21.15.45:0/3506042875 conn(0x55ba2ff4d990 msgr2=0x55ba2ff58e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.436+0000 7f66c8ff9640 0 client.26744 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.436+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 shutdown_connections 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.436+0000 7f66ea179740 1 -- 172.21.15.45:0/3506042875 wait complete. 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.436+0000 7f66ea179740 20 client.26744 trim_cache size 0 max 16384 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stdout: "f21ff1ac-60e2-46de-afd0-687ea39da1cb": { 2024-03-12T05:17:26.434 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:17:26.435 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "f21ff1ac-60e2-46de-afd0-687ea39da1cb", 2024-03-12T05:17:26.435 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:17:26.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:17:26.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:17:26.435 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.437+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 msgr2=0x7f66cc0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 0x7f66cc0a41a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f66dc096c40 tx=0x7f66dc0049c0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 msgr2=0x7f66e0102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00e9f0 tx=0x7f66dc00edf0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 shutdown_connections 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66cc0a1ce0 0x7f66cc0a41a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66e010dad0 0x7f66e01066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66e0100840 0x7f66e0102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 --2- 172.21.15.45:0/2789105549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f66e00fff20 0x7f66e0102310 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 >> 172.21.15.45:0/2789105549 conn(0x7f66e0077230 msgr2=0x7f66e010efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 shutdown_connections 2024-03-12T05:17:26.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:26.438+0000 7f66e75a1640 1 -- 172.21.15.45:0/2789105549 wait complete. 2024-03-12T05:17:26.456 INFO:tasks.cephfs.filesystem:scrub status for tag:f21ff1ac-60e2-46de-afd0-687ea39da1cb - {'path': '/', 'tag': 'f21ff1ac-60e2-46de-afd0-687ea39da1cb', 'options': 'recursive,force'} 2024-03-12T05:17:26.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:26 smithi045 ceph-mon[32174]: pgmap v201: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:26.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:26 smithi130 ceph-mon[35398]: pgmap v201: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:26.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:26 smithi116 ceph-mon[35486]: pgmap v201: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:27.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:17:27.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:17:27.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:17:27.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:27.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:17:28.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:28 smithi130 ceph-mon[35398]: pgmap v202: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:28.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:28 smithi116 ceph-mon[35486]: pgmap v202: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:28.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:28 smithi045 ceph-mon[32174]: pgmap v202: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 17 MiB/s wr, 1.53k op/s 2024-03-12T05:17:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 325.0 s, 1394.4 tps, lat 71.757 ms stddev 20.272 2024-03-12T05:17:30.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:30 smithi130 ceph-mon[35398]: pgmap v203: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-12T05:17:30.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:30 smithi116 ceph-mon[35486]: pgmap v203: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-12T05:17:30.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:30 smithi045 ceph-mon[32174]: pgmap v203: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-12T05:17:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:32 smithi130 ceph-mon[35398]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 14 MiB/s wr, 1.27k op/s 2024-03-12T05:17:32.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:32 smithi116 ceph-mon[35486]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 14 MiB/s wr, 1.27k op/s 2024-03-12T05:17:32.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:32 smithi045 ceph-mon[32174]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 14 MiB/s wr, 1.27k op/s 2024-03-12T05:17:34.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:34 smithi130 ceph-mon[35398]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-03-12T05:17:34.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:34 smithi116 ceph-mon[35486]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-03-12T05:17:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:34 smithi045 ceph-mon[32174]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-03-12T05:17:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 330.0 s, 1327.2 tps, lat 75.482 ms stddev 42.353 2024-03-12T05:17:36.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:36 smithi130 ceph-mon[35398]: pgmap v206: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 19 MiB/s wr, 1.78k op/s 2024-03-12T05:17:36.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:36 smithi116 ceph-mon[35486]: pgmap v206: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 19 MiB/s wr, 1.78k op/s 2024-03-12T05:17:36.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:36 smithi045 ceph-mon[32174]: pgmap v206: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 19 MiB/s wr, 1.78k op/s 2024-03-12T05:17:38.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:38 smithi130 ceph-mon[35398]: pgmap v207: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-03-12T05:17:38.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:38.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:38.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:38.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:17:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:38 smithi116 ceph-mon[35486]: pgmap v207: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-03-12T05:17:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:17:38.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:38 smithi045 ceph-mon[32174]: pgmap v207: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 13 MiB/s wr, 1.21k op/s 2024-03-12T05:17:38.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:38.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:38.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:38.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:17:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 335.0 s, 1540.2 tps, lat 64.867 ms stddev 16.734 2024-03-12T05:17:40.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:40 smithi130 ceph-mon[35398]: pgmap v208: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 19 MiB/s wr, 1.87k op/s 2024-03-12T05:17:40.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:40 smithi116 ceph-mon[35486]: pgmap v208: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 19 MiB/s wr, 1.87k op/s 2024-03-12T05:17:40.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:40 smithi045 ceph-mon[32174]: pgmap v208: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 19 MiB/s wr, 1.87k op/s 2024-03-12T05:17:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:42 smithi116 ceph-mon[35486]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:42.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:42 smithi045 ceph-mon[32174]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:42 smithi130 ceph-mon[35398]: pgmap v209: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:44.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:44 smithi116 ceph-mon[35486]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:44.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:44 smithi045 ceph-mon[32174]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 340.0 s, 1383.8 tps, lat 72.318 ms stddev 21.852 2024-03-12T05:17:44.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:44 smithi130 ceph-mon[35398]: pgmap v210: 97 pgs: 97 active+clean; 3.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 13 MiB/s wr, 1.26k op/s 2024-03-12T05:17:46.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:46 smithi116 ceph-mon[35486]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 19 MiB/s wr, 1.89k op/s 2024-03-12T05:17:46.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:46 smithi045 ceph-mon[32174]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 19 MiB/s wr, 1.89k op/s 2024-03-12T05:17:46.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:46 smithi130 ceph-mon[35398]: pgmap v211: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 19 MiB/s wr, 1.89k op/s 2024-03-12T05:17:48.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:48 smithi116 ceph-mon[35486]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:48.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:48 smithi045 ceph-mon[32174]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:48 smithi130 ceph-mon[35398]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 345.0 s, 1517.8 tps, lat 65.790 ms stddev 18.019 2024-03-12T05:17:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:50 smithi116 ceph-mon[35486]: pgmap v213: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:50.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:50 smithi045 ceph-mon[32174]: pgmap v213: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:50 smithi130 ceph-mon[35398]: pgmap v213: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:52.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:52 smithi116 ceph-mon[35486]: pgmap v214: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:52.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:52 smithi045 ceph-mon[32174]: pgmap v214: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:52.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:52 smithi130 ceph-mon[35398]: pgmap v214: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:54.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:54 smithi116 ceph-mon[35486]: pgmap v215: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:54.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:54 smithi045 ceph-mon[32174]: pgmap v215: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 350.0 s, 1427.4 tps, lat 70.073 ms stddev 18.597 2024-03-12T05:17:54.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:54 smithi130 ceph-mon[35398]: pgmap v215: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:17:55.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:55 smithi045 ceph-mon[32174]: pgmap v216: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:55.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:55 smithi130 ceph-mon[35398]: pgmap v216: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:55.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:55 smithi116 ceph-mon[35486]: pgmap v216: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 18 MiB/s wr, 1.90k op/s 2024-03-12T05:17:56.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:17:56.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:17:56.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 -- 172.21.15.45:0/2840773050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc100270 msgr2=0x7f8ddc100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 --2- 172.21.15.45:0/2840773050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc100270 0x7f8ddc100650 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009cd0 tx=0x7f8dcc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 -- 172.21.15.45:0/2840773050 shutdown_connections 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 --2- 172.21.15.45:0/2840773050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc105340 0x7f8ddc10cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 --2- 172.21.15.45:0/2840773050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100c20 0x7f8ddc104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 --2- 172.21.15.45:0/2840773050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc100270 0x7f8ddc100650 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 -- 172.21.15.45:0/2840773050 >> 172.21.15.45:0/2840773050 conn(0x7f8ddc077230 msgr2=0x7f8ddc077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 -- 172.21.15.45:0/2840773050 shutdown_connections 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.597+0000 7f8de124e640 1 -- 172.21.15.45:0/2840773050 wait complete. 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 Processor -- start 2024-03-12T05:17:56.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 -- start start 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100270 0x7f8ddc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc198fe0 0x7f8ddc1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1a3c60 con 0x7f8ddc105340 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1a3dd0 con 0x7f8ddc100270 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8de124e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1a3f90 con 0x7f8ddc198fe0 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.598+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46724/0 (socket says 172.21.15.45:46724) 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 -- 172.21.15.45:0/846403632 learned_addr learned my addr 172.21.15.45:0/846403632 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100270 0x7f8ddc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc198fe0 0x7f8ddc1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 -- 172.21.15.45:0/846403632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc198fe0 msgr2=0x7f8ddc1a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc198fe0 0x7f8ddc1a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 -- 172.21.15.45:0/846403632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100270 msgr2=0x7f8ddc198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100270 0x7f8ddc198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 -- 172.21.15.45:0/846403632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc009900 con 0x7f8ddc105340 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dda59c640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4005fd0 tx=0x7f8dc400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dd3fff640 1 -- 172.21.15.45:0/846403632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4009c30 con 0x7f8ddc105340 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dd3fff640 1 -- 172.21.15.45:0/846403632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8dc40044d0 con 0x7f8ddc105340 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc1a4270 con 0x7f8ddc105340 2024-03-12T05:17:56.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8dd3fff640 1 -- 172.21.15.45:0/846403632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4003e50 con 0x7f8ddc105340 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.599+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc1a4780 con 0x7f8ddc105340 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.601+0000 7f8dd3fff640 1 -- 172.21.15.45:0/846403632 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8dc4004000 con 0x7f8ddc105340 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 Processor -- start 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 -- start start 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8dd3fff640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 0x7f8dc80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 0x7f8dc80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8dd3fff640 1 -- 172.21.15.45:0/846403632 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8dc40cd5c0 con 0x7f8ddc105340 2024-03-12T05:17:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d39270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d397b0 0x561226d3bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226c9f280 con 0x561226d302b0 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226c9c7b0 con 0x561226d30bd0 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226c97f90 con 0x561226d397b0 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46730/0 (socket says 172.21.15.45:46730) 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 -- 172.21.15.45:0/1830995717 learned_addr learned my addr 172.21.15.45:0/1830995717 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d397b0 0x561226d3bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8dda59c640 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d39270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 0x7f8dc80a4060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005fd0 tx=0x7f8dd400d4b0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 -- 172.21.15.45:0/1830995717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d397b0 msgr2=0x561226d3bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d397b0 0x561226d3bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 -- 172.21.15.45:0/1830995717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 msgr2=0x561226d39270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d39270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.602+0000 7f8ddad9d640 1 -- 172.21.15.45:0/1830995717 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561226c56570 con 0x561226d302b0 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000f80 tx=0x7f8dcc0093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afcb9e0a82bdd00c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8dd17fa640 1 -- 172.21.15.45:0/1830995717 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc032c90 con 0x561226d302b0 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8dd17fa640 1 -- 172.21.15.45:0/1830995717 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8dcc032df0 con 0x561226d302b0 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8dd17fa640 1 -- 172.21.15.45:0/1830995717 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc032a90 con 0x561226d302b0 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8de3e1d740 1 -- 172.21.15.45:0/1830995717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 msgr2=0x561226d30690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.603+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc000f80 tx=0x7f8dcc0093e0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- 172.21.15.45:0/1830995717 shutdown_connections 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d397b0 0x561226d3bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d39270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/1830995717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d302b0 0x561226d30690 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- 172.21.15.45:0/1830995717 >> 172.21.15.45:0/1830995717 conn(0x561226d2a990 msgr2=0x561226d2ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- 172.21.15.45:0/1830995717 shutdown_connections 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- 172.21.15.45:0/1830995717 wait complete. 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- start start 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 Processor -- start 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d39240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56122696b430 con 0x561226d397b0 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226927e00 con 0x561226d30bd0 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226c1cfd0 con 0x561226d302b0 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8de3e1d740 10 client.0 _ll_register_callbacks cb 0x561226c72fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8ddb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8ddb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46742/0 (socket says 172.21.15.45:46742) 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8ddb59e640 1 -- 172.21.15.45:0/2513200616 learned_addr learned my addr 172.21.15.45:0/2513200616 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.604+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d39240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dda59c640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 -- 172.21.15.45:0/2513200616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 msgr2=0x561226d39240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d39240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 -- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 msgr2=0x561226d365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 -- 172.21.15.45:0/2513200616 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc10d720 con 0x561226d397b0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dda59c640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4011670 tx=0x7f8dd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596aaa95ee2173ad server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dc27fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd401e900 con 0x561226d397b0 2024-03-12T05:17:56.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8dd401ea60 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5612269977e0 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dd401ec20 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8de3e1d740 10 client.17277 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561226c96610 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.605+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8dd4032020 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8dd4019070 con 0x561226d397b0 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 10 client.17277 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 10 client.17277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 20 client.17277 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d49a50 0x561226d4be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 4 client.17277 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612269d4720 con 0x561226d49a50 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d49a50 0x561226d4be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8dc27fc640 10 client.17277 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.606+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d49a50 0x561226d4be70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31d15418848b8efb server_cookie=b7ece74464d5b127 in_seq=0 out_seq=0 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8dc27fc640 1 -- 172.21.15.45:0/2513200616 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5612269d4720 con 0x561226d49a50 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8dc27fc640 10 client.17277 handle_command_reply: tid=0 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 client.17277 shutdown 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 msgr2=0x561226d36cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4011670 tx=0x7f8dd40046c0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 shutdown_connections 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d49a50 0x561226d4be70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d36cc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/2513200616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d39240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 >> 172.21.15.45:0/2513200616 conn(0x561226d2a990 msgr2=0x561226d2c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8dc27fc640 0 client.17277 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 shutdown_connections 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 1 -- 172.21.15.45:0/2513200616 wait complete. 2024-03-12T05:17:56.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.607+0000 7f8de3e1d740 20 client.17277 trim_cache size 0 max 16384 2024-03-12T05:17:56.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 Processor -- start 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 -- start start 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d38d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 0x561226d39240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 0x561226d34ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d2ba60 con 0x561226d30bd0 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d510b0 con 0x561226d397b0 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.639+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d49830 con 0x561226d302b0 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8ddb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 0x561226d34ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8ddb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 0x561226d34ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38350/0 (socket says 172.21.15.45:38350) 2024-03-12T05:17:56.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8ddb59e640 1 -- 172.21.15.45:0/4137384977 learned_addr learned my addr 172.21.15.45:0/4137384977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 0x561226d39240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d38d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 -- 172.21.15.45:0/4137384977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 msgr2=0x561226d38d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d38d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 -- 172.21.15.45:0/4137384977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 msgr2=0x561226d34ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 0x561226d34ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.640+0000 7f8dda59c640 1 -- 172.21.15.45:0/4137384977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561226c32ac0 con 0x561226d30bd0 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.641+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d38d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.641+0000 7f8dda59c640 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 0x561226d39240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cac80 tx=0x7f8dc4099dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9b8bcabb827df server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.641+0000 7f8dd0ff9640 1 -- 172.21.15.45:0/4137384977 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc409e2b0 con 0x561226d30bd0 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.641+0000 7f8dd0ff9640 1 -- 172.21.15.45:0/4137384977 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8dc40a2900 con 0x561226d30bd0 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.641+0000 7f8dd0ff9640 1 -- 172.21.15.45:0/4137384977 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40967d0 con 0x561226d30bd0 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 -- 172.21.15.45:0/4137384977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 msgr2=0x561226d39240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 0x561226d39240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40cac80 tx=0x7f8dc4099dd0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 -- 172.21.15.45:0/4137384977 shutdown_connections 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d30bd0 0x561226d39240 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d397b0 0x561226d34ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/4137384977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d38d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.642+0000 7f8de3e1d740 1 -- 172.21.15.45:0/4137384977 >> 172.21.15.45:0/4137384977 conn(0x561226d2a990 msgr2=0x561226d318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- 172.21.15.45:0/4137384977 shutdown_connections 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- 172.21.15.45:0/4137384977 wait complete. 2024-03-12T05:17:56.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- start start 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 Processor -- start 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d3d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d98af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d991a0 con 0x561226d397b0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d99310 con 0x561226d30bd0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561226d994d0 con 0x561226d302b0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.643+0000 7f8de3e1d740 10 client.0 _ll_register_callbacks cb 0x7f8dcc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8ddb59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d98af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8ddad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d3d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38362/0 (socket says 172.21.15.45:38362) 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 -- 172.21.15.45:0/647977172 learned_addr learned my addr 172.21.15.45:0/647977172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 -- 172.21.15.45:0/647977172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 msgr2=0x561226d3d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d3d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 -- 172.21.15.45:0/647977172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 msgr2=0x561226d98af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d98af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 -- 172.21.15.45:0/647977172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561226c56570 con 0x561226d30bd0 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8ddb59e640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d98af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dda59c640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4099e60 tx=0x7f8dc4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646676dff474abfb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.644+0000 7f8dc27fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:56.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40967d0 con 0x561226d30bd0 2024-03-12T05:17:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8dc4096930 con 0x561226d30bd0 2024-03-12T05:17:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8dcc032df0 con 0x561226d30bd0 2024-03-12T05:17:56.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dc4022bb0 con 0x561226d30bd0 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 10 client.26762 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f8ddc1a4780 con 0x561226d30bd0 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8dcc032df0 con 0x561226d30bd0 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8dc40b1020 con 0x561226d30bd0 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 10 client.26762 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 20 client.26762 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d40350 0x561226d40760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 4 client.26762 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.645+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561226a23890 con 0x561226d40350 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.646+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d40350 0x561226d40760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.646+0000 7f8dc27fc640 10 client.26762 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:56.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8ddad9d640 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d40350 0x561226d40760 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd0500ad52ce5d71 server_cookie=2301dcfb929f3a68 in_seq=0 out_seq=0 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8dc27fc640 1 -- 172.21.15.45:0/647977172 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561226a23890 con 0x561226d40350 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8dc27fc640 10 client.26762 handle_command_reply: tid=0 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 client.26762 shutdown 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 msgr2=0x561226d3de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4099e60 tx=0x7f8dc4005be0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 shutdown_connections 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561226d40350 0x561226d40760 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561226d397b0 0x561226d98af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561226d30bd0 0x561226d3de60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 --2- 172.21.15.45:0/647977172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561226d302b0 0x561226d3d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 >> 172.21.15.45:0/647977172 conn(0x561226d2a990 msgr2=0x561226d35e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8dc27fc640 0 client.26762 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:56.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 shutdown_connections 2024-03-12T05:17:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.647+0000 7f8de3e1d740 1 -- 172.21.15.45:0/647977172 wait complete. 2024-03-12T05:17:56.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de3e1d740 20 client.26762 trim_cache size 0 max 16384 2024-03-12T05:17:56.648 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 msgr2=0x7f8dc80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 0x7f8dc80a4060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005fd0 tx=0x7f8dd400d4b0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 msgr2=0x7f8ddc198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4005fd0 tx=0x7f8dc400dde0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 shutdown_connections 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8dc80a1ba0 0x7f8dc80a4060 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ddc105340 0x7f8ddc198a50 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ddc198fe0 0x7f8ddc1a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 --2- 172.21.15.45:0/846403632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ddc100270 0x7f8ddc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.648+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 >> 172.21.15.45:0/846403632 conn(0x7f8ddc077230 msgr2=0x7f8ddc0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.649+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 shutdown_connections 2024-03-12T05:17:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.649+0000 7f8de124e640 1 -- 172.21.15.45:0/846403632 wait complete. 2024-03-12T05:17:56.663 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:17:56.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:17:56.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 -- 172.21.15.45:0/4142815864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108b40 msgr2=0x7fc9dc114a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/4142815864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108b40 0x7fc9dc114a10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4013400 tx=0x7fc9d4034260 comp rx=0 tx=0).stop 2024-03-12T05:17:56.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 -- 172.21.15.45:0/4142815864 shutdown_connections 2024-03-12T05:17:56.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/4142815864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9dc108180 0x7fc9dc108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/4142815864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108b40 0x7fc9dc114a10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/4142815864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc10cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 -- 172.21.15.45:0/4142815864 >> 172.21.15.45:0/4142815864 conn(0x7fc9dc06c6d0 msgr2=0x7fc9dc06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.843+0000 7fc9e0e64640 1 -- 172.21.15.45:0/4142815864 shutdown_connections 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.844+0000 7fc9e0e64640 1 -- 172.21.15.45:0/4142815864 wait complete. 2024-03-12T05:17:56.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.845+0000 7fc9e0e64640 1 Processor -- start 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.845+0000 7fc9e0e64640 1 -- start start 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108180 0x7fc9dc1b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9dc108b40 0x7fc9dc1b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc1aef10 con 0x7fc9dc108b40 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc1af080 con 0x7fc9dc10cb80 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9e0e64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc1af240 con 0x7fc9dc108180 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9dad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9dad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:38378/0 (socket says 172.21.15.45:38378) 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9dad9d640 1 -- 172.21.15.45:0/2384027564 learned_addr learned my addr 172.21.15.45:0/2384027564 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.846+0000 7fc9da59c640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108180 0x7fc9dc1b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 -- 172.21.15.45:0/2384027564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108180 msgr2=0x7fc9dc1b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108180 0x7fc9dc1b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 -- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9dc108b40 msgr2=0x7fc9dc1b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9dc108b40 0x7fc9dc1b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 -- 172.21.15.45:0/2384027564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d40130b0 con 0x7fc9dc10cb80 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9dad9d640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4034870 tx=0x7fc9d40048a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d400fea0 con 0x7fc9dc10cb80 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9d40117d0 con 0x7fc9dc10cb80 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.847+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc1af4c0 con 0x7fc9dc10cb80 2024-03-12T05:17:56.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.848+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc1bb7b0 con 0x7fc9dc10cb80 2024-03-12T05:17:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.848+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d403dd90 con 0x7fc9dc10cb80 2024-03-12T05:17:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.850+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc005290 con 0x7fc9dc10cb80 2024-03-12T05:17:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.851+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc9d403d560 con 0x7fc9dc10cb80 2024-03-12T05:17:56.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.852+0000 7fc9cb7fe640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 0x7fc9c40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:56.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.853+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc9d40ef300 con 0x7fc9dc10cb80 2024-03-12T05:17:56.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.853+0000 7fc9da59c640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 0x7fc9c40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:56.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.854+0000 7fc9da59c640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 0x7fc9c40a4160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc1afa00 tx=0x7fc9d0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:56.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:56.857+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc9d40b8250 con 0x7fc9dc10cb80 2024-03-12T05:17:57.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.276+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9bc005810 con 0x7fc9dc10cb80 2024-03-12T05:17:57.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.277+0000 7fc9cb7fe640 1 -- 172.21.15.45:0/2384027564 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fc9d40bb910 con 0x7fc9dc10cb80 2024-03-12T05:17:57.273 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:17:57.273 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:17:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:17:57.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 msgr2=0x7fc9c40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 0x7fc9c40a4160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc1afa00 tx=0x7fc9d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:17:57.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 msgr2=0x7fc9dc1ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4034870 tx=0x7fc9d40048a0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 shutdown_connections 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9c40a1ca0 0x7fc9c40a4160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9dc108b40 0x7fc9dc1b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9dc10cb80 0x7fc9dc1ae8c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 --2- 172.21.15.45:0/2384027564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9dc108180 0x7fc9dc1b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 >> 172.21.15.45:0/2384027564 conn(0x7fc9dc06c6d0 msgr2=0x7fc9dc070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 shutdown_connections 2024-03-12T05:17:57.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.281+0000 7fc9e0e64640 1 -- 172.21.15.45:0/2384027564 wait complete. 2024-03-12T05:17:57.295 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:57.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:17:57.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2488324008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 msgr2=0x7f60bc104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2488324008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc104f50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009c10 tx=0x7f60ac02bfc0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2488324008 shutdown_connections 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2488324008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2488324008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc104f50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2488324008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2488324008 >> 172.21.15.45:0/2488324008 conn(0x7f60bc076e30 msgr2=0x7f60bc0fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.467+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2488324008 shutdown_connections 2024-03-12T05:17:57.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.468+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2488324008 wait complete. 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 Processor -- start 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 -- start start 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc1a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc19e140 con 0x7f60bc100170 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc19e2b0 con 0x7f60bc100a90 2024-03-12T05:17:57.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60c0f6c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc19e470 con 0x7f60bc105570 2024-03-12T05:17:57.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.469+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34136/0 (socket says 172.21.15.45:34136) 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60ba59c640 1 -- 172.21.15.45:0/2578836005 learned_addr learned my addr 172.21.15.45:0/2578836005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc1a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 -- 172.21.15.45:0/2578836005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 msgr2=0x7f60bc1a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc1a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 -- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 msgr2=0x7f60bc1a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60bad9d640 1 -- 172.21.15.45:0/2578836005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009840 con 0x7f60bc105570 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.470+0000 7f60ba59c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc19daf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009fd0 tx=0x7f60b000eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60ab7fe640 1 -- 172.21.15.45:0/2578836005 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0004220 con 0x7f60bc105570 2024-03-12T05:17:57.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc19e750 con 0x7f60bc105570 2024-03-12T05:17:57.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc1aa9e0 con 0x7f60bc105570 2024-03-12T05:17:57.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60ab7fe640 1 -- 172.21.15.45:0/2578836005 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f60b000cd60 con 0x7f60bc105570 2024-03-12T05:17:57.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.471+0000 7f60ab7fe640 1 -- 172.21.15.45:0/2578836005 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00047a0 con 0x7f60bc105570 2024-03-12T05:17:57.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.475+0000 7f60c3c19740 1 Processor -- start 2024-03-12T05:17:57.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60ab7fe640 1 -- 172.21.15.45:0/2578836005 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f60b000c8b0 con 0x7f60bc105570 2024-03-12T05:17:57.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 -- start start 2024-03-12T05:17:57.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f486630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f491b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f441960 con 0x558c1f486250 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f3f5310 con 0x558c1f48f750 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60c3c19740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f3f2840 con 0x558c1f486b70 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60b9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f486630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51082/0 (socket says 172.21.15.45:51082) 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60ba59c640 1 -- 172.21.15.45:0/366198577 learned_addr learned my addr 172.21.15.45:0/366198577 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.476+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f486630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34144/0 (socket says 172.21.15.45:34144) 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f491b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 -- 172.21.15.45:0/366198577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 msgr2=0x558c1f491b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f491b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 -- 172.21.15.45:0/366198577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 msgr2=0x558c1f486630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f486630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 -- 172.21.15.45:0/366198577 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c1f3b0060 con 0x558c1f486b70 2024-03-12T05:17:57.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ab7fe640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 0x7f60a40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ab7fe640 1 -- 172.21.15.45:0/2578836005 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f60b0014070 con 0x7f60bc105570 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60ba59c640 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011650 tx=0x7f60b4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1581b1db1580d204 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.477+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 0x7f60a40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.478+0000 7f60a8ff9640 1 -- 172.21.15.45:0/366198577 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b400f6c0 con 0x558c1f486b70 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.478+0000 7f60a8ff9640 1 -- 172.21.15.45:0/366198577 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f60b400fcd0 con 0x558c1f486b70 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.478+0000 7f60a8ff9640 1 -- 172.21.15.45:0/366198577 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4019d60 con 0x558c1f486b70 2024-03-12T05:17:57.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 -- 172.21.15.45:0/366198577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 msgr2=0x558c1f48f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 0x7f60a40a4160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f60ac034fd0 tx=0x7f60ac000f40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f60b4011650 tx=0x7f60b4009f90 comp rx=0 tx=0).stop 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 -- 172.21.15.45:0/366198577 shutdown_connections 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f491b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f48f210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 --2- 172.21.15.45:0/366198577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f486630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 -- 172.21.15.45:0/366198577 >> 172.21.15.45:0/366198577 conn(0x558c1f480980 msgr2=0x558c1f480d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 -- 172.21.15.45:0/366198577 shutdown_connections 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.479+0000 7f60c3c19740 1 -- 172.21.15.45:0/366198577 wait complete. 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 -- start start 2024-03-12T05:17:57.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 Processor -- start 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f48f750 0x558c1f48d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51092/0 (socket says 172.21.15.45:51092) 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60ba59c640 1 -- 172.21.15.45:0/4211214985 learned_addr learned my addr 172.21.15.45:0/4211214985 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f3ee020 con 0x558c1f48f750 2024-03-12T05:17:57.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f0c1430 con 0x558c1f486b70 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.480+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f07de00 con 0x558c1f486250 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60bad9d640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f48f750 0x558c1f48d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60c3c19740 10 client.0 _ll_register_callbacks cb 0x558c1f395cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 -- 172.21.15.45:0/4211214985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 msgr2=0x558c1f48cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 -- 172.21.15.45:0/4211214985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f48f750 msgr2=0x558c1f48d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f48f750 0x558c1f48d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b4009e00 con 0x558c1f486250 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60ba59c640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60b40097f0 tx=0x7f60b4005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30a26439a83b1a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.481+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b400fe40 con 0x558c1f486250 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.482+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f60b400f8d0 con 0x558c1f486250 2024-03-12T05:17:57.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.482+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c1f0ed7e0 con 0x558c1f486250 2024-03-12T05:17:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.482+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60b400fa90 con 0x558c1f486250 2024-03-12T05:17:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.482+0000 7f60c3c19740 10 client.26755 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.482+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558c1f3ec6a0 con 0x558c1f486250 2024-03-12T05:17:57.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558c1f0ed7e0 con 0x558c1f486250 2024-03-12T05:17:57.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f60b40046c0 con 0x558c1f486250 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 10 client.26755 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 10 client.26755 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 20 client.26755 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 4 client.26755 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c1f12a720 con 0x558c1f49fbc0 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.483+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.484+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4a1fe0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=909a54de460cc3b0 server_cookie=56f78404cc1c1840 in_seq=0 out_seq=0 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.484+0000 7f60a1ffb640 10 client.26755 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.485+0000 7f60a1ffb640 1 -- 172.21.15.45:0/4211214985 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558c1f12a720 con 0x558c1f49fbc0 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.485+0000 7f60a1ffb640 10 client.26755 handle_command_reply: tid=0 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.485+0000 7f60c3c19740 1 client.26755 shutdown 2024-03-12T05:17:57.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.485+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 msgr2=0x558c1f48c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.485+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60b40097f0 tx=0x7f60b4005db0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 shutdown_connections 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4a1fe0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f48f750 0x558c1f48d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60a1ffb640 0 client.26755 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 --2- 172.21.15.45:0/4211214985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486250 0x558c1f48c6d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 >> 172.21.15.45:0/4211214985 conn(0x558c1f480980 msgr2=0x558c1f4826d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 shutdown_connections 2024-03-12T05:17:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.486+0000 7f60c3c19740 1 -- 172.21.15.45:0/4211214985 wait complete. 2024-03-12T05:17:57.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.487+0000 7f60c3c19740 20 client.26755 trim_cache size 0 max 16384 2024-03-12T05:17:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 Processor -- start 2024-03-12T05:17:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 -- start start 2024-03-12T05:17:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f48ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f373130 con 0x558c1f486250 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f4a7250 con 0x558c1f486b70 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60c3c19740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f480830 con 0x558c1f48f750 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60bad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60bad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51096/0 (socket says 172.21.15.45:51096) 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60bad9d640 1 -- 172.21.15.45:0/2176053973 learned_addr learned my addr 172.21.15.45:0/2176053973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.509+0000 7f60ba59c640 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f48ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 -- 172.21.15.45:0/2176053973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 msgr2=0x558c1f48cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 -- 172.21.15.45:0/2176053973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 msgr2=0x558c1f48ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f48ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 -- 172.21.15.45:0/2176053973 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c1f388c20 con 0x558c1f48f750 2024-03-12T05:17:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60bad9d640 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f60b0096410 tx=0x7f60b00092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc73e250c651ace9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60a3fff640 1 -- 172.21.15.45:0/2176053973 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00a27a0 con 0x558c1f48f750 2024-03-12T05:17:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.510+0000 7f60a3fff640 1 -- 172.21.15.45:0/2176053973 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f60b0004380 con 0x558c1f48f750 2024-03-12T05:17:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.511+0000 7f60a3fff640 1 -- 172.21.15.45:0/2176053973 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0096800 con 0x558c1f48f750 2024-03-12T05:17:57.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.511+0000 7f60c3c19740 1 -- 172.21.15.45:0/2176053973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 msgr2=0x558c1f48d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.511+0000 7f60c3c19740 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f60b0096410 tx=0x7f60b00092d0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 -- 172.21.15.45:0/2176053973 shutdown_connections 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f48f750 0x558c1f48d180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f486b70 0x558c1f48cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 --2- 172.21.15.45:0/2176053973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f48ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 -- 172.21.15.45:0/2176053973 >> 172.21.15.45:0/2176053973 conn(0x558c1f480980 msgr2=0x558c1f4879f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 -- 172.21.15.45:0/2176053973 shutdown_connections 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 -- 172.21.15.45:0/2176053973 wait complete. 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.512+0000 7f60c3c19740 1 -- start start 2024-03-12T05:17:57.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.513+0000 7f60c3c19740 1 Processor -- start 2024-03-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.513+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.513+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f494ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.513+0000 7f60c3c19740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f495600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60c3c19740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f495cd0 con 0x558c1f486250 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60c3c19740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f4f60d0 con 0x558c1f48f750 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60c3c19740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c1f4f6210 con 0x558c1f486b70 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60c3c19740 10 client.0 _ll_register_callbacks cb 0x7f60b4009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60ba59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34164/0 (socket says 172.21.15.45:34164) 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60ba59c640 1 -- 172.21.15.45:0/3874001722 learned_addr learned my addr 172.21.15.45:0/3874001722 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f495600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60b9d9b640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f494ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 -- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 msgr2=0x558c1f494ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f494ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 -- 172.21.15.45:0/3874001722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 msgr2=0x558c1f496b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60bad9d640 1 -- 172.21.15.45:0/3874001722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc1aa9e0 con 0x558c1f48f750 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.514+0000 7f60ba59c640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.515+0000 7f60bad9d640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f495600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60b00982b0 tx=0x7f60b0005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c86a5724596e084 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.515+0000 7f60a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.515+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00a27a0 con 0x558c1f48f750 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.515+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f60b007e070 con 0x558c1f48f750 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.515+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60b400fcd0 con 0x558c1f48f750 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.516+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60b0083480 con 0x558c1f48f750 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.516+0000 7f60c3c19740 10 client.26786 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:57.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.516+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558c1f3ae8c0 con 0x558c1f48f750 2024-03-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.518+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f60b00a9030 con 0x558c1f48f750 2024-03-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.518+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f60b00b83e0 con 0x558c1f48f750 2024-03-12T05:17:57.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 10 client.26786 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 10 client.26786 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 20 client.26786 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4f9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 4 client.26786 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:57.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f60b400fcd0 con 0x558c1f49fbc0 2024-03-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.519+0000 7f60ba59c640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60a1ffb640 10 client.26786 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60ba59c640 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4f9560 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=923d3f599d217deb server_cookie=73b0c434e581d6f in_seq=0 out_seq=0 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60a1ffb640 1 -- 172.21.15.45:0/3874001722 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f60b400fcd0 con 0x558c1f49fbc0 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60a1ffb640 10 client.26786 handle_command_reply: tid=0 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 client.26786 shutdown 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 msgr2=0x558c1f495600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f495600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60b00982b0 tx=0x7f60b0005250 comp rx=0 tx=0).stop 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 shutdown_connections 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558c1f49fbc0 0x558c1f4f9560 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1f48f750 0x558c1f495600 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60a1ffb640 0 client.26786 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c1f486b70 0x558c1f494ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.520+0000 7f60c3c19740 1 --2- 172.21.15.45:0/3874001722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558c1f486250 0x558c1f496b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.521+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 >> 172.21.15.45:0/3874001722 conn(0x558c1f480980 msgr2=0x558c1f47f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.521+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 shutdown_connections 2024-03-12T05:17:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.521+0000 7f60c3c19740 1 -- 172.21.15.45:0/3874001722 wait complete. 2024-03-12T05:17:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.521+0000 7f60c3c19740 20 client.26786 trim_cache size 0 max 16384 2024-03-12T05:17:57.519 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.523+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 msgr2=0x7f60a40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.523+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 0x7f60a40a4160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f60ac034fd0 tx=0x7f60ac000f40 comp rx=0 tx=0).stop 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.523+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 msgr2=0x7f60bc19daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.523+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc19daf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009fd0 tx=0x7f60b000eba0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 shutdown_connections 2024-03-12T05:17:57.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f60a40a1ca0 0x7f60a40a4160 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60bc105570 0x7f60bc19daf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60bc100a90 0x7f60bc1a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 --2- 172.21.15.45:0/2578836005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60bc100170 0x7f60bc1a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 >> 172.21.15.45:0/2578836005 conn(0x7f60bc076e30 msgr2=0x7f60bc102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 shutdown_connections 2024-03-12T05:17:57.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.524+0000 7f60c0f6c640 1 -- 172.21.15.45:0/2578836005 wait complete. 2024-03-12T05:17:57.539 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:57.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 -- 172.21.15.45:0/3207536148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 msgr2=0x7f85a40ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/3207536148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a40ffc30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8594009900 tx=0x7f859403cc40 comp rx=0 tx=0).stop 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 -- 172.21.15.45:0/3207536148 shutdown_connections 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/3207536148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/3207536148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a40ffc30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/3207536148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85a41021a0 0x7f85a40ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 -- 172.21.15.45:0/3207536148 >> 172.21.15.45:0/3207536148 conn(0x7f85a40766b0 msgr2=0x7f85a40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 -- 172.21.15.45:0/3207536148 shutdown_connections 2024-03-12T05:17:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.718+0000 7f85ab5d7640 1 -- 172.21.15.45:0/3207536148 wait complete. 2024-03-12T05:17:57.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 Processor -- start 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 -- start start 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85a41021a0 0x7f85a4105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a41099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4109f00 con 0x7f85a410de70 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a410a070 con 0x7f85a41021a0 2024-03-12T05:17:57.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85ab5d7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a410a230 con 0x7f85a4102610 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85a8b72640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85a8b72640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51128/0 (socket says 172.21.15.45:51128) 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85a8b72640 1 -- 172.21.15.45:0/2111580462 learned_addr learned my addr 172.21.15.45:0/2111580462 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.719+0000 7f85a9b74640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a41099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 -- 172.21.15.45:0/2111580462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85a41021a0 msgr2=0x7f85a4105110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85a41021a0 0x7f85a4105110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 -- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 msgr2=0x7f85a41099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a41099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 -- 172.21.15.45:0/2111580462 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594009560 con 0x7f85a4102610 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a9b74640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f85a8b72640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f859403d250 tx=0x7f8594004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.720+0000 7f859a7fc640 1 -- 172.21.15.45:0/2111580462 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594004480 con 0x7f85a4102610 2024-03-12T05:17:57.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.721+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a410a450 con 0x7f85a4102610 2024-03-12T05:17:57.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.721+0000 7f859a7fc640 1 -- 172.21.15.45:0/2111580462 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8594003aa0 con 0x7f85a4102610 2024-03-12T05:17:57.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.721+0000 7f859a7fc640 1 -- 172.21.15.45:0/2111580462 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940053d0 con 0x7f85a4102610 2024-03-12T05:17:57.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.721+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a41a2260 con 0x7f85a4102610 2024-03-12T05:17:57.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.723+0000 7f859a7fc640 1 -- 172.21.15.45:0/2111580462 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8594005530 con 0x7f85a4102610 2024-03-12T05:17:57.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.724+0000 7f859a7fc640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 0x7f85900a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.724+0000 7f859a7fc640 1 -- 172.21.15.45:0/2111580462 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8594100f50 con 0x7f85a4102610 2024-03-12T05:17:57.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.724+0000 7f85ae1ac740 1 Processor -- start 2024-03-12T05:17:57.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 -- start start 2024-03-12T05:17:57.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85a9373640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 0x7f85900a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def35630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35b70 0x55e2def3e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2deef0960 con 0x55e2def35b70 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2deea4310 con 0x55e2def35250 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2deea1840 con 0x55e2def3e750 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85a9b74640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85a9b74640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51132/0 (socket says 172.21.15.45:51132) 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.725+0000 7f85a9b74640 1 -- 172.21.15.45:0/1818676923 learned_addr learned my addr 172.21.15.45:0/1818676923 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9373640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 0x7f85900a41f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009a60 tx=0x7f85a0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 -- 172.21.15.45:0/1818676923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 msgr2=0x55e2def35630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def35630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 -- 172.21.15.45:0/1818676923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35b70 msgr2=0x55e2def3e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35b70 0x55e2def3e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 -- 172.21.15.45:0/1818676923 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2dee5f060 con 0x55e2def3e750 2024-03-12T05:17:57.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f85a9b74640 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f859c010740 tx=0x7f859c010b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1582b5b41c102 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.726+0000 7f858f7fe640 1 -- 172.21.15.45:0/1818676923 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00e8d0 con 0x55e2def3e750 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.727+0000 7f858f7fe640 1 -- 172.21.15.45:0/1818676923 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f859c012450 con 0x55e2def3e750 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.727+0000 7f858f7fe640 1 -- 172.21.15.45:0/1818676923 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00eee0 con 0x55e2def3e750 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1818676923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 msgr2=0x55e2def40b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f859c010740 tx=0x7f859c010b40 comp rx=0 tx=0).stop 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1818676923 shutdown_connections 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35b70 0x55e2def3e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def40b40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1818676923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def35630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1818676923 >> 172.21.15.45:0/1818676923 conn(0x55e2def2f980 msgr2=0x55e2def2fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1818676923 shutdown_connections 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1818676923 wait complete. 2024-03-12T05:17:57.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.728+0000 7f85ae1ac740 1 -- start start 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 Processor -- start 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def3b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def3c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2dee9d020 con 0x55e2def35250 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2deb70430 con 0x55e2def35b70 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2deb2ce00 con 0x55e2def3e750 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85ae1ac740 10 client.0 _ll_register_callbacks cb 0x55e2dee44cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85a8b72640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85a8b72640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45902/0 (socket says 172.21.15.45:45902) 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.729+0000 7f85a8b72640 1 -- 172.21.15.45:0/4179947927 learned_addr learned my addr 172.21.15.45:0/4179947927 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a9b74640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def3c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a9373640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def3b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a8b72640 1 -- 172.21.15.45:0/4179947927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 msgr2=0x55e2def3c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a8b72640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def3c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a8b72640 1 -- 172.21.15.45:0/4179947927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 msgr2=0x55e2def3b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a8b72640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def3b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a8b72640 1 -- 172.21.15.45:0/4179947927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c010370 con 0x55e2def35b70 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a9b74640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def3c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.730+0000 7f85a9373640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def3b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f85a8b72640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8594006010 tx=0x7f85940ce700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb47bf11da058 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f858cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940d7010 con 0x55e2def35b70 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f85940ceb00 con 0x55e2def35b70 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2deb9c7e0 con 0x55e2def35b70 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.731+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85940d2c50 con 0x55e2def35b70 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.732+0000 7f85ae1ac740 10 client.26792 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.732+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e2dee9b6a0 con 0x55e2def35b70 2024-03-12T05:17:57.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.732+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e2deb9c7e0 con 0x55e2def35b70 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.732+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f85940ea070 con 0x55e2def35b70 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 10 client.26792 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 10 client.26792 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 20 client.26792 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2def4e9e0 0x55e2def50e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 4 client.26792 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2debd9720 con 0x55e2def4e9e0 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.733+0000 7f85a9373640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2def4e9e0 0x55e2def50e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.734+0000 7f85a9373640 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2def4e9e0 0x55e2def50e00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ab4cc8f3cf096dc9 server_cookie=ecfccc28dd8fc414 in_seq=0 out_seq=0 2024-03-12T05:17:57.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.734+0000 7f858cff9640 10 client.26792 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.734+0000 7f858cff9640 1 -- 172.21.15.45:0/4179947927 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e2debd9720 con 0x55e2def4e9e0 2024-03-12T05:17:57.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.734+0000 7f858cff9640 10 client.26792 handle_command_reply: tid=0 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.734+0000 7f85ae1ac740 1 client.26792 shutdown 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 msgr2=0x55e2def3bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8594006010 tx=0x7f85940ce700 comp rx=0 tx=0).stop 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 shutdown_connections 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2def4e9e0 0x55e2def50e00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def3c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def3bda0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/4179947927 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def3b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 >> 172.21.15.45:0/4179947927 conn(0x55e2def2f980 msgr2=0x55e2def316d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f858cff9640 0 client.26792 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 shutdown_connections 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.735+0000 7f85ae1ac740 1 -- 172.21.15.45:0/4179947927 wait complete. 2024-03-12T05:17:57.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.736+0000 7f85ae1ac740 20 client.26792 trim_cache size 0 max 16384 2024-03-12T05:17:57.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.757+0000 7f85ae1ac740 1 Processor -- start 2024-03-12T05:17:57.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.757+0000 7f85ae1ac740 1 -- start start 2024-03-12T05:17:57.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def35b70 0x55e2def3c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def3e750 0x55e2def38ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85a9373640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85a9373640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45904/0 (socket says 172.21.15.45:45904) 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85a9373640 1 -- 172.21.15.45:0/106538977 learned_addr learned my addr 172.21.15.45:0/106538977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2dee22130 con 0x55e2def3e750 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.758+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2def4b740 con 0x55e2def35250 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2def309b0 con 0x55e2def35b70 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9b74640 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def3e750 0x55e2def38ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a8b72640 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def35b70 0x55e2def3c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9373640 1 -- 172.21.15.45:0/106538977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def35b70 msgr2=0x55e2def3c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9373640 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def35b70 0x55e2def3c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9373640 1 -- 172.21.15.45:0/106538977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def3e750 msgr2=0x55e2def38ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9373640 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def3e750 0x55e2def38ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.759+0000 7f85a9373640 1 -- 172.21.15.45:0/106538977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2dee37c20 con 0x55e2def35250 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.760+0000 7f85a9373640 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009840 tx=0x7f85a0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87efb4d274d45354 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.760+0000 7f858effd640 1 -- 172.21.15.45:0/106538977 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0022040 con 0x55e2def35250 2024-03-12T05:17:57.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.760+0000 7f858effd640 1 -- 172.21.15.45:0/106538977 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85a0005020 con 0x55e2def35250 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.760+0000 7f858effd640 1 -- 172.21.15.45:0/106538977 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00246d0 con 0x55e2def35250 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 msgr2=0x55e2def3df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009840 tx=0x7f85a0015000 comp rx=0 tx=0).stop 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 shutdown_connections 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def3e750 0x55e2def38ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def35b70 0x55e2def3c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/106538977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35250 0x55e2def3df90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 >> 172.21.15.45:0/106538977 conn(0x55e2def2f980 msgr2=0x55e2def36af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 shutdown_connections 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- 172.21.15.45:0/106538977 wait complete. 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.761+0000 7f85ae1ac740 1 -- start start 2024-03-12T05:17:57.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 Processor -- start 2024-03-12T05:17:57.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def44690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def44bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2defa4f30 con 0x55e2def35250 2024-03-12T05:17:57.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2defa50a0 con 0x55e2def35b70 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2defa5260 con 0x55e2def3e750 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85ae1ac740 10 client.0 _ll_register_callbacks cb 0x7f859c009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85a9373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85a8b72640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def44690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85a9373640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34212/0 (socket says 172.21.15.45:34212) 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85a9373640 1 -- 172.21.15.45:0/1044452335 learned_addr learned my addr 172.21.15.45:0/1044452335 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.762+0000 7f85a9b74640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def44bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 -- 172.21.15.45:0/1044452335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 msgr2=0x55e2def44bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def44bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 -- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 msgr2=0x55e2def44690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def44690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 -- 172.21.15.45:0/1044452335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a00096b0 con 0x55e2def35250 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f85a9373640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009350 tx=0x7f85a0005190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960b3a4f9e17eeee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f858cff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0005440 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.763+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f85a0010cf0 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.764+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f859c012450 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.764+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85a0024450 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.764+0000 7f85ae1ac740 10 client.17319 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.764+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e2dee5d8c0 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.764+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f859c012450 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f85a0018040 con 0x55e2def35250 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 10 client.17319 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 10 client.17319 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 20 client.17319 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2defa69e0 0x55e2defa8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 4 client.17319 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:57.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e2dec28890 con 0x55e2defa69e0 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85a8b72640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2defa69e0 0x55e2defa8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f858cff9640 10 client.17319 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.765+0000 7f85a8b72640 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2defa69e0 0x55e2defa8e00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d08b08b6eda9904d server_cookie=53c77bfd4c1dec50 in_seq=0 out_seq=0 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f858cff9640 1 -- 172.21.15.45:0/1044452335 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e2dec28890 con 0x55e2defa69e0 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f858cff9640 10 client.17319 handle_command_reply: tid=0 2024-03-12T05:17:57.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 client.17319 shutdown 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 msgr2=0x55e2def44150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009350 tx=0x7f85a0005190 comp rx=0 tx=0).stop 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 shutdown_connections 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55e2defa69e0 0x55e2defa8e00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2def3e750 0x55e2def44bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2def35b70 0x55e2def44690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 --2- 172.21.15.45:0/1044452335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e2def35250 0x55e2def44150 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.766+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 >> 172.21.15.45:0/1044452335 conn(0x55e2def2f980 msgr2=0x55e2def30db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.767+0000 7f858cff9640 0 client.17319 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.767+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 shutdown_connections 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.767+0000 7f85ae1ac740 1 -- 172.21.15.45:0/1044452335 wait complete. 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.767+0000 7f85ae1ac740 20 client.17319 trim_cache size 0 max 16384 2024-03-12T05:17:57.766 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:57.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 msgr2=0x7f85900a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 0x7f85900a41f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009a60 tx=0x7f85a0005f90 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 msgr2=0x7f85a4105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f859403d250 tx=0x7f8594004040 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 shutdown_connections 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85900a1d30 0x7f85900a41f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85a410de70 0x7f85a41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.768+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85a4102610 0x7f85a4105650 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.769+0000 7f85ab5d7640 1 --2- 172.21.15.45:0/2111580462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85a41021a0 0x7f85a4105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.769+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 >> 172.21.15.45:0/2111580462 conn(0x7f85a40766b0 msgr2=0x7f85a410f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.769+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 shutdown_connections 2024-03-12T05:17:57.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.769+0000 7f85ab5d7640 1 -- 172.21.15.45:0/2111580462 wait complete. 2024-03-12T05:17:57.783 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:57.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:17:57.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:57 smithi045 ceph-mon[32174]: pgmap v217: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:57.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:57 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2384027564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.964+0000 7fbce3fd0640 1 -- 172.21.15.45:0/2304187175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc0ff5d0 msgr2=0x7fbcdc0ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.964+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/2304187175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc0ff9b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009900 tx=0x7fbcd802bc40 comp rx=0 tx=0).stop 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 -- 172.21.15.45:0/2304187175 shutdown_connections 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/2304187175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0fff80 0x7fbcdc103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/2304187175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/2304187175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc0ff9b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 -- 172.21.15.45:0/2304187175 >> 172.21.15.45:0/2304187175 conn(0x7fbcdc0761d0 msgr2=0x7fbcdc0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 -- 172.21.15.45:0/2304187175 shutdown_connections 2024-03-12T05:17:57.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.965+0000 7fbce3fd0640 1 -- 172.21.15.45:0/2304187175 wait complete. 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.966+0000 7fbce3fd0640 1 Processor -- start 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.967+0000 7fbce3fd0640 1 -- start start 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.967+0000 7fbce3fd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.967+0000 7fbce3fd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.967+0000 7fbce1d6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.967+0000 7fbce1d6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34232/0 (socket says 172.21.15.45:34232) 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce1d6c640 1 -- 172.21.15.45:0/970404638 learned_addr learned my addr 172.21.15.45:0/970404638 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 0x7fbcdc19c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19c880 con 0x7fbcdc0ff5d0 2024-03-12T05:17:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce156b640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19c9f0 con 0x7fbcdc1046a0 2024-03-12T05:17:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19cb60 con 0x7fbcdc19be90 2024-03-12T05:17:57.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.968+0000 7fbce256d640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 0x7fbcdc19c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 -- 172.21.15.45:0/970404638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 msgr2=0x7fbcdc197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 -- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 msgr2=0x7fbcdc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8009560 con 0x7fbcdc19be90 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce156b640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:17:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce256d640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 0x7fbcdc19c340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009fd0 tx=0x7fbcd400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbcd2ffd640 1 -- 172.21.15.45:0/970404638 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400ce20 con 0x7fbcdc19be90 2024-03-12T05:17:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc1a4140 con 0x7fbcdc19be90 2024-03-12T05:17:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.969+0000 7fbcd2ffd640 1 -- 172.21.15.45:0/970404638 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd4010430 con 0x7fbcdc19be90 2024-03-12T05:17:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.970+0000 7fbcd2ffd640 1 -- 172.21.15.45:0/970404638 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400f620 con 0x7fbcdc19be90 2024-03-12T05:17:57.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.970+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc1a4590 con 0x7fbcdc19be90 2024-03-12T05:17:57.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:57 smithi130 ceph-mon[35398]: pgmap v217: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:57.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:57 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2384027564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.973+0000 7fbcd2ffd640 1 -- 172.21.15.45:0/970404638 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbcd40105a0 con 0x7fbcdc19be90 2024-03-12T05:17:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.973+0000 7fbce6ba7740 1 Processor -- start 2024-03-12T05:17:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 -- start start 2024-03-12T05:17:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a2490a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a249af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbcd2ffd640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 0x7fbcc80a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a204960 con 0x56164a249670 2024-03-12T05:17:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a1b8310 con 0x56164a25ae80 2024-03-12T05:17:57.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a1b5840 con 0x56164a248cc0 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 0x7fbcc80a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbcd2ffd640 1 -- 172.21.15.45:0/970404638 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbcd4014070 con 0x7fbcdc19be90 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce256d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce156b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a249af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce156b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a2490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce156b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a249af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34234/0 (socket says 172.21.15.45:34234) 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.974+0000 7fbce156b640 1 -- 172.21.15.45:0/839581644 learned_addr learned my addr 172.21.15.45:0/839581644 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 -- 172.21.15.45:0/839581644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 msgr2=0x56164a25d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 -- 172.21.15.45:0/839581644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 msgr2=0x56164a249af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a249af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 -- 172.21.15.45:0/839581644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56164a173060 con 0x56164a248cc0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbce156b640 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a2490a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0058c0 tx=0x7fbccc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acda9d0fd593551f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbcc7fff640 1 -- 172.21.15.45:0/839581644 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc004020 con 0x56164a248cc0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbcc7fff640 1 -- 172.21.15.45:0/839581644 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbccc005020 con 0x56164a248cc0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.975+0000 7fbcc7fff640 1 -- 172.21.15.45:0/839581644 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc00fb30 con 0x56164a248cc0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 0x7fbcc80a4120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009850 tx=0x7fbcd8000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 -- 172.21.15.45:0/839581644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 msgr2=0x56164a2490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a2490a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0058c0 tx=0x7fbccc004730 comp rx=0 tx=0).stop 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 -- 172.21.15.45:0/839581644 shutdown_connections 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a249af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/839581644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a2490a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.977+0000 7fbce6ba7740 1 -- 172.21.15.45:0/839581644 >> 172.21.15.45:0/839581644 conn(0x56164a243980 msgr2=0x56164a243d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.978+0000 7fbce6ba7740 1 -- 172.21.15.45:0/839581644 shutdown_connections 2024-03-12T05:17:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.978+0000 7fbce6ba7740 1 -- 172.21.15.45:0/839581644 wait complete. 2024-03-12T05:17:57.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.978+0000 7fbce6ba7740 1 -- start start 2024-03-12T05:17:57.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 Processor -- start 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 0x56164a24b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a249670 0x56164a24b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a1b1020 con 0x56164a248cc0 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561649e84430 con 0x56164a25ae80 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561649e40e00 con 0x56164a249670 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce6ba7740 10 client.0 _ll_register_callbacks cb 0x56164a158cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce256d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.979+0000 7fbce256d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45932/0 (socket says 172.21.15.45:45932) 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 -- 172.21.15.45:0/2707852777 learned_addr learned my addr 172.21.15.45:0/2707852777 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 0x56164a24b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 -- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a249670 msgr2=0x56164a24b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a249670 0x56164a24b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 -- 172.21.15.45:0/2707852777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 msgr2=0x56164a24b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 0x56164a24b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 -- 172.21.15.45:0/2707852777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc01a040 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 0x56164a24b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce256d640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcd409a6b0 tx=0x7fbcd40b3580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d767fde97b014f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbcc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4083ea0 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbcd409e870 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.980+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561649eb07e0 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcd4010b10 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 10 client.26810 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56164a1af6a0 con 0x56164a25ae80 2024-03-12T05:17:57.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbcd4065020 con 0x56164a25ae80 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbcd4085070 con 0x56164a25ae80 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 10 client.26810 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 10 client.26810 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 20 client.26810 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a26b310 0x56164a26d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 4 client.26810 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561649eb07e0 con 0x56164a26b310 2024-03-12T05:17:57.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.981+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a26b310 0x56164a26d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.982+0000 7fbcc57fa640 10 client.26810 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.982+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a26b310 0x56164a26d750 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=725b1925ac6256 server_cookie=e319b9e84f953025 in_seq=0 out_seq=0 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2707852777 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561649eb07e0 con 0x56164a26b310 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbcc57fa640 10 client.26810 handle_command_reply: tid=0 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 client.26810 shutdown 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 msgr2=0x56164a24bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcd409a6b0 tx=0x7fbcd40b3580 comp rx=0 tx=0).stop 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 shutdown_connections 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a26b310 0x56164a26d750 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a24bcb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbcc57fa640 0 client.26810 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a249670 0x56164a24b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2707852777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a248cc0 0x56164a24b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 >> 172.21.15.45:0/2707852777 conn(0x56164a243980 msgr2=0x56164a2456d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.983+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 shutdown_connections 2024-03-12T05:17:57.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.984+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2707852777 wait complete. 2024-03-12T05:17:57.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:57.984+0000 7fbce6ba7740 20 client.26810 trim_cache size 0 max 16384 2024-03-12T05:17:57.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:57 smithi116 ceph-mon[35486]: pgmap v217: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 1.28k op/s 2024-03-12T05:17:57.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:57 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2384027564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 Processor -- start 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 -- start start 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a24d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a136130 con 0x56164a249670 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561649eb07e0 con 0x56164a25ae80 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a235a40 con 0x56164a248cc0 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce156b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce156b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34262/0 (socket says 172.21.15.45:34262) 2024-03-12T05:17:58.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce156b640 1 -- 172.21.15.45:0/2001702376 learned_addr learned my addr 172.21.15.45:0/2001702376 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a24d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.006+0000 7fbce256d640 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 -- 172.21.15.45:0/2001702376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 msgr2=0x56164a24d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a24d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 -- 172.21.15.45:0/2001702376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 msgr2=0x56164a25c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 -- 172.21.15.45:0/2001702376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56164a14bc20 con 0x56164a249670 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbce156b640 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00ba60 tx=0x7fbccc0046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ded3acf6bb74a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbcc77fe640 1 -- 172.21.15.45:0/2001702376 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0126f0 con 0x56164a249670 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbcc77fe640 1 -- 172.21.15.45:0/2001702376 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbccc0044d0 con 0x56164a249670 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.007+0000 7fbcc77fe640 1 -- 172.21.15.45:0/2001702376 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc00fae0 con 0x56164a249670 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.008+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2001702376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 msgr2=0x56164a242a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.008+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00ba60 tx=0x7fbccc0046a0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2001702376 shutdown_connections 2024-03-12T05:17:58.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a249670 0x56164a242a00 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a25ae80 0x56164a25c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2001702376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a24d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2001702376 >> 172.21.15.45:0/2001702376 conn(0x56164a243980 msgr2=0x56164a2422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2001702376 shutdown_connections 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2001702376 wait complete. 2024-03-12T05:17:58.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 -- start start 2024-03-12T05:17:58.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.009+0000 7fbce6ba7740 1 Processor -- start 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a25f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a25ae80 0x56164a260440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a2c28b0 con 0x56164a25ae80 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a2c2a20 con 0x56164a249670 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164a2c2be0 con 0x56164a248cc0 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce6ba7740 10 client.0 _ll_register_callbacks cb 0x7fbccc00be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.011+0000 7fbce1d6c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a25f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45944/0 (socket says 172.21.15.45:45944) 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 -- 172.21.15.45:0/2673258311 learned_addr learned my addr 172.21.15.45:0/2673258311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce256d640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a25ae80 0x56164a260440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 -- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 msgr2=0x56164a25f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a25f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 -- 172.21.15.45:0/2673258311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a25ae80 msgr2=0x56164a260440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a25ae80 0x56164a260440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 -- 172.21.15.45:0/2673258311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56164a173060 con 0x56164a249670 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.012+0000 7fbce156b640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56164a24c610 tx=0x7fbccc0053c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38df55e2b982ba41 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbcc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0126f0 con 0x56164a249670 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbccc0042e0 con 0x56164a249670 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbccc005020 con 0x56164a249670 2024-03-12T05:17:58.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbccc00fa00 con 0x56164a249670 2024-03-12T05:17:58.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.013+0000 7fbce6ba7740 10 client.26816 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:58.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56164a1718c0 con 0x56164a249670 2024-03-12T05:17:58.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbccc004710 con 0x56164a249670 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbccc0035f0 con 0x56164a249670 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 10 client.26816 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 10 client.26816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 20 client.26816 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a2c3fc0 0x56164a2c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 4 client.26816 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.014+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fbccc005020 con 0x56164a2c3fc0 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a2c3fc0 0x56164a2c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbce1d6c640 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a2c3fc0 0x56164a2c63b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e79cecf240a013aa server_cookie=2e19d9370da0a17c in_seq=0 out_seq=0 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbcc57fa640 10 client.26816 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbcc57fa640 1 -- 172.21.15.45:0/2673258311 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fbccc005020 con 0x56164a2c3fc0 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbcc57fa640 10 client.26816 handle_command_reply: tid=0 2024-03-12T05:17:58.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.015+0000 7fbce6ba7740 1 client.26816 shutdown 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 msgr2=0x56164a25ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56164a24c610 tx=0x7fbccc0053c0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 shutdown_connections 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x56164a2c3fc0 0x56164a2c63b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56164a25ae80 0x56164a260440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56164a249670 0x56164a25ff00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 --2- 172.21.15.45:0/2673258311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56164a248cc0 0x56164a25f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 >> 172.21.15.45:0/2673258311 conn(0x56164a243980 msgr2=0x56164a242b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbcc57fa640 0 client.26816 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 shutdown_connections 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.016+0000 7fbce6ba7740 1 -- 172.21.15.45:0/2673258311 wait complete. 2024-03-12T05:17:58.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce6ba7740 20 client.26816 trim_cache size 0 max 16384 2024-03-12T05:17:58.015 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 msgr2=0x7fbcc80a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 0x7fbcc80a4120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009850 tx=0x7fbcd8000bc0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 msgr2=0x7fbcdc19c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 0x7fbcdc19c340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009fd0 tx=0x7fbcd400ef10 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 shutdown_connections 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbcc80a1c60 0x7fbcc80a4120 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbcdc19be90 0x7fbcdc19c340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcdc1046a0 0x7fbcdc197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 --2- 172.21.15.45:0/970404638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcdc0ff5d0 0x7fbcdc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 >> 172.21.15.45:0/970404638 conn(0x7fbcdc0761d0 msgr2=0x7fbcdc1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.017+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 shutdown_connections 2024-03-12T05:17:58.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.018+0000 7fbce3fd0640 1 -- 172.21.15.45:0/970404638 wait complete. 2024-03-12T05:17:58.031 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:58.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:17:58.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.221+0000 7f5a22264640 1 -- 172.21.15.45:0/1424622553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c107720 msgr2=0x7f5a1c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.221+0000 7f5a22264640 1 --2- 172.21.15.45:0/1424622553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c107720 0x7f5a1c107ba0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5a1001a910 tx=0x7f5a1003cc60 comp rx=0 tx=0).stop 2024-03-12T05:17:58.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.221+0000 7f5a22264640 1 -- 172.21.15.45:0/1424622553 shutdown_connections 2024-03-12T05:17:58.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 --2- 172.21.15.45:0/1424622553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c1145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 --2- 172.21.15.45:0/1424622553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c107720 0x7f5a1c107ba0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 --2- 172.21.15.45:0/1424622553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c10d730 0x7f5a1c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 -- 172.21.15.45:0/1424622553 >> 172.21.15.45:0/1424622553 conn(0x7f5a1c0756c0 msgr2=0x7f5a1c077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 -- 172.21.15.45:0/1424622553 shutdown_connections 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.222+0000 7f5a22264640 1 -- 172.21.15.45:0/1424622553 wait complete. 2024-03-12T05:17:58.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.223+0000 7f5a22264640 1 Processor -- start 2024-03-12T05:17:58.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.223+0000 7f5a22264640 1 -- start start 2024-03-12T05:17:58.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a22264640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 0x7f5a1c110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a22264640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a22264640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c10d730 0x7f5a1c19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34272/0 (socket says 172.21.15.45:34272) 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.224+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 0x7f5a1c110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 0x7f5a1c110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51224/0 (socket says 172.21.15.45:51224) 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 -- 172.21.15.45:0/1834417603 learned_addr learned my addr 172.21.15.45:0/1834417603 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c19b470 con 0x7f5a1c1080e0 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c19b5e0 con 0x7f5a1c10d730 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c19b7a0 con 0x7f5a1c107720 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a20801640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c10d730 0x7f5a1c19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 -- 172.21.15.45:0/1834417603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 msgr2=0x7f5a1c110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 0x7f5a1c110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 -- 172.21.15.45:0/1834417603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c10d730 msgr2=0x7f5a1c19ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c10d730 0x7f5a1c19ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.225+0000 7f5a1b7fe640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1001a570 con 0x7f5a1c1080e0 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a1b7fe640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5a100057c0 tx=0x7f5a1003da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a197fa640 1 -- 172.21.15.45:0/1834417603 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10004510 con 0x7f5a1c1080e0 2024-03-12T05:17:58.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a197fa640 1 -- 172.21.15.45:0/1834417603 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a10005310 con 0x7f5a1c1080e0 2024-03-12T05:17:58.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a197fa640 1 -- 172.21.15.45:0/1834417603 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10003ea0 con 0x7f5a1c1080e0 2024-03-12T05:17:58.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c19ba20 con 0x7f5a1c1080e0 2024-03-12T05:17:58.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.226+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c1a2100 con 0x7f5a1c1080e0 2024-03-12T05:17:58.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.229+0000 7f5a197fa640 1 -- 172.21.15.45:0/1834417603 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5a10004e60 con 0x7f5a1c1080e0 2024-03-12T05:17:58.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 Processor -- start 2024-03-12T05:17:58.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 -- start start 2024-03-12T05:17:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f162b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f112960 con 0x559d8f157b70 2024-03-12T05:17:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f0c6310 con 0x559d8f160750 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a24e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f0c3840 con 0x559d8f157250 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.230+0000 7f5a197fa640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 0x7f5a040a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a20801640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a197fa640 1 -- 172.21.15.45:0/1834417603 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5a10100b50 con 0x7f5a1c1080e0 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a20801640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51232/0 (socket says 172.21.15.45:51232) 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a20801640 1 -- 172.21.15.45:0/572265313 learned_addr learned my addr 172.21.15.45:0/572265313 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a20801640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 0x7f5a040a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 -- 172.21.15.45:0/572265313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 msgr2=0x559d8f157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 -- 172.21.15.45:0/572265313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 msgr2=0x559d8f162b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f162b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 -- 172.21.15.45:0/572265313 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d8f081060 con 0x559d8f157b70 2024-03-12T05:17:58.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.231+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f160210 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009c00 tx=0x7f5a0c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82e8db40ffbe2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.232+0000 7f5a027fc640 1 -- 172.21.15.45:0/572265313 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c009db0 con 0x559d8f157b70 2024-03-12T05:17:58.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.232+0000 7f5a027fc640 1 -- 172.21.15.45:0/572265313 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a0c0054d0 con 0x559d8f157b70 2024-03-12T05:17:58.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.232+0000 7f5a027fc640 1 -- 172.21.15.45:0/572265313 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c019670 con 0x559d8f157b70 2024-03-12T05:17:58.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.233+0000 7f5a24e39740 1 -- 172.21.15.45:0/572265313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 msgr2=0x559d8f160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.233+0000 7f5a24e39740 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f160210 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c009c00 tx=0x7f5a0c011810 comp rx=0 tx=0).stop 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.233+0000 7f5a24e39740 1 -- 172.21.15.45:0/572265313 shutdown_connections 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f160210 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f162b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 --2- 172.21.15.45:0/572265313 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 -- 172.21.15.45:0/572265313 >> 172.21.15.45:0/572265313 conn(0x559d8f151980 msgr2=0x559d8f151d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 -- 172.21.15.45:0/572265313 shutdown_connections 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a24e39740 1 -- 172.21.15.45:0/572265313 wait complete. 2024-03-12T05:17:58.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.234+0000 7f5a20801640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 0x7f5a040a41a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009fd0 tx=0x7f5a14009600 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 -- start start 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 Processor -- start 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f15e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f0bf020 con 0x559d8f157b70 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8ed92430 con 0x559d8f160750 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8ed4ee00 con 0x559d8f157250 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.235+0000 7f5a24e39740 10 client.0 _ll_register_callbacks cb 0x559d8f066cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f15dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34294/0 (socket says 172.21.15.45:34294) 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a1b7fe640 1 -- 172.21.15.45:0/1405213050 learned_addr learned my addr 172.21.15.45:0/1405213050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f15e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 -- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 msgr2=0x559d8f15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 -- 172.21.15.45:0/1405213050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 msgr2=0x559d8f15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 -- 172.21.15.45:0/1405213050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c009830 con 0x559d8f160750 2024-03-12T05:17:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.236+0000 7f5a20801640 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f15e150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009240 tx=0x7f5a14004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16205b1df3fb0905 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f59fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1401ea00 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d8edbe7e0 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5a1401eb60 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a1401ed20 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f5a24e39740 10 client.26825 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559d8f0bd6a0 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.237+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a14035020 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a140103d0 con 0x559d8f160750 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 10 client.26825 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 10 client.26825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 20 client.26825 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f170be0 0x559d8f172fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 4 client.26825 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:58.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.238+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d8edfb720 con 0x559d8f170be0 2024-03-12T05:17:58.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.239+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f170be0 0x559d8f172fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.239+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f170be0 0x559d8f172fd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a197016afb62dcdf server_cookie=123a7cb2b2b7b772 in_seq=0 out_seq=0 2024-03-12T05:17:58.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.239+0000 7f59fb7fe640 10 client.26825 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.240+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1405213050 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d8edfb720 con 0x559d8f170be0 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.240+0000 7f59fb7fe640 10 client.26825 handle_command_reply: tid=0 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.240+0000 7f5a24e39740 1 client.26825 shutdown 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.240+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 msgr2=0x559d8f15e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.240+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f15e150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009240 tx=0x7f5a14004ab0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 shutdown_connections 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f170be0 0x559d8f172fd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f15e150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1405213050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 >> 172.21.15.45:0/1405213050 conn(0x559d8f151980 msgr2=0x559d8f1536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f59fb7fe640 0 client.26825 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:58.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 shutdown_connections 2024-03-12T05:17:58.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 1 -- 172.21.15.45:0/1405213050 wait complete. 2024-03-12T05:17:58.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.241+0000 7f5a24e39740 20 client.26825 trim_cache size 0 max 16384 2024-03-12T05:17:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 Processor -- start 2024-03-12T05:17:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 -- start start 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 0x559d8f15fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 0x559d8f15bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 0x559d8f15fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f044130 con 0x559d8f157250 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f152980 con 0x559d8f157b70 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 0x559d8f15bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a20801640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51248/0 (socket says 172.21.15.45:51248) 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 0x559d8f15bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45976/0 (socket says 172.21.15.45:45976) 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 -- 172.21.15.45:0/233384403 learned_addr learned my addr 172.21.15.45:0/233384403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a24e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f14d320 con 0x559d8f160750 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.263+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 0x559d8f15fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34300/0 (socket says 172.21.15.45:34300) 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 -- 172.21.15.45:0/233384403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 msgr2=0x559d8f15bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 0x559d8f15bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 -- 172.21.15.45:0/233384403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 msgr2=0x559d8f15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 0x559d8f15fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 -- 172.21.15.45:0/233384403 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d8f059c20 con 0x559d8f160750 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.264+0000 7f5a20801640 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009be0 tx=0x7f5a140125a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79c4f936fea3441 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.265+0000 7f5a01ffb640 1 -- 172.21.15.45:0/233384403 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14010aa0 con 0x559d8f160750 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.265+0000 7f5a01ffb640 1 -- 172.21.15.45:0/233384403 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a14035040 con 0x559d8f160750 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.265+0000 7f5a01ffb640 1 -- 172.21.15.45:0/233384403 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1401ea00 con 0x559d8f160750 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 -- 172.21.15.45:0/233384403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 msgr2=0x559d8f15c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009be0 tx=0x7f5a140125a0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 -- 172.21.15.45:0/233384403 shutdown_connections 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f160750 0x559d8f15c4e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f157b70 0x559d8f15bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 --2- 172.21.15.45:0/233384403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157250 0x559d8f15fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.266+0000 7f5a24e39740 1 -- 172.21.15.45:0/233384403 >> 172.21.15.45:0/233384403 conn(0x559d8f151980 msgr2=0x559d8f177930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.267+0000 7f5a24e39740 1 -- 172.21.15.45:0/233384403 shutdown_connections 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.267+0000 7f5a24e39740 1 -- 172.21.15.45:0/233384403 wait complete. 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.267+0000 7f5a24e39740 1 -- start start 2024-03-12T05:17:58.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.267+0000 7f5a24e39740 1 Processor -- start 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f167830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f1666e0 con 0x559d8f157b70 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f166850 con 0x559d8f160750 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8f166a10 con 0x559d8f157250 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a24e39740 10 client.0 _ll_register_callbacks cb 0x7f5a0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:58.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f167830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:45990/0 (socket says 172.21.15.45:45990) 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 -- 172.21.15.45:0/1101444009 learned_addr learned my addr 172.21.15.45:0/1101444009 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a1b7fe640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 -- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 msgr2=0x559d8f167830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f167830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 -- 172.21.15.45:0/1101444009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 msgr2=0x559d8f165c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.268+0000 7f5a20801640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f165c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f5a20801640 1 -- 172.21.15.45:0/1101444009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d8f0bd6a0 con 0x559d8f160750 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f5a1b7fe640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:58.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f5a20801640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400ab20 tx=0x7f5a140129b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2255c8dfd2d2527 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f59fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140103d0 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5a14036080 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a0c0054d0 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a14010530 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.269+0000 7f5a24e39740 10 client.26831 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559d8f07f8c0 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5a0c0054d0 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a14003b70 con 0x559d8f160750 2024-03-12T05:17:58.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 10 client.26831 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 10 client.26831 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 20 client.26831 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f1c8250 0x559d8f1ca670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 4 client.26831 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.270+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d8ee4a890 con 0x559d8f1c8250 2024-03-12T05:17:58.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.271+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f1c8250 0x559d8f1ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f5a1bfff640 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f1c8250 0x559d8f1ca670 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=652d544af13e6eb5 server_cookie=5116fd7ab45c4eaf in_seq=0 out_seq=0 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f59fb7fe640 10 client.26831 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f59fb7fe640 1 -- 172.21.15.45:0/1101444009 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d8ee4a890 con 0x559d8f1c8250 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f59fb7fe640 10 client.26831 handle_command_reply: tid=0 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f5a24e39740 1 client.26831 shutdown 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 msgr2=0x559d8f1661a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.272+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400ab20 tx=0x7f5a140129b0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 shutdown_connections 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d8f157b70 0x559d8f165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d8f1c8250 0x559d8f1ca670 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d8f160750 0x559d8f1661a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 --2- 172.21.15.45:0/1101444009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d8f157250 0x559d8f167830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 >> 172.21.15.45:0/1101444009 conn(0x559d8f151980 msgr2=0x559d8f15bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f59fb7fe640 0 client.26831 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 shutdown_connections 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 1 -- 172.21.15.45:0/1101444009 wait complete. 2024-03-12T05:17:58.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.273+0000 7f5a24e39740 20 client.26831 trim_cache size 0 max 16384 2024-03-12T05:17:58.273 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.274+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 msgr2=0x7f5a040a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.274+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 0x7f5a040a41a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009fd0 tx=0x7f5a14009600 comp rx=0 tx=0).stop 2024-03-12T05:17:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.274+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 msgr2=0x7f5a1c110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.274+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5a100057c0 tx=0x7f5a1003da20 comp rx=0 tx=0).stop 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 shutdown_connections 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a040a1ce0 0x7f5a040a41a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a1c1080e0 0x7f5a1c110560 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a1c10d730 0x7f5a1c19ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 --2- 172.21.15.45:0/1834417603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a1c107720 0x7f5a1c110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.275+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 >> 172.21.15.45:0/1834417603 conn(0x7f5a1c0756c0 msgr2=0x7f5a1c075b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.276+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 shutdown_connections 2024-03-12T05:17:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.276+0000 7f5a22264640 1 -- 172.21.15.45:0/1834417603 wait complete. 2024-03-12T05:17:58.293 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:17:58.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:17:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 -- 172.21.15.45:0/2636360295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 msgr2=0x7f68f011a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 --2- 172.21.15.45:0/2636360295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f011a2e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f68e4009900 tx=0x7f68e402bc40 comp rx=0 tx=0).stop 2024-03-12T05:17:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 -- 172.21.15.45:0/2636360295 shutdown_connections 2024-03-12T05:17:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 --2- 172.21.15.45:0/2636360295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f011a820 0x7f68f011cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 --2- 172.21.15.45:0/2636360295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f011a2e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 --2- 172.21.15.45:0/2636360295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f006cb10 0x7f68f006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 -- 172.21.15.45:0/2636360295 >> 172.21.15.45:0/2636360295 conn(0x7f68f006c180 msgr2=0x7f68f010a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 -- 172.21.15.45:0/2636360295 shutdown_connections 2024-03-12T05:17:58.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.468+0000 7f68f61b5640 1 -- 172.21.15.45:0/2636360295 wait complete. 2024-03-12T05:17:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.469+0000 7f68f61b5640 1 Processor -- start 2024-03-12T05:17:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.469+0000 7f68f61b5640 1 -- start start 2024-03-12T05:17:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f006cb10 0x7f68f01a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f01a6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01ab3d0 con 0x7f68f006cb10 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01ab540 con 0x7f68f006d430 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68f61b5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01ab700 con 0x7f68f011a820 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f006cb10 0x7f68f01a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f01a6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51276/0 (socket says 172.21.15.45:51276) 2024-03-12T05:17:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f01a6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46000/0 (socket says 172.21.15.45:46000) 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 -- 172.21.15.45:0/3418871104 learned_addr learned my addr 172.21.15.45:0/3418871104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 -- 172.21.15.45:0/3418871104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 msgr2=0x7f68f01a6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f01a6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 -- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f006cb10 msgr2=0x7f68f01a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f006cb10 0x7f68f01a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.470+0000 7f68effff640 1 -- 172.21.15.45:0/3418871104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4009560 con 0x7f68f011a820 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68effff640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f68e800e9d0 tx=0x7f68e800ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68ecff9640 1 -- 172.21.15.45:0/3418871104 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e800cd80 con 0x7f68f011a820 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68ecff9640 1 -- 172.21.15.45:0/3418871104 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f68e8005020 con 0x7f68f011a820 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68ecff9640 1 -- 172.21.15.45:0/3418871104 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e80116d0 con 0x7f68f011a820 2024-03-12T05:17:58.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f01ab9e0 con 0x7f68f011a820 2024-03-12T05:17:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.471+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f01b3260 con 0x7f68f011a820 2024-03-12T05:17:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.475+0000 7f68ecff9640 1 -- 172.21.15.45:0/3418871104 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f68e8005190 con 0x7f68f011a820 2024-03-12T05:17:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.475+0000 7f68f8d88740 1 Processor -- start 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.475+0000 7f68f8d88740 1 -- start start 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ecff9640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 0x7f68d80a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a98210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a9ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6a4a960 con 0x5594d6a8fb70 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d69fe310 con 0x5594d6a98750 2024-03-12T05:17:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68f8d88740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d69fb840 con 0x5594d6a8f250 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ecff9640 1 -- 172.21.15.45:0/3418871104 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f68e80cd910 con 0x7f68f011a820 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51280/0 (socket says 172.21.15.45:51280) 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2715234608 learned_addr learned my addr 172.21.15.45:0/2715234608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68effff640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a9ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68eeffd640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a98210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 0x7f68d80a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2715234608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 msgr2=0x5594d6a9ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a9ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2715234608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 msgr2=0x5594d6a98210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a98210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.476+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2715234608 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594d69b9060 con 0x5594d6a8f250 2024-03-12T05:17:58.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.477+0000 7f68eeffd640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a98210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.477+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009cf0 tx=0x7f68e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=147e742421fc5514 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.477+0000 7f68d5ffb640 1 -- 172.21.15.45:0/2715234608 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0009ea0 con 0x5594d6a8f250 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.477+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 0x7f68d80a4160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f68e4005ab0 tx=0x7f68e4005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.477+0000 7f68d5ffb640 1 -- 172.21.15.45:0/2715234608 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f68e0004230 con 0x5594d6a8f250 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68d5ffb640 1 -- 172.21.15.45:0/2715234608 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0019800 con 0x5594d6a8f250 2024-03-12T05:17:58.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 -- 172.21.15.45:0/2715234608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 msgr2=0x5594d6a8f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009cf0 tx=0x7f68e0011810 comp rx=0 tx=0).stop 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 -- 172.21.15.45:0/2715234608 shutdown_connections 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a98210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a9ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2715234608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a8f630 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.478+0000 7f68f8d88740 1 -- 172.21.15.45:0/2715234608 >> 172.21.15.45:0/2715234608 conn(0x5594d6a89980 msgr2=0x5594d6a89d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.479+0000 7f68f8d88740 1 -- 172.21.15.45:0/2715234608 shutdown_connections 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.479+0000 7f68f8d88740 1 -- 172.21.15.45:0/2715234608 wait complete. 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.479+0000 7f68f8d88740 1 -- start start 2024-03-12T05:17:58.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.479+0000 7f68f8d88740 1 Processor -- start 2024-03-12T05:17:58.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a95700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a95c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d69f7020 con 0x5594d6a8f250 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d66ca430 con 0x5594d6a98750 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6686e00 con 0x5594d6a8fb70 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68f8d88740 10 client.0 _ll_register_callbacks cb 0x5594d699ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46018/0 (socket says 172.21.15.45:46018) 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.480+0000 7f68effff640 1 -- 172.21.15.45:0/647969047 learned_addr learned my addr 172.21.15.45:0/647969047 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a95700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 -- 172.21.15.45:0/647969047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 msgr2=0x5594d6a95c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a95c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 -- 172.21.15.45:0/647969047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 msgr2=0x5594d6a95700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a95700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 -- 172.21.15.45:0/647969047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e0009950 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a95700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68effff640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f68e8096f30 tx=0x7f68e809a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=effb8f531912af4a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.481+0000 7f68ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e8005540 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f68e80cb250 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594d66f67e0 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68e80cb410 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68f8d88740 10 client.26843 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5594d69f56a0 con 0x5594d6a98750 2024-03-12T05:17:58.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.482+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5594d66f67e0 con 0x5594d6a98750 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f68e8018030 con 0x5594d6a98750 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 10 client.26843 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 10 client.26843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 20 client.26843 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6aa8bc0 0x5594d6aaafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 4 client.26843 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594d6733720 con 0x5594d6aa8bc0 2024-03-12T05:17:58.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.483+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6aa8bc0 0x5594d6aaafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.484+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6aa8bc0 0x5594d6aaafe0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5accb880019838bc server_cookie=f00119ae70798043 in_seq=0 out_seq=0 2024-03-12T05:17:58.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.484+0000 7f68ceffd640 10 client.26843 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:58.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.484+0000 7f68ceffd640 1 -- 172.21.15.45:0/647969047 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5594d6733720 con 0x5594d6aa8bc0 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.484+0000 7f68ceffd640 10 client.26843 handle_command_reply: tid=0 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 client.26843 shutdown 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 msgr2=0x5594d6a96180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f68e8096f30 tx=0x7f68e809a6a0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 shutdown_connections 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6aa8bc0 0x5594d6aaafe0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a96180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a95c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 --2- 172.21.15.45:0/647969047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a95700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68ceffd640 0 client.26843 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 >> 172.21.15.45:0/647969047 conn(0x5594d6a89980 msgr2=0x5594d6a8b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 shutdown_connections 2024-03-12T05:17:58.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.485+0000 7f68f8d88740 1 -- 172.21.15.45:0/647969047 wait complete. 2024-03-12T05:17:58.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.486+0000 7f68f8d88740 20 client.26843 trim_cache size 0 max 16384 2024-03-12T05:17:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.508+0000 7f68f8d88740 1 Processor -- start 2024-03-12T05:17:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 -- start start 2024-03-12T05:17:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a92960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51294/0 (socket says 172.21.15.45:51294) 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a92ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2189645165 learned_addr learned my addr 172.21.15.45:0/2189645165 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d697c130 con 0x5594d6a8fb70 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6733720 con 0x5594d6a98750 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.509+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d66f67e0 con 0x5594d6a8f250 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68effff640 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a92ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68eeffd640 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a92960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2189645165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 msgr2=0x5594d6a92ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a92ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2189645165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 msgr2=0x5594d6a92960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a92960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 -- 172.21.15.45:0/2189645165 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594d6991c20 con 0x5594d6a8f250 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009c40 tx=0x7f68e0019420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f89a848cee9cb2b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68d57fa640 1 -- 172.21.15.45:0/2189645165 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0029070 con 0x5594d6a8f250 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68d57fa640 1 -- 172.21.15.45:0/2189645165 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f68e0009ea0 con 0x5594d6a8f250 2024-03-12T05:17:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.510+0000 7f68d57fa640 1 -- 172.21.15.45:0/2189645165 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0005200 con 0x5594d6a8f250 2024-03-12T05:17:58.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.511+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 msgr2=0x5594d6a97e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.511+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009c40 tx=0x7f68e0019420 comp rx=0 tx=0).stop 2024-03-12T05:17:58.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 shutdown_connections 2024-03-12T05:17:58.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8fb70 0x5594d6a92960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6a92ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 --2- 172.21.15.45:0/2189645165 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8f250 0x5594d6a97e30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 >> 172.21.15.45:0/2189645165 conn(0x5594d6a89980 msgr2=0x5594d6a909f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 shutdown_connections 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.512+0000 7f68f8d88740 1 -- 172.21.15.45:0/2189645165 wait complete. 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.513+0000 7f68f8d88740 1 -- start start 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.513+0000 7f68f8d88740 1 Processor -- start 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.513+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a9e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a9e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68ef7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a9e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34370/0 (socket says 172.21.15.45:34370) 2024-03-12T05:17:58.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68f8d88740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a9ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68ef7fe640 1 -- 172.21.15.45:0/3992150121 learned_addr learned my addr 172.21.15.45:0/3992150121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6aff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6a9f380 con 0x5594d6a8f250 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a9ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6aff650 con 0x5594d6a98750 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.514+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594d6aff790 con 0x5594d6a8fb70 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68f8d88740 10 client.0 _ll_register_callbacks cb 0x7f68e00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68effff640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6aff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 -- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 msgr2=0x5594d6aff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6aff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 -- 172.21.15.45:0/3992150121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 msgr2=0x5594d6a9e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:58.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a9e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594d69b9060 con 0x5594d6a8fb70 2024-03-12T05:17:58.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.515+0000 7f68eeffd640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a9ec80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f68e402c180 tx=0x7f68e402bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=246d1c6bdd15e23c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:58.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.516+0000 7f68ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.516+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e40038c0 con 0x5594d6a8fb70 2024-03-12T05:17:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.516+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f68e4003a20 con 0x5594d6a8fb70 2024-03-12T05:17:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.516+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68e0004230 con 0x5594d6a8fb70 2024-03-12T05:17:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68e4003be0 con 0x5594d6a8fb70 2024-03-12T05:17:58.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 10 client.26824 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5594d69b78c0 con 0x5594d6a8fb70 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f68e0004230 con 0x5594d6a8fb70 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f68e403c040 con 0x5594d6a8fb70 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 10 client.26824 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 20 client.26824 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.517+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6b00fc0 0x5594d6b033e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.518+0000 7f68f8d88740 4 client.26824 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.518+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5594d6782890 con 0x5594d6b00fc0 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.518+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6b00fc0 0x5594d6b033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:58.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.519+0000 7f68ceffd640 10 client.26824 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.519+0000 7f68ef7fe640 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6b00fc0 0x5594d6b033e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31c33d014ad1ea0 server_cookie=4338f285129174ff in_seq=0 out_seq=0 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.519+0000 7f68ceffd640 1 -- 172.21.15.45:0/3992150121 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5594d6782890 con 0x5594d6b00fc0 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.519+0000 7f68ceffd640 10 client.26824 handle_command_reply: tid=0 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.519+0000 7f68f8d88740 1 client.26824 shutdown 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 msgr2=0x5594d6a9ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a9ec80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f68e402c180 tx=0x7f68e402bc40 comp rx=0 tx=0).stop 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 shutdown_connections 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5594d6b00fc0 0x5594d6b033e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5594d6a98750 0x5594d6aff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594d6a8fb70 0x5594d6a9ec80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 --2- 172.21.15.45:0/3992150121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5594d6a8f250 0x5594d6a9e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 >> 172.21.15.45:0/3992150121 conn(0x5594d6a89980 msgr2=0x5594d6a8adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68ceffd640 0 client.26824 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:58.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 shutdown_connections 2024-03-12T05:17:58.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.520+0000 7f68f8d88740 1 -- 172.21.15.45:0/3992150121 wait complete. 2024-03-12T05:17:58.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.521+0000 7f68f8d88740 20 client.26824 trim_cache size 0 max 16384 2024-03-12T05:17:58.519 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:17:58.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.522+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 msgr2=0x7f68d80a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.522+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 0x7f68d80a4160 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f68e4005ab0 tx=0x7f68e4005a20 comp rx=0 tx=0).stop 2024-03-12T05:17:58.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.522+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 msgr2=0x7f68f01aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:58.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.522+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f68e800e9d0 tx=0x7f68e800ee90 comp rx=0 tx=0).stop 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 shutdown_connections 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f68d80a1ca0 0x7f68d80a4160 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68f011a820 0x7f68f01aadd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68f006d430 0x7f68f01a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 --2- 172.21.15.45:0/3418871104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68f006cb10 0x7f68f01a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.523+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 >> 172.21.15.45:0/3418871104 conn(0x7f68f006c180 msgr2=0x7f68f010a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.524+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 shutdown_connections 2024-03-12T05:17:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:58.524+0000 7f68f61b5640 1 -- 172.21.15.45:0/3418871104 wait complete. 2024-03-12T05:17:59.540 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:17:59.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:17:59.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:17:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 -- 172.21.15.45:0/3428795356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2794108c80 msgr2=0x7f2794113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 --2- 172.21.15.45:0/3428795356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2794108c80 0x7f2794113d90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2784009860 tx=0x7f278402bb70 comp rx=0 tx=0).stop 2024-03-12T05:17:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 -- 172.21.15.45:0/3428795356 shutdown_connections 2024-03-12T05:17:59.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 --2- 172.21.15.45:0/3428795356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27941142d0 0x7f27941166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 --2- 172.21.15.45:0/3428795356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2794108c80 0x7f2794113d90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 --2- 172.21.15.45:0/3428795356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108360 0x7f2794108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 -- 172.21.15.45:0/3428795356 >> 172.21.15.45:0/3428795356 conn(0x7f279406c260 msgr2=0x7f279406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 -- 172.21.15.45:0/3428795356 shutdown_connections 2024-03-12T05:17:59.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.732+0000 7f279c24e640 1 -- 172.21.15.45:0/3428795356 wait complete. 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.733+0000 7f279c24e640 1 Processor -- start 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.733+0000 7f279c24e640 1 -- start start 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 0x7f27941a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 0x7f27941a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27941142d0 0x7f27941a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a7070 con 0x7f2794108c80 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a71e0 con 0x7f27941142d0 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279c24e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a73a0 con 0x7f2794108360 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f27997e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 0x7f27941a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 0x7f27941a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f27997e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 0x7f27941a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34398/0 (socket says 172.21.15.45:34398) 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f27997e9640 1 -- 172.21.15.45:0/809888726 learned_addr learned my addr 172.21.15.45:0/809888726 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f279a7eb640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27941142d0 0x7f27941a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 -- 172.21.15.45:0/809888726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27941142d0 msgr2=0x7f27941a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27941142d0 0x7f27941a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 -- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 msgr2=0x7f27941a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 0x7f27941a26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.734+0000 7f2799fea640 1 -- 172.21.15.45:0/809888726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784009490 con 0x7f2794108360 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f2799fea640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 0x7f27941a21b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f279000e9f0 tx=0x7f279000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f278affd640 1 -- 172.21.15.45:0/809888726 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279000ccc0 con 0x7f2794108360 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f278affd640 1 -- 172.21.15.45:0/809888726 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f279000ce20 con 0x7f2794108360 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27941a7680 con 0x7f2794108360 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27941aef00 con 0x7f2794108360 2024-03-12T05:17:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.735+0000 7f278affd640 1 -- 172.21.15.45:0/809888726 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27900187c0 con 0x7f2794108360 2024-03-12T05:17:59.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.738+0000 7f278affd640 1 -- 172.21.15.45:0/809888726 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2790010430 con 0x7f2794108360 2024-03-12T05:17:59.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.738+0000 7f279ee1d740 1 Processor -- start 2024-03-12T05:17:59.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 -- start start 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79f380 0x55870f79f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 0x55870f7a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f70df90 con 0x55870f79fd30 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f70b4c0 con 0x55870f7a88b0 2024-03-12T05:17:59.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279ee1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f706ca0 con 0x55870f79f380 2024-03-12T05:17:59.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f2799fea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 0x55870f7a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f27997e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f27997e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46052/0 (socket says 172.21.15.45:46052) 2024-03-12T05:17:59.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f27997e9640 1 -- 172.21.15.45:0/1872338035 learned_addr learned my addr 172.21.15.45:0/1872338035 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.739+0000 7f279a7eb640 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79f380 0x55870f79f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 -- 172.21.15.45:0/1872338035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79f380 msgr2=0x55870f79f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79f380 0x55870f79f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 -- 172.21.15.45:0/1872338035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 msgr2=0x55870f7a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f278affd640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 0x7f27800a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 0x55870f7a82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 -- 172.21.15.45:0/1872338035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55870f6c90b0 con 0x55870f7a88b0 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f278affd640 1 -- 172.21.15.45:0/809888726 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2790014070 con 0x7f2794108360 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f2799fea640 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 0x55870f7a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:17:59.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f279a7eb640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 0x7f27800a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.740+0000 7f27997e9640 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2784009980 tx=0x7f2784032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87fd327fe00a2e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.741+0000 7f277ffff640 1 -- 172.21.15.45:0/1872338035 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784034df0 con 0x55870f7a88b0 2024-03-12T05:17:59.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.741+0000 7f277ffff640 1 -- 172.21.15.45:0/1872338035 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f278402ce90 con 0x55870f7a88b0 2024-03-12T05:17:59.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.741+0000 7f277ffff640 1 -- 172.21.15.45:0/1872338035 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278402c770 con 0x55870f7a88b0 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 -- 172.21.15.45:0/1872338035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 msgr2=0x55870f7aad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2784009980 tx=0x7f2784032430 comp rx=0 tx=0).stop 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279a7eb640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 0x7f27800a4120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f279406a5a0 tx=0x7f278c010040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 -- 172.21.15.45:0/1872338035 shutdown_connections 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f79fd30 0x55870f7a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f7a88b0 0x55870f7aad00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1872338035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79f380 0x55870f79f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 -- 172.21.15.45:0/1872338035 >> 172.21.15.45:0/1872338035 conn(0x55870f799840 msgr2=0x55870f799c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 -- 172.21.15.45:0/1872338035 shutdown_connections 2024-03-12T05:17:59.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.742+0000 7f279ee1d740 1 -- 172.21.15.45:0/1872338035 wait complete. 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 -- start start 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 Processor -- start 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f6abf40 con 0x55870f7a88b0 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f3da4b0 con 0x55870f79f380 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f68c670 con 0x55870f79fd30 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f279ee1d740 10 client.0 _ll_register_callbacks cb 0x55870f6f9f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.743+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51334/0 (socket says 172.21.15.45:51334) 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 -- 172.21.15.45:0/2852056201 learned_addr learned my addr 172.21.15.45:0/2852056201 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f279a7eb640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f2799fea640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 -- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 msgr2=0x55870f7a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 -- 172.21.15.45:0/2852056201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 msgr2=0x55870f7a6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 -- 172.21.15.45:0/2852056201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2794116ff0 con 0x55870f79fd30 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f27997e9640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2784034920 tx=0x7f2784004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7babafaf1a895e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f277d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.744+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784034740 con 0x55870f79fd30 2024-03-12T05:17:59.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2784034440 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55870f6e0f40 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f278402c510 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 10 client.26836 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55870f705320 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f278405f020 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2784032c00 con 0x55870f79fd30 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 10 client.26836 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 10 client.26836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 20 client.26836 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f7c1080 0x55870f7c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 4 client.26836 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:17:59.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.745+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55870f6fd310 con 0x55870f7c1080 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.746+0000 7f2799fea640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f7c1080 0x55870f7c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.746+0000 7f277d7fa640 10 client.26836 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.746+0000 7f2799fea640 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f7c1080 0x55870f7c34a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a395724d42b3263e server_cookie=af84748083ea2fdf in_seq=0 out_seq=0 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f277d7fa640 1 -- 172.21.15.45:0/2852056201 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55870f6fd310 con 0x55870f7c1080 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f277d7fa640 10 client.26836 handle_command_reply: tid=0 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 client.26836 shutdown 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 msgr2=0x55870f7a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2784034920 tx=0x7f2784004270 comp rx=0 tx=0).stop 2024-03-12T05:17:59.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 shutdown_connections 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f7c1080 0x55870f7c34a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a5cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 --2- 172.21.15.45:0/2852056201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 >> 172.21.15.45:0/2852056201 conn(0x55870f799840 msgr2=0x55870f79b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.747+0000 7f277d7fa640 0 client.26836 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.748+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 shutdown_connections 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.748+0000 7f279ee1d740 1 -- 172.21.15.45:0/2852056201 wait complete. 2024-03-12T05:17:59.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.748+0000 7f279ee1d740 20 client.26836 trim_cache size 0 max 16384 2024-03-12T05:17:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.776+0000 7f279ee1d740 1 Processor -- start 2024-03-12T05:17:59.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.776+0000 7f279ee1d740 1 -- start start 2024-03-12T05:17:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.776+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.776+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.776+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279a7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279a7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34430/0 (socket says 172.21.15.45:34430) 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51350/0 (socket says 172.21.15.45:51350) 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279a7eb640 1 -- 172.21.15.45:0/828306339 learned_addr learned my addr 172.21.15.45:0/828306339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f79a640 con 0x55870f7a88b0 2024-03-12T05:17:59.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f7a1390 con 0x55870f79f380 2024-03-12T05:17:59.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f7be0c0 con 0x55870f79fd30 2024-03-12T05:17:59.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f2799fea640 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.777+0000 7f279a7eb640 1 -- 172.21.15.45:0/828306339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 msgr2=0x55870f7a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.778+0000 7f279a7eb640 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.778+0000 7f279a7eb640 1 -- 172.21.15.45:0/828306339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 msgr2=0x55870f7a7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.778+0000 7f279a7eb640 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.778+0000 7f279a7eb640 1 -- 172.21.15.45:0/828306339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55870f6a1c00 con 0x55870f7a88b0 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.778+0000 7f279a7eb640 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f278c003d60 tx=0x7f278c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c959dd61b04641b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.779+0000 7f277f7fe640 1 -- 172.21.15.45:0/828306339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c013040 con 0x55870f7a88b0 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.779+0000 7f277f7fe640 1 -- 172.21.15.45:0/828306339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f278c026b20 con 0x55870f7a88b0 2024-03-12T05:17:59.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.779+0000 7f277f7fe640 1 -- 172.21.15.45:0/828306339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c00dbf0 con 0x55870f7a88b0 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.780+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 msgr2=0x55870f7a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.780+0000 7f279ee1d740 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f278c003d60 tx=0x7f278c016000 comp rx=0 tx=0).stop 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 shutdown_connections 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7a29e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f7a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 --2- 172.21.15.45:0/828306339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f7a7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 >> 172.21.15.45:0/828306339 conn(0x55870f799840 msgr2=0x55870f7c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 shutdown_connections 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.781+0000 7f279ee1d740 1 -- 172.21.15.45:0/828306339 wait complete. 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.782+0000 7f279ee1d740 1 -- start start 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.782+0000 7f279ee1d740 1 Processor -- start 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f810900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f279ee1d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7b7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f2799fea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f810900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f2799fea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f810900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46070/0 (socket says 172.21.15.45:46070) 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f27997e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51360/0 (socket says 172.21.15.45:51360) 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f2799fea640 1 -- 172.21.15.45:0/1288984541 learned_addr learned my addr 172.21.15.45:0/1288984541 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.783+0000 7f279a7eb640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f7b7a20 con 0x55870f7a88b0 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f7b7b90 con 0x55870f79f380 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870f7b7d50 con 0x55870f79fd30 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f279ee1d740 10 client.0 _ll_register_callbacks cb 0x7f2790005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f27997e9640 1 -- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 msgr2=0x55870f810900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f27997e9640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f810900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f27997e9640 1 -- 172.21.15.45:0/1288984541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 msgr2=0x55870f7b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f27997e9640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7b7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.784+0000 7f27997e9640 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c003a10 con 0x55870f79fd30 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.785+0000 7f27997e9640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2784009830 tx=0x7f2784003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=928f93b1d61871a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.785+0000 7f277d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.785+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784005c90 con 0x55870f79fd30 2024-03-12T05:17:59.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.785+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2784000e40 con 0x55870f79fd30 2024-03-12T05:17:59.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.786+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55870f72c900 con 0x55870f79fd30 2024-03-12T05:17:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.786+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2784034440 con 0x55870f79fd30 2024-03-12T05:17:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.786+0000 7f279ee1d740 10 client.26842 fetch_fsmap learned FSMap version 33 2024-03-12T05:17:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.786+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55870f6c7910 con 0x55870f79fd30 2024-03-12T05:17:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f278405f040 con 0x55870f79fd30 2024-03-12T05:17:59.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f278402c7a0 con 0x55870f79fd30 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 10 client.26842 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 10 client.26842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 20 client.26842 populate_metadata read hostname 'smithi045' 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f8193f0 0x55870f81b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 4 client.26842 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55870f72c900 con 0x55870f8193f0 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.787+0000 7f2799fea640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f8193f0 0x55870f81b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.788+0000 7f277d7fa640 10 client.26842 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.788+0000 7f2799fea640 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f8193f0 0x55870f81b810 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f4e0e8d9e035139 server_cookie=4e67e16dd263cae in_seq=0 out_seq=0 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.789+0000 7f277d7fa640 1 -- 172.21.15.45:0/1288984541 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55870f72c900 con 0x55870f8193f0 2024-03-12T05:17:59.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.789+0000 7f277d7fa640 10 client.26842 handle_command_reply: tid=0 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.789+0000 7f279ee1d740 1 client.26842 shutdown 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.789+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 msgr2=0x55870f810e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.789+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2784009830 tx=0x7f2784003700 comp rx=0 tx=0).stop 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 shutdown_connections 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55870f8193f0 0x55870f81b810 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55870f7a88b0 0x55870f7b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55870f79fd30 0x55870f810e40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 --2- 172.21.15.45:0/1288984541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55870f79f380 0x55870f810900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 >> 172.21.15.45:0/1288984541 conn(0x55870f799840 msgr2=0x55870f798050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f277d7fa640 0 client.26842 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 shutdown_connections 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 1 -- 172.21.15.45:0/1288984541 wait complete. 2024-03-12T05:17:59.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.790+0000 7f279ee1d740 20 client.26842 trim_cache size 0 max 16384 2024-03-12T05:17:59.789 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:17:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:17:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "20df9a45-ffd9-43ae-80e4-a06f73ebd619", 2024-03-12T05:17:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:17:59.789 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:17:59.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.791+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 msgr2=0x7f27800a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.791+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 0x7f27800a4120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f279406a5a0 tx=0x7f278c010040 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.791+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 msgr2=0x7f27941a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.791+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 0x7f27941a21b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f279000e9f0 tx=0x7f279000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.791+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 shutdown_connections 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f27800a1c60 0x7f27800a4120 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2794108c80 0x7f27941a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27941142d0 0x7f27941a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 --2- 172.21.15.45:0/809888726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2794108360 0x7f27941a21b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 >> 172.21.15.45:0/809888726 conn(0x7f279406c260 msgr2=0x7f2794109c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 shutdown_connections 2024-03-12T05:17:59.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.792+0000 7f279c24e640 1 -- 172.21.15.45:0/809888726 wait complete. 2024-03-12T05:17:59.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:17:59.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:17:59.849 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:17:59 smithi045 ceph-mon[32174]: pgmap v218: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 17 MiB/s wr, 1.85k op/s 2024-03-12T05:17:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 355.0 s, 1355.4 tps, lat 73.646 ms stddev 22.363 2024-03-12T05:17:59.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:17:59 smithi130 ceph-mon[35398]: pgmap v218: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 17 MiB/s wr, 1.85k op/s 2024-03-12T05:17:59.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 -- 172.21.15.45:0/333210749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 msgr2=0x7f8b9c0b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/333210749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c0b3c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4066e50 tx=0x7f8ba408eaf0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 -- 172.21.15.45:0/333210749 shutdown_connections 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/333210749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c0a5950 0x7f8b9c0ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/333210749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c0b3c40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/333210749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c0a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 -- 172.21.15.45:0/333210749 >> 172.21.15.45:0/333210749 conn(0x7f8b9c01a620 msgr2=0x7f8b9c01aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 -- 172.21.15.45:0/333210749 shutdown_connections 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.992+0000 7f8ba94d8640 1 -- 172.21.15.45:0/333210749 wait complete. 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.993+0000 7f8ba94d8640 1 Processor -- start 2024-03-12T05:17:59.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.993+0000 7f8ba94d8640 1 -- start start 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.993+0000 7f8ba94d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba94d8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba94d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 0x7f8b9c141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba94d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c142430 con 0x7f8b9c141a40 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba94d8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c1425a0 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba94d8640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c142710 con 0x7f8b9c0ac0e0 2024-03-12T05:17:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 0x7f8b9c141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:46094/0 (socket says 172.21.15.45:46094) 2024-03-12T05:17:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 0x7f8b9c141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34456/0 (socket says 172.21.15.45:34456) 2024-03-12T05:17:59.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 -- 172.21.15.45:0/1243321610 learned_addr learned my addr 172.21.15.45:0/1243321610 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 -- 172.21.15.45:0/1243321610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 msgr2=0x7f8b9c13d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c13d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 -- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 msgr2=0x7f8b9c141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 0x7f8b9c141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.994+0000 7f8ba3fff640 1 -- 172.21.15.45:0/1243321610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba404f750 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800e9f0 tx=0x7f8b9800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba17fa640 1 -- 172.21.15.45:0/1243321610 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98009700 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c0a66d0 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba17fa640 1 -- 172.21.15.45:0/1243321610 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b9800cd80 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba17fa640 1 -- 172.21.15.45:0/1243321610 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98018840 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.995+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c0a6bd0 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:17:59 smithi116 ceph-mon[35486]: pgmap v218: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 17 MiB/s wr, 1.85k op/s 2024-03-12T05:17:59.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.998+0000 7f8ba17fa640 1 -- 172.21.15.45:0/1243321610 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8b980189a0 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.999+0000 7f8bad3e8740 1 Processor -- start 2024-03-12T05:17:59.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:17:59.999+0000 7f8bad3e8740 1 -- start start 2024-03-12T05:17:59.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eef600 0x5645f6ef7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6e55280 con 0x5645f6eef600 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6e527b0 con 0x5645f6ef81e0 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6e4df90 con 0x5645f6eeece0 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.000+0000 7f8ba17fa640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 0x7f8b940a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba17fa640 1 -- 172.21.15.45:0/1243321610 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8b98014070 con 0x7f8b9c0a4fa0 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51378/0 (socket says 172.21.15.45:51378) 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba8cd7640 1 -- 172.21.15.45:0/1098302905 learned_addr learned my addr 172.21.15.45:0/1098302905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eef600 0x5645f6ef7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.001+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 0x7f8b940a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 -- 172.21.15.45:0/1098302905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 msgr2=0x5645f6efa5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efa5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 -- 172.21.15.45:0/1098302905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eef600 msgr2=0x5645f6ef7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eef600 0x5645f6ef7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:17:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 -- 172.21.15.45:0/1098302905 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645f6e0c570 con 0x5645f6eeece0 2024-03-12T05:18:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4064670 tx=0x7f8ba4067b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34bcdd18e3bc6d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 0x7f8b940a40e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4066920 tx=0x7f8ba40a7000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/1098302905 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40a0070 con 0x5645f6eeece0 2024-03-12T05:18:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/1098302905 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ba40ac920 con 0x5645f6eeece0 2024-03-12T05:18:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.002+0000 7f8b8e7fc640 1 -- 172.21.15.45:0/1098302905 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40b4760 con 0x5645f6eeece0 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 -- 172.21.15.45:0/1098302905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 msgr2=0x5645f6eef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4064670 tx=0x7f8ba4067b60 comp rx=0 tx=0).stop 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 -- 172.21.15.45:0/1098302905 shutdown_connections 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eef600 0x5645f6ef7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/1098302905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6eef0c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 -- 172.21.15.45:0/1098302905 >> 172.21.15.45:0/1098302905 conn(0x5645f6ee9410 msgr2=0x5645f6ee97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 -- 172.21.15.45:0/1098302905 shutdown_connections 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.003+0000 7f8bad3e8740 1 -- 172.21.15.45:0/1098302905 wait complete. 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 -- start start 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 Processor -- start 2024-03-12T05:18:00.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 0x5645f6ef7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6ef5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6b21430 con 0x5645f6ef81e0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6adde00 con 0x5645f6eeece0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6dd2fd0 con 0x5645f6eef600 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.004+0000 7f8bad3e8740 10 client.0 _ll_register_callbacks cb 0x5645f6e28fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 0x5645f6ef7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba8cd7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34482/0 (socket says 172.21.15.45:34482) 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba8cd7640 1 -- 172.21.15.45:0/3246665365 learned_addr learned my addr 172.21.15.45:0/3246665365 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6ef5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3246665365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 msgr2=0x5645f6ef5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6ef5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3246665365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 msgr2=0x5645f6ef56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 -- 172.21.15.45:0/3246665365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba40a3040 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 0x5645f6ef7c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8b980cb1e0 tx=0x7f8b9809a830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6353d3881b7b6b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.005+0000 7f8b837fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980cb460 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8b980cb5c0 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5645f6b4d7e0 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b980a0d90 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8bad3e8740 10 client.26876 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5645f6e4c610 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b980b7070 con 0x5645f6eeece0 2024-03-12T05:18:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.006+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8b9809a9d0 con 0x5645f6eeece0 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 10 client.26876 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 10 client.26876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 20 client.26876 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f08390 0x5645f6f0a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 4 client.26876 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645f6b4d7e0 con 0x5645f6f08390 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.007+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f08390 0x5645f6f0a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.008+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f08390 0x5645f6f0a7f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d4c6d7dd420e931 server_cookie=ca3d3c304b8355cf in_seq=0 out_seq=0 2024-03-12T05:18:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.008+0000 7f8b837fe640 10 client.26876 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:00.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.009+0000 7f8b837fe640 1 -- 172.21.15.45:0/3246665365 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5645f6b4d7e0 con 0x5645f6f08390 2024-03-12T05:18:00.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.009+0000 7f8b837fe640 10 client.26876 handle_command_reply: tid=0 2024-03-12T05:18:00.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.009+0000 7f8bad3e8740 1 client.26876 shutdown 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.009+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 msgr2=0x5645f6ef7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.009+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 0x5645f6ef7c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8b980cb1e0 tx=0x7f8b9809a830 comp rx=0 tx=0).stop 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 shutdown_connections 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f08390 0x5645f6f0a7f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6ef56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8b837fe640 0 client.26876 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6ef5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/3246665365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eeece0 0x5645f6ef7c70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 >> 172.21.15.45:0/3246665365 conn(0x5645f6ee9410 msgr2=0x5645f6edfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 shutdown_connections 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 1 -- 172.21.15.45:0/3246665365 wait complete. 2024-03-12T05:18:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.010+0000 7f8bad3e8740 20 client.26876 trim_cache size 0 max 16384 2024-03-12T05:18:00.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.038+0000 7f8bad3e8740 1 Processor -- start 2024-03-12T05:18:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.038+0000 7f8bad3e8740 1 -- start start 2024-03-12T05:18:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eef600 0x5645f6edbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51388/0 (socket says 172.21.15.45:51388) 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8ba3fff640 1 -- 172.21.15.45:0/194551288 learned_addr learned my addr 172.21.15.45:0/194551288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6eff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eef600 0x5645f6edbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6eff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6eea410 con 0x5645f6ef81e0 2024-03-12T05:18:00.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6eeddd0 con 0x5645f6eef600 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.039+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6eee940 con 0x5645f6eeece0 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 -- 172.21.15.45:0/194551288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eef600 msgr2=0x5645f6edbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eef600 0x5645f6edbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 -- 172.21.15.45:0/194551288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 msgr2=0x5645f6eff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6eff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 -- 172.21.15.45:0/194551288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645f6de8ac0 con 0x5645f6eeece0 2024-03-12T05:18:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a2930 tx=0x7f8b980a2fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9fdd2b3464a718 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.040+0000 7f8b8dffb640 1 -- 172.21.15.45:0/194551288 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9809e810 con 0x5645f6eeece0 2024-03-12T05:18:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.041+0000 7f8b8dffb640 1 -- 172.21.15.45:0/194551288 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8b980b7030 con 0x5645f6eeece0 2024-03-12T05:18:00.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.041+0000 7f8b8dffb640 1 -- 172.21.15.45:0/194551288 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980850b0 con 0x5645f6eeece0 2024-03-12T05:18:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 msgr2=0x5645f6f0a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a2930 tx=0x7f8b980a2fe0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 shutdown_connections 2024-03-12T05:18:00.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6ef81e0 0x5645f6eff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6eef600 0x5645f6edbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/194551288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eeece0 0x5645f6f0a780 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 >> 172.21.15.45:0/194551288 conn(0x5645f6ee9410 msgr2=0x5645f6ee97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 shutdown_connections 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- 172.21.15.45:0/194551288 wait complete. 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.042+0000 7f8bad3e8740 1 -- start start 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8bad3e8740 1 Processor -- start 2024-03-12T05:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8bad3e8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34500/0 (socket says 172.21.15.45:34500) 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8ba3fff640 1 -- 172.21.15.45:0/2355028241 learned_addr learned my addr 172.21.15.45:0/2355028241 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6efd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.043+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6efd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6efe290 con 0x5645f6eeece0 2024-03-12T05:18:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6efe400 con 0x5645f6ef81e0 2024-03-12T05:18:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645f6f5e920 con 0x5645f6eef600 2024-03-12T05:18:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba8cd7640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8bad3e8740 10 client.0 _ll_register_callbacks cb 0x7f8b98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 -- 172.21.15.45:0/2355028241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 msgr2=0x5645f6efd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6efd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 -- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 msgr2=0x5645f6efdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645f6e4c610 con 0x5645f6eeece0 2024-03-12T05:18:00.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8ba3fff640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a0120 tx=0x7f8b9809a830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4017541476c5d21e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8b837fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.044+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980a2c40 con 0x5645f6eeece0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8b9809e810 con 0x5645f6eeece0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ba40ac920 con 0x5645f6eeece0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b98099bc0 con 0x5645f6eeece0 2024-03-12T05:18:00.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 10 client.17403 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5645f6e0e6a0 con 0x5645f6eeece0 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ba40ac920 con 0x5645f6eeece0 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8b98099630 con 0x5645f6eeece0 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 10 client.17403 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:00.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.045+0000 7f8bad3e8740 20 client.17403 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f5fb30 0x5645f6f61f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8bad3e8740 4 client.17403 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5645f6b8a720 con 0x5645f6f5fb30 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f5fb30 0x5645f6f61f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8ba37fe640 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f5fb30 0x5645f6f61f20 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca4bd19e4631d1aa server_cookie=150ec6baa5f738a9 in_seq=0 out_seq=0 2024-03-12T05:18:00.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.046+0000 7f8b837fe640 10 client.17403 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:00.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8b837fe640 1 -- 172.21.15.45:0/2355028241 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5645f6b8a720 con 0x5645f6f5fb30 2024-03-12T05:18:00.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8b837fe640 10 client.17403 handle_command_reply: tid=0 2024-03-12T05:18:00.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 client.17403 shutdown 2024-03-12T05:18:00.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 msgr2=0x5645f6eff330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a0120 tx=0x7f8b9809a830 comp rx=0 tx=0).stop 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 shutdown_connections 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5645f6f5fb30 0x5645f6f61f20 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f6ef81e0 0x5645f6efdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8b837fe640 0 client.17403 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645f6eef600 0x5645f6efd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 --2- 172.21.15.45:0/2355028241 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5645f6eeece0 0x5645f6eff330 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 >> 172.21.15.45:0/2355028241 conn(0x5645f6ee9410 msgr2=0x5645f6edbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 shutdown_connections 2024-03-12T05:18:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 1 -- 172.21.15.45:0/2355028241 wait complete. 2024-03-12T05:18:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.047+0000 7f8bad3e8740 20 client.17403 trim_cache size 0 max 16384 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: "20df9a45-ffd9-43ae-80e4-a06f73ebd619": { 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:18:00.061 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "20df9a45-ffd9-43ae-80e4-a06f73ebd619", 2024-03-12T05:18:00.062 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:18:00.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:18:00.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:18:00.062 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 msgr2=0x7f8b940a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 0x7f8b940a40e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4066920 tx=0x7f8ba40a7000 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 msgr2=0x7f8b9c13d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800e9f0 tx=0x7f8b9800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 shutdown_connections 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8b940a1c20 0x7f8b940a40e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b9c141a40 0x7f8b9c141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c0ac0e0 0x7f8b9c13d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.048+0000 7f8ba94d8640 1 --2- 172.21.15.45:0/1243321610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b9c0a4fa0 0x7f8b9c13d1d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.049+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 >> 172.21.15.45:0/1243321610 conn(0x7f8b9c01a620 msgr2=0x7f8b9c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.049+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 shutdown_connections 2024-03-12T05:18:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:00.049+0000 7f8ba94d8640 1 -- 172.21.15.45:0/1243321610 wait complete. 2024-03-12T05:18:00.065 INFO:tasks.cephfs.filesystem:scrub status for tag:20df9a45-ffd9-43ae-80e4-a06f73ebd619 - {'path': '/', 'tag': '20df9a45-ffd9-43ae-80e4-a06f73ebd619', 'options': 'recursive,force'} 2024-03-12T05:18:00.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:00 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:00.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:00 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:00.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:00 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:00.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:00 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:00.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:00 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:00.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:00 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:00.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:00 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:00.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:00 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:00 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:00 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:00 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:00 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:18:01.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:01.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:01.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:01 smithi130 ceph-mon[35398]: pgmap v219: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:01.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:01 smithi116 ceph-mon[35486]: pgmap v219: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:01 smithi045 ceph-mon[32174]: pgmap v219: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:03.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:03 smithi130 ceph-mon[35398]: pgmap v220: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:03.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:03 smithi116 ceph-mon[35486]: pgmap v220: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:04.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:03 smithi045 ceph-mon[32174]: pgmap v220: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:18:04.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 360.0 s, 1375.8 tps, lat 72.778 ms stddev 33.287 2024-03-12T05:18:05.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:05 smithi130 ceph-mon[35398]: pgmap v221: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.83k op/s 2024-03-12T05:18:05.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:05 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:18:05.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:05 smithi116 ceph-mon[35486]: pgmap v221: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.83k op/s 2024-03-12T05:18:05.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:05 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:18:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:05 smithi045 ceph-mon[32174]: pgmap v221: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 17 MiB/s wr, 1.83k op/s 2024-03-12T05:18:06.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:05 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:18:06.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:06 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:18:06.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:06 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:18:06.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:06 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:18:06.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:06 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:18:06.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:06 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:18:06.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:06 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:18:07.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:06 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:18:07.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:06 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:18:07.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:06 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:18:07.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:07 smithi130 ceph-mon[35398]: pgmap v222: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-03-12T05:18:07.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:07 smithi116 ceph-mon[35486]: pgmap v222: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-03-12T05:18:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:07 smithi045 ceph-mon[32174]: pgmap v222: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 1.17k op/s 2024-03-12T05:18:09.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:09 smithi045 ceph-mon[32174]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T05:18:09.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:09 smithi045 ceph-mon[32174]: pgmap v224: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:18:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 365.0 s, 1448.0 tps, lat 69.046 ms stddev 21.505 2024-03-12T05:18:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:09 smithi130 ceph-mon[35398]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T05:18:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:09 smithi130 ceph-mon[35398]: pgmap v224: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:18:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:09 smithi116 ceph-mon[35486]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T05:18:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:09 smithi116 ceph-mon[35486]: pgmap v224: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-03-12T05:18:11.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:11 smithi130 ceph-mon[35398]: pgmap v225: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:11.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:11 smithi116 ceph-mon[35486]: pgmap v225: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:12.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:11 smithi045 ceph-mon[32174]: pgmap v225: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:13.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:13 smithi130 ceph-mon[35398]: pgmap v226: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:13.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:13.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:13.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:13.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:13 smithi116 ceph-mon[35486]: pgmap v226: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:13 smithi045 ceph-mon[32174]: pgmap v226: 97 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 90 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:18:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:14.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 370.0 s, 1512.8 tps, lat 66.083 ms stddev 20.475 2024-03-12T05:18:15.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:15 smithi130 ceph-mon[35398]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:15.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:15 smithi116 ceph-mon[35486]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:16.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:15 smithi045 ceph-mon[32174]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:17.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:17 smithi130 ceph-mon[35398]: pgmap v228: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:17.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:17 smithi116 ceph-mon[35486]: pgmap v228: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:18.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:17 smithi045 ceph-mon[32174]: pgmap v228: 97 pgs: 97 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 13 MiB/s wr, 1.49k op/s 2024-03-12T05:18:19.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 375.0 s, 1273.4 tps, lat 78.476 ms stddev 49.045 2024-03-12T05:18:20.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:20 smithi130 ceph-mon[35398]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:18:20.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:20 smithi130 ceph-mon[35398]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T05:18:20.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:20 smithi116 ceph-mon[35486]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:18:20.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:20 smithi116 ceph-mon[35486]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T05:18:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:20 smithi045 ceph-mon[32174]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:18:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:20 smithi045 ceph-mon[32174]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T05:18:22.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:22 smithi130 ceph-mon[35398]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:22.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:22 smithi116 ceph-mon[35486]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:22 smithi045 ceph-mon[32174]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:24.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:24 smithi130 ceph-mon[35398]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:24.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:24 smithi116 ceph-mon[35486]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:24 smithi045 ceph-mon[32174]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:18:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 380.0 s, 1445.4 tps, lat 68.997 ms stddev 20.440 2024-03-12T05:18:26.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:26 smithi130 ceph-mon[35398]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:26.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:26 smithi116 ceph-mon[35486]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:26.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:26 smithi045 ceph-mon[32174]: pgmap v233: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:28.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:28 smithi130 ceph-mon[35398]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:28.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:28 smithi116 ceph-mon[35486]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:28 smithi045 ceph-mon[32174]: pgmap v234: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 13 MiB/s wr, 1.36k op/s 2024-03-12T05:18:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 385.0 s, 1333.2 tps, lat 75.236 ms stddev 24.889 2024-03-12T05:18:30.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:18:30.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:18:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 -- 172.21.15.45:0/3103631832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 msgr2=0x7f7f04073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/3103631832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f04073900 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009930 tx=0x7f7ef402bc30 comp rx=0 tx=0).stop 2024-03-12T05:18:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 -- 172.21.15.45:0/3103631832 shutdown_connections 2024-03-12T05:18:30.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/3103631832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04073e40 0x7f7f04111bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/3103631832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04112100 0x7f7f041144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/3103631832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f04073900 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.296+0000 7f7f0c62e640 1 -- 172.21.15.45:0/3103631832 >> 172.21.15.45:0/3103631832 conn(0x7f7f040fcc20 msgr2=0x7f7f040ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.297+0000 7f7f0c62e640 1 -- 172.21.15.45:0/3103631832 shutdown_connections 2024-03-12T05:18:30.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.297+0000 7f7f0c62e640 1 -- 172.21.15.45:0/3103631832 wait complete. 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.297+0000 7f7f0c62e640 1 Processor -- start 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.297+0000 7f7f0c62e640 1 -- start start 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04073e40 0x7f7f041a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 0x7f7f0419db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0419e2a0 con 0x7f7f04112100 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0419e410 con 0x7f7f04073e40 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0c62e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0419e5d0 con 0x7f7f04073520 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46906/0 (socket says 172.21.15.45:46906) 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/2418722158 learned_addr learned my addr 172.21.15.45:0/2418722158 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04073e40 0x7f7f041a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 0x7f7f0419db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/2418722158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04073e40 msgr2=0x7f7f041a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04073e40 0x7f7f041a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 msgr2=0x7f7f0419db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 0x7f7f0419db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.298+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/2418722158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4009560 con 0x7f7f04073520 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.299+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 0x7f7f0419db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:18:30.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.299+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4005ab0 tx=0x7f7ef4005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.300+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2418722158 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4004350 con 0x7f7f04073520 2024-03-12T05:18:30.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.300+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2418722158 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ef40044b0 con 0x7f7f04073520 2024-03-12T05:18:30.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.300+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2418722158 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40053d0 con 0x7f7f04073520 2024-03-12T05:18:30.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.300+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0419e850 con 0x7f7f04073520 2024-03-12T05:18:30.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.300+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f041aa9d0 con 0x7f7f04073520 2024-03-12T05:18:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.304+0000 7f7f0f202740 1 Processor -- start 2024-03-12T05:18:30.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.305+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2418722158 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7ef4005530 con 0x7f7f04073520 2024-03-12T05:18:30.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.305+0000 7f7f0f202740 1 -- start start 2024-03-12T05:18:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7b7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc726280 con 0x5563dc7c07b0 2024-03-12T05:18:30.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc7237b0 con 0x5563dc7b72b0 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0f202740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc71ef90 con 0x5563dc7b7bd0 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0abcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0abcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7b7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52040/0 (socket says 172.21.15.45:52040) 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0abcb640 1 -- 172.21.15.45:0/914383720 learned_addr learned my addr 172.21.15.45:0/914383720 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7efb7fe640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 0x7f7ef00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/914383720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 msgr2=0x5563dc7b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.306+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7b7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/914383720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 msgr2=0x5563dc7c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 0x7f7ef00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/914383720 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563dc6dd570 con 0x5563dc7b7bd0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7efb7fe640 1 -- 172.21.15.45:0/2418722158 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7ef402de40 con 0x7f7f04073520 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c0270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40ed6d0 tx=0x7f7ef407d6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2daebe3e9fddabb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7ef8bfe640 1 -- 172.21.15.45:0/914383720 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40f32f0 con 0x5563dc7b7bd0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7ef8bfe640 1 -- 172.21.15.45:0/914383720 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ef4036d30 con 0x5563dc7b7bd0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 0x7f7ef00a40e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0125f0 tx=0x7f7efc004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.307+0000 7f7ef8bfe640 1 -- 172.21.15.45:0/914383720 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40c5350 con 0x5563dc7b7bd0 2024-03-12T05:18:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.308+0000 7f7f0f202740 1 -- 172.21.15.45:0/914383720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 msgr2=0x5563dc7c0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.308+0000 7f7f0f202740 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c0270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40ed6d0 tx=0x7f7ef407d6c0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.309+0000 7f7f0f202740 1 -- 172.21.15.45:0/914383720 shutdown_connections 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.309+0000 7f7f0f202740 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.309+0000 7f7f0f202740 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c0270 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.309+0000 7f7f0f202740 1 --2- 172.21.15.45:0/914383720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.309+0000 7f7f0f202740 1 -- 172.21.15.45:0/914383720 >> 172.21.15.45:0/914383720 conn(0x5563dc7b1990 msgr2=0x5563dc7b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.310+0000 7f7f0f202740 1 -- 172.21.15.45:0/914383720 shutdown_connections 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.310+0000 7f7f0f202740 1 -- 172.21.15.45:0/914383720 wait complete. 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.310+0000 7f7f0f202740 1 -- start start 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 Processor -- start 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7c0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 0x5563dc7bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc3f2430 con 0x5563dc7b72b0 2024-03-12T05:18:30.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.311+0000 7f7f0f202740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc3aee00 con 0x5563dc7c07b0 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0f202740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc6a3fd0 con 0x5563dc7b7bd0 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f09bc9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f09bc9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7bd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46914/0 (socket says 172.21.15.45:46914) 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f09bc9640 1 -- 172.21.15.45:0/1727521385 learned_addr learned my addr 172.21.15.45:0/1727521385 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0f202740 10 client.0 _ll_register_callbacks cb 0x5563dc6f9fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7c0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 0x5563dc7bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/1727521385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 msgr2=0x5563dc7bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7bd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 msgr2=0x5563dc7bdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 0x5563dc7bdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/1727521385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f041aa9d0 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.312+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 0x5563dc7bdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.313+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7c0250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40bae60 tx=0x7f7ef40eda70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae1a47e3c69b913 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.313+0000 7f7eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.313+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40d4070 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.313+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563dc41e7e0 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.313+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7ef40bcbb0 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ef40bd2b0 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 10 client.17409 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5563dc71d610 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ef409f080 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7ef403c070 con 0x5563dc7b72b0 2024-03-12T05:18:30.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 10 client.17409 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 10 client.17409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 20 client.17409 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc7cff90 0x5563dc7d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.314+0000 7f7f0f202740 4 client.17409 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.315+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563dc45b720 con 0x5563dc7cff90 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.315+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc7cff90 0x5563dc7d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.315+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc7cff90 0x5563dc7d2380 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=467e5439859f1407 server_cookie=42939f5fc93b3afe in_seq=0 out_seq=0 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.315+0000 7f7eedffb640 10 client.17409 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.316+0000 7f7eedffb640 1 -- 172.21.15.45:0/1727521385 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5563dc45b720 con 0x5563dc7cff90 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.316+0000 7f7eedffb640 10 client.17409 handle_command_reply: tid=0 2024-03-12T05:18:30.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.316+0000 7f7f0f202740 1 client.17409 shutdown 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.316+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 msgr2=0x5563dc7c0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.316+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7c0250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40bae60 tx=0x7f7ef40eda70 comp rx=0 tx=0).stop 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 shutdown_connections 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc7cff90 0x5563dc7d2380 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7c07b0 0x5563dc7bdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7bd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 --2- 172.21.15.45:0/1727521385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7c0250 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 >> 172.21.15.45:0/1727521385 conn(0x5563dc7b1990 msgr2=0x5563dc7b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7eedffb640 0 client.17409 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 shutdown_connections 2024-03-12T05:18:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 1 -- 172.21.15.45:0/1727521385 wait complete. 2024-03-12T05:18:30.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.317+0000 7f7f0f202740 20 client.17409 trim_cache size 0 max 16384 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.345+0000 7f7f0f202740 1 Processor -- start 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.345+0000 7f7f0f202740 1 -- start start 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7bd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b7bd0 0x5563dc7b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc7b6cf0 con 0x5563dc7b72b0 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc7ad0d0 con 0x5563dc7b7bd0 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0abcb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0abcb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46916/0 (socket says 172.21.15.45:46916) 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0abcb640 1 -- 172.21.15.45:0/286174810 learned_addr learned my addr 172.21.15.45:0/286174810 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7bd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.346+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc7b2a10 con 0x5563dc7c07b0 2024-03-12T05:18:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 -- 172.21.15.45:0/286174810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b7bd0 msgr2=0x5563dc7b88b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:18:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b7bd0 0x5563dc7b88b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 -- 172.21.15.45:0/286174810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 msgr2=0x5563dc7bd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7bd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 -- 172.21.15.45:0/286174810 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563dc6b9ac0 con 0x5563dc7c07b0 2024-03-12T05:18:30.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.347+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00f8e0 tx=0x7f7efc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=744f603b3d320340 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.348+0000 7f7eeffff640 1 -- 172.21.15.45:0/286174810 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc022040 con 0x5563dc7c07b0 2024-03-12T05:18:30.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.348+0000 7f7eeffff640 1 -- 172.21.15.45:0/286174810 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7efc01b930 con 0x5563dc7c07b0 2024-03-12T05:18:30.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.348+0000 7f7eeffff640 1 -- 172.21.15.45:0/286174810 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc026980 con 0x5563dc7c07b0 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 msgr2=0x5563dc7b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00f8e0 tx=0x7f7efc017000 comp rx=0 tx=0).stop 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 shutdown_connections 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7c07b0 0x5563dc7b8df0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b7bd0 0x5563dc7b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 --2- 172.21.15.45:0/286174810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7b72b0 0x5563dc7bd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 >> 172.21.15.45:0/286174810 conn(0x5563dc7b1990 msgr2=0x5563dc7d6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 shutdown_connections 2024-03-12T05:18:30.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.349+0000 7f7f0f202740 1 -- 172.21.15.45:0/286174810 wait complete. 2024-03-12T05:18:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 -- start start 2024-03-12T05:18:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 Processor -- start 2024-03-12T05:18:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc7c7d10 con 0x5563dc7c07b0 2024-03-12T05:18:30.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc837490 con 0x5563dc7b72b0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563dc8375d0 con 0x5563dc7b7bd0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0f202740 10 client.0 _ll_register_callbacks cb 0x7f7ef40b8ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0a3ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0a3ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52066/0 (socket says 172.21.15.45:52066) 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/418069916 learned_addr learned my addr 172.21.15.45:0/418069916 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.350+0000 7f7f0abcb640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/418069916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 msgr2=0x5563dc7c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/418069916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 msgr2=0x5563dc7c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 -- 172.21.15.45:0/418069916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc00f5c0 con 0x5563dc7b72b0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0a3ca640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40bcb40 tx=0x7f7ef40092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536296688d784fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:30.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40d4070 con 0x5563dc7b72b0 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7ef40c0d00 con 0x5563dc7b72b0 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ef4036d30 con 0x5563dc7b72b0 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ef40bd2b0 con 0x5563dc7b72b0 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0f202740 10 client.26897 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.351+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5563dc6df6a0 con 0x5563dc7b72b0 2024-03-12T05:18:30.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7ef409f020 con 0x5563dc7b72b0 2024-03-12T05:18:30.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7ef403c040 con 0x5563dc7b72b0 2024-03-12T05:18:30.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 10 client.26897 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 10 client.26897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 20 client.26897 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc8388b0 0x5563dc83aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 4 client.26897 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5563dc4aa890 con 0x5563dc8388b0 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.352+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc8388b0 0x5563dc83aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.353+0000 7f7eedffb640 10 client.26897 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.353+0000 7f7f09bc9640 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc8388b0 0x5563dc83aca0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd79c763563f00ee server_cookie=f7ffdec59ef9461c in_seq=0 out_seq=0 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.353+0000 7f7eedffb640 1 -- 172.21.15.45:0/418069916 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5563dc4aa890 con 0x5563dc8388b0 2024-03-12T05:18:30.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.353+0000 7f7eedffb640 10 client.26897 handle_command_reply: tid=0 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.353+0000 7f7f0f202740 1 client.26897 shutdown 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 msgr2=0x5563dc7c3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40bcb40 tx=0x7f7ef40092d0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 shutdown_connections 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5563dc8388b0 0x5563dc83aca0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5563dc7c07b0 0x5563dc7c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7eedffb640 0 client.26897 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5563dc7b7bd0 0x5563dc7c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 --2- 172.21.15.45:0/418069916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563dc7b72b0 0x5563dc7c3d50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 >> 172.21.15.45:0/418069916 conn(0x5563dc7b1990 msgr2=0x5563dc7b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 shutdown_connections 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.354+0000 7f7f0f202740 1 -- 172.21.15.45:0/418069916 wait complete. 2024-03-12T05:18:30.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.355+0000 7f7f0f202740 20 client.26897 trim_cache size 0 max 16384 2024-03-12T05:18:30.353 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:18:30.353 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:18:30.353 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:18:30.353 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:18:30.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 msgr2=0x7f7ef00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 0x7f7ef00a40e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0125f0 tx=0x7f7efc004a40 comp rx=0 tx=0).stop 2024-03-12T05:18:30.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 msgr2=0x7f7f041a3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4005ab0 tx=0x7f7ef4005e00 comp rx=0 tx=0).stop 2024-03-12T05:18:30.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 shutdown_connections 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ef00a1c20 0x7f7ef00a40e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f04112100 0x7f7f0419db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f04073e40 0x7f7f041a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 --2- 172.21.15.45:0/2418722158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f04073520 0x7f7f041a3a50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 >> 172.21.15.45:0/2418722158 conn(0x7f7f040fcc20 msgr2=0x7f7f040fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.356+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 shutdown_connections 2024-03-12T05:18:30.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.357+0000 7f7f0c62e640 1 -- 172.21.15.45:0/2418722158 wait complete. 2024-03-12T05:18:30.380 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:18:30.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.560+0000 7f95adadf640 1 -- 172.21.15.45:0/809337260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108110 msgr2=0x7f95a8114620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.560+0000 7f95adadf640 1 --2- 172.21.15.45:0/809337260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108110 0x7f95a8114620 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009cd0 tx=0x7f95a002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 -- 172.21.15.45:0/809337260 shutdown_connections 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 --2- 172.21.15.45:0/809337260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108110 0x7f95a8114620 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 --2- 172.21.15.45:0/809337260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8107750 0x7f95a8107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 --2- 172.21.15.45:0/809337260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a810d760 0x7f95a810db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 -- 172.21.15.45:0/809337260 >> 172.21.15.45:0/809337260 conn(0x7f95a80756d0 msgr2=0x7f95a8077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 -- 172.21.15.45:0/809337260 shutdown_connections 2024-03-12T05:18:30.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.561+0000 7f95adadf640 1 -- 172.21.15.45:0/809337260 wait complete. 2024-03-12T05:18:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.562+0000 7f95adadf640 1 Processor -- start 2024-03-12T05:18:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.562+0000 7f95adadf640 1 -- start start 2024-03-12T05:18:30.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.562+0000 7f95adadf640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95adadf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8108110 0x7f95a81a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95adadf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 0x7f95a819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46944/0 (socket says 172.21.15.45:46944) 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 0x7f95a819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 0x7f95a819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55802/0 (socket says 172.21.15.45:55802) 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 -- 172.21.15.45:0/3426384759 learned_addr learned my addr 172.21.15.45:0/3426384759 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819e150 con 0x7f95a810d760 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819e2c0 con 0x7f95a8108110 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819e480 con 0x7f95a8107750 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 -- 172.21.15.45:0/3426384759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8108110 msgr2=0x7f95a81a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8108110 0x7f95a81a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 -- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 msgr2=0x7f95a819db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 0x7f95a819db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.563+0000 7f95a77fe640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a0009900 con 0x7f95a8107750 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.564+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f959800e9f0 tx=0x7f959800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.564+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598009700 con 0x7f95a8107750 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.564+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a819e760 con 0x7f95a8107750 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.564+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f959800cd80 con 0x7f95a8107750 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.565+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598018840 con 0x7f95a8107750 2024-03-12T05:18:30.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.565+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a81aa9f0 con 0x7f95a8107750 2024-03-12T05:18:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.567+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f95980189a0 con 0x7f95a8107750 2024-03-12T05:18:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.567+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a8067f70 con 0x7f95a8107750 2024-03-12T05:18:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.568+0000 7f95a4ff9640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 0x7f95900a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:30.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.568+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9598014070 con 0x7f95a8107750 2024-03-12T05:18:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.568+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 0x7f95900a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.569+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 0x7f95900a40e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f959c005fd0 tx=0x7f959c005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:30.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.573+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9598052d20 con 0x7f95a8107750 2024-03-12T05:18:30.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:30 smithi130 ceph-mon[35398]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:18:30.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:30 smithi116 ceph-mon[35486]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:18:30.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:30 smithi045 ceph-mon[32174]: pgmap v235: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:18:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.988+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a810fcc0 con 0x7f95a8107750 2024-03-12T05:18:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.988+0000 7f95a4ff9640 1 -- 172.21.15.45:0/3426384759 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f9598096a10 con 0x7f95a8107750 2024-03-12T05:18:30.984 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:18:30.984 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:18:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:18:30.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 msgr2=0x7f95900a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 0x7f95900a40e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f959c005fd0 tx=0x7f959c005850 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 msgr2=0x7f95a81a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f959800e9f0 tx=0x7f959800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 shutdown_connections 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1c20 0x7f95900a40e0 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f959c005fd0 tx=0x7f959c005850 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a810d760 0x7f95a819db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8108110 0x7f95a81a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 --2- 172.21.15.45:0/3426384759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8107750 0x7f95a81a3950 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:30.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.992+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 >> 172.21.15.45:0/3426384759 conn(0x7f95a80756d0 msgr2=0x7f95a8075ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.993+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 shutdown_connections 2024-03-12T05:18:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:30.993+0000 7f95adadf640 1 -- 172.21.15.45:0/3426384759 wait complete. 2024-03-12T05:18:31.006 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:18:31.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.180+0000 7fb57e7cc640 1 -- 172.21.15.45:0/2740392505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb570096930 msgr2=0x7fb57009a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/2740392505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb570096930 0x7fb57009a970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb574009930 tx=0x7fb57402bc30 comp rx=0 tx=0).stop 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 -- 172.21.15.45:0/2740392505 shutdown_connections 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/2740392505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb57009b050 0x7fb5700a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/2740392505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb570096930 0x7fb57009a970 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/2740392505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb570095f80 0x7fb570096360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.181+0000 7fb57e7cc640 1 -- 172.21.15.45:0/2740392505 >> 172.21.15.45:0/2740392505 conn(0x7fb57000ce50 msgr2=0x7fb57000d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.182+0000 7fb57e7cc640 1 -- 172.21.15.45:0/2740392505 shutdown_connections 2024-03-12T05:18:31.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.183+0000 7fb57e7cc640 1 -- 172.21.15.45:0/2740392505 wait complete. 2024-03-12T05:18:31.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.184+0000 7fb57e7cc640 1 Processor -- start 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.184+0000 7fb57e7cc640 1 -- start start 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.184+0000 7fb57e7cc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.184+0000 7fb57e7cc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 0x7fb5701377d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57e7cc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5700a0a50 0x7fb5700a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55816/0 (socket says 172.21.15.45:55816) 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57d7ca640 1 -- 172.21.15.45:0/692253315 learned_addr learned my addr 172.21.15.45:0/692253315 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 0x7fb5701377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700a14d0 con 0x7fb570095f80 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700a1640 con 0x7fb5700a0a50 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5700a1800 con 0x7fb57009b050 2024-03-12T05:18:31.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.185+0000 7fb57dfcb640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5700a0a50 0x7fb5700a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 -- 172.21.15.45:0/692253315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5700a0a50 msgr2=0x7fb5700a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5700a0a50 0x7fb5700a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 -- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 msgr2=0x7fb570137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568009490 con 0x7fb57009b050 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 0x7fb5701377d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb57402c240 tx=0x7fb574004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb56e7fc640 1 -- 172.21.15.45:0/692253315 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57402c5d0 con 0x7fb57009b050 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb56e7fc640 1 -- 172.21.15.45:0/692253315 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb574003aa0 con 0x7fb57009b050 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.186+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb574009560 con 0x7fb57009b050 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.187+0000 7fb56e7fc640 1 -- 172.21.15.45:0/692253315 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574005470 con 0x7fb57009b050 2024-03-12T05:18:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.188+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57013f600 con 0x7fb57009b050 2024-03-12T05:18:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.190+0000 7fb56e7fc640 1 -- 172.21.15.45:0/692253315 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb574003c10 con 0x7fb57009b050 2024-03-12T05:18:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.191+0000 7fb56e7fc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 0x7fb5640a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.191+0000 7fb56e7fc640 1 -- 172.21.15.45:0/692253315 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb57402de40 con 0x7fb57009b050 2024-03-12T05:18:31.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.191+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 0x7fb5640a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.192+0000 7fb582602740 1 Processor -- start 2024-03-12T05:18:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.192+0000 7fb582602740 1 -- start start 2024-03-12T05:18:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.192+0000 7fb582602740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e0a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb582602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb582602740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e15d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb582602740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8dc5960 con 0x55a5e8e13960 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb582602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8d79310 con 0x55a5e8e0ad80 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb582602740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8d76840 con 0x55a5e8e0a460 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52122/0 (socket says 172.21.15.45:52122) 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1643175783 learned_addr learned my addr 172.21.15.45:0/1643175783 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e0a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57dfcb640 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e15d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 0x7fb5640a4120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5700a1f80 tx=0x7fb568005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1643175783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 msgr2=0x55a5e8e0a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e0a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1643175783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 msgr2=0x55a5e8e15d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e15d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.193+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1643175783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5e8d34060 con 0x55a5e8e0ad80 2024-03-12T05:18:31.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.194+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb574009900 tx=0x7fb5740bbf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc2b70631f45f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.195+0000 7fb5637fe640 1 -- 172.21.15.45:0/1643175783 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740bc340 con 0x55a5e8e0ad80 2024-03-12T05:18:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.195+0000 7fb5637fe640 1 -- 172.21.15.45:0/1643175783 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5740c0490 con 0x55a5e8e0ad80 2024-03-12T05:18:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.195+0000 7fb5637fe640 1 -- 172.21.15.45:0/1643175783 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740c4650 con 0x55a5e8e0ad80 2024-03-12T05:18:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- 172.21.15.45:0/1643175783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 msgr2=0x55a5e8e13420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb574009900 tx=0x7fb5740bbf40 comp rx=0 tx=0).stop 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- 172.21.15.45:0/1643175783 shutdown_connections 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e15d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e13420 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 --2- 172.21.15.45:0/1643175783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e0a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- 172.21.15.45:0/1643175783 >> 172.21.15.45:0/1643175783 conn(0x55a5e8e04b90 msgr2=0x55a5e8e04f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- 172.21.15.45:0/1643175783 shutdown_connections 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- 172.21.15.45:0/1643175783 wait complete. 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.196+0000 7fb582602740 1 -- start start 2024-03-12T05:18:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.197+0000 7fb582602740 1 Processor -- start 2024-03-12T05:18:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e10ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e13960 0x55a5e8e11420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8d72020 con 0x55a5e8e0a460 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8a45430 con 0x55a5e8e0ad80 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8a01e00 con 0x55a5e8e13960 2024-03-12T05:18:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb582602740 10 client.0 _ll_register_callbacks cb 0x55a5e8d19cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb57cfc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e10ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55822/0 (socket says 172.21.15.45:55822) 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb57dfcb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e13960 0x55a5e8e11420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.198+0000 7fb57d7ca640 1 -- 172.21.15.45:0/1355166213 learned_addr learned my addr 172.21.15.45:0/1355166213 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e13960 msgr2=0x55a5e8e11420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e13960 0x55a5e8e11420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1355166213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 msgr2=0x55a5e8e109a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57cfc9640 1 -- 172.21.15.45:0/1355166213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57013f600 con 0x55a5e8e0ad80 2024-03-12T05:18:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.199+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.200+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e10ee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb574030040 tx=0x7fb5740bbfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bbe16dd2aa7072c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.200+0000 7fb560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.200+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740da070 con 0x55a5e8e0ad80 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.200+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5e8a717e0 con 0x55a5e8e0ad80 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.200+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb5740c4ae0 con 0x55a5e8e0ad80 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.201+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5740bc340 con 0x55a5e8e0ad80 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.201+0000 7fb582602740 10 client.26912 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.201+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a5e8d706a0 con 0x55a5e8e0ad80 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.201+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5740c4860 con 0x55a5e8e0ad80 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.201+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb57403c070 con 0x55a5e8e0ad80 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 10 client.26912 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 10 client.26912 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 20 client.26912 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e34860 0x55a5e8e36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 4 client.26912 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5e8a717e0 con 0x55a5e8e34860 2024-03-12T05:18:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.202+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e34860 0x55a5e8e36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.203+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e34860 0x55a5e8e36cc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3959d1e20a5e0f74 server_cookie=e701e8924093af4b in_seq=0 out_seq=0 2024-03-12T05:18:31.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.205+0000 7fb560ff9640 10 client.26912 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:18:31.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.205+0000 7fb560ff9640 1 -- 172.21.15.45:0/1355166213 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a5e8a717e0 con 0x55a5e8e34860 2024-03-12T05:18:31.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.205+0000 7fb560ff9640 10 client.26912 handle_command_reply: tid=0 2024-03-12T05:18:31.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.205+0000 7fb582602740 1 client.26912 shutdown 2024-03-12T05:18:31.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.207+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 msgr2=0x55a5e8e10ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.207+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e10ee0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb574030040 tx=0x7fb5740bbfb0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 shutdown_connections 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e34860 0x55a5e8e36cc0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e13960 0x55a5e8e11420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e10ee0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 --2- 172.21.15.45:0/1355166213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e0a460 0x55a5e8e109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 >> 172.21.15.45:0/1355166213 conn(0x55a5e8e04b90 msgr2=0x55a5e8e068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.208+0000 7fb560ff9640 0 client.26912 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.209+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 shutdown_connections 2024-03-12T05:18:31.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.209+0000 7fb582602740 1 -- 172.21.15.45:0/1355166213 wait complete. 2024-03-12T05:18:31.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.210+0000 7fb582602740 20 client.26912 trim_cache size 0 max 16384 2024-03-12T05:18:31.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.237+0000 7fb582602740 1 Processor -- start 2024-03-12T05:18:31.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.237+0000 7fb582602740 1 -- start start 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e11240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e0f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8cf7130 con 0x55a5e8e13960 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8e09890 con 0x55a5e8e0ad80 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb582602740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8e046c0 con 0x55a5e8e0a460 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46984/0 (socket says 172.21.15.45:46984) 2024-03-12T05:18:31.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 -- 172.21.15.45:0/480309245 learned_addr learned my addr 172.21.15.45:0/480309245 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57dfcb640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e0f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e11240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 -- 172.21.15.45:0/480309245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 msgr2=0x55a5e8e11240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e11240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 -- 172.21.15.45:0/480309245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 msgr2=0x55a5e8e0f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e0f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57d7ca640 1 -- 172.21.15.45:0/480309245 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5e8d0cc20 con 0x55a5e8e0a460 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.238+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e11240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:31.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.239+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb568011550 tx=0x7fb568014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c7ff776d8d6bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.239+0000 7fb562ffd640 1 -- 172.21.15.45:0/480309245 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568021070 con 0x55a5e8e0a460 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.239+0000 7fb562ffd640 1 -- 172.21.15.45:0/480309245 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb568022040 con 0x55a5e8e0a460 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.239+0000 7fb562ffd640 1 -- 172.21.15.45:0/480309245 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb568004a00 con 0x55a5e8e0a460 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.240+0000 7fb582602740 1 -- 172.21.15.45:0/480309245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 msgr2=0x55a5e8e130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.240+0000 7fb582602740 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb568011550 tx=0x7fb568014000 comp rx=0 tx=0).stop 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.240+0000 7fb582602740 1 -- 172.21.15.45:0/480309245 shutdown_connections 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e0f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e11240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 --2- 172.21.15.45:0/480309245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0a460 0x55a5e8e130d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 -- 172.21.15.45:0/480309245 >> 172.21.15.45:0/480309245 conn(0x55a5e8e04b90 msgr2=0x55a5e8e3b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 -- 172.21.15.45:0/480309245 shutdown_connections 2024-03-12T05:18:31.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.241+0000 7fb582602740 1 -- 172.21.15.45:0/480309245 wait complete. 2024-03-12T05:18:31.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 -- start start 2024-03-12T05:18:31.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 Processor -- start 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0a460 0x55a5e8e2ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e2a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8e2aa80 con 0x55a5e8e13960 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8e2abf0 con 0x55a5e8e0a460 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5e8e8adb0 con 0x55a5e8e0ad80 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.242+0000 7fb582602740 10 client.0 _ll_register_callbacks cb 0x7fb57402c170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:46992/0 (socket says 172.21.15.45:46992) 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 -- 172.21.15.45:0/521846347 learned_addr learned my addr 172.21.15.45:0/521846347 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0a460 0x55a5e8e2ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 -- 172.21.15.45:0/521846347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0a460 msgr2=0x55a5e8e2ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0a460 0x55a5e8e2ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 -- 172.21.15.45:0/521846347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 msgr2=0x55a5e8e2a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e2a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 -- 172.21.15.45:0/521846347 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb568017050 con 0x55a5e8e0ad80 2024-03-12T05:18:31.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb57cfc9640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb5740b8c80 tx=0x7fb574005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=284ec79d76ebf8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.243+0000 7fb560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740da040 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb5740bca40 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5740c0490 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5740bc110 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb582602740 10 client.26899 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.244+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a5e8d328c0 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5740c0490 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb57403c070 con 0x55a5e8e0ad80 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 10 client.26899 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 10 client.26899 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:18:31.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 20 client.26899 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e8c0c0 0x55a5e8e8e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 4 client.26899 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a5e8aae720 con 0x55a5e8e8c0c0 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.245+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e8c0c0 0x55a5e8e8e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb57d7ca640 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e8c0c0 0x55a5e8e8e4e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7473e8ec79b4198f server_cookie=dbbd5d846b9b47b0 in_seq=0 out_seq=0 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb560ff9640 10 client.26899 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb560ff9640 1 -- 172.21.15.45:0/521846347 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a5e8aae720 con 0x55a5e8e8c0c0 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb560ff9640 10 client.26899 handle_command_reply: tid=0 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb582602740 1 client.26899 shutdown 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 msgr2=0x55a5e8e29de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.246+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb5740b8c80 tx=0x7fb574005fd0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 shutdown_connections 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55a5e8e8c0c0 0x55a5e8e8e4e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a5e8e13960 0x55a5e8e2a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5e8e0ad80 0x55a5e8e29de0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 --2- 172.21.15.45:0/521846347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e8e0a460 0x55a5e8e2ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 >> 172.21.15.45:0/521846347 conn(0x55a5e8e04b90 msgr2=0x55a5e8e0f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb560ff9640 0 client.26899 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 shutdown_connections 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 1 -- 172.21.15.45:0/521846347 wait complete. 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.247+0000 7fb582602740 20 client.26899 trim_cache size 0 max 16384 2024-03-12T05:18:31.247 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:18:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 msgr2=0x7fb5640a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 0x7fb5640a4120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5700a1f80 tx=0x7fb568005f90 comp rx=0 tx=0).stop 2024-03-12T05:18:31.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 msgr2=0x7fb5701377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 0x7fb5701377d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb57402c240 tx=0x7fb574004be0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 shutdown_connections 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5640a1c60 0x7fb5640a4120 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5700a1f80 tx=0x7fb568005f90 comp rx=0 tx=0).stop 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5700a0a50 0x7fb5700a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb57009b050 0x7fb5701377d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 --2- 172.21.15.45:0/692253315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb570095f80 0x7fb570137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.249+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 >> 172.21.15.45:0/692253315 conn(0x7fb57000ce50 msgr2=0x7fb570093cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.250+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 shutdown_connections 2024-03-12T05:18:31.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.250+0000 7fb57e7cc640 1 -- 172.21.15.45:0/692253315 wait complete. 2024-03-12T05:18:31.271 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:18:31.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:18:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.451+0000 7fa8d5d9b640 1 -- 172.21.15.45:0/3341461558 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c002cb60 con 0x7fa8d8107710 2024-03-12T05:18:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.452+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/3341461558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 msgr2=0x7fa8d8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.452+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/3341461558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d8107b90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009900 tx=0x7fa8c002bc30 comp rx=0 tx=0).stop 2024-03-12T05:18:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/3341461558 shutdown_connections 2024-03-12T05:18:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/3341461558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 0x7fa8d81145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/3341461558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d8107b90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/3341461558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d810db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/3341461558 >> 172.21.15.45:0/3341461558 conn(0x7fa8d8075620 msgr2=0x7fa8d8077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/3341461558 shutdown_connections 2024-03-12T05:18:31.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.456+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/3341461558 wait complete. 2024-03-12T05:18:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.457+0000 7fa8dcfd0640 1 Processor -- start 2024-03-12T05:18:31.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 -- start start 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d81a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 0x7fa8d81a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d819e2a0 con 0x7fa8d810d720 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d819e410 con 0x7fa8d8107710 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8dcfd0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d819e5d0 con 0x7fa8d81080d0 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d81a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 0x7fa8d81a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55838/0 (socket says 172.21.15.45:55838) 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d6d9d640 1 -- 172.21.15.45:0/2340897050 learned_addr learned my addr 172.21.15.45:0/2340897050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d659c640 1 -- 172.21.15.45:0/2340897050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 msgr2=0x7fa8d81a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d659c640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 0x7fa8d81a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d659c640 1 -- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 msgr2=0x7fa8d819db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.458+0000 7fa8d659c640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8d659c640 1 -- 172.21.15.45:0/2340897050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009560 con 0x7fa8d8107710 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8d659c640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d81a3a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800e9f0 tx=0x7fa8c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8cf7fe640 1 -- 172.21.15.45:0/2340897050 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800ccc0 con 0x7fa8d8107710 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d819e8b0 con 0x7fa8d8107710 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d81aa9d0 con 0x7fa8d8107710 2024-03-12T05:18:31.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.459+0000 7fa8cf7fe640 1 -- 172.21.15.45:0/2340897050 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8c800ce20 con 0x7fa8d8107710 2024-03-12T05:18:31.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.460+0000 7fa8cf7fe640 1 -- 172.21.15.45:0/2340897050 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c80187c0 con 0x7fa8d8107710 2024-03-12T05:18:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.463+0000 7fa8dfb9c740 1 Processor -- start 2024-03-12T05:18:31.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 -- start start 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8cf7fe640 1 -- 172.21.15.45:0/2340897050 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa8c8018920 con 0x7fa8d8107710 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a426630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a3e1960 con 0x55754a426b70 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a395310 con 0x55754a426250 2024-03-12T05:18:31.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a392840 con 0x55754a42f750 2024-03-12T05:18:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8d6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a426630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47002/0 (socket says 172.21.15.45:47002) 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.464+0000 7fa8cffff640 1 -- 172.21.15.45:0/2576767577 learned_addr learned my addr 172.21.15.45:0/2576767577 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 -- 172.21.15.45:0/2576767577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 msgr2=0x55754a426630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a426630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 -- 172.21.15.45:0/2576767577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 msgr2=0x55754a42f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 -- 172.21.15.45:0/2576767577 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55754a350060 con 0x55754a42f750 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cf7fe640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 0x7fa8c40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cffff640 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0000c00 tx=0x7fa8c0005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c472818f62af2c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.465+0000 7fa8cf7fe640 1 -- 172.21.15.45:0/2340897050 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa8c8014070 con 0x7fa8d8107710 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.466+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 0x7fa8c40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.466+0000 7fa8ccff9640 1 -- 172.21.15.45:0/2576767577 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0004880 con 0x55754a42f750 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.467+0000 7fa8ccff9640 1 -- 172.21.15.45:0/2576767577 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8c00049e0 con 0x55754a42f750 2024-03-12T05:18:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.467+0000 7fa8ccff9640 1 -- 172.21.15.45:0/2576767577 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0003970 con 0x55754a42f750 2024-03-12T05:18:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.468+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/2576767577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 msgr2=0x55754a431b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.468+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0000c00 tx=0x7fa8c0005df0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.469+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/2576767577 shutdown_connections 2024-03-12T05:18:31.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.469+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.469+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a431b40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.469+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/2576767577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a426630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.469+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/2576767577 >> 172.21.15.45:0/2576767577 conn(0x55754a420980 msgr2=0x55754a420d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.472+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 0x7fa8c40a4160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00049d0 tx=0x7fa8d0009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.475+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/2576767577 shutdown_connections 2024-03-12T05:18:31.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.475+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/2576767577 wait complete. 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.476+0000 7fa8dfb9c740 1 -- start start 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.476+0000 7fa8dfb9c740 1 Processor -- start 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a42c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a42d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a38e020 con 0x55754a426b70 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8d659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a42c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8d659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a42c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52158/0 (socket says 172.21.15.45:52158) 2024-03-12T05:18:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8d659c640 1 -- 172.21.15.45:0/4094650584 learned_addr learned my addr 172.21.15.45:0/4094650584 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a061430 con 0x55754a426250 2024-03-12T05:18:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a01de00 con 0x55754a42f750 2024-03-12T05:18:31.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.477+0000 7fa8dfb9c740 10 client.0 _ll_register_callbacks cb 0x55754a335cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.478+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a42d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.480+0000 7fa8d6d9d640 1 -- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 msgr2=0x55754a42c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.480+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a42c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.480+0000 7fa8d6d9d640 1 -- 172.21.15.45:0/4094650584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 msgr2=0x55754a42cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.480+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.480+0000 7fa8d6d9d640 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d8114ee0 con 0x55754a42f750 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.481+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a42d1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009ed0 tx=0x7fa8d0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bb1d90c037c0b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.481+0000 7fa8bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.481+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d001f970 con 0x55754a42f750 2024-03-12T05:18:31.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.481+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa8d001fad0 con 0x55754a42f750 2024-03-12T05:18:31.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.481+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55754a08d7e0 con 0x55754a42f750 2024-03-12T05:18:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8d001fc90 con 0x55754a42f750 2024-03-12T05:18:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 10 client.26911 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55754a38c6a0 con 0x55754a42f750 2024-03-12T05:18:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8d0035020 con 0x55754a42f750 2024-03-12T05:18:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa8d0029510 con 0x55754a42f750 2024-03-12T05:18:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 10 client.26911 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 10 client.26911 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:18:31.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 20 client.26911 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.484+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a43fc50 0x55754a442070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.485+0000 7fa8dfb9c740 4 client.26911 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:31.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.485+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55754a0ca720 con 0x55754a43fc50 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.485+0000 7fa8d659c640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a43fc50 0x55754a442070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.486+0000 7fa8d659c640 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a43fc50 0x55754a442070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=93aeac6692b8ac28 server_cookie=6b7aa095d1a0a234 in_seq=0 out_seq=0 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.486+0000 7fa8bdffb640 10 client.26911 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.487+0000 7fa8bdffb640 1 -- 172.21.15.45:0/4094650584 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55754a0ca720 con 0x55754a43fc50 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.487+0000 7fa8bdffb640 10 client.26911 handle_command_reply: tid=0 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.487+0000 7fa8dfb9c740 1 client.26911 shutdown 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.487+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 msgr2=0x55754a42d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.487+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a42d1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009ed0 tx=0x7fa8d0004a40 comp rx=0 tx=0).stop 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 shutdown_connections 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a43fc50 0x55754a442070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a42cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a42d1a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/4094650584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a42c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 >> 172.21.15.45:0/4094650584 conn(0x55754a420980 msgr2=0x55754a4226d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8bdffb640 0 client.26911 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:18:31.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 shutdown_connections 2024-03-12T05:18:31.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/4094650584 wait complete. 2024-03-12T05:18:31.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.488+0000 7fa8dfb9c740 20 client.26911 trim_cache size 0 max 16384 2024-03-12T05:18:31.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.511+0000 7fa8dfb9c740 1 Processor -- start 2024-03-12T05:18:31.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.511+0000 7fa8dfb9c740 1 -- start start 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a426250 0x55754a42ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 0x55754a4469f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a313130 con 0x55754a42f750 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a41c2b0 con 0x55754a426b70 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a431e70 con 0x55754a426250 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52172/0 (socket says 172.21.15.45:52172) 2024-03-12T05:18:31.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 -- 172.21.15.45:0/1572252458 learned_addr learned my addr 172.21.15.45:0/1572252458 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 0x55754a4469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 -- 172.21.15.45:0/1572252458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a426250 msgr2=0x55754a42ee50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a426250 0x55754a42ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 -- 172.21.15.45:0/1572252458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 msgr2=0x55754a4469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.512+0000 7fa8cffff640 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 0x55754a4469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8cffff640 1 -- 172.21.15.45:0/1572252458 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55754a328c20 con 0x55754a426b70 2024-03-12T05:18:31.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 0x55754a4469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8cffff640 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002c240 tx=0x7fa8c0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f7f005b781f331 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8bffff640 1 -- 172.21.15.45:0/1572252458 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0004880 con 0x55754a426b70 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8bffff640 1 -- 172.21.15.45:0/1572252458 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8c00044f0 con 0x55754a426b70 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.513+0000 7fa8bffff640 1 -- 172.21.15.45:0/1572252458 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0003df0 con 0x55754a426b70 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.514+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/1572252458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 msgr2=0x55754a42cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.514+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002c240 tx=0x7fa8c0005850 comp rx=0 tx=0).stop 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/1572252458 shutdown_connections 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a42f750 0x55754a4469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426b70 0x55754a42cfc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/1572252458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a426250 0x55754a42ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/1572252458 >> 172.21.15.45:0/1572252458 conn(0x55754a420980 msgr2=0x55754a427b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/1572252458 shutdown_connections 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/1572252458 wait complete. 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.515+0000 7fa8dfb9c740 1 -- start start 2024-03-12T05:18:31.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 Processor -- start 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a4368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a435190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a435720 con 0x55754a426b70 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a435890 con 0x55754a426250 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55754a435a50 con 0x55754a42f750 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8dfb9c740 10 client.0 _ll_register_callbacks cb 0x7fa8d00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55872/0 (socket says 172.21.15.45:55872) 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8cffff640 1 -- 172.21.15.45:0/505689974 learned_addr learned my addr 172.21.15.45:0/505689974 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8d659c640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a4368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.516+0000 7fa8d6d9d640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a435190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 -- 172.21.15.45:0/505689974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 msgr2=0x55754a435190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a435190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 -- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 msgr2=0x55754a4368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a4368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 -- 172.21.15.45:0/505689974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55754a350060 con 0x55754a426b70 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8cffff640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0004c20 tx=0x7fa8c002c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02065f2fd8c99b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0003d70 con 0x55754a426b70 2024-03-12T05:18:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.517+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa8c0003ed0 con 0x55754a426b70 2024-03-12T05:18:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8c00049e0 con 0x55754a426b70 2024-03-12T05:18:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8c00040d0 con 0x55754a426b70 2024-03-12T05:18:31.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8dfb9c740 10 client.17445 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55754a34e8c0 con 0x55754a426b70 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa8c00049e0 con 0x55754a426b70 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.518+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa8c003c050 con 0x55754a426b70 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 10 client.17445 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 10 client.17445 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 20 client.17445 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a497370 0x55754a499790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 4 client.17445 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55754a119890 con 0x55754a497370 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8d659c640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a497370 0x55754a499790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.519+0000 7fa8bdffb640 10 client.17445 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8d659c640 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a497370 0x55754a499790 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=621b1520b7a569c5 server_cookie=1cb673be29f84fac in_seq=0 out_seq=0 2024-03-12T05:18:31.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8bdffb640 1 -- 172.21.15.45:0/505689974 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55754a119890 con 0x55754a497370 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8bdffb640 10 client.17445 handle_command_reply: tid=0 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8dfb9c740 1 client.17445 shutdown 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 msgr2=0x55754a434c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.520+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0004c20 tx=0x7fa8c002c9d0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 shutdown_connections 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55754a497370 0x55754a499790 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55754a426b70 0x55754a434c50 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55754a42f750 0x55754a435190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 --2- 172.21.15.45:0/505689974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55754a426250 0x55754a4368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 >> 172.21.15.45:0/505689974 conn(0x55754a420980 msgr2=0x55754a4469f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8bdffb640 0 client.17445 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 shutdown_connections 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 1 -- 172.21.15.45:0/505689974 wait complete. 2024-03-12T05:18:31.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.521+0000 7fa8dfb9c740 20 client.17445 trim_cache size 0 max 16384 2024-03-12T05:18:31.519 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 msgr2=0x7fa8c40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 0x7fa8c40a4160 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00049d0 tx=0x7fa8d0009500 comp rx=0 tx=0).stop 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 msgr2=0x7fa8d81a3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d81a3a50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa8c800e9f0 tx=0x7fa8c800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 shutdown_connections 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa8c40a1ca0 0x7fa8c40a4160 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8d810d720 0x7fa8d819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8d81080d0 0x7fa8d81a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 --2- 172.21.15.45:0/2340897050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8d8107710 0x7fa8d81a3a50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 >> 172.21.15.45:0/2340897050 conn(0x7fa8d8075620 msgr2=0x7fa8d8075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 shutdown_connections 2024-03-12T05:18:31.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.522+0000 7fa8dcfd0640 1 -- 172.21.15.45:0/2340897050 wait complete. 2024-03-12T05:18:31.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:31 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3426384759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:18:31.536 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:18:31.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.716+0000 7f66a372c640 1 -- 172.21.15.45:0/2664501670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c100c10 msgr2=0x7f669c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.716+0000 7f66a372c640 1 --2- 172.21.15.45:0/2664501670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c100c10 0x7f669c104c50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f668c00d940 tx=0x7f668c02dbb0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 -- 172.21.15.45:0/2664501670 shutdown_connections 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 --2- 172.21.15.45:0/2664501670 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c105330 0x7f669c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 --2- 172.21.15.45:0/2664501670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c100c10 0x7f669c104c50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 --2- 172.21.15.45:0/2664501670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c100260 0x7f669c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 -- 172.21.15.45:0/2664501670 >> 172.21.15.45:0/2664501670 conn(0x7f669c077180 msgr2=0x7f669c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 -- 172.21.15.45:0/2664501670 shutdown_connections 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.717+0000 7f66a372c640 1 -- 172.21.15.45:0/2664501670 wait complete. 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 Processor -- start 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 -- start start 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c100260 0x7f669c10cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c10a340 0x7f669c10a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1aa2b0 con 0x7f669c105330 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1aa420 con 0x7f669c10a340 2024-03-12T05:18:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.718+0000 7f66a372c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c1aa5e0 con 0x7f669c100260 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55890/0 (socket says 172.21.15.45:55890) 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3827439141 learned_addr learned my addr 172.21.15.45:0/3827439141 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a14c8640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c100260 0x7f669c10cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c10a340 0x7f669c10a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3827439141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c100260 msgr2=0x7f669c10cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c100260 0x7f669c10cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3827439141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c10a340 msgr2=0x7f669c10a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c10a340 0x7f669c10a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3827439141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c00d570 con 0x7f669c105330 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.719+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f668c000c00 tx=0x7f668c004fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.720+0000 7f66927fc640 1 -- 172.21.15.45:0/3827439141 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c02e550 con 0x7f669c105330 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.720+0000 7f66927fc640 1 -- 172.21.15.45:0/3827439141 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f668c02eb60 con 0x7f669c105330 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.720+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c1aa860 con 0x7f669c105330 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.720+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c1aad00 con 0x7f669c105330 2024-03-12T05:18:31.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.720+0000 7f66927fc640 1 -- 172.21.15.45:0/3827439141 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c03ead0 con 0x7f669c105330 2024-03-12T05:18:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.724+0000 7f66927fc640 1 -- 172.21.15.45:0/3827439141 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f668c03ec30 con 0x7f669c105330 2024-03-12T05:18:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.724+0000 7f66a63dc740 1 Processor -- start 2024-03-12T05:18:31.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.724+0000 7f66a63dc740 1 -- start start 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204c6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da20481960 con 0x55da204c6250 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da20435310 con 0x55da204cf750 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a63dc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da20432840 con 0x55da204c6b70 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66927fc640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 0x7f66880a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47040/0 (socket says 172.21.15.45:47040) 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 -- 172.21.15.45:0/1945457544 learned_addr learned my addr 172.21.15.45:0/1945457544 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 0x7f66880a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 -- 172.21.15.45:0/1945457544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 msgr2=0x55da204d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 -- 172.21.15.45:0/1945457544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 msgr2=0x55da204c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.725+0000 7f66a14c8640 1 -- 172.21.15.45:0/1945457544 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da203f0060 con 0x55da204c6b70 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66927fc640 1 -- 172.21.15.45:0/3827439141 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f668c0ef560 con 0x7f669c105330 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66a14c8640 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6694009fc0 tx=0x7f669400f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c5ff0944bd8edc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 0x7f66880a40e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6698005fd0 tx=0x7f6698005da0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66877fe640 1 -- 172.21.15.45:0/1945457544 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694004d20 con 0x55da204c6b70 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66877fe640 1 -- 172.21.15.45:0/1945457544 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6694004e80 con 0x55da204c6b70 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.726+0000 7f66877fe640 1 -- 172.21.15.45:0/1945457544 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694005420 con 0x55da204c6b70 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 -- 172.21.15.45:0/1945457544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 msgr2=0x55da204cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6694009fc0 tx=0x7f669400f7f0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 -- 172.21.15.45:0/1945457544 shutdown_connections 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204cf210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 --2- 172.21.15.45:0/1945457544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 -- 172.21.15.45:0/1945457544 >> 172.21.15.45:0/1945457544 conn(0x55da204c0980 msgr2=0x55da204c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 -- 172.21.15.45:0/1945457544 shutdown_connections 2024-03-12T05:18:31.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.727+0000 7f66a63dc740 1 -- 172.21.15.45:0/1945457544 wait complete. 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.728+0000 7f66a63dc740 1 -- start start 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.728+0000 7f66a63dc740 1 Processor -- start 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55904/0 (socket says 172.21.15.45:55904) 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a0cc7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a0cc7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47048/0 (socket says 172.21.15.45:47048) 2024-03-12T05:18:31.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da2042e020 con 0x55da204c6250 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a14c8640 1 -- 172.21.15.45:0/3950655644 learned_addr learned my addr 172.21.15.45:0/3950655644 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da20101430 con 0x55da204cf750 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da200bde00 con 0x55da204c6b70 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.729+0000 7f66a63dc740 10 client.0 _ll_register_callbacks cb 0x55da203d5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3950655644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 msgr2=0x55da204cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3950655644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 msgr2=0x55da204cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6694009c20 con 0x55da204c6b70 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f668c0ecd30 tx=0x7f668c03c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aada8a4d37f41db3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f6684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c0c10b0 con 0x55da204c6b70 2024-03-12T05:18:31.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f668c0bd1a0 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da2016a720 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f668c0c12f0 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a63dc740 10 client.26923 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.730+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55da2042c6a0 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f668c0ad020 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f668c0c54b0 con 0x55da204c6b70 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 10 client.26923 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 10 client.26923 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 20 client.26923 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da204e19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 4 client.26923 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.731+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da2016a720 con 0x55da204df580 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.732+0000 7f66a14c8640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da204e19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.732+0000 7f66a14c8640 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da204e19e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cafc404cc1d51c50 server_cookie=9bb84a92d17132c4 in_seq=0 out_seq=0 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.732+0000 7f6684ff9640 10 client.26923 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:18:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.733+0000 7f6684ff9640 1 -- 172.21.15.45:0/3950655644 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55da2016a720 con 0x55da204df580 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.733+0000 7f6684ff9640 10 client.26923 handle_command_reply: tid=0 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.733+0000 7f66a63dc740 1 client.26923 shutdown 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.733+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 msgr2=0x55da204ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.733+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f668c0ecd30 tx=0x7f668c03c000 comp rx=0 tx=0).stop 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 shutdown_connections 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da204e19e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204ccd20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f6684ff9640 0 client.26923 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 --2- 172.21.15.45:0/3950655644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6250 0x55da204cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 >> 172.21.15.45:0/3950655644 conn(0x55da204c0980 msgr2=0x55da204c26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 shutdown_connections 2024-03-12T05:18:31.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.734+0000 7f66a63dc740 1 -- 172.21.15.45:0/3950655644 wait complete. 2024-03-12T05:18:31.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.735+0000 7f66a63dc740 20 client.26923 trim_cache size 0 max 16384 2024-03-12T05:18:31.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:31 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3426384759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:18:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.757+0000 7f66a63dc740 1 Processor -- start 2024-03-12T05:18:31.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.757+0000 7f66a63dc740 1 -- start start 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204cf750 0x55da204c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a0cc7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52208/0 (socket says 172.21.15.45:52208) 2024-03-12T05:18:31.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a1cc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204cf750 0x55da204c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da203b3130 con 0x55da204cf750 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a14c8640 1 -- 172.21.15.45:0/928070830 learned_addr learned my addr 172.21.15.45:0/928070830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da204cd7b0 con 0x55da204c6250 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.758+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da204c5c90 con 0x55da204c6b70 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 -- 172.21.15.45:0/928070830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 msgr2=0x55da204c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 -- 172.21.15.45:0/928070830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204cf750 msgr2=0x55da204c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204cf750 0x55da204c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 -- 172.21.15.45:0/928070830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da203c8c20 con 0x55da204c6250 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f66a14c8640 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f669400b010 tx=0x7f669400ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83ee0b2ef2b56ec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f6686ffd640 1 -- 172.21.15.45:0/928070830 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669402f070 con 0x55da204c6250 2024-03-12T05:18:31.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f6686ffd640 1 -- 172.21.15.45:0/928070830 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6694022040 con 0x55da204c6250 2024-03-12T05:18:31.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.759+0000 7f6686ffd640 1 -- 172.21.15.45:0/928070830 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694021040 con 0x55da204c6250 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.760+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 msgr2=0x55da204cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.760+0000 7f66a63dc740 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f669400b010 tx=0x7f669400ff60 comp rx=0 tx=0).stop 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.760+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 shutdown_connections 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.760+0000 7f66a63dc740 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204cf750 0x55da204c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.760+0000 7f66a63dc740 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6b70 0x55da204c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.761+0000 7f66a63dc740 1 --2- 172.21.15.45:0/928070830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204c6250 0x55da204cef10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.761+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 >> 172.21.15.45:0/928070830 conn(0x55da204c0980 msgr2=0x55da204e6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.761+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 shutdown_connections 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.761+0000 7f66a63dc740 1 -- 172.21.15.45:0/928070830 wait complete. 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.761+0000 7f66a63dc740 1 -- start start 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.762+0000 7f66a63dc740 1 Processor -- start 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.762+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.762+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6b70 0x55da205307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.762+0000 7f66a63dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a63dc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da204d6270 con 0x55da204c6b70 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a63dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da204d63e0 con 0x55da204cf750 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a63dc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da204d65a0 con 0x55da204c6250 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47062/0 (socket says 172.21.15.45:47062) 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 -- 172.21.15.45:0/4269906124 learned_addr learned my addr 172.21.15.45:0/4269906124 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a63dc740 10 client.0 _ll_register_callbacks cb 0x7f6694009760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a1cc9640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6b70 0x55da205307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 -- 172.21.15.45:0/4269906124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 msgr2=0x55da204d5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 -- 172.21.15.45:0/4269906124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6b70 msgr2=0x55da205307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6b70 0x55da205307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 -- 172.21.15.45:0/4269906124 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da203f0060 con 0x55da204c6250 2024-03-12T05:18:31.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f66a14c8640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6694004d70 tx=0x7f6694009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d1d03fc636e0e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.763+0000 7f6684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669402f070 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6694003c00 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da2012d7e0 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6694021040 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f66a63dc740 10 client.26929 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.764+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55da203ee8c0 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6694003e00 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66940044b0 con 0x55da204c6250 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 10 client.26929 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 10 client.26929 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 20 client.26929 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da20539c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 4 client.26929 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:18:31.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.765+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da2012d7e0 con 0x55da204df580 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.766+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da20539c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.767+0000 7f66a0cc7640 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da20539c90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53d3fdccb628a0d8 server_cookie=c893833ff6d10ebb in_seq=0 out_seq=0 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.767+0000 7f6684ff9640 10 client.26929 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.767+0000 7f6684ff9640 1 -- 172.21.15.45:0/4269906124 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da2012d7e0 con 0x55da204df580 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.767+0000 7f6684ff9640 10 client.26929 handle_command_reply: tid=0 2024-03-12T05:18:31.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.767+0000 7f66a63dc740 1 client.26929 shutdown 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 msgr2=0x55da20530260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6694004d70 tx=0x7f6694009840 comp rx=0 tx=0).stop 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 shutdown_connections 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55da204df580 0x55da20539c90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55da204c6b70 0x55da205307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da204cf750 0x55da204d5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 --2- 172.21.15.45:0/4269906124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da204c6250 0x55da20530260 secure :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6694004d70 tx=0x7f6694009840 comp rx=0 tx=0).stop 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 >> 172.21.15.45:0/4269906124 conn(0x55da204c0980 msgr2=0x55da204cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.768+0000 7f6684ff9640 0 client.26929 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.770+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 shutdown_connections 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.770+0000 7f66a63dc740 1 -- 172.21.15.45:0/4269906124 wait complete. 2024-03-12T05:18:31.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.770+0000 7f66a63dc740 20 client.26929 trim_cache size 0 max 16384 2024-03-12T05:18:31.768 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.771+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 msgr2=0x7f66880a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.771+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 0x7f66880a40e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6698005fd0 tx=0x7f6698005da0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.771+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 msgr2=0x7f669c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.771+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f668c000c00 tx=0x7f668c004fc0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 shutdown_connections 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f66880a1c20 0x7f66880a40e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669c105330 0x7f669c109e00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f669c10a340 0x7f669c10a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 --2- 172.21.15.45:0/3827439141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f669c100260 0x7f669c10cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 >> 172.21.15.45:0/3827439141 conn(0x7f669c077180 msgr2=0x7f669c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 shutdown_connections 2024-03-12T05:18:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.772+0000 7f66a372c640 1 -- 172.21.15.45:0/3827439141 wait complete. 2024-03-12T05:18:31.788 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:18:31.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:18:31.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.962+0000 7f629f13a640 1 -- 172.21.15.45:0/834036136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100c10 msgr2=0x7f6298104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.962+0000 7f629f13a640 1 --2- 172.21.15.45:0/834036136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100c10 0x7f6298104c50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f628c009860 tx=0x7f628c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 -- 172.21.15.45:0/834036136 shutdown_connections 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 --2- 172.21.15.45:0/834036136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298105330 0x7f629810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 --2- 172.21.15.45:0/834036136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298100c10 0x7f6298104c50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 --2- 172.21.15.45:0/834036136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6298100260 0x7f6298100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 -- 172.21.15.45:0/834036136 >> 172.21.15.45:0/834036136 conn(0x7f6298077180 msgr2=0x7f6298077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 -- 172.21.15.45:0/834036136 shutdown_connections 2024-03-12T05:18:31.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.963+0000 7f629f13a640 1 -- 172.21.15.45:0/834036136 wait complete. 2024-03-12T05:18:31.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.964+0000 7f629f13a640 1 Processor -- start 2024-03-12T05:18:31.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.964+0000 7f629f13a640 1 -- start start 2024-03-12T05:18:31.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.964+0000 7f629f13a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.964+0000 7f629f13a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298105330 0x7f6298199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.964+0000 7f629f13a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 0x7f629819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629f13a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629819e9e0 con 0x7f629819dff0 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47082/0 (socket says 172.21.15.45:47082) 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 -- 172.21.15.45:0/1612691256 learned_addr learned my addr 172.21.15.45:0/1612691256 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 0x7f629819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629819eb50 con 0x7f6298105330 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629819ecc0 con 0x7f6298100260 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f6297fff640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298105330 0x7f6298199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 -- 172.21.15.45:0/1612691256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298105330 msgr2=0x7f6298199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298105330 0x7f6298199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.965+0000 7f629ced6640 1 -- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 msgr2=0x7f629819e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.966+0000 7f629ced6640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 0x7f629819e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.966+0000 7f629ced6640 1 -- 172.21.15.45:0/1612691256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c009490 con 0x7f6298100260 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.966+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 0x7f629819e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.966+0000 7f629ced6640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f628800e9c0 tx=0x7f628800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.967+0000 7f6295ffb640 1 -- 172.21.15.45:0/1612691256 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628800ccc0 con 0x7f6298100260 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.967+0000 7f6295ffb640 1 -- 172.21.15.45:0/1612691256 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f628800ce20 con 0x7f6298100260 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.967+0000 7f6295ffb640 1 -- 172.21.15.45:0/1612691256 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288010430 con 0x7f6298100260 2024-03-12T05:18:31.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.967+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62981a62a0 con 0x7f6298100260 2024-03-12T05:18:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.967+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62981a6670 con 0x7f6298100260 2024-03-12T05:18:31.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.970+0000 7f6295ffb640 1 -- 172.21.15.45:0/1612691256 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6288010590 con 0x7f6298100260 2024-03-12T05:18:31.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 Processor -- start 2024-03-12T05:18:31.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 -- start start 2024-03-12T05:18:31.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:31 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3426384759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811530630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530b70 0x558811539210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588114eb960 con 0x558811530250 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881149f310 con 0x558811539750 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f62a1de6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881149c840 con 0x558811530b70 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f6295ffb640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 0x7f62800ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f6297fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f629d6d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811530630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f6297fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52250/0 (socket says 172.21.15.45:52250) 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.971+0000 7f6297fff640 1 -- 172.21.15.45:0/1325662281 learned_addr learned my addr 172.21.15.45:0/1325662281 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6295ffb640 1 -- 172.21.15.45:0/1612691256 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6288014070 con 0x7f6298100260 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 0x7f62800ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f629ced6640 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530b70 0x558811539210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 -- 172.21.15.45:0/1325662281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530b70 msgr2=0x558811539210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530b70 0x558811539210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 -- 172.21.15.45:0/1325662281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 msgr2=0x558811530630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811530630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 -- 172.21.15.45:0/1325662281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55881145a060 con 0x558811539750 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811530630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.972+0000 7f6297fff640 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f629806a8e0 tx=0x7f628c032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa7101ae38b4a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.973+0000 7f627effd640 1 -- 172.21.15.45:0/1325662281 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c034440 con 0x558811539750 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.973+0000 7f627effd640 1 -- 172.21.15.45:0/1325662281 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f628c02ce90 con 0x558811539750 2024-03-12T05:18:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.973+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 0x7f62800ac970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6290009fd0 tx=0x7f6290004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.973+0000 7f627effd640 1 -- 172.21.15.45:0/1325662281 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c005430 con 0x558811539750 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 -- 172.21.15.45:0/1325662281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 msgr2=0x55881153bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f629806a8e0 tx=0x7f628c032450 comp rx=0 tx=0).stop 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 -- 172.21.15.45:0/1325662281 shutdown_connections 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x55881153bb40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530b70 0x558811539210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1325662281 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811530630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 -- 172.21.15.45:0/1325662281 >> 172.21.15.45:0/1325662281 conn(0x55881152a980 msgr2=0x55881152ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 -- 172.21.15.45:0/1325662281 shutdown_connections 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.974+0000 7f62a1de6740 1 -- 172.21.15.45:0/1325662281 wait complete. 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.975+0000 7f62a1de6740 1 -- start start 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.975+0000 7f62a1de6740 1 Processor -- start 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.975+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x558811536c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811537150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558811498020 con 0x558811530b70 2024-03-12T05:18:31.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881116b430 con 0x558811530250 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558811127e00 con 0x558811539750 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f6297fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x558811536c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629ced6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629d6d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811537150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629ced6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52256/0 (socket says 172.21.15.45:52256) 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629ced6640 1 -- 172.21.15.45:0/1851587065 learned_addr learned my addr 172.21.15.45:0/1851587065 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f62a1de6740 10 client.0 _ll_register_callbacks cb 0x55881143fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:31.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629d6d7640 1 -- 172.21.15.45:0/1851587065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 msgr2=0x5588115366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.976+0000 7f629d6d7640 1 -- 172.21.15.45:0/1851587065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 msgr2=0x558811536c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x558811536c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f629d6d7640 1 -- 172.21.15.45:0/1851587065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6298068660 con 0x558811539750 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f6297fff640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x558811536c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f629ced6640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811537150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6290005fd0 tx=0x7f6290004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae33cf62de5442e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f6277fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290004b20 con 0x558811539750 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588111977e0 con 0x558811539750 2024-03-12T05:18:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.977+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6290004c80 con 0x558811539750 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6290004e40 con 0x558811539750 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 10 client.26938 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5588114966a0 con 0x558811539750 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5588111977e0 con 0x558811539750 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f62900051c0 con 0x558811539750 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 10 client.26938 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 10 client.26938 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 20 client.26938 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x55881154bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 4 client.26938 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:31.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.978+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588111d4720 con 0x558811549b00 2024-03-12T05:18:31.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.979+0000 7f629ced6640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x55881154bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.979+0000 7f629ced6640 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x55881154bf20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea8ad1a4d56b89c4 server_cookie=10119f04a97796ef in_seq=0 out_seq=0 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.979+0000 7f6277fff640 10 client.26938 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f6277fff640 1 -- 172.21.15.45:0/1851587065 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588111d4720 con 0x558811549b00 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f6277fff640 10 client.26938 handle_command_reply: tid=0 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 client.26938 shutdown 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 msgr2=0x558811537150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:31.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811537150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6290005fd0 tx=0x7f6290004a40 comp rx=0 tx=0).stop 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 shutdown_connections 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x558811536c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x55881154bf20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811537150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1851587065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530250 0x5588115366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f6277fff640 0 client.26938 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.980+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 >> 172.21.15.45:0/1851587065 conn(0x55881152a980 msgr2=0x55881152c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.981+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 shutdown_connections 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.981+0000 7f62a1de6740 1 -- 172.21.15.45:0/1851587065 wait complete. 2024-03-12T05:18:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:31.981+0000 7f62a1de6740 20 client.26938 trim_cache size 0 max 16384 2024-03-12T05:18:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.002+0000 7f62a1de6740 1 Processor -- start 2024-03-12T05:18:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.002+0000 7f62a1de6740 1 -- start start 2024-03-12T05:18:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811538e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811540670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881141d130 con 0x558811530250 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558811532060 con 0x558811530b70 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f62a1de6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588111d4720 con 0x558811539750 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811538e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f6297fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f6297fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52268/0 (socket says 172.21.15.45:52268) 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f6297fff640 1 -- 172.21.15.45:0/258737924 learned_addr learned my addr 172.21.15.45:0/258737924 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629d6d7640 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811540670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 -- 172.21.15.45:0/258737924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 msgr2=0x558811540670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811540670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 -- 172.21.15.45:0/258737924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 msgr2=0x55881152c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f629ced6640 1 -- 172.21.15.45:0/258737924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558811432c20 con 0x558811530250 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.003+0000 7f6297fff640 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.004+0000 7f629ced6640 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811538e00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f628809a3d0 tx=0x7f628800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8557e0ce839d09e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.004+0000 7f627e7fc640 1 -- 172.21.15.45:0/258737924 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628809add0 con 0x558811530250 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.004+0000 7f627e7fc640 1 -- 172.21.15.45:0/258737924 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f628809a6d0 con 0x558811530250 2024-03-12T05:18:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.004+0000 7f627e7fc640 1 -- 172.21.15.45:0/258737924 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288096d40 con 0x558811530250 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 -- 172.21.15.45:0/258737924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 msgr2=0x558811538e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811538e00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f628809a3d0 tx=0x7f628800cf90 comp rx=0 tx=0).stop 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 -- 172.21.15.45:0/258737924 shutdown_connections 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811539750 0x558811540670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811530b70 0x55881152c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 --2- 172.21.15.45:0/258737924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530250 0x558811538e00 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.006+0000 7f62a1de6740 1 -- 172.21.15.45:0/258737924 >> 172.21.15.45:0/258737924 conn(0x55881152a980 msgr2=0x55881152bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.007+0000 7f62a1de6740 1 -- 172.21.15.45:0/258737924 shutdown_connections 2024-03-12T05:18:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.007+0000 7f62a1de6740 1 -- 172.21.15.45:0/258737924 wait complete. 2024-03-12T05:18:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.007+0000 7f62a1de6740 1 -- start start 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.008+0000 7f62a1de6740 1 Processor -- start 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 0x55881153f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x55881153fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f629ced6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 0x55881153f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f629ced6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 0x55881153f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47110/0 (socket says 172.21.15.45:47110) 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f6297fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x55881153fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f629ced6640 1 -- 172.21.15.45:0/1824544018 learned_addr learned my addr 172.21.15.45:0/1824544018 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x5588115a0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588115a0590 con 0x558811530b70 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588115a06d0 con 0x558811539750 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588115a0890 con 0x558811530250 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.009+0000 7f62a1de6740 10 client.0 _ll_register_callbacks cb 0x7f628c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f629d6d7640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x5588115a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 -- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 msgr2=0x55881153f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 0x55881153f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 -- 172.21.15.45:0/1824544018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 msgr2=0x5588115a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x5588115a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55881145a060 con 0x558811530b70 2024-03-12T05:18:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.010+0000 7f6297fff640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x55881153fda0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f628c02bbb0 tx=0x7f628c034980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452da360d9cce8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f6277fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c005450 con 0x558811530b70 2024-03-12T05:18:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f628c0055b0 con 0x558811530b70 2024-03-12T05:18:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f628c02ce90 con 0x558811530b70 2024-03-12T05:18:32.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f628c02c510 con 0x558811530b70 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f62a1de6740 10 client.17481 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.011+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f62981a6670 con 0x558811530b70 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f628c02cb20 con 0x558811530b70 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f628c0035f0 con 0x558811530b70 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 10 client.17481 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 10 client.17481 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 20 client.17481 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x5588115a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 4 client.17481 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:18:32.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.012+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f628c02ce90 con 0x558811549b00 2024-03-12T05:18:32.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.013+0000 7f629ced6640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x5588115a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.014+0000 7f629ced6640 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x5588115a3dd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4566ae941b058091 server_cookie=8e7e6d6062865d1a in_seq=0 out_seq=0 2024-03-12T05:18:32.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.014+0000 7f6277fff640 10 client.17481 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:18:32.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f6277fff640 1 -- 172.21.15.45:0/1824544018 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f628c02ce90 con 0x558811549b00 2024-03-12T05:18:32.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f6277fff640 10 client.17481 handle_command_reply: tid=0 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 client.17481 shutdown 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 msgr2=0x55881153fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x55881153fda0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f628c02bbb0 tx=0x7f628c034980 comp rx=0 tx=0).stop 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 shutdown_connections 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558811530b70 0x55881153fda0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.015+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558811549b00 0x5588115a3dd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f6277fff640 0 client.17481 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558811539750 0x5588115a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 1 --2- 172.21.15.45:0/1824544018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558811530250 0x55881153f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 >> 172.21.15.45:0/1824544018 conn(0x55881152a980 msgr2=0x55881152c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 shutdown_connections 2024-03-12T05:18:32.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 1 -- 172.21.15.45:0/1824544018 wait complete. 2024-03-12T05:18:32.017 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.016+0000 7f62a1de6740 20 client.17481 trim_cache size 0 max 16384 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.018+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 msgr2=0x7f62800ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.018+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 0x7f62800ac970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6290009fd0 tx=0x7f6290004ab0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.018+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 msgr2=0x7f6298199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.018+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f628800e9c0 tx=0x7f628800ee80 comp rx=0 tx=0).stop 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 shutdown_connections 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f62800aa4b0 0x7f62800ac970 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f629819dff0 0x7f629819e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6298105330 0x7f6298199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 --2- 172.21.15.45:0/1612691256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6298100260 0x7f6298199730 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 >> 172.21.15.45:0/1612691256 conn(0x7f6298077180 msgr2=0x7f62980fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.019+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 shutdown_connections 2024-03-12T05:18:32.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.020+0000 7f629f13a640 1 -- 172.21.15.45:0/1612691256 wait complete. 2024-03-12T05:18:32.034 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:18:32.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:18:32.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.206+0000 7fae29ffb640 1 -- 172.21.15.45:0/3519683185 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c032340 con 0x7fae2c10d720 2024-03-12T05:18:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.206+0000 7fae31447640 1 -- 172.21.15.45:0/3519683185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c10d720 msgr2=0x7fae2c10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 --2- 172.21.15.45:0/3519683185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c10d720 0x7fae2c10db00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009890 tx=0x7fae1c02bb90 comp rx=0 tx=0).stop 2024-03-12T05:18:32.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 -- 172.21.15.45:0/3519683185 shutdown_connections 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 --2- 172.21.15.45:0/3519683185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c1080d0 0x7fae2c1145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 --2- 172.21.15.45:0/3519683185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c107710 0x7fae2c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 --2- 172.21.15.45:0/3519683185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c10d720 0x7fae2c10db00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 -- 172.21.15.45:0/3519683185 >> 172.21.15.45:0/3519683185 conn(0x7fae2c075620 msgr2=0x7fae2c077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 -- 172.21.15.45:0/3519683185 shutdown_connections 2024-03-12T05:18:32.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.207+0000 7fae31447640 1 -- 172.21.15.45:0/3519683185 wait complete. 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 Processor -- start 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 -- start start 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 0x7fae2c19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c1080d0 0x7fae2c19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c1a2ca0 con 0x7fae2c107710 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c1a2e10 con 0x7fae2c1080d0 2024-03-12T05:18:32.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.208+0000 7fae31447640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c1a2fd0 con 0x7fae2c10d720 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47116/0 (socket says 172.21.15.45:47116) 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 -- 172.21.15.45:0/1720336685 learned_addr learned my addr 172.21.15.45:0/1720336685 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2affd640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 0x7fae2c19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c1080d0 0x7fae2c19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 -- 172.21.15.45:0/1720336685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c1080d0 msgr2=0x7fae2c19e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c1080d0 0x7fae2c19e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 -- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 msgr2=0x7fae2c19dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 0x7fae2c19dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2b7fe640 1 -- 172.21.15.45:0/1720336685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c009490 con 0x7fae2c10d720 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.209+0000 7fae2affd640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 0x7fae2c19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fae24009fd0 tx=0x7fae2400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae23fff640 1 -- 172.21.15.45:0/1720336685 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24009b50 con 0x7fae2c10d720 2024-03-12T05:18:32.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c1a32b0 con 0x7fae2c10d720 2024-03-12T05:18:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae23fff640 1 -- 172.21.15.45:0/1720336685 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae24003970 con 0x7fae2c10d720 2024-03-12T05:18:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c1aab30 con 0x7fae2c10d720 2024-03-12T05:18:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.210+0000 7fae23fff640 1 -- 172.21.15.45:0/1720336685 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24004730 con 0x7fae2c10d720 2024-03-12T05:18:32.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.213+0000 7fae23fff640 1 -- 172.21.15.45:0/1720336685 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fae2400a040 con 0x7fae2c10d720 2024-03-12T05:18:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.214+0000 7fae34018740 1 Processor -- start 2024-03-12T05:18:32.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.214+0000 7fae34018740 1 -- start start 2024-03-12T05:18:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.214+0000 7fae23fff640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 0x7fae180a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae23fff640 1 -- 172.21.15.45:0/1720336685 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae240cd8a0 con 0x7fae2c10d720 2024-03-12T05:18:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 0x7fae180a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.214+0000 7fae34018740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f908630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae34018740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae34018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f913b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae34018740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f8c3960 con 0x555e4f911750 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae34018740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f877310 con 0x555e4f908250 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae34018740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f874840 con 0x555e4f908b70 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47122/0 (socket says 172.21.15.45:47122) 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2a7fc640 1 -- 172.21.15.45:0/2034082505 learned_addr learned my addr 172.21.15.45:0/2034082505 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f913b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.215+0000 7fae2affd640 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f908630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 0x7fae180a4160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae14009eb0 tx=0x7fae140156d0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 -- 172.21.15.45:0/2034082505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 msgr2=0x555e4f908630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f908630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 -- 172.21.15.45:0/2034082505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 msgr2=0x555e4f913b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f913b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 -- 172.21.15.45:0/2034082505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e4f832060 con 0x555e4f908b70 2024-03-12T05:18:32.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fae14009840 tx=0x7fae1401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0a5977d2c65731b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae217fa640 1 -- 172.21.15.45:0/2034082505 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae140177f0 con 0x555e4f908b70 2024-03-12T05:18:32.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.216+0000 7fae217fa640 1 -- 172.21.15.45:0/2034082505 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae14006c50 con 0x555e4f908b70 2024-03-12T05:18:32.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.217+0000 7fae217fa640 1 -- 172.21.15.45:0/2034082505 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14033d70 con 0x555e4f908b70 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.217+0000 7fae34018740 1 -- 172.21.15.45:0/2034082505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 msgr2=0x555e4f911210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.217+0000 7fae34018740 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fae14009840 tx=0x7fae1401d000 comp rx=0 tx=0).stop 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 -- 172.21.15.45:0/2034082505 shutdown_connections 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f913b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f911210 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 --2- 172.21.15.45:0/2034082505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f908630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 -- 172.21.15.45:0/2034082505 >> 172.21.15.45:0/2034082505 conn(0x555e4f902980 msgr2=0x555e4f902d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 -- 172.21.15.45:0/2034082505 shutdown_connections 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 -- 172.21.15.45:0/2034082505 wait complete. 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 -- start start 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.218+0000 7fae34018740 1 Processor -- start 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 0x555e4f90e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f90f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 0x555e4f90e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 0x555e4f90e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47128/0 (socket says 172.21.15.45:47128) 2024-03-12T05:18:32.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52306/0 (socket says 172.21.15.45:52306) 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f90f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f90f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56014/0 (socket says 172.21.15.45:56014) 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae2affd640 1 -- 172.21.15.45:0/537186042 learned_addr learned my addr 172.21.15.45:0/537186042 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f870020 con 0x555e4f911750 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f543430 con 0x555e4f908b70 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.219+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f4ffe00 con 0x555e4f908250 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae34018740 10 client.0 _ll_register_callbacks cb 0x555e4f817cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 -- 172.21.15.45:0/537186042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 msgr2=0x555e4f90e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 0x555e4f90e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 -- 172.21.15.45:0/537186042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 msgr2=0x555e4f90f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f90f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae14018480 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fae14009840 tx=0x7fae14000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10950a440f561651 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.220+0000 7fae127fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14033ea0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae140175f0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e4f56f7e0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae140177b0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae34018740 10 client.26978 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555e4f86e6a0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555e4f56f7e0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.221+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae140508f0 con 0x555e4f908b70 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 10 client.26978 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 10 client.26978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 20 client.26978 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:32.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f923e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 4 client.26978 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e4f5ac720 con 0x555e4f9219e0 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae2affd640 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f923e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae127fc640 10 client.26978 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.222+0000 7fae2affd640 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f923e20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5daa2b0d277166d server_cookie=ad2d81ec58a652b1 in_seq=0 out_seq=0 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.223+0000 7fae127fc640 1 -- 172.21.15.45:0/537186042 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e4f5ac720 con 0x555e4f9219e0 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.223+0000 7fae127fc640 10 client.26978 handle_command_reply: tid=0 2024-03-12T05:18:32.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.223+0000 7fae34018740 1 client.26978 shutdown 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.223+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 msgr2=0x555e4f90eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.223+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fae14009840 tx=0x7fae14000bc0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 shutdown_connections 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f923e20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae127fc640 0 client.26978 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f90f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90eca0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 --2- 172.21.15.45:0/537186042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908250 0x555e4f90e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 >> 172.21.15.45:0/537186042 conn(0x555e4f902980 msgr2=0x555e4f9046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 shutdown_connections 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 1 -- 172.21.15.45:0/537186042 wait complete. 2024-03-12T05:18:32.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.224+0000 7fae34018740 20 client.26978 trim_cache size 0 max 16384 2024-03-12T05:18:32.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.247+0000 7fae34018740 1 Processor -- start 2024-03-12T05:18:32.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.248+0000 7fae34018740 1 -- start start 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.248+0000 7fae34018740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.248+0000 7fae34018740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.248+0000 7fae34018740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f911750 0x555e4f90b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.248+0000 7fae34018740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f7f5130 con 0x555e4f908250 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae34018740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f929070 con 0x555e4f908b70 2024-03-12T05:18:32.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56026/0 (socket says 172.21.15.45:56026) 2024-03-12T05:18:32.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae34018740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f8fe080 con 0x555e4f911750 2024-03-12T05:18:32.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 -- 172.21.15.45:0/3887917609 learned_addr learned my addr 172.21.15.45:0/3887917609 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f911750 0x555e4f90b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 -- 172.21.15.45:0/3887917609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f911750 msgr2=0x555e4f90b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f911750 0x555e4f90b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 -- 172.21.15.45:0/3887917609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 msgr2=0x555e4f90f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.249+0000 7fae2affd640 1 -- 172.21.15.45:0/3887917609 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e4f80ac20 con 0x555e4f908250 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.250+0000 7fae2affd640 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fae1c037040 tx=0x7fae1c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e06f9d00311d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.250+0000 7fae20ff9640 1 -- 172.21.15.45:0/3887917609 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c044070 con 0x555e4f908250 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.250+0000 7fae20ff9640 1 -- 172.21.15.45:0/3887917609 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae1c034cc0 con 0x555e4f908250 2024-03-12T05:18:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.250+0000 7fae20ff9640 1 -- 172.21.15.45:0/3887917609 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c02c790 con 0x555e4f908250 2024-03-12T05:18:32.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.253+0000 7fae34018740 1 -- 172.21.15.45:0/3887917609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 msgr2=0x555e4f910e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.253+0000 7fae34018740 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fae1c037040 tx=0x7fae1c005720 comp rx=0 tx=0).stop 2024-03-12T05:18:32.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.253+0000 7fae34018740 1 -- 172.21.15.45:0/3887917609 shutdown_connections 2024-03-12T05:18:32.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.253+0000 7fae34018740 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f911750 0x555e4f90b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908b70 0x555e4f90f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 --2- 172.21.15.45:0/3887917609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f908250 0x555e4f910e90 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 -- 172.21.15.45:0/3887917609 >> 172.21.15.45:0/3887917609 conn(0x555e4f902980 msgr2=0x555e4f909a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 -- 172.21.15.45:0/3887917609 shutdown_connections 2024-03-12T05:18:32.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 -- 172.21.15.45:0/3887917609 wait complete. 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.254+0000 7fae34018740 1 -- start start 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.255+0000 7fae34018740 1 Processor -- start 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.255+0000 7fae34018740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.255+0000 7fae34018740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f9176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.255+0000 7fae2affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.255+0000 7fae2affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:52324/0 (socket says 172.21.15.45:52324) 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2affd640 1 -- 172.21.15.45:0/750271195 learned_addr learned my addr 172.21.15.45:0/750271195 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:32.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f9176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f917c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f977f50 con 0x555e4f911750 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f9780c0 con 0x555e4f908250 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4f978280 con 0x555e4f908b70 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2b7fe640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f917c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae34018740 10 client.0 _ll_register_callbacks cb 0x7fae14009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2affd640 1 -- 172.21.15.45:0/750271195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 msgr2=0x555e4f9176d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2affd640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f9176d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.256+0000 7fae2affd640 1 -- 172.21.15.45:0/750271195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 msgr2=0x555e4f917c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae2affd640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f917c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae2affd640 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c114ee0 con 0x555e4f908250 2024-03-12T05:18:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae2affd640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0057c0 tx=0x7fae1c035c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73c1ff5a3bc0dea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae127fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c044050 con 0x555e4f908250 2024-03-12T05:18:32.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae1c004350 con 0x555e4f908250 2024-03-12T05:18:32.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae14006c50 con 0x555e4f908250 2024-03-12T05:18:32.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.257+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae1c004510 con 0x555e4f908250 2024-03-12T05:18:32.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.258+0000 7fae34018740 10 client.26984 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:32.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.258+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555e4f8308c0 con 0x555e4f908250 2024-03-12T05:18:32.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.258+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae14006c50 con 0x555e4f908250 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.258+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae1c032600 con 0x555e4f908250 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 10 client.26984 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 10 client.26984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 20 client.26984 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f97b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 4 client.26984 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:18:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.259+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e4f5fb890 con 0x555e4f9219e0 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.261+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f97b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.261+0000 7fae2a7fc640 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f97b650 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c12d675e130851d0 server_cookie=74a6dd391b71dd60 in_seq=0 out_seq=0 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.261+0000 7fae127fc640 10 client.26984 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.262+0000 7fae127fc640 1 -- 172.21.15.45:0/750271195 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e4f5fb890 con 0x555e4f9219e0 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.262+0000 7fae127fc640 10 client.26984 handle_command_reply: tid=0 2024-03-12T05:18:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.262+0000 7fae34018740 1 client.26984 shutdown 2024-03-12T05:18:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 msgr2=0x555e4f917190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0057c0 tx=0x7fae1c035c70 comp rx=0 tx=0).stop 2024-03-12T05:18:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 shutdown_connections 2024-03-12T05:18:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555e4f9219e0 0x555e4f97b650 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae127fc640 0 client.26984 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e4f911750 0x555e4f917c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e4f908b70 0x555e4f9176d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 --2- 172.21.15.45:0/750271195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e4f908250 0x555e4f917190 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 >> 172.21.15.45:0/750271195 conn(0x555e4f902980 msgr2=0x555e4f9012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.263+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 shutdown_connections 2024-03-12T05:18:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.264+0000 7fae34018740 1 -- 172.21.15.45:0/750271195 wait complete. 2024-03-12T05:18:32.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.264+0000 7fae34018740 20 client.26984 trim_cache size 0 max 16384 2024-03-12T05:18:32.261 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:18:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.267+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 msgr2=0x7fae180a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.267+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 0x7fae180a4160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae14009eb0 tx=0x7fae140156d0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.267+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 msgr2=0x7fae2c1a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:32.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.267+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fae24009fd0 tx=0x7fae2400c800 comp rx=0 tx=0).stop 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 shutdown_connections 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae180a1ca0 0x7fae180a4160 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae2c10d720 0x7fae2c1a26a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2c1080d0 0x7fae2c19e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 --2- 172.21.15.45:0/1720336685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae2c107710 0x7fae2c19dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:32.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.268+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 >> 172.21.15.45:0/1720336685 conn(0x7fae2c075620 msgr2=0x7fae2c075fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.269+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 shutdown_connections 2024-03-12T05:18:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:32.269+0000 7fae31447640 1 -- 172.21.15.45:0/1720336685 wait complete. 2024-03-12T05:18:32.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:32 smithi116 ceph-mon[35486]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:18:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:32 smithi045 ceph-mon[32174]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:18:32.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:32 smithi130 ceph-mon[35398]: pgmap v236: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:18:33.283 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:18:33.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:18:33.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:18:33.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1986425029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c103f40 msgr2=0x7f6d2c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1986425029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c103f40 0x7f6d2c107f80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009930 tx=0x7f6d2002bc50 comp rx=0 tx=0).stop 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1986425029 shutdown_connections 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1986425029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c103f40 0x7f6d2c107f80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1986425029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c108660 0x7f6d2c110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1986425029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c103590 0x7f6d2c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1986425029 >> 172.21.15.45:0/1986425029 conn(0x7f6d2c0fedf0 msgr2=0x7f6d2c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1986425029 shutdown_connections 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.517+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1986425029 wait complete. 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 Processor -- start 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 -- start start 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c103590 0x7f6d2c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c199830 0x7f6d2c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c19a2e0 con 0x7f6d2c108660 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c19a450 con 0x7f6d2c103590 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d31e5b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c19a5c0 con 0x7f6d2c199830 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56048/0 (socket says 172.21.15.45:56048) 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d2affd640 1 -- 172.21.15.45:0/1690085213 learned_addr learned my addr 172.21.15.45:0/1690085213 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c199830 0x7f6d2c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.518+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c103590 0x7f6d2c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 -- 172.21.15.45:0/1690085213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c199830 msgr2=0x7f6d2c199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c199830 0x7f6d2c199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 -- 172.21.15.45:0/1690085213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c103590 msgr2=0x7f6d2c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c103590 0x7f6d2c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 -- 172.21.15.45:0/1690085213 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20009560 con 0x7f6d2c108660 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009230 tx=0x7f6d2002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d28ff9640 1 -- 172.21.15.45:0/1690085213 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20004410 con 0x7f6d2c108660 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c1a64c0 con 0x7f6d2c108660 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c1a69d0 con 0x7f6d2c108660 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d28ff9640 1 -- 172.21.15.45:0/1690085213 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d2002de90 con 0x7f6d2c108660 2024-03-12T05:18:33.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.519+0000 7f6d28ff9640 1 -- 172.21.15.45:0/1690085213 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20003a50 con 0x7f6d2c108660 2024-03-12T05:18:33.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.521+0000 7f6d28ff9640 1 -- 172.21.15.45:0/1690085213 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6d20005400 con 0x7f6d2c108660 2024-03-12T05:18:33.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.521+0000 7f6d34a2a740 1 Processor -- start 2024-03-12T05:18:33.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 -- start start 2024-03-12T05:18:33.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d28ff9640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 0x7f6d140a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 0x55a83cfaf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d28ff9640 1 -- 172.21.15.45:0/1690085213 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6d200ef7d0 con 0x7f6d2c108660 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 0x7f6d140a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfbad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cf1df90 con 0x55a83cfafd30 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cf1b4c0 con 0x55a83cfaf380 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cf16ca0 con 0x55a83cfb88b0 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56060/0 (socket says 172.21.15.45:56060) 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 -- 172.21.15.45:0/1975050143 learned_addr learned my addr 172.21.15.45:0/1975050143 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 0x55a83cfaf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfbad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 -- 172.21.15.45:0/1975050143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 msgr2=0x55a83cfbad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfbad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 -- 172.21.15.45:0/1975050143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 msgr2=0x55a83cfaf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.522+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 0x55a83cfaf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d2affd640 1 -- 172.21.15.45:0/1975050143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a83ced90b0 con 0x55a83cfafd30 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfbad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 0x55a83cfaf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 0x7f6d140a4060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009390 tx=0x7f6d24009650 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d2affd640 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6d200b8e50 tx=0x7f6d200b80e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cde52211c4d4ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1975050143 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200bcb80 con 0x55a83cfafd30 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1975050143 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d200bc480 con 0x55a83cfafd30 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.523+0000 7f6d11ffb640 1 -- 172.21.15.45:0/1975050143 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200c4790 con 0x55a83cfafd30 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- 172.21.15.45:0/1975050143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 msgr2=0x55a83cfb82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6d200b8e50 tx=0x7f6d200b80e0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- 172.21.15.45:0/1975050143 shutdown_connections 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb82e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfbad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/1975050143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfaf380 0x55a83cfaf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- 172.21.15.45:0/1975050143 >> 172.21.15.45:0/1975050143 conn(0x55a83cfa9840 msgr2=0x55a83cfa9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- 172.21.15.45:0/1975050143 shutdown_connections 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- 172.21.15.45:0/1975050143 wait complete. 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 -- start start 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.524+0000 7f6d34a2a740 1 Processor -- start 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfb5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 0x55a83cfb6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cebbf40 con 0x55a83cfafd30 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cbea4b0 con 0x55a83cfb88b0 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83ce9c670 con 0x55a83cfaf380 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56076/0 (socket says 172.21.15.45:56076) 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2affd640 1 -- 172.21.15.45:0/207894086 learned_addr learned my addr 172.21.15.45:0/207894086 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d34a2a740 10 client.0 _ll_register_callbacks cb 0x55a83cf09f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 0x55a83cfb6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfb5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 -- 172.21.15.45:0/207894086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 msgr2=0x55a83cfb5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfb5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 -- 172.21.15.45:0/207894086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 msgr2=0x55a83cfb5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 -- 172.21.15.45:0/207894086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c1a69d0 con 0x55a83cfb88b0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2affd640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 0x55a83cfb6310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d24018e30 tx=0x7f6d24013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952c9dca73a6a098 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.525+0000 7f6d0affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24010050 con 0x55a83cfb88b0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6d2401d040 con 0x55a83cfb88b0 2024-03-12T05:18:33.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a83cef0f40 con 0x55a83cfb88b0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d24021640 con 0x55a83cfb88b0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 10 client.26993 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a83cf15320 con 0x55a83cfb88b0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d2402e020 con 0x55a83cfb88b0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6d24029970 con 0x55a83cfb88b0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 10 client.26993 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 10 client.26993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 20 client.26993 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83cfc8ae0 0x55a83cfcaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 4 client.26993 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.526+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a83cf0d310 con 0x55a83cfc8ae0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83cfc8ae0 0x55a83cfcaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d0affd640 10 client.26993 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83cfc8ae0 0x55a83cfcaf00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c6fdd105677b3de server_cookie=4bc28b7073983454 in_seq=0 out_seq=0 2024-03-12T05:18:33.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d0affd640 1 -- 172.21.15.45:0/207894086 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a83cf0d310 con 0x55a83cfc8ae0 2024-03-12T05:18:33.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d0affd640 10 client.26993 handle_command_reply: tid=0 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.527+0000 7f6d34a2a740 1 client.26993 shutdown 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 msgr2=0x55a83cfb6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 0x55a83cfb6310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d24018e30 tx=0x7f6d24013000 comp rx=0 tx=0).stop 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 shutdown_connections 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83cfc8ae0 0x55a83cfcaf00 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfafd30 0x55a83cfb5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfb88b0 0x55a83cfb6310 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d0affd640 0 client.26993 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/207894086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfb5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 >> 172.21.15.45:0/207894086 conn(0x55a83cfa9840 msgr2=0x55a83cfab7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 shutdown_connections 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 1 -- 172.21.15.45:0/207894086 wait complete. 2024-03-12T05:18:33.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.528+0000 7f6d34a2a740 20 client.26993 trim_cache size 0 max 16384 2024-03-12T05:18:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.551+0000 7f6d34a2a740 1 Processor -- start 2024-03-12T05:18:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 -- start start 2024-03-12T05:18:33.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfb6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfb2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cfa9370 con 0x55a83cfaf380 2024-03-12T05:18:33.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cfaa410 con 0x55a83cfafd30 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cfd00f0 con 0x55a83cfb88b0 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d2b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56084/0 (socket says 172.21.15.45:56084) 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.552+0000 7f6d2b7fe640 1 -- 172.21.15.45:0/2316848440 learned_addr learned my addr 172.21.15.45:0/2316848440 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfb6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfb2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 -- 172.21.15.45:0/2316848440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 msgr2=0x55a83cfb2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfb2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 -- 172.21.15.45:0/2316848440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 msgr2=0x55a83cfb7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 -- 172.21.15.45:0/2316848440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a83ceb1c00 con 0x55a83cfafd30 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:33.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d2affd640 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfb6130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6d2004aff0 tx=0x7f6d200dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=451d8aa0987b8163 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.553+0000 7f6d117fa640 1 -- 172.21.15.45:0/2316848440 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200bcb80 con 0x55a83cfafd30 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.554+0000 7f6d117fa640 1 -- 172.21.15.45:0/2316848440 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d200c05d0 con 0x55a83cfafd30 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.554+0000 7f6d117fa640 1 -- 172.21.15.45:0/2316848440 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d200c49f0 con 0x55a83cfafd30 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 -- 172.21.15.45:0/2316848440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 msgr2=0x55a83cfb6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfb6130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6d2004aff0 tx=0x7f6d200dc040 comp rx=0 tx=0).stop 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 -- 172.21.15.45:0/2316848440 shutdown_connections 2024-03-12T05:18:33.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfb88b0 0x55a83cfb2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfb6130 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/2316848440 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfaf380 0x55a83cfb7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 -- 172.21.15.45:0/2316848440 >> 172.21.15.45:0/2316848440 conn(0x55a83cfa9840 msgr2=0x55a83cfb4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 -- 172.21.15.45:0/2316848440 shutdown_connections 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.555+0000 7f6d34a2a740 1 -- 172.21.15.45:0/2316848440 wait complete. 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.556+0000 7f6d34a2a740 1 -- start start 2024-03-12T05:18:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.556+0000 7f6d34a2a740 1 Processor -- start 2024-03-12T05:18:33.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.556+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfbe020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.556+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfbe560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d34a2a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83cfbffc0 con 0x55a83cfb88b0 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83d027800 con 0x55a83cfafd30 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d34a2a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83d027940 con 0x55a83cfaf380 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56096/0 (socket says 172.21.15.45:56096) 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 -- 172.21.15.45:0/3637886377 learned_addr learned my addr 172.21.15.45:0/3637886377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2affd640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfbe560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d34a2a740 10 client.0 _ll_register_callbacks cb 0x7f6d20005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfbe020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 -- 172.21.15.45:0/3637886377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 msgr2=0x55a83cfbe020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfbe020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 -- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 msgr2=0x55a83cfbe560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfbe560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2bfff640 1 -- 172.21.15.45:0/3637886377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c00bb00 con 0x55a83cfb88b0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.557+0000 7f6d2affd640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfbe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d2bfff640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6d24018850 tx=0x7f6d24029bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ba7480878624ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d0affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24010040 con 0x55a83cfb88b0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a83cf3c900 con 0x55a83cfb88b0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6d2401d050 con 0x55a83cfb88b0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d2400ace0 con 0x55a83cfb88b0 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d34a2a740 10 client.17517 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:33.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.558+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a83ced90b0 con 0x55a83cfb88b0 2024-03-12T05:18:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6d2402e040 con 0x55a83cfb88b0 2024-03-12T05:18:33.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6d2402a510 con 0x55a83cfb88b0 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 10 client.17517 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 20 client.17517 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83d029120 0x55a83d02b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 4 client.17517 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a83cf3c900 con 0x55a83d029120 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.559+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83d029120 0x55a83d02b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.560+0000 7f6d0affd640 10 client.17517 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.560+0000 7f6d2b7fe640 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83d029120 0x55a83d02b540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d742c120e22323d5 server_cookie=c5b1410fd98b1840 in_seq=0 out_seq=0 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.561+0000 7f6d0affd640 1 -- 172.21.15.45:0/3637886377 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a83cf3c900 con 0x55a83d029120 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.561+0000 7f6d0affd640 10 client.17517 handle_command_reply: tid=0 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.561+0000 7f6d34a2a740 1 client.17517 shutdown 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.561+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 msgr2=0x55a83cfbeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.561+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6d24018850 tx=0x7f6d24029bb0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 shutdown_connections 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a83d029120 0x55a83d02b540 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a83cfb88b0 0x55a83cfbeaa0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a83cfafd30 0x55a83cfbe560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 --2- 172.21.15.45:0/3637886377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a83cfaf380 0x55a83cfbe020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d0affd640 0 client.17517 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 >> 172.21.15.45:0/3637886377 conn(0x55a83cfa9840 msgr2=0x55a83cfaae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 shutdown_connections 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 1 -- 172.21.15.45:0/3637886377 wait complete. 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.562+0000 7f6d34a2a740 20 client.17517 trim_cache size 0 max 16384 2024-03-12T05:18:33.559 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:18:33.560 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:18:33.560 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "cad1a2c2-cb22-401f-baff-926095405622", 2024-03-12T05:18:33.560 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:18:33.560 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 msgr2=0x7f6d140a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 0x7f6d140a4060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009390 tx=0x7f6d24009650 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 msgr2=0x7f6d2c19fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009230 tx=0x7f6d2002c680 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 shutdown_connections 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6d140a1ba0 0x7f6d140a4060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d2c108660 0x7f6d2c19fbb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d2c199830 0x7f6d2c199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 --2- 172.21.15.45:0/1690085213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d2c103590 0x7f6d2c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 >> 172.21.15.45:0/1690085213 conn(0x7f6d2c0fedf0 msgr2=0x7f6d2c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 shutdown_connections 2024-03-12T05:18:33.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.563+0000 7f6d31e5b640 1 -- 172.21.15.45:0/1690085213 wait complete. 2024-03-12T05:18:33.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:18:33.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 -- 172.21.15.45:0/2119359221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28105340 msgr2=0x7fbd280682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/2119359221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28105340 0x7fbd280682e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009be0 tx=0x7fbd2002bef0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 -- 172.21.15.45:0/2119359221 shutdown_connections 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/2119359221 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd28105340 0x7fbd280682e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/2119359221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100c20 0x7fbd28104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/2119359221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28100270 0x7fbd28100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 -- 172.21.15.45:0/2119359221 >> 172.21.15.45:0/2119359221 conn(0x7fbd28077230 msgr2=0x7fbd28077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 -- 172.21.15.45:0/2119359221 shutdown_connections 2024-03-12T05:18:33.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.757+0000 7fbd2e37a640 1 -- 172.21.15.45:0/2119359221 wait complete. 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.758+0000 7fbd2e37a640 1 Processor -- start 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.758+0000 7fbd2e37a640 1 -- start start 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28105340 0x7fbd28072d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2806c4f0 con 0x7fbd2806b8d0 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2806c660 con 0x7fbd28105340 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2e37a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28070320 con 0x7fbd28100270 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2c917640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47214/0 (socket says 172.21.15.45:47214) 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 -- 172.21.15.45:0/1115637063 learned_addr learned my addr 172.21.15.45:0/1115637063 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2c917640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56122/0 (socket says 172.21.15.45:56122) 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28105340 0x7fbd28072d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 -- 172.21.15.45:0/1115637063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28105340 msgr2=0x7fbd28072d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28105340 0x7fbd28072d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 -- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 msgr2=0x7fbd2806bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd27fff640 1 -- 172.21.15.45:0/1115637063 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18009560 con 0x7fbd28100270 2024-03-12T05:18:33.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.759+0000 7fbd2c917640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800e9c0 tx=0x7fbd1800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd25ffb640 1 -- 172.21.15.45:0/1115637063 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1800cde0 con 0x7fbd28100270 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd25ffb640 1 -- 172.21.15.45:0/1115637063 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd180044d0 con 0x7fbd28100270 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd25ffb640 1 -- 172.21.15.45:0/1115637063 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18010430 con 0x7fbd28100270 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd20009840 con 0x7fbd28100270 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.760+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280708d0 con 0x7fbd28100270 2024-03-12T05:18:33.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.763+0000 7fbd25ffb640 1 -- 172.21.15.45:0/1115637063 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbd18004020 con 0x7fbd28100270 2024-03-12T05:18:33.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.764+0000 7fbd25ffb640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 0x7fbd140a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.764+0000 7fbd25ffb640 1 -- 172.21.15.45:0/1115637063 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbd18014070 con 0x7fbd28100270 2024-03-12T05:18:33.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.764+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 0x7fbd140a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.765+0000 7fbd3102e740 1 Processor -- start 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.765+0000 7fbd3102e740 1 -- start start 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.765+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 0x7fbd140a4120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd2806d040 tx=0x7fbd10005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.765+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.765+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd1ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47222/0 (socket says 172.21.15.45:47222) 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd1ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56136/0 (socket says 172.21.15.45:56136) 2024-03-12T05:18:33.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd27fff640 1 -- 172.21.15.45:0/1178205806 learned_addr learned my addr 172.21.15.45:0/1178205806 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc0ba280 con 0x5568dc14bbd0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc0b77b0 con 0x5568dc1547b0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc0b2f90 con 0x5568dc14b2b0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd2c917640 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd27fff640 1 -- 172.21.15.45:0/1178205806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 msgr2=0x5568dc156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.766+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd27fff640 1 -- 172.21.15.45:0/1178205806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 msgr2=0x5568dc154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd27fff640 1 -- 172.21.15.45:0/1178205806 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568dc071570 con 0x5568dc14b2b0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd18097300 tx=0x7fbd1809a7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecce64df914b4c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd1e7fc640 1 -- 172.21.15.45:0/1178205806 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1809b030 con 0x5568dc14b2b0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd1e7fc640 1 -- 172.21.15.45:0/1178205806 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd1809a930 con 0x5568dc14b2b0 2024-03-12T05:18:33.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.767+0000 7fbd1e7fc640 1 -- 172.21.15.45:0/1178205806 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18096fa0 con 0x5568dc14b2b0 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 msgr2=0x5568dc14b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbd18097300 tx=0x7fbd1809a7c0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 shutdown_connections 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14bbd0 0x5568dc154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1178205806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14b2b0 0x5568dc14b690 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 >> 172.21.15.45:0/1178205806 conn(0x5568dc145990 msgr2=0x5568dc145d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 shutdown_connections 2024-03-12T05:18:33.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.769+0000 7fbd3102e740 1 -- 172.21.15.45:0/1178205806 wait complete. 2024-03-12T05:18:33.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.770+0000 7fbd3102e740 1 -- start start 2024-03-12T05:18:33.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.770+0000 7fbd3102e740 1 Processor -- start 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc154250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dbd86430 con 0x5568dc14b2b0 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dbd42e00 con 0x5568dc1547b0 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc037fd0 con 0x5568dc14bbd0 2024-03-12T05:18:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd3102e740 10 client.0 _ll_register_callbacks cb 0x5568dc08dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56144/0 (socket says 172.21.15.45:56144) 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd27fff640 1 -- 172.21.15.45:0/1786319051 learned_addr learned my addr 172.21.15.45:0/1786319051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd2c917640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 msgr2=0x5568dc151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1786319051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 msgr2=0x5568dc154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.771+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1786319051 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280708d0 con 0x5568dc14bbd0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.772+0000 7fbd077fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.772+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc1515e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5568dc14ad80 tx=0x7fbd10009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd9b6fbdf949756 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.772+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1002a040 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.772+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568dbdb27e0 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.772+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbd1002e420 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd100232d0 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 10 client.26992 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5568dc0b1610 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd1003b020 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbd100216b0 con 0x5568dc14bbd0 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 10 client.26992 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 10 client.26992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 20 client.26992 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc166d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 4 client.26992 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.773+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568dbdef720 con 0x5568dc164970 2024-03-12T05:18:33.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.774+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc166d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.774+0000 7fbd077fe640 10 client.26992 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.774+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc166d90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5ef8fb7cdcdcd10 server_cookie=76020f8daa8fea57 in_seq=0 out_seq=0 2024-03-12T05:18:33.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd077fe640 1 -- 172.21.15.45:0/1786319051 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5568dbdef720 con 0x5568dc164970 2024-03-12T05:18:33.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd077fe640 10 client.26992 handle_command_reply: tid=0 2024-03-12T05:18:33.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 client.26992 shutdown 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 msgr2=0x5568dc1515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc1515e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x5568dc14ad80 tx=0x7fbd10009f90 comp rx=0 tx=0).stop 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 shutdown_connections 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc166d90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc1515e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.776+0000 7fbd077fe640 0 client.26992 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.775+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1786319051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.776+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 >> 172.21.15.45:0/1786319051 conn(0x5568dc145990 msgr2=0x5568dc147730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.776+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 shutdown_connections 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.776+0000 7fbd3102e740 1 -- 172.21.15.45:0/1786319051 wait complete. 2024-03-12T05:18:33.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.776+0000 7fbd3102e740 20 client.26992 trim_cache size 0 max 16384 2024-03-12T05:18:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.799+0000 7fbd3102e740 1 Processor -- start 2024-03-12T05:18:33.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.799+0000 7fbd3102e740 1 -- start start 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc153d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc15b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc1410f0 con 0x5568dc14b2b0 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc14a3a0 con 0x5568dc1547b0 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd3102e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc14a5d0 con 0x5568dc14bbd0 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:47246/0 (socket says 172.21.15.45:47246) 2024-03-12T05:18:33.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1637074995 learned_addr learned my addr 172.21.15.45:0/1637074995 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd2c917640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc15b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1637074995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 msgr2=0x5568dc15b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc15b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1637074995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 msgr2=0x5568dc153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd1ffff640 1 -- 172.21.15.45:0/1637074995 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568dc04dac0 con 0x5568dc14bbd0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.800+0000 7fbd27fff640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.801+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbd100180a0 tx=0x7fbd10023cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc5d9aced94d540 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.801+0000 7fbd1dffb640 1 -- 172.21.15.45:0/1637074995 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000ba20 con 0x5568dc14bbd0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.801+0000 7fbd1dffb640 1 -- 172.21.15.45:0/1637074995 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbd1003b040 con 0x5568dc14bbd0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.801+0000 7fbd1dffb640 1 -- 172.21.15.45:0/1637074995 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10005590 con 0x5568dc14bbd0 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- 172.21.15.45:0/1637074995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 msgr2=0x5568dc154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbd100180a0 tx=0x7fbd10023cc0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- 172.21.15.45:0/1637074995 shutdown_connections 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc1547b0 0x5568dc15b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc154250 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 --2- 172.21.15.45:0/1637074995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc14b2b0 0x5568dc153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- 172.21.15.45:0/1637074995 >> 172.21.15.45:0/1637074995 conn(0x5568dc145990 msgr2=0x5568dc14c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- 172.21.15.45:0/1637074995 shutdown_connections 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- 172.21.15.45:0/1637074995 wait complete. 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.802+0000 7fbd3102e740 1 -- start start 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 Processor -- start 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc14b2b0 0x5568dc15a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc15b030 con 0x5568dc1547b0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc14b2b0 0x5568dc15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd2c917640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd2c917640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:56168/0 (socket says 172.21.15.45:56168) 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd2c917640 1 -- 172.21.15.45:0/3265840481 learned_addr learned my addr 172.21.15.45:0/3265840481 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc15b1a0 con 0x5568dc14b2b0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.803+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568dc1bb410 con 0x5568dc14bbd0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd1ffff640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd3102e740 10 client.0 _ll_register_callbacks cb 0x7fbd180046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 -- 172.21.15.45:0/3265840481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 msgr2=0x5568dc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 -- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc14b2b0 msgr2=0x5568dc15a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc14b2b0 0x5568dc15a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568dc0b1610 con 0x5568dc1547b0 2024-03-12T05:18:33.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd1809a930 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd2c917640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009be0 tx=0x7fbd20033830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c92f9147825262 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.804+0000 7fbd077fe640 10 client.17535 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd077fe640 1 -- 172.21.15.45:0/3265840481 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20043040 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd077fe640 1 -- 172.21.15.45:0/3265840481 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbd20003b70 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd077fe640 1 -- 172.21.15.45:0/3265840481 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd20003d30 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 10 client.17535 fetch_fsmap learned FSMap version 33 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5568dc0736a0 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd077fe640 1 -- 172.21.15.45:0/3265840481 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbd20046030 con 0x5568dc1547b0 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 10 client.17535 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 20 client.17535 populate_metadata read hostname 'smithi045' 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc1bea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 4 client.17535 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:18:33.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.805+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5568dbe3e890 con 0x5568dc164970 2024-03-12T05:18:33.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd27fff640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc1bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:18:33.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd27fff640 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc1bea30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50d14120710d0c server_cookie=d3aaa970040cb3fb in_seq=0 out_seq=0 2024-03-12T05:18:33.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd077fe640 10 client.17535 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd077fe640 1 -- 172.21.15.45:0/3265840481 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5568dbe3e890 con 0x5568dc164970 2024-03-12T05:18:33.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd077fe640 10 client.17535 handle_command_reply: tid=0 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.806+0000 7fbd3102e740 1 client.17535 shutdown 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 msgr2=0x5568dc1baea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009be0 tx=0x7fbd20033830 comp rx=0 tx=0).stop 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 shutdown_connections 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5568dc164970 0x5568dc1bea30 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5568dc1547b0 0x5568dc1baea0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5568dc14bbd0 0x5568dc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 --2- 172.21.15.45:0/3265840481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568dc14b2b0 0x5568dc15a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 >> 172.21.15.45:0/3265840481 conn(0x5568dc145990 msgr2=0x5568dc14fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd077fe640 0 client.17535 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 shutdown_connections 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 1 -- 172.21.15.45:0/3265840481 wait complete. 2024-03-12T05:18:33.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.807+0000 7fbd3102e740 20 client.17535 trim_cache size 0 max 16384 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "cad1a2c2-cb22-401f-baff-926095405622": { 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "cad1a2c2-cb22-401f-baff-926095405622", 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:18:33.809 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:18:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 msgr2=0x7fbd140a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 0x7fbd140a4120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbd2806d040 tx=0x7fbd10005f90 comp rx=0 tx=0).stop 2024-03-12T05:18:33.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 msgr2=0x7fbd28072800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800e9c0 tx=0x7fbd1800ee80 comp rx=0 tx=0).stop 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 shutdown_connections 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbd140a1c60 0x7fbd140a4120 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbd2806b8d0 0x7fbd2806bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd28105340 0x7fbd28072d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 --2- 172.21.15.45:0/1115637063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd28100270 0x7fbd28072800 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 >> 172.21.15.45:0/1115637063 conn(0x7fbd28077230 msgr2=0x7fbd281020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.808+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 shutdown_connections 2024-03-12T05:18:33.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:18:33.809+0000 7fbd2e37a640 1 -- 172.21.15.45:0/1115637063 wait complete. 2024-03-12T05:18:33.820 INFO:tasks.cephfs.filesystem:scrub status for tag:cad1a2c2-cb22-401f-baff-926095405622 - {'path': '/', 'tag': 'cad1a2c2-cb22-401f-baff-926095405622', 'options': 'recursive,force'} 2024-03-12T05:18:34.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-03-12T05:18:34.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:34.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:18:34.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:34 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:18:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:34 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 390.0 s, 1356.2 tps, lat 73.727 ms stddev 27.777 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:34.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:34 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:18:36.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:36 smithi116 ceph-mon[35486]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 15 MiB/s wr, 1.71k op/s 2024-03-12T05:18:36.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:36 smithi045 ceph-mon[32174]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 15 MiB/s wr, 1.71k op/s 2024-03-12T05:18:36.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:36 smithi130 ceph-mon[35398]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 15 MiB/s wr, 1.71k op/s 2024-03-12T05:18:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:38 smithi116 ceph-mon[35486]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:18:38.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:38 smithi045 ceph-mon[32174]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:18:38.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:38 smithi130 ceph-mon[35398]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:18:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 395.0 s, 1434.8 tps, lat 69.665 ms stddev 21.136 2024-03-12T05:18:40.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:40 smithi116 ceph-mon[35486]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.58k op/s 2024-03-12T05:18:40.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:40 smithi045 ceph-mon[32174]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.58k op/s 2024-03-12T05:18:40.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:40 smithi130 ceph-mon[35398]: pgmap v240: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.58k op/s 2024-03-12T05:18:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:42 smithi116 ceph-mon[35486]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:42.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:42 smithi045 ceph-mon[32174]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:42 smithi130 ceph-mon[35398]: pgmap v241: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:44.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:44 smithi045 ceph-mon[32174]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.8 MiB/s wr, 1.11k op/s 2024-03-12T05:18:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 400.0 s, 1328.6 tps, lat 75.307 ms stddev 27.264 2024-03-12T05:18:44.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:44 smithi130 ceph-mon[35398]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.8 MiB/s wr, 1.11k op/s 2024-03-12T05:18:44.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:44 smithi116 ceph-mon[35486]: pgmap v242: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.8 MiB/s wr, 1.11k op/s 2024-03-12T05:18:45.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:45 smithi045 ceph-mon[32174]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 1.66k op/s 2024-03-12T05:18:45.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:45 smithi130 ceph-mon[35398]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 1.66k op/s 2024-03-12T05:18:45.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:45 smithi116 ceph-mon[35486]: pgmap v243: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 15 MiB/s wr, 1.66k op/s 2024-03-12T05:18:47.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:47 smithi130 ceph-mon[35398]: pgmap v244: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.4 MiB/s wr, 1.07k op/s 2024-03-12T05:18:47.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:47 smithi116 ceph-mon[35486]: pgmap v244: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.4 MiB/s wr, 1.07k op/s 2024-03-12T05:18:48.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:47 smithi045 ceph-mon[32174]: pgmap v244: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.4 MiB/s wr, 1.07k op/s 2024-03-12T05:18:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:48 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:48 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:48 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:48 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:18:48.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:48 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:48.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:48 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:48.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:48 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:48.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:48 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:18:49.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:48 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:49.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:48 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:49.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:48 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:49.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:48 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:18:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 405.0 s, 1326.6 tps, lat 75.258 ms stddev 22.527 2024-03-12T05:18:49.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:49 smithi130 ceph-mon[35398]: pgmap v245: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:18:49.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:49 smithi116 ceph-mon[35486]: pgmap v245: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:18:50.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:49 smithi045 ceph-mon[32174]: pgmap v245: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-03-12T05:18:51.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:51 smithi130 ceph-mon[35398]: pgmap v246: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:51.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:51 smithi116 ceph-mon[35486]: pgmap v246: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:52.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:51 smithi045 ceph-mon[32174]: pgmap v246: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:18:53.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:53 smithi130 ceph-mon[35398]: pgmap v247: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.5 MiB/s wr, 1.08k op/s 2024-03-12T05:18:53.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:53 smithi116 ceph-mon[35486]: pgmap v247: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.5 MiB/s wr, 1.08k op/s 2024-03-12T05:18:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:53 smithi045 ceph-mon[32174]: pgmap v247: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 9.5 MiB/s wr, 1.08k op/s 2024-03-12T05:18:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 410.0 s, 1314.2 tps, lat 76.103 ms stddev 27.202 2024-03-12T05:18:55.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:55 smithi130 ceph-mon[35398]: pgmap v248: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:18:55.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:55 smithi116 ceph-mon[35486]: pgmap v248: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:18:56.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:55 smithi045 ceph-mon[32174]: pgmap v248: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:18:57.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:57 smithi130 ceph-mon[35398]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.3 MiB/s wr, 1.07k op/s 2024-03-12T05:18:57.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:57 smithi116 ceph-mon[35486]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.3 MiB/s wr, 1.07k op/s 2024-03-12T05:18:58.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:57 smithi045 ceph-mon[32174]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 9.3 MiB/s wr, 1.07k op/s 2024-03-12T05:18:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 415.0 s, 1415.4 tps, lat 70.769 ms stddev 20.850 2024-03-12T05:18:59.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:18:59 smithi130 ceph-mon[35398]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:18:59.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:18:59 smithi116 ceph-mon[35486]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:19:00.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:18:59 smithi045 ceph-mon[32174]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:19:01.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:01 smithi130 ceph-mon[35398]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:19:01.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:01 smithi116 ceph-mon[35486]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:19:02.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:01 smithi045 ceph-mon[32174]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:19:03.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:19:03.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:19:03.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:03 smithi130 ceph-mon[35398]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:19:03.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:03 smithi116 ceph-mon[35486]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:19:04.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:03 smithi045 ceph-mon[32174]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:19:04.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 -- 172.21.15.45:0/4254161359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c101370 msgr2=0x7f017c101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 --2- 172.21.15.45:0/4254161359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c101370 0x7f017c101750 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f017800fb90 tx=0x7f017802fcf0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 -- 172.21.15.45:0/4254161359 shutdown_connections 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 --2- 172.21.15.45:0/4254161359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c106440 0x7f017c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 --2- 172.21.15.45:0/4254161359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101d20 0x7f017c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 --2- 172.21.15.45:0/4254161359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c101370 0x7f017c101750 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 -- 172.21.15.45:0/4254161359 >> 172.21.15.45:0/4254161359 conn(0x7f017c0fcc30 msgr2=0x7f017c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 -- 172.21.15.45:0/4254161359 shutdown_connections 2024-03-12T05:19:04.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.491+0000 7f0183927640 1 -- 172.21.15.45:0/4254161359 wait complete. 2024-03-12T05:19:04.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 Processor -- start 2024-03-12T05:19:04.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 -- start start 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 0x7f017c072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c06bf90 0x7f017c06c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0731b0 con 0x7f017c06bf90 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0704f0 con 0x7f017c106440 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0183927640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0706b0 con 0x7f017c101370 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.492+0000 7f0180ec2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40880/0 (socket says 172.21.15.45:40880) 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 -- 172.21.15.45:0/2153444171 learned_addr learned my addr 172.21.15.45:0/2153444171 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f01816c3640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 0x7f017c072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c06bf90 0x7f017c06c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 -- 172.21.15.45:0/2153444171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 msgr2=0x7f017c072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 0x7f017c072960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 -- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c06bf90 msgr2=0x7f017c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c06bf90 0x7f017c06c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 -- 172.21.15.45:0/2153444171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174009900 con 0x7f017c106440 2024-03-12T05:19:04.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f01816c3640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 0x7f017c072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:04.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.493+0000 7f0180ec2640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e9f0 tx=0x7f016c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.511+0000 7f01727fc640 1 -- 172.21.15.45:0/2153444171 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0096c0 con 0x7f017c106440 2024-03-12T05:19:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.511+0000 7f01727fc640 1 -- 172.21.15.45:0/2153444171 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f016c0054d0 con 0x7f017c106440 2024-03-12T05:19:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.511+0000 7f01727fc640 1 -- 172.21.15.45:0/2153444171 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c010430 con 0x7f017c106440 2024-03-12T05:19:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.511+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017800f790 con 0x7f017c106440 2024-03-12T05:19:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.512+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c070c60 con 0x7f017c106440 2024-03-12T05:19:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.518+0000 7f01727fc640 1 -- 172.21.15.45:0/2153444171 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f016c005020 con 0x7f017c106440 2024-03-12T05:19:04.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.523+0000 7f01727fc640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 0x7f01680a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.523+0000 7f01727fc640 1 -- 172.21.15.45:0/2153444171 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f016c014070 con 0x7f017c106440 2024-03-12T05:19:04.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.523+0000 7f01816c3640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 0x7f01680a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.524+0000 7f01816c3640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 0x7f01680a4230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0178000c00 tx=0x7f0178004320 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.557+0000 7f01865d0740 1 Processor -- start 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.557+0000 7f01865d0740 1 -- start start 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c141e3e0 0x5650c1426a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c1426fc0 0x5650c14293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c1384280 con 0x5650c1426fc0 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c13817b0 con 0x5650c141dac0 2024-03-12T05:19:04.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01865d0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c137cf90 con 0x5650c141e3e0 2024-03-12T05:19:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01816c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01816c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40892/0 (socket says 172.21.15.45:40892) 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f01816c3640 1 -- 172.21.15.45:0/2976092643 learned_addr learned my addr 172.21.15.45:0/2976092643 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f0180ec2640 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c141e3e0 0x5650c1426a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.558+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c1426fc0 0x5650c14293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 -- 172.21.15.45:0/2976092643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c141e3e0 msgr2=0x5650c1426a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c141e3e0 0x5650c1426a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 -- 172.21.15.45:0/2976092643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c1426fc0 msgr2=0x5650c14293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c1426fc0 0x5650c14293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 -- 172.21.15.45:0/2976092643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650c133b570 con 0x5650c141dac0 2024-03-12T05:19:04.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01816c3640 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f017803a000 tx=0x7f0178041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21805dfc5f33c481 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.559+0000 7f01677fe640 1 -- 172.21.15.45:0/2976092643 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178005250 con 0x5650c141dac0 2024-03-12T05:19:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.560+0000 7f01677fe640 1 -- 172.21.15.45:0/2976092643 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0178038a60 con 0x5650c141dac0 2024-03-12T05:19:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.560+0000 7f01677fe640 1 -- 172.21.15.45:0/2976092643 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178004ac0 con 0x5650c141dac0 2024-03-12T05:19:04.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.560+0000 7f01865d0740 1 -- 172.21.15.45:0/2976092643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 msgr2=0x5650c141dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.560+0000 7f01865d0740 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f017803a000 tx=0x7f0178041000 comp rx=0 tx=0).stop 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 -- 172.21.15.45:0/2976092643 shutdown_connections 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c1426fc0 0x5650c14293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c141e3e0 0x5650c1426a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 --2- 172.21.15.45:0/2976092643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c141dea0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 -- 172.21.15.45:0/2976092643 >> 172.21.15.45:0/2976092643 conn(0x5650c14181a0 msgr2=0x5650c1418580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 -- 172.21.15.45:0/2976092643 shutdown_connections 2024-03-12T05:19:04.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.561+0000 7f01865d0740 1 -- 172.21.15.45:0/2976092643 wait complete. 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.563+0000 7f01865d0740 1 -- start start 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.563+0000 7f01865d0740 1 Processor -- start 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c1426a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c1423db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c1050430 con 0x5650c141e3e0 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c100ce00 con 0x5650c141dac0 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f01865d0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c1301fd0 con 0x5650c1426fc0 2024-03-12T05:19:04.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f0181ec4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f0181ec4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:55978/0 (socket says 172.21.15.45:55978) 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f0181ec4640 1 -- 172.21.15.45:0/2805897510 learned_addr learned my addr 172.21.15.45:0/2805897510 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.564+0000 7f0180ec2640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c1423db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f01865d0740 10 client.0 _ll_register_callbacks cb 0x5650c1357fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f01816c3640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c1426a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 -- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 msgr2=0x5650c1426a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c1426a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 -- 172.21.15.45:0/2805897510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 msgr2=0x5650c1423db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c1423db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 -- 172.21.15.45:0/2805897510 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c10e820 con 0x5650c1426fc0 2024-03-12T05:19:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f017400ee90 tx=0x7f017400d7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5fa1f1a983b5b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0164ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0164ff9640 1 -- 172.21.15.45:0/2805897510 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174013040 con 0x5650c1426fc0 2024-03-12T05:19:04.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.565+0000 7f0164ff9640 1 -- 172.21.15.45:0/2805897510 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0174004b20 con 0x5650c1426fc0 2024-03-12T05:19:04.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.572+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650c107c7e0 con 0x5650c1426fc0 2024-03-12T05:19:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.573+0000 7f0164ff9640 1 -- 172.21.15.45:0/2805897510 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0174004ce0 con 0x5650c1426fc0 2024-03-12T05:19:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.573+0000 7f01865d0740 10 client.27010 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:04.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.573+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5650c137b610 con 0x5650c1426fc0 2024-03-12T05:19:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f0164ff9640 1 -- 172.21.15.45:0/2805897510 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0174016070 con 0x5650c1426fc0 2024-03-12T05:19:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 10 client.27010 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 10 client.27010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 20 client.27010 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c1436f80 0x5650c14393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 4 client.27010 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650c10b9720 con 0x5650c1436f80 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.574+0000 7f01816c3640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c1436f80 0x5650c14393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.575+0000 7f01816c3640 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c1436f80 0x5650c14393e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2100b5f8e7a6c1 server_cookie=3816fb16f922167b in_seq=0 out_seq=0 2024-03-12T05:19:04.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.575+0000 7f0164ff9640 10 client.27010 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.575+0000 7f0164ff9640 1 -- 172.21.15.45:0/2805897510 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5650c10b9720 con 0x5650c1436f80 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.575+0000 7f0164ff9640 10 client.27010 handle_command_reply: tid=0 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 client.27010 shutdown 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 msgr2=0x5650c14244d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f017400ee90 tx=0x7f017400d7b0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 shutdown_connections 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c1436f80 0x5650c14393e0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c1423db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c14244d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 --2- 172.21.15.45:0/2805897510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c1426a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f0164ff9640 0 client.27010 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:04.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 >> 172.21.15.45:0/2805897510 conn(0x5650c14181a0 msgr2=0x5650c1419f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 shutdown_connections 2024-03-12T05:19:04.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 1 -- 172.21.15.45:0/2805897510 wait complete. 2024-03-12T05:19:04.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.576+0000 7f01865d0740 20 client.27010 trim_cache size 0 max 16384 2024-03-12T05:19:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 Processor -- start 2024-03-12T05:19:04.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 -- start start 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141e3e0 0x5650c1426a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c1421010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c141ce40 con 0x5650c141dac0 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c141cbb0 con 0x5650c141e3e0 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.606+0000 7f01865d0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c1418f80 con 0x5650c1426fc0 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f01816c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f01816c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33790/0 (socket says 172.21.15.45:33790) 2024-03-12T05:19:04.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f01816c3640 1 -- 172.21.15.45:0/2244010661 learned_addr learned my addr 172.21.15.45:0/2244010661 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c1421010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 -- 172.21.15.45:0/2244010661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141e3e0 msgr2=0x5650c1426a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:19:04.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141e3e0 0x5650c1426a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 -- 172.21.15.45:0/2244010661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 msgr2=0x5650c14264e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f0181ec4640 1 -- 172.21.15.45:0/2244010661 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650c1317ac0 con 0x5650c1426fc0 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.607+0000 7f01816c3640 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.608+0000 7f0181ec4640 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c1421010 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0174009d40 tx=0x7f0174016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da8573f074eb05a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.608+0000 7f0166ffd640 1 -- 172.21.15.45:0/2244010661 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01740174b0 con 0x5650c1426fc0 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.608+0000 7f0166ffd640 1 -- 172.21.15.45:0/2244010661 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0174024030 con 0x5650c1426fc0 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.608+0000 7f0166ffd640 1 -- 172.21.15.45:0/2244010661 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174017e00 con 0x5650c1426fc0 2024-03-12T05:19:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.609+0000 7f01865d0740 1 -- 172.21.15.45:0/2244010661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 msgr2=0x5650c1421010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.609+0000 7f01865d0740 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c1421010 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0174009d40 tx=0x7f0174016000 comp rx=0 tx=0).stop 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 -- 172.21.15.45:0/2244010661 shutdown_connections 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c1421010 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141e3e0 0x5650c1426a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 --2- 172.21.15.45:0/2244010661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141dac0 0x5650c14264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 -- 172.21.15.45:0/2244010661 >> 172.21.15.45:0/2244010661 conn(0x5650c14181a0 msgr2=0x5650c141f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 -- 172.21.15.45:0/2244010661 shutdown_connections 2024-03-12T05:19:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.610+0000 7f01865d0740 1 -- 172.21.15.45:0/2244010661 wait complete. 2024-03-12T05:19:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.611+0000 7f01865d0740 1 -- start start 2024-03-12T05:19:04.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.611+0000 7f01865d0740 1 Processor -- start 2024-03-12T05:19:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c142db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c142e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c142b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c142b710 con 0x5650c141e3e0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c142b880 con 0x5650c141dac0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c142ba40 con 0x5650c1426fc0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01865d0740 10 client.0 _ll_register_callbacks cb 0x7f017803b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f0180ec2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c142e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f0181ec4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c142b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01816c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c142db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f0181ec4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c142b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56008/0 (socket says 172.21.15.45:56008) 2024-03-12T05:19:04.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f0181ec4640 1 -- 172.21.15.45:0/4204646844 learned_addr learned my addr 172.21.15.45:0/4204646844 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.612+0000 7f01816c3640 1 -- 172.21.15.45:0/4204646844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 msgr2=0x5650c142b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01816c3640 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c142b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01816c3640 1 -- 172.21.15.45:0/4204646844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 msgr2=0x5650c142e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01816c3640 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c142e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01816c3640 1 -- 172.21.15.45:0/4204646844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650c133b570 con 0x5650c141dac0 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01816c3640 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c142db90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f01780042f0 tx=0x7f01780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c799359698b8de3b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f0164ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017805e030 con 0x5650c141dac0 2024-03-12T05:19:04.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f017805f070 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.613+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0178038a60 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0178030690 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 10 client.27032 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5650c137b610 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0178030ca0 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0178003b20 con 0x5650c141dac0 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 10 client.27032 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 10 client.27032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 20 client.27032 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c148f310 0x5650c1491730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 4 client.27032 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.614+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f0178038a60 con 0x5650c148f310 2024-03-12T05:19:04.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.615+0000 7f0180ec2640 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c148f310 0x5650c1491730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.615+0000 7f0164ff9640 10 client.27032 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.615+0000 7f0180ec2640 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c148f310 0x5650c1491730 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86d966553a37696f server_cookie=e4b8b03c1f956217 in_seq=0 out_seq=0 2024-03-12T05:19:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.616+0000 7f0164ff9640 1 -- 172.21.15.45:0/4204646844 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f0178038a60 con 0x5650c148f310 2024-03-12T05:19:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.616+0000 7f0164ff9640 10 client.27032 handle_command_reply: tid=0 2024-03-12T05:19:04.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.616+0000 7f01865d0740 1 client.27032 shutdown 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 msgr2=0x5650c142db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c142db90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f01780042f0 tx=0x7f01780046c0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 shutdown_connections 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5650c148f310 0x5650c1491730 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5650c141e3e0 0x5650c142e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f0164ff9640 0 client.27032 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650c1426fc0 0x5650c142b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.617+0000 7f01865d0740 1 --2- 172.21.15.45:0/4204646844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650c141dac0 0x5650c142db90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.618+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 >> 172.21.15.45:0/4204646844 conn(0x5650c14181a0 msgr2=0x5650c1422680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.618+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 shutdown_connections 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.618+0000 7f01865d0740 1 -- 172.21.15.45:0/4204646844 wait complete. 2024-03-12T05:19:04.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.618+0000 7f01865d0740 20 client.27032 trim_cache size 0 max 16384 2024-03-12T05:19:04.615 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:04.615 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:19:04.615 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:19:04.615 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.619+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 msgr2=0x7f01680a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.619+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 0x7f01680a4230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0178000c00 tx=0x7f0178004320 comp rx=0 tx=0).stop 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.619+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 msgr2=0x7f017c06ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.619+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e9f0 tx=0x7f016c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 shutdown_connections 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01680a1d70 0x7f01680a4230 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f017c06bf90 0x7f017c06c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f017c106440 0x7f017c06ba50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 --2- 172.21.15.45:0/2153444171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f017c101370 0x7f017c072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 >> 172.21.15.45:0/2153444171 conn(0x7f017c0fcc30 msgr2=0x7f017c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 shutdown_connections 2024-03-12T05:19:04.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.620+0000 7f0183927640 1 -- 172.21.15.45:0/2153444171 wait complete. 2024-03-12T05:19:04.655 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:19:04.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:19:04.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 -- 172.21.15.45:0/1348123550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f810cb80 msgr2=0x7f87f810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/1348123550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f810cb80 0x7f87f810cf60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009370 tx=0x7f87e802ce60 comp rx=0 tx=0).stop 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 -- 172.21.15.45:0/1348123550 shutdown_connections 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/1348123550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f8114a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/1348123550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f8108180 0x7f87f8108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/1348123550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f810cb80 0x7f87f810cf60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.835+0000 7f87fd9e2640 1 -- 172.21.15.45:0/1348123550 >> 172.21.15.45:0/1348123550 conn(0x7f87f806c6d0 msgr2=0x7f87f806cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.836+0000 7f87fd9e2640 1 -- 172.21.15.45:0/1348123550 shutdown_connections 2024-03-12T05:19:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.836+0000 7f87fd9e2640 1 -- 172.21.15.45:0/1348123550 wait complete. 2024-03-12T05:19:04.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 Processor -- start 2024-03-12T05:19:04.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 -- start start 2024-03-12T05:19:04.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f8108180 0x7f87f81a3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f810cb80 0x7f87f819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f819e090 con 0x7f87f8108180 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f819e200 con 0x7f87f810cb80 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.837+0000 7f87fd9e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f819e3c0 con 0x7f87f8108b40 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56014/0 (socket says 172.21.15.45:56014) 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 -- 172.21.15.45:0/3962647098 learned_addr learned my addr 172.21.15.45:0/3962647098 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f6ffd640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f8108180 0x7f87f81a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f77fe640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f810cb80 0x7f87f819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 -- 172.21.15.45:0/3962647098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f810cb80 msgr2=0x7f87f819da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f810cb80 0x7f87f819da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 -- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f8108180 msgr2=0x7f87f81a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f8108180 0x7f87f81a3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.838+0000 7f87f67fc640 1 -- 172.21.15.45:0/3962647098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0009840 con 0x7f87f8108b40 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87f67fc640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009f20 tx=0x7f87ec00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00bd70 con 0x7f87f8108b40 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e8009df0 con 0x7f87f8108b40 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f819e890 con 0x7f87f8108b40 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f87ec015980 con 0x7f87f8108b40 2024-03-12T05:19:04.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.839+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec01d9a0 con 0x7f87f8108b40 2024-03-12T05:19:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.841+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e4005290 con 0x7f87f8108b40 2024-03-12T05:19:04.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.842+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f87ec01db00 con 0x7f87f8108b40 2024-03-12T05:19:04.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.843+0000 7f87fc9e0640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 0x7f87e00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:04.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.843+0000 7f87f6ffd640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 0x7f87e00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:04.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.843+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f87ec0d19a0 con 0x7f87f8108b40 2024-03-12T05:19:04.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.844+0000 7f87f6ffd640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 0x7f87e00a40e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f87e8006010 tx=0x7f87e8000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:04.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:04.849+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f87ec09a840 con 0x7f87f8108b40 2024-03-12T05:19:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 420.0 s, 1258.0 tps, lat 79.319 ms stddev 37.831 2024-03-12T05:19:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.265+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e4005810 con 0x7f87f8108b40 2024-03-12T05:19:05.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.265+0000 7f87fc9e0640 1 -- 172.21.15.45:0/3962647098 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f87ec09df00 con 0x7f87f8108b40 2024-03-12T05:19:05.262 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:19:05.262 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:19:05.262 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 msgr2=0x7f87e00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 0x7f87e00a40e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f87e8006010 tx=0x7f87e8000bc0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 msgr2=0x7f87f81a3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009f20 tx=0x7f87ec00ef90 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 shutdown_connections 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f87e00a1c20 0x7f87e00a40e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87f810cb80 0x7f87f819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87f8108b40 0x7f87f81a3d80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 --2- 172.21.15.45:0/3962647098 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87f8108180 0x7f87f81a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.269+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 >> 172.21.15.45:0/3962647098 conn(0x7f87f806c6d0 msgr2=0x7f87f8070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.270+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 shutdown_connections 2024-03-12T05:19:05.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.270+0000 7f87fd9e2640 1 -- 172.21.15.45:0/3962647098 wait complete. 2024-03-12T05:19:05.335 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:05.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 -- 172.21.15.45:0/2356400430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 msgr2=0x7f88a406cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/2356400430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a406cec0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f889c009be0 tx=0x7f889c02bf80 comp rx=0 tx=0).stop 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 -- 172.21.15.45:0/2356400430 shutdown_connections 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/2356400430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a406d400 0x7f88a411a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/2356400430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a411a6c0 0x7f88a411cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/2356400430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a406cec0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 -- 172.21.15.45:0/2356400430 >> 172.21.15.45:0/2356400430 conn(0x7f88a406bfb0 msgr2=0x7f88a410a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 -- 172.21.15.45:0/2356400430 shutdown_connections 2024-03-12T05:19:05.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.519+0000 7f88ab3a0640 1 -- 172.21.15.45:0/2356400430 wait complete. 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.520+0000 7f88ab3a0640 1 Processor -- start 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.520+0000 7f88ab3a0640 1 -- start start 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.520+0000 7f88ab3a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a41a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.520+0000 7f88ab3a0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.520+0000 7f88ab3a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a411a6c0 0x7f88a41aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88ab3a0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41ab240 con 0x7f88a411a6c0 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88ab3a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41ab3b0 con 0x7f88a406cae0 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88ab3a0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41ab570 con 0x7f88a406d400 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56024/0 (socket says 172.21.15.45:56024) 2024-03-12T05:19:05.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aab9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a411a6c0 0x7f88a41aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88a9b9d640 1 -- 172.21.15.45:0/3858136940 learned_addr learned my addr 172.21.15.45:0/3858136940 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a41a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 -- 172.21.15.45:0/3858136940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 msgr2=0x7f88a41a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 -- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a411a6c0 msgr2=0x7f88a41aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a411a6c0 0x7f88a41aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88aa39e640 1 -- 172.21.15.45:0/3858136940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c009840 con 0x7f88a406cae0 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.521+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:05.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f88aa39e640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a41a63d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f889c02c590 tx=0x7f889c009330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f889b7fe640 1 -- 172.21.15.45:0/3858136940 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c004680 con 0x7f88a406cae0 2024-03-12T05:19:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f889b7fe640 1 -- 172.21.15.45:0/3858136940 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f889c0047e0 con 0x7f88a406cae0 2024-03-12T05:19:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f889b7fe640 1 -- 172.21.15.45:0/3858136940 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c03e640 con 0x7f88a406cae0 2024-03-12T05:19:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a41ab7f0 con 0x7f88a406cae0 2024-03-12T05:19:05.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.522+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a41abaa0 con 0x7f88a406cae0 2024-03-12T05:19:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.526+0000 7f889b7fe640 1 -- 172.21.15.45:0/3858136940 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f889c034b00 con 0x7f88a406cae0 2024-03-12T05:19:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.526+0000 7f88af1d5740 1 Processor -- start 2024-03-12T05:19:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.527+0000 7f889b7fe640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 0x7f88900a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.527+0000 7f889b7fe640 1 -- 172.21.15.45:0/3858136940 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f889c0f01e0 con 0x7f88a406cae0 2024-03-12T05:19:05.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88aab9f640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 0x7f88900a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 -- start start 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0440 0x5561099e0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55610999b960 con 0x5561099e0d60 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55610994f310 con 0x5561099e9940 2024-03-12T05:19:05.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.528+0000 7f88af1d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55610994c840 con 0x5561099e0440 2024-03-12T05:19:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33842/0 (socket says 172.21.15.45:33842) 2024-03-12T05:19:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1839141104 learned_addr learned my addr 172.21.15.45:0/1839141104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88aab9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0440 0x5561099e0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88aab9f640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 0x7f88900a40e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8894009b50 tx=0x7f8894005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1839141104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0440 msgr2=0x5561099e0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0440 0x5561099e0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1839141104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 msgr2=0x5561099ebd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099ebd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1839141104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55610990a060 con 0x5561099e0d60 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f88a00115f0 tx=0x7f88a0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bc44c875a02b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.529+0000 7f8898bfe640 1 -- 172.21.15.45:0/1839141104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0004030 con 0x5561099e0d60 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.530+0000 7f8898bfe640 1 -- 172.21.15.45:0/1839141104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88a000f640 con 0x5561099e0d60 2024-03-12T05:19:05.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.530+0000 7f8898bfe640 1 -- 172.21.15.45:0/1839141104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0003850 con 0x5561099e0d60 2024-03-12T05:19:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 -- 172.21.15.45:0/1839141104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 msgr2=0x5561099e9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f88a00115f0 tx=0x7f88a0009f90 comp rx=0 tx=0).stop 2024-03-12T05:19:05.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 -- 172.21.15.45:0/1839141104 shutdown_connections 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0d60 0x5561099e9400 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099ebd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1839141104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0440 0x5561099e0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.531+0000 7f88af1d5740 1 -- 172.21.15.45:0/1839141104 >> 172.21.15.45:0/1839141104 conn(0x5561099dab70 msgr2=0x5561099daf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.532+0000 7f88af1d5740 1 -- 172.21.15.45:0/1839141104 shutdown_connections 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.532+0000 7f88af1d5740 1 -- 172.21.15.45:0/1839141104 wait complete. 2024-03-12T05:19:05.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.532+0000 7f88af1d5740 1 -- start start 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 Processor -- start 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x5561099e7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556109948020 con 0x5561099e0440 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55610961b430 con 0x5561099e0d60 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561095d7e00 con 0x5561099e9940 2024-03-12T05:19:05.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88af1d5740 10 client.0 _ll_register_callbacks cb 0x5561098efcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33846/0 (socket says 172.21.15.45:33846) 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88aa39e640 1 -- 172.21.15.45:0/1395007997 learned_addr learned my addr 172.21.15.45:0/1395007997 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.533+0000 7f88aab9f640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x5561099e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 -- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 msgr2=0x5561099e7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x5561099e7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 -- 172.21.15.45:0/1395007997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 msgr2=0x5561099e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099e6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 -- 172.21.15.45:0/1395007997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0009d70 con 0x5561099e0440 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aab9f640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x5561099e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:05.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f889c0bcb90 tx=0x7f889c0bc8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490da7861aad5044 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f888dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0d5dc0 con 0x5561099e0440 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561096477e0 con 0x5561099e0440 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.534+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f889c0c5630 con 0x5561099e0440 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f889c0db340 con 0x5561099e0440 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 10 client.17571 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:05.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5561099466a0 con 0x5561099e0440 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f889c09f070 con 0x5561099e0440 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f889c03c070 con 0x5561099e0440 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 10 client.17571 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 10 client.17571 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 20 client.17571 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5561099f9bd0 0x5561099fbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 4 client.17571 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.535+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556109684720 con 0x5561099f9bd0 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.536+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5561099f9bd0 0x5561099fbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.536+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5561099f9bd0 0x5561099fbfc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=113123529d47c0a2 server_cookie=d1b08325f025c290 in_seq=0 out_seq=0 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.536+0000 7f888dffb640 10 client.17571 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.537+0000 7f888dffb640 1 -- 172.21.15.45:0/1395007997 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556109684720 con 0x5561099f9bd0 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.537+0000 7f888dffb640 10 client.17571 handle_command_reply: tid=0 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.537+0000 7f88af1d5740 1 client.17571 shutdown 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.537+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 msgr2=0x5561099e69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.537+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f889c0bcb90 tx=0x7f889c0bc8e0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 shutdown_connections 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5561099f9bd0 0x5561099fbfc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x5561099e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1395007997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e69f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f888dffb640 0 client.17571 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:05.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 >> 172.21.15.45:0/1395007997 conn(0x5561099dab70 msgr2=0x5561099dc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 shutdown_connections 2024-03-12T05:19:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 1 -- 172.21.15.45:0/1395007997 wait complete. 2024-03-12T05:19:05.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.538+0000 7f88af1d5740 20 client.17571 trim_cache size 0 max 16384 2024-03-12T05:19:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 Processor -- start 2024-03-12T05:19:05.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 -- start start 2024-03-12T05:19:05.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561098cd130 con 0x5561099e0440 2024-03-12T05:19:05.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561099d6550 con 0x5561099e9940 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.558+0000 7f88af1d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561099dbb70 con 0x5561099e0d60 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33862/0 (socket says 172.21.15.45:33862) 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56054/0 (socket says 172.21.15.45:56054) 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88aa39e640 1 -- 172.21.15.45:0/1584982711 learned_addr learned my addr 172.21.15.45:0/1584982711 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88aab9f640 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1584982711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 msgr2=0x5561099e41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099e41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1584982711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 msgr2=0x5561099e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 -- 172.21.15.45:0/1584982711 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561098e2c20 con 0x5561099e0d60 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.559+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f88a001c040 tx=0x7f88a0005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe9be5035f758e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.560+0000 7f888ffff640 1 -- 172.21.15.45:0/1584982711 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000fd60 con 0x5561099e0d60 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.560+0000 7f888ffff640 1 -- 172.21.15.45:0/1584982711 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88a0005d10 con 0x5561099e0d60 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.560+0000 7f888ffff640 1 -- 172.21.15.45:0/1584982711 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0003a10 con 0x5561099e0d60 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 -- 172.21.15.45:0/1584982711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 msgr2=0x5561099e3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f88a001c040 tx=0x7f88a0005040 comp rx=0 tx=0).stop 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 -- 172.21.15.45:0/1584982711 shutdown_connections 2024-03-12T05:19:05.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e9940 0x5561099e41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e0d60 0x5561099e3c70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1584982711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 -- 172.21.15.45:0/1584982711 >> 172.21.15.45:0/1584982711 conn(0x5561099dab70 msgr2=0x5561099e52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 -- 172.21.15.45:0/1584982711 shutdown_connections 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.561+0000 7f88af1d5740 1 -- 172.21.15.45:0/1584982711 wait complete. 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88af1d5740 1 -- start start 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88af1d5740 1 Processor -- start 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88af1d5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88aa39e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33868/0 (socket says 172.21.15.45:33868) 2024-03-12T05:19:05.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88aa39e640 1 -- 172.21.15.45:0/1194322897 learned_addr learned my addr 172.21.15.45:0/1194322897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.562+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x556109a4a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556109a4a6c0 con 0x5561099e0440 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556109a4a830 con 0x5561099e0d60 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556109a4a9f0 con 0x5561099e9940 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88af1d5740 10 client.0 _ll_register_callbacks cb 0x7f88a0009b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 -- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 msgr2=0x556109a4a180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x556109a4a180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 -- 172.21.15.45:0/1194322897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 msgr2=0x5561099f00b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099f00b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55610990a060 con 0x5561099e0440 2024-03-12T05:19:05.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88a000f640 con 0x5561099e0440 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.563+0000 7f88aa39e640 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f889c006010 tx=0x7f889c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4afe599bd59f9a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f888dffb640 10 client.17577 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f888dffb640 1 -- 172.21.15.45:0/1194322897 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c0bcc60 con 0x5561099e0440 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f888dffb640 1 -- 172.21.15.45:0/1194322897 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f889c0c5630 con 0x5561099e0440 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f888dffb640 1 -- 172.21.15.45:0/1194322897 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f889c0bc9d0 con 0x5561099e0440 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f88af1d5740 10 client.17577 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:05.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.564+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5561099088c0 con 0x5561099e0440 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f888dffb640 1 -- 172.21.15.45:0/1194322897 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f889c03c040 con 0x5561099e0440 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 10 client.17577 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 10 client.17577 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 20 client.17577 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556109a51bf0 0x556109a53fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 4 client.17577 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5561096d3890 con 0x556109a51bf0 2024-03-12T05:19:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.565+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556109a51bf0 0x556109a53fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.566+0000 7f88a9b9d640 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556109a51bf0 0x556109a53fe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ccb58d5613d4d67 server_cookie=58ee5b55ada4caae in_seq=0 out_seq=0 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.566+0000 7f888dffb640 10 client.17577 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.566+0000 7f888dffb640 1 -- 172.21.15.45:0/1194322897 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5561096d3890 con 0x556109a51bf0 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.566+0000 7f888dffb640 10 client.17577 handle_command_reply: tid=0 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.566+0000 7f88af1d5740 1 client.17577 shutdown 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 msgr2=0x5561099efb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f889c006010 tx=0x7f889c02bf80 comp rx=0 tx=0).stop 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 shutdown_connections 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x556109a51bf0 0x556109a53fe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f888dffb640 0 client.17577 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561099e9940 0x556109a4a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561099e0d60 0x5561099f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 --2- 172.21.15.45:0/1194322897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5561099e0440 0x5561099efb70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 >> 172.21.15.45:0/1194322897 conn(0x5561099dab70 msgr2=0x5561099d9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.567+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 shutdown_connections 2024-03-12T05:19:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.568+0000 7f88af1d5740 1 -- 172.21.15.45:0/1194322897 wait complete. 2024-03-12T05:19:05.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.568+0000 7f88af1d5740 20 client.17577 trim_cache size 0 max 16384 2024-03-12T05:19:05.567 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.569+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 msgr2=0x7f88900a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.569+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 0x7f88900a40e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8894009b50 tx=0x7f8894005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.569+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 msgr2=0x7f88a41a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.569+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a41a63d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f889c02c590 tx=0x7f889c009330 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 shutdown_connections 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88900a1c20 0x7f88900a40e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88a411a6c0 0x7f88a41aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f88a406d400 0x7f88a41a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 --2- 172.21.15.45:0/3858136940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88a406cae0 0x7f88a41a63d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 >> 172.21.15.45:0/3858136940 conn(0x7f88a406bfb0 msgr2=0x7f88a410a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 shutdown_connections 2024-03-12T05:19:05.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.570+0000 7f88ab3a0640 1 -- 172.21.15.45:0/3858136940 wait complete. 2024-03-12T05:19:05.584 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:05.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:19:05.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.779+0000 7f206188d640 1 -- 172.21.15.45:0/945294680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c108840 msgr2=0x7f205c108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.779+0000 7f206188d640 1 --2- 172.21.15.45:0/945294680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c108840 0x7f205c108c20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2054005fd0 tx=0x7f205402cc30 comp rx=0 tx=0).stop 2024-03-12T05:19:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.780+0000 7f206188d640 1 -- 172.21.15.45:0/945294680 shutdown_connections 2024-03-12T05:19:05.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.780+0000 7f206188d640 1 --2- 172.21.15.45:0/945294680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c10cfd0 0x7f205c079490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.780+0000 7f206188d640 1 --2- 172.21.15.45:0/945294680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c10c440 0x7f205c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.780+0000 7f206188d640 1 --2- 172.21.15.45:0/945294680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c108840 0x7f205c108c20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.780+0000 7f206188d640 1 -- 172.21.15.45:0/945294680 >> 172.21.15.45:0/945294680 conn(0x7f205c06c260 msgr2=0x7f205c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.781+0000 7f206188d640 1 -- 172.21.15.45:0/945294680 shutdown_connections 2024-03-12T05:19:05.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.781+0000 7f206188d640 1 -- 172.21.15.45:0/945294680 wait complete. 2024-03-12T05:19:05.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.782+0000 7f206188d640 1 Processor -- start 2024-03-12T05:19:05.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.782+0000 7f206188d640 1 -- start start 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c10c440 0x7f205c0732f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c073830 0x7f205c073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c079840 con 0x7f205c10c440 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1bb310 con 0x7f205c073830 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f206188d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1bb4d0 con 0x7f205c108840 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c073830 0x7f205c073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56082/0 (socket says 172.21.15.45:56082) 2024-03-12T05:19:05.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 -- 172.21.15.45:0/1441505479 learned_addr learned my addr 172.21.15.45:0/1441505479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c10c440 0x7f205c0732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 -- 172.21.15.45:0/1441505479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c073830 msgr2=0x7f205c073cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c073830 0x7f205c073cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 -- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c10c440 msgr2=0x7f205c0732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c10c440 0x7f205c0732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.783+0000 7f205affd640 1 -- 172.21.15.45:0/1441505479 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054007490 con 0x7f205c108840 2024-03-12T05:19:05.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f205affd640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2054000f80 tx=0x7f205400aaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f206088b640 1 -- 172.21.15.45:0/1441505479 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205402d5d0 con 0x7f205c108840 2024-03-12T05:19:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f206088b640 1 -- 172.21.15.45:0/1441505479 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f205402da00 con 0x7f205c108840 2024-03-12T05:19:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f206088b640 1 -- 172.21.15.45:0/1441505479 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054008710 con 0x7f205c108840 2024-03-12T05:19:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205c1bb750 con 0x7f205c108840 2024-03-12T05:19:05.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.784+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c1bbc10 con 0x7f205c108840 2024-03-12T05:19:05.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.787+0000 7f206088b640 1 -- 172.21.15.45:0/1441505479 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2054008870 con 0x7f205c108840 2024-03-12T05:19:05.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.788+0000 7f206088b640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 0x7f20440a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.788+0000 7f206088b640 1 -- 172.21.15.45:0/1441505479 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f20540eeab0 con 0x7f205c108840 2024-03-12T05:19:05.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.788+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 0x7f20440a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.790+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 0x7f20440a40e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f205c0743c0 tx=0x7f2050005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.790+0000 7f206445a740 1 Processor -- start 2024-03-12T05:19:05.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.791+0000 7f206445a740 1 -- start start 2024-03-12T05:19:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.791+0000 7f206445a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.791+0000 7f206445a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.791+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.791+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56096/0 (socket says 172.21.15.45:56096) 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f205a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f205a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33890/0 (socket says 172.21.15.45:33890) 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f205affd640 1 -- 172.21.15.45:0/293449913 learned_addr learned my addr 172.21.15.45:0/293449913 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f206445a740 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f822186960 con 0x55f8221cbb70 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f82213a310 con 0x55f8221d4750 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f205b7fe640 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.792+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f822137840 con 0x55f8221cb250 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.793+0000 7f205a7fc640 1 -- 172.21.15.45:0/293449913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 msgr2=0x55f8221cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.793+0000 7f205a7fc640 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.793+0000 7f205a7fc640 1 -- 172.21.15.45:0/293449913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 msgr2=0x55f8221d6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.793+0000 7f205a7fc640 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.793+0000 7f205a7fc640 1 -- 172.21.15.45:0/293449913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8220f5060 con 0x55f8221cbb70 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.794+0000 7f205a7fc640 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f20500118d0 tx=0x7f2050014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de4a53b44a6ad05 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.794+0000 7f2041ffb640 1 -- 172.21.15.45:0/293449913 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205000ee80 con 0x55f8221cbb70 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.794+0000 7f2041ffb640 1 -- 172.21.15.45:0/293449913 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2050021070 con 0x55f8221cbb70 2024-03-12T05:19:05.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.794+0000 7f2041ffb640 1 -- 172.21.15.45:0/293449913 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205001c410 con 0x55f8221cbb70 2024-03-12T05:19:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.795+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 msgr2=0x55f8221d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.795+0000 7f206445a740 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f20500118d0 tx=0x7f2050014000 comp rx=0 tx=0).stop 2024-03-12T05:19:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 shutdown_connections 2024-03-12T05:19:05.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d4210 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 --2- 172.21.15.45:0/293449913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 >> 172.21.15.45:0/293449913 conn(0x55f8221c5980 msgr2=0x55f8221c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 shutdown_connections 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.796+0000 7f206445a740 1 -- 172.21.15.45:0/293449913 wait complete. 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.797+0000 7f206445a740 1 -- start start 2024-03-12T05:19:05.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.797+0000 7f206445a740 1 Processor -- start 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f205affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56102/0 (socket says 172.21.15.45:56102) 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f205affd640 1 -- 172.21.15.45:0/1035076295 learned_addr learned my addr 172.21.15.45:0/1035076295 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f205b7fe640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f822133020 con 0x55f8221cbb70 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f821e06430 con 0x55f8221d4750 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f821dc2e00 con 0x55f8221cb250 2024-03-12T05:19:05.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.798+0000 7f206445a740 10 client.0 _ll_register_callbacks cb 0x55f8220dacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:05.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 -- 172.21.15.45:0/1035076295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 msgr2=0x55f8221d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 -- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 msgr2=0x55f8221d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2050017070 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205affd640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d1d60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2050009a90 tx=0x7f20500035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d85db87683957c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.799+0000 7f203affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050004000 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2050004160 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f821e327e0 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2050004320 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f206445a740 10 client.17589 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f8221316a0 con 0x55f8221cbb70 2024-03-12T05:19:05.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f821e327e0 con 0x55f8221cbb70 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f205001d530 con 0x55f8221cbb70 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f206445a740 10 client.17589 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.800+0000 7f206445a740 10 client.17589 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.801+0000 7f206445a740 20 client.17589 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.801+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f8221e4840 0x55f8221e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.801+0000 7f206445a740 4 client.17589 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.801+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f821e6f720 con 0x55f8221e4840 2024-03-12T05:19:05.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.801+0000 7f205affd640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f8221e4840 0x55f8221e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.802+0000 7f203affd640 10 client.17589 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:05.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.802+0000 7f205affd640 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f8221e4840 0x55f8221e6cc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d2f5d73448337b8f server_cookie=5fa77b52da8687b5 in_seq=0 out_seq=0 2024-03-12T05:19:05.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.803+0000 7f203affd640 1 -- 172.21.15.45:0/1035076295 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f821e6f720 con 0x55f8221e4840 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.803+0000 7f203affd640 10 client.17589 handle_command_reply: tid=0 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.803+0000 7f206445a740 1 client.17589 shutdown 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.803+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 msgr2=0x55f8221d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.803+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d1d60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2050009a90 tx=0x7f20500035f0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 shutdown_connections 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f8221e4840 0x55f8221e6cc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d1d60 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f203affd640 0 client.17589 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221d4750 0x55f8221d22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 --2- 172.21.15.45:0/1035076295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221cb250 0x55f8221d1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.804+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 >> 172.21.15.45:0/1035076295 conn(0x55f8221c5980 msgr2=0x55f8221c76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.805+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 shutdown_connections 2024-03-12T05:19:05.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.805+0000 7f206445a740 1 -- 172.21.15.45:0/1035076295 wait complete. 2024-03-12T05:19:05.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.805+0000 7f206445a740 20 client.17589 trim_cache size 0 max 16384 2024-03-12T05:19:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 Processor -- start 2024-03-12T05:19:05.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 -- start start 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cb250 0x55f8221d3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8220b8130 con 0x55f8221cbb70 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8221e45c0 con 0x55f8221cb250 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.828+0000 7f206445a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8221caeb0 con 0x55f8221d4750 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33906/0 (socket says 172.21.15.45:33906) 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205a7fc640 1 -- 172.21.15.45:0/2390364680 learned_addr learned my addr 172.21.15.45:0/2390364680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205affd640 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cb250 0x55f8221d3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 -- 172.21.15.45:0/2390364680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cb250 msgr2=0x55f8221d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cb250 0x55f8221d3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 -- 172.21.15.45:0/2390364680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 msgr2=0x55f8221d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 -- 172.21.15.45:0/2390364680 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8220cdc20 con 0x55f8221d4750 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205a7fc640 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.829+0000 7f205b7fe640 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221ceaa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f204c009ca0 tx=0x7f204c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5f439191200fb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.830+0000 7f20417fa640 1 -- 172.21.15.45:0/2390364680 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c017050 con 0x55f8221d4750 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.830+0000 7f20417fa640 1 -- 172.21.15.45:0/2390364680 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f204c018040 con 0x55f8221d4750 2024-03-12T05:19:05.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.830+0000 7f20417fa640 1 -- 172.21.15.45:0/2390364680 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c003c90 con 0x55f8221d4750 2024-03-12T05:19:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 -- 172.21.15.45:0/2390364680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 msgr2=0x55f8221ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221ceaa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f204c009ca0 tx=0x7f204c005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 -- 172.21.15.45:0/2390364680 shutdown_connections 2024-03-12T05:19:05.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cbb70 0x55f8221d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221ceaa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 --2- 172.21.15.45:0/2390364680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cb250 0x55f8221d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 -- 172.21.15.45:0/2390364680 >> 172.21.15.45:0/2390364680 conn(0x55f8221c5980 msgr2=0x55f8221eb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 -- 172.21.15.45:0/2390364680 shutdown_connections 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.831+0000 7f206445a740 1 -- 172.21.15.45:0/2390364680 wait complete. 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.832+0000 7f206445a740 1 -- start start 2024-03-12T05:19:05.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.832+0000 7f206445a740 1 Processor -- start 2024-03-12T05:19:05.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.832+0000 7f206445a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cb250 0x55f8221da0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.832+0000 7f206445a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cbb70 0x55f8221da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f206445a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f206445a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f82223adf0 con 0x55f8221cb250 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f206445a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f82223af30 con 0x55f8221cbb70 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f206445a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f82223b0f0 con 0x55f8221d4750 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56128/0 (socket says 172.21.15.45:56128) 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 -- 172.21.15.45:0/1468485525 learned_addr learned my addr 172.21.15.45:0/1468485525 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205a7fc640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cbb70 0x55f8221da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f206445a740 10 client.0 _ll_register_callbacks cb 0x7f20500099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205affd640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cb250 0x55f8221da0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 -- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cbb70 msgr2=0x55f8221da5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cbb70 0x55f8221da5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 -- 172.21.15.45:0/1468485525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cb250 msgr2=0x55f8221da0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cb250 0x55f8221da0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.833+0000 7f205b7fe640 1 -- 172.21.15.45:0/1468485525 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8221316a0 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f205b7fe640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f204c005bc0 tx=0x7f204c01dac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ad756d673d0d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f203affd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2050021070 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f203affd640 1 -- 172.21.15.45:0/1468485525 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c012110 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f203affd640 1 -- 172.21.15.45:0/1468485525 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f204c01dc30 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f203affd640 1 -- 172.21.15.45:0/1468485525 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f204c012270 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f206445a740 10 client.27055 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.834+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x7f204c009980 con 0x55f8221d4750 2024-03-12T05:19:05.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f203affd640 1 -- 172.21.15.45:0/1468485525 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f204c005030 con 0x55f8221d4750 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 10 client.27055 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 10 client.27055 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 20 client.27055 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f82223c590 0x55f82223e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 4 client.27055 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f821ebe890 con 0x55f82223c590 2024-03-12T05:19:05.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.835+0000 7f205affd640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f82223c590 0x55f82223e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.836+0000 7f205affd640 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f82223c590 0x55f82223e9b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c87fab7d49abe748 server_cookie=ee7184e803d6e9c8 in_seq=0 out_seq=0 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.836+0000 7f203affd640 10 client.27055 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.836+0000 7f203affd640 1 -- 172.21.15.45:0/1468485525 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f821ebe890 con 0x55f82223c590 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.836+0000 7f203affd640 10 client.27055 handle_command_reply: tid=0 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.836+0000 7f206445a740 1 client.27055 shutdown 2024-03-12T05:19:05.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 msgr2=0x55f8221dab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f204c005bc0 tx=0x7f204c01dac0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 shutdown_connections 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55f82223c590 0x55f82223e9b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8221d4750 0x55f8221dab20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f8221cbb70 0x55f8221da5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 --2- 172.21.15.45:0/1468485525 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f8221cb250 0x55f8221da0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 >> 172.21.15.45:0/1468485525 conn(0x55f8221c5980 msgr2=0x55f8221c42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f203affd640 0 client.27055 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 shutdown_connections 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.837+0000 7f206445a740 1 -- 172.21.15.45:0/1468485525 wait complete. 2024-03-12T05:19:05.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.838+0000 7f206445a740 20 client.27055 trim_cache size 0 max 16384 2024-03-12T05:19:05.836 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 msgr2=0x7f20440a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 0x7f20440a40e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f205c0743c0 tx=0x7f2050005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 msgr2=0x7f205c079090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2054000f80 tx=0x7f205400aaf0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 shutdown_connections 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f20440a1c20 0x7f20440a40e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f205c10c440 0x7f205c0732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f205c073830 0x7f205c073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 --2- 172.21.15.45:0/1441505479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c108840 0x7f205c079090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.839+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 >> 172.21.15.45:0/1441505479 conn(0x7f205c06c260 msgr2=0x7f205c082210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.840+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 shutdown_connections 2024-03-12T05:19:05.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:05.840+0000 7f206188d640 1 -- 172.21.15.45:0/1441505479 wait complete. 2024-03-12T05:19:05.855 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:05.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:19:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:05 smithi045 ceph-mon[32174]: pgmap v253: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:05.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:05 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3962647098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:05.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:05 smithi130 ceph-mon[35398]: pgmap v253: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:05.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:05 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3962647098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:05 smithi116 ceph-mon[35486]: pgmap v253: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:05 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3962647098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.035+0000 7eff745f6640 1 -- 172.21.15.45:0/534348180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c10d9d0 msgr2=0x7eff6c10de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.035+0000 7eff745f6640 1 --2- 172.21.15.45:0/534348180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c10d9d0 0x7eff6c10de50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7eff68009900 tx=0x7eff6802bc30 comp rx=0 tx=0).stop 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.037+0000 7eff745f6640 1 -- 172.21.15.45:0/534348180 shutdown_connections 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 --2- 172.21.15.45:0/534348180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c11a960 0x7eff6c11cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 --2- 172.21.15.45:0/534348180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c10d9d0 0x7eff6c10de50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 --2- 172.21.15.45:0/534348180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 -- 172.21.15.45:0/534348180 >> 172.21.15.45:0/534348180 conn(0x7eff6c06c260 msgr2=0x7eff6c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 -- 172.21.15.45:0/534348180 shutdown_connections 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 -- 172.21.15.45:0/534348180 wait complete. 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.038+0000 7eff745f6640 1 Processor -- start 2024-03-12T05:19:06.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 -- start start 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 0x7eff6c1a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c1a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 0x7eff6c1aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1ab1d0 con 0x7eff6c10f9a0 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1ab340 con 0x7eff6c10d9d0 2024-03-12T05:19:06.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff745f6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1ab500 con 0x7eff6c11a960 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff72392640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 0x7eff6c1a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff71b91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c1a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff72b93640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 0x7eff6c1aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff72b93640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 0x7eff6c1aabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56144/0 (socket says 172.21.15.45:56144) 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff71b91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c1a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33942/0 (socket says 172.21.15.45:33942) 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.039+0000 7eff72b93640 1 -- 172.21.15.45:0/433832432 learned_addr learned my addr 172.21.15.45:0/433832432 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 -- 172.21.15.45:0/433832432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 msgr2=0x7eff6c1aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 0x7eff6c1aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 -- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 msgr2=0x7eff6c1a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c1a6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 -- 172.21.15.45:0/433832432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68009560 con 0x7eff6c10d9d0 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.040+0000 7eff72392640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 0x7eff6c1a6310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9f0 tx=0x7eff5c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.041+0000 7eff637fe640 1 -- 172.21.15.45:0/433832432 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00ccc0 con 0x7eff6c10d9d0 2024-03-12T05:19:06.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.041+0000 7eff637fe640 1 -- 172.21.15.45:0/433832432 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff5c00ce20 con 0x7eff6c10d9d0 2024-03-12T05:19:06.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.041+0000 7eff637fe640 1 -- 172.21.15.45:0/433832432 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0187a0 con 0x7eff6c10d9d0 2024-03-12T05:19:06.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.041+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c1ab7e0 con 0x7eff6c10d9d0 2024-03-12T05:19:06.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.041+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c1b2e80 con 0x7eff6c10d9d0 2024-03-12T05:19:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.045+0000 7eff637fe640 1 -- 172.21.15.45:0/433832432 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7eff5c010430 con 0x7eff6c10d9d0 2024-03-12T05:19:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.046+0000 7eff637fe640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 0x7eff580a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.046+0000 7eff637fe640 1 -- 172.21.15.45:0/433832432 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7eff5c014070 con 0x7eff6c10d9d0 2024-03-12T05:19:06.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.046+0000 7eff771c5740 1 Processor -- start 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.047+0000 7eff71b91640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 0x7eff580a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.047+0000 7eff771c5740 1 -- start start 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.047+0000 7eff771c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a98e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff771c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 0x55728a997210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff771c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a997750 0x55728a999b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff771c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a949960 con 0x55728a997750 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff771c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a8fd310 con 0x55728a98eb70 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff771c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a8fa840 con 0x55728a98e250 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff71b91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 0x55728a997210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a98e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff71b91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 0x55728a997210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41050/0 (socket says 172.21.15.45:41050) 2024-03-12T05:19:06.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff71b91640 1 -- 172.21.15.45:0/874846391 learned_addr learned my addr 172.21.15.45:0/874846391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72b93640 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a997750 0x55728a999b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 -- 172.21.15.45:0/874846391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 msgr2=0x55728a997210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 0x55728a997210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 -- 172.21.15.45:0/874846391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a997750 msgr2=0x55728a999b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a997750 0x55728a999b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff72392640 1 -- 172.21.15.45:0/874846391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55728a8b8060 con 0x55728a98e250 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.048+0000 7eff71b91640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 0x7eff580a41a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7eff68009900 tx=0x7eff6802c770 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.049+0000 7eff72392640 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a98e630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0cb1c0 tx=0x7eff5c09a650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42571efe420c3a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.049+0000 7eff60bfe640 1 -- 172.21.15.45:0/874846391 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0a2e40 con 0x55728a98e250 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.049+0000 7eff60bfe640 1 -- 172.21.15.45:0/874846391 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff5c010b00 con 0x55728a98e250 2024-03-12T05:19:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.049+0000 7eff60bfe640 1 -- 172.21.15.45:0/874846391 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c09ac10 con 0x55728a98e250 2024-03-12T05:19:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 -- 172.21.15.45:0/874846391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 msgr2=0x55728a98e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a98e630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0cb1c0 tx=0x7eff5c09a650 comp rx=0 tx=0).stop 2024-03-12T05:19:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 -- 172.21.15.45:0/874846391 shutdown_connections 2024-03-12T05:19:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a997750 0x55728a999b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98eb70 0x55728a997210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 --2- 172.21.15.45:0/874846391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a98e630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 -- 172.21.15.45:0/874846391 >> 172.21.15.45:0/874846391 conn(0x55728a988980 msgr2=0x55728a988d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 -- 172.21.15.45:0/874846391 shutdown_connections 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.050+0000 7eff771c5740 1 -- 172.21.15.45:0/874846391 wait complete. 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.051+0000 7eff771c5740 1 -- start start 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.051+0000 7eff771c5740 1 Processor -- start 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.051+0000 7eff771c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 0x55728a994800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a997750 0x55728a995280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff71b91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff71b91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33946/0 (socket says 172.21.15.45:33946) 2024-03-12T05:19:06.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff71b91640 1 -- 172.21.15.45:0/3980317486 learned_addr learned my addr 172.21.15.45:0/3980317486 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a8f6020 con 0x55728a98eb70 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a5c9430 con 0x55728a98e250 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a585e00 con 0x55728a997750 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff771c5740 10 client.0 _ll_register_callbacks cb 0x55728a89dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.052+0000 7eff72392640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 0x55728a994800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff72b93640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a997750 0x55728a995280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 -- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a997750 msgr2=0x55728a995280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a997750 0x55728a995280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 -- 172.21.15.45:0/3980317486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 msgr2=0x55728a994800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 0x55728a994800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c1b2e80 con 0x55728a98eb70 2024-03-12T05:19:06.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff72392640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 0x55728a994800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff71b91640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7eff68005ec0 tx=0x7eff6802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d31536c8e00062 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.053+0000 7eff55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.054+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff680039b0 con 0x55728a98eb70 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.054+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7eff6802de40 con 0x55728a98eb70 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.054+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55728a5f57e0 con 0x55728a98eb70 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff68003b10 con 0x55728a98eb70 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 10 client.17607 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55728a8f46a0 con 0x55728a98eb70 2024-03-12T05:19:06.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55728a5f57e0 con 0x55728a98eb70 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff68042470 con 0x55728a98eb70 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 10 client.17607 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 10 client.17607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 20 client.17607 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9a7bf0 0x55728a9aa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 4 client.17607 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.055+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55728a632720 con 0x55728a9a7bf0 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.056+0000 7eff72392640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9a7bf0 0x55728a9aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.056+0000 7eff72392640 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9a7bf0 0x55728a9aa010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9fcdd05f03218992 server_cookie=af1e15ac83cc4a2e in_seq=0 out_seq=0 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.056+0000 7eff55ffb640 10 client.17607 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff55ffb640 1 -- 172.21.15.45:0/3980317486 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55728a632720 con 0x55728a9a7bf0 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff55ffb640 10 client.17607 handle_command_reply: tid=0 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 client.17607 shutdown 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 msgr2=0x55728a994d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7eff68005ec0 tx=0x7eff6802bc30 comp rx=0 tx=0).stop 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 shutdown_connections 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9a7bf0 0x55728a9aa010 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a994d40 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a997750 0x55728a995280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 --2- 172.21.15.45:0/3980317486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a98e250 0x55728a994800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.057+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 >> 172.21.15.45:0/3980317486 conn(0x55728a988980 msgr2=0x55728a98a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.058+0000 7eff55ffb640 0 client.17607 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.058+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 shutdown_connections 2024-03-12T05:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.058+0000 7eff771c5740 1 -- 172.21.15.45:0/3980317486 wait complete. 2024-03-12T05:19:06.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.058+0000 7eff771c5740 20 client.17607 trim_cache size 0 max 16384 2024-03-12T05:19:06.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.080+0000 7eff771c5740 1 Processor -- start 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.080+0000 7eff771c5740 1 -- start start 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a98fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a990070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a87b130 con 0x55728a98eb70 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff71b91640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a98fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff72392640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff72392640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56164/0 (socket says 172.21.15.45:56164) 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff72392640 1 -- 172.21.15.45:0/3012301254 learned_addr learned my addr 172.21.15.45:0/3012301254 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff72b93640 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a990070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a632720 con 0x55728a997750 2024-03-12T05:19:06.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.081+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a5f57e0 con 0x55728a98e250 2024-03-12T05:19:06.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 -- 172.21.15.45:0/3012301254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 msgr2=0x55728a990070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a990070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 -- 172.21.15.45:0/3012301254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 msgr2=0x55728a98fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a98fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 -- 172.21.15.45:0/3012301254 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55728a890c20 con 0x55728a98e250 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff71b91640 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a98fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff72392640 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c6a0 tx=0x7eff5c009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=358eafb67f29e87d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.082+0000 7eff57fff640 1 -- 172.21.15.45:0/3012301254 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0b3190 con 0x55728a98e250 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.083+0000 7eff57fff640 1 -- 172.21.15.45:0/3012301254 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7eff5c0a2e40 con 0x55728a98e250 2024-03-12T05:19:06.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.083+0000 7eff57fff640 1 -- 172.21.15.45:0/3012301254 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0a29b0 con 0x55728a98e250 2024-03-12T05:19:06.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 msgr2=0x55728a996f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00c6a0 tx=0x7eff5c009700 comp rx=0 tx=0).stop 2024-03-12T05:19:06.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 shutdown_connections 2024-03-12T05:19:06.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98eb70 0x55728a98fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a990070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 --2- 172.21.15.45:0/3012301254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98e250 0x55728a996f30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 >> 172.21.15.45:0/3012301254 conn(0x55728a988980 msgr2=0x55728a983a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 shutdown_connections 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.084+0000 7eff771c5740 1 -- 172.21.15.45:0/3012301254 wait complete. 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.085+0000 7eff771c5740 1 -- start start 2024-03-12T05:19:06.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.085+0000 7eff771c5740 1 Processor -- start 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.085+0000 7eff771c5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 0x55728a99c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.085+0000 7eff771c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 0x55728a99cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.085+0000 7eff771c5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a9f7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff771c5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a9f76b0 con 0x55728a98e250 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff771c5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a9f7820 con 0x55728a997750 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff771c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55728a9f79e0 con 0x55728a98eb70 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff71b91640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 0x55728a99cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff71b91640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 0x55728a99cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56166/0 (socket says 172.21.15.45:56166) 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff71b91640 1 -- 172.21.15.45:0/2813969505 learned_addr learned my addr 172.21.15.45:0/2813969505 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72392640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 0x55728a99c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a9f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff771c5740 10 client.0 _ll_register_callbacks cb 0x7eff5c096390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 -- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 msgr2=0x55728a99cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 0x55728a99cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 -- 172.21.15.45:0/2813969505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 msgr2=0x55728a99c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 0x55728a99c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.086+0000 7eff72b93640 1 -- 172.21.15.45:0/2813969505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55728a8f46a0 con 0x55728a997750 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.087+0000 7eff72392640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 0x55728a99c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.087+0000 7eff72b93640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a9f7170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55728a98d3c0 tx=0x7eff6400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64138abc6da5fd3d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.087+0000 7eff55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.088+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64004b20 con 0x55728a997750 2024-03-12T05:19:06.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.088+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7eff64004c80 con 0x55728a997750 2024-03-12T05:19:06.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.088+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55728a994630 con 0x55728a997750 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.089+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff64004e40 con 0x55728a997750 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.089+0000 7eff771c5740 10 client.27086 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.089+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55728a8b8060 con 0x55728a997750 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.089+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55728a994630 con 0x55728a997750 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff6401b070 con 0x55728a997750 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 10 client.27086 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 10 client.27086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 20 client.27086 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9ff330 0x55728aa01750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 4 client.27086 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:06.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7eff5c010b00 con 0x55728a9ff330 2024-03-12T05:19:06.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.090+0000 7eff72392640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9ff330 0x55728aa01750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.091+0000 7eff55ffb640 10 client.27086 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.091+0000 7eff72392640 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9ff330 0x55728aa01750 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5229f005362a2c2 server_cookie=dc9052571bde0687 in_seq=0 out_seq=0 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.091+0000 7eff55ffb640 1 -- 172.21.15.45:0/2813969505 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7eff5c010b00 con 0x55728a9ff330 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.091+0000 7eff55ffb640 10 client.27086 handle_command_reply: tid=0 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.092+0000 7eff771c5740 1 client.27086 shutdown 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.092+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 msgr2=0x55728a9f7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.092+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a9f7170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55728a98d3c0 tx=0x7eff6400f790 comp rx=0 tx=0).stop 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 shutdown_connections 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55728a9ff330 0x55728aa01750 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55728a997750 0x55728a9f7170 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55728a98eb70 0x55728a99cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 --2- 172.21.15.45:0/2813969505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55728a98e250 0x55728a99c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 >> 172.21.15.45:0/2813969505 conn(0x55728a988980 msgr2=0x55728a98fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.093+0000 7eff55ffb640 0 client.27086 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.095+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 shutdown_connections 2024-03-12T05:19:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.095+0000 7eff771c5740 1 -- 172.21.15.45:0/2813969505 wait complete. 2024-03-12T05:19:06.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.096+0000 7eff771c5740 20 client.27086 trim_cache size 0 max 16384 2024-03-12T05:19:06.091 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.097+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 msgr2=0x7eff580a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.097+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 0x7eff580a41a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7eff68009900 tx=0x7eff6802c770 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.097+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 msgr2=0x7eff6c1a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.097+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 0x7eff6c1a6310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9f0 tx=0x7eff5c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 shutdown_connections 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7eff580a1ce0 0x7eff580a41a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff6c10f9a0 0x7eff6c1a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff6c11a960 0x7eff6c1aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 --2- 172.21.15.45:0/433832432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff6c10d9d0 0x7eff6c1a6310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.098+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 >> 172.21.15.45:0/433832432 conn(0x7eff6c06c260 msgr2=0x7eff6c11b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.102+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 shutdown_connections 2024-03-12T05:19:06.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.102+0000 7eff745f6640 1 -- 172.21.15.45:0/433832432 wait complete. 2024-03-12T05:19:06.118 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:06.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:19:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 -- 172.21.15.45:0/4246602748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da410d9d0 msgr2=0x7f5da410de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 --2- 172.21.15.45:0/4246602748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da410d9d0 0x7f5da410de50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009900 tx=0x7f5d9402bc60 comp rx=0 tx=0).stop 2024-03-12T05:19:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 -- 172.21.15.45:0/4246602748 shutdown_connections 2024-03-12T05:19:06.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 --2- 172.21.15.45:0/4246602748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da411a960 0x7f5da411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 --2- 172.21.15.45:0/4246602748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da410d9d0 0x7f5da410de50 secure :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009900 tx=0x7f5d9402bc60 comp rx=0 tx=0).stop 2024-03-12T05:19:06.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 --2- 172.21.15.45:0/4246602748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da410f9a0 0x7f5da410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 -- 172.21.15.45:0/4246602748 >> 172.21.15.45:0/4246602748 conn(0x7f5da406c260 msgr2=0x7f5da406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 -- 172.21.15.45:0/4246602748 shutdown_connections 2024-03-12T05:19:06.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.349+0000 7f5da935e640 1 -- 172.21.15.45:0/4246602748 wait complete. 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.350+0000 7f5da935e640 1 Processor -- start 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.350+0000 7f5da935e640 1 -- start start 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.350+0000 7f5da935e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da410f9a0 0x7f5da41ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.350+0000 7f5da935e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 0x7f5da41ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da935e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 0x7f5da41a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5d9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 0x7f5da41ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5d9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 0x7f5da41ac9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41104/0 (socket says 172.21.15.45:41104) 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5d9bfff640 1 -- 172.21.15.45:0/661032567 learned_addr learned my addr 172.21.15.45:0/661032567 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da935e640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41a7000 con 0x7f5da410f9a0 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da935e640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41a7170 con 0x7f5da411a960 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da935e640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41a7330 con 0x7f5da41a6610 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 0x7f5da41a6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 -- 172.21.15.45:0/661032567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 msgr2=0x7f5da41ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 0x7f5da41ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 -- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da410f9a0 msgr2=0x7f5da41ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da410f9a0 0x7f5da41ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.351+0000 7f5da37fe640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94009560 con 0x7f5da41a6610 2024-03-12T05:19:06.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.352+0000 7f5da37fe640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 0x7f5da41a6ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00be20 tx=0x7f5d9c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.353+0000 7f5da0ff9640 1 -- 172.21.15.45:0/661032567 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00ce70 con 0x7f5da41a6610 2024-03-12T05:19:06.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.353+0000 7f5da0ff9640 1 -- 172.21.15.45:0/661032567 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d9c0148f0 con 0x7f5da41a6610 2024-03-12T05:19:06.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.353+0000 7f5da0ff9640 1 -- 172.21.15.45:0/661032567 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c012a20 con 0x7f5da41a6610 2024-03-12T05:19:06.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.355+0000 7f5da935e640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41b3320 con 0x7f5da41a6610 2024-03-12T05:19:06.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.355+0000 7f5da935e640 1 -- 172.21.15.45:0/661032567 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41b37b0 con 0x7f5da41a6610 2024-03-12T05:19:06.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.358+0000 7f5da0ff9640 1 -- 172.21.15.45:0/661032567 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5d9c014a60 con 0x7f5da41a6610 2024-03-12T05:19:06.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.358+0000 7f5dac00d740 1 Processor -- start 2024-03-12T05:19:06.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.358+0000 7f5dac00d740 1 -- start start 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145ed530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41459f960 con 0x55b4145eda70 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b414553310 con 0x55b4145e4ca0 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.359+0000 7f5dac00d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b414550840 con 0x55b4145e4380 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34000/0 (socket says 172.21.15.45:34000) 2024-03-12T05:19:06.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 -- 172.21.15.45:0/1915103996 learned_addr learned my addr 172.21.15.45:0/1915103996 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 -- 172.21.15.45:0/1915103996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 msgr2=0x55b4145e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 -- 172.21.15.45:0/1915103996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 msgr2=0x55b4145ed530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145ed530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 -- 172.21.15.45:0/1915103996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b41450e060 con 0x55b4145eda70 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.360+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009230 tx=0x7f5d9402c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c77f1745b3b2a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.361+0000 7f5da0ff9640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 0x7f5d900a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.361+0000 7f5da935e640 1 -- 172.21.15.45:0/1915103996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9402ca90 con 0x55b4145eda70 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.361+0000 7f5da935e640 1 -- 172.21.15.45:0/1915103996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d940044f0 con 0x55b4145eda70 2024-03-12T05:19:06.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.361+0000 7f5da935e640 1 -- 172.21.15.45:0/1915103996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94003de0 con 0x55b4145eda70 2024-03-12T05:19:06.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.361+0000 7f5da0ff9640 1 -- 172.21.15.45:0/661032567 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5d9c0cce60 con 0x7f5da41a6610 2024-03-12T05:19:06.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 0x7f5d900a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 -- 172.21.15.45:0/1915103996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 msgr2=0x55b4145efe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009230 tx=0x7f5d9402c6e0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 -- 172.21.15.45:0/1915103996 shutdown_connections 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145efe60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145ed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1915103996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 -- 172.21.15.45:0/1915103996 >> 172.21.15.45:0/1915103996 conn(0x55b4145deca0 msgr2=0x55b4145df080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.362+0000 7f5dac00d740 1 -- 172.21.15.45:0/1915103996 shutdown_connections 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.363+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 0x7f5d900a4120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009460 tx=0x7f5d8c0093b0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.363+0000 7f5dac00d740 1 -- 172.21.15.45:0/1915103996 wait complete. 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.363+0000 7f5dac00d740 1 -- start start 2024-03-12T05:19:06.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.363+0000 7f5dac00d740 1 Processor -- start 2024-03-12T05:19:06.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41454c020 con 0x55b4145eda70 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41421f430 con 0x55b4145e4ca0 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4141dbe00 con 0x55b4145e4380 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5dac00d740 10 client.0 _ll_register_callbacks cb 0x55b4144f3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56202/0 (socket says 172.21.15.45:56202) 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 -- 172.21.15.45:0/1385853601 learned_addr learned my addr 172.21.15.45:0/1385853601 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da37fe640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 -- 172.21.15.45:0/1385853601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 msgr2=0x55b4145eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 -- 172.21.15.45:0/1385853601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 msgr2=0x55b4145eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 -- 172.21.15.45:0/1385853601 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9402de70 con 0x55b4145e4380 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.364+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c010f70 tx=0x7f5d8c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d55b58456c243d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5d8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c01f860 con 0x55b4145e4380 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5d8c01f9c0 con 0x55b4145e4380 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b41424b7e0 con 0x55b4145e4380 2024-03-12T05:19:06.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d8c01fb80 con 0x55b4145e4380 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.365+0000 7f5dac00d740 10 client.27079 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b41454a6a0 con 0x55b4145e4380 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b41424b7e0 con 0x55b4145e4380 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5d8c011d20 con 0x55b4145e4380 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 10 client.27079 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 10 client.27079 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:19:06.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 20 client.27079 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b4145fdc40 0x55b414600060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 4 client.27079 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.366+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b414288720 con 0x55b4145fdc40 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.367+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b4145fdc40 0x55b414600060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.367+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b4145fdc40 0x55b414600060 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7095d8d1b9459cc9 server_cookie=58d134f8ba961cae in_seq=0 out_seq=0 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.367+0000 7f5d8a7fc640 10 client.27079 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/1385853601 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b414288720 con 0x55b4145fdc40 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5d8a7fc640 10 client.27079 handle_command_reply: tid=0 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 client.27079 shutdown 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 msgr2=0x55b4145eaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c010f70 tx=0x7f5d8c005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 shutdown_connections 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b4145fdc40 0x55b414600060 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4ca0 0x55b4145eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 --2- 172.21.15.45:0/1385853601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4380 0x55b4145eaa70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 >> 172.21.15.45:0/1385853601 conn(0x55b4145deca0 msgr2=0x55b4145e09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.368+0000 7f5d8a7fc640 0 client.27079 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.369+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 shutdown_connections 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.369+0000 7f5dac00d740 1 -- 172.21.15.45:0/1385853601 wait complete. 2024-03-12T05:19:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.369+0000 7f5dac00d740 20 client.27079 trim_cache size 0 max 16384 2024-03-12T05:19:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.391+0000 7f5dac00d740 1 Processor -- start 2024-03-12T05:19:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.392+0000 7f5dac00d740 1 -- start start 2024-03-12T05:19:06.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.392+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5da2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34012/0 (socket says 172.21.15.45:34012) 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5da2ffd640 1 -- 172.21.15.45:0/2502076341 learned_addr learned my addr 172.21.15.45:0/2502076341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145eb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 0x55b4145e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.393+0000 7f5da37fe640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 0x55b4145e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4144d1130 con 0x55b4145e4380 2024-03-12T05:19:06.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b414288720 con 0x55b4145eda70 2024-03-12T05:19:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b41424b7e0 con 0x55b4145e4ca0 2024-03-12T05:19:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da2ffd640 1 -- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 msgr2=0x55b4145eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145eb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da2ffd640 1 -- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 msgr2=0x55b4145e7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 0x55b4145e7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da2ffd640 1 -- 172.21.15.45:0/2502076341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4144e6c20 con 0x55b4145e4380 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.394+0000 7f5da37fe640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 0x55b4145e7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.395+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0058b0 tx=0x7f5d8c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc5c69323c9a5da server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.395+0000 7f5d98ff9640 1 -- 172.21.15.45:0/2502076341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c010b50 con 0x55b4145e4380 2024-03-12T05:19:06.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.396+0000 7f5d98ff9640 1 -- 172.21.15.45:0/2502076341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d8c027750 con 0x55b4145e4380 2024-03-12T05:19:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.396+0000 7f5d98ff9640 1 -- 172.21.15.45:0/2502076341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c01fdf0 con 0x55b4145e4380 2024-03-12T05:19:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 msgr2=0x55b4145ed1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0058b0 tx=0x7f5d8c0058e0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 shutdown_connections 2024-03-12T05:19:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145eda70 0x55b4145e7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.397+0000 7f5dac00d740 1 --2- 172.21.15.45:0/2502076341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145e4380 0x55b4145ed1a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.398+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 >> 172.21.15.45:0/2502076341 conn(0x55b4145deca0 msgr2=0x55b4145e5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.398+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 shutdown_connections 2024-03-12T05:19:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.398+0000 7f5dac00d740 1 -- 172.21.15.45:0/2502076341 wait complete. 2024-03-12T05:19:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 -- start start 2024-03-12T05:19:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 Processor -- start 2024-03-12T05:19:06.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4380 0x55b4145f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4145f3d70 con 0x55b4145eda70 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b414654150 con 0x55b4145e4380 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5dac00d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b414654290 con 0x55b4145e4ca0 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5da37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5da37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34020/0 (socket says 172.21.15.45:34020) 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.399+0000 7f5da37fe640 1 -- 172.21.15.45:0/3287923865 learned_addr learned my addr 172.21.15.45:0/3287923865 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5dac00d740 10 client.0 _ll_register_callbacks cb 0x7f5d94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4380 0x55b4145f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 -- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 msgr2=0x55b4145f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 -- 172.21.15.45:0/3287923865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4380 msgr2=0x55b4145f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4380 0x55b4145f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 -- 172.21.15.45:0/3287923865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41b37b0 con 0x55b4145eda70 2024-03-12T05:19:06.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5d9bfff640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:06.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5da37fe640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c099920 tx=0x7f5d9c099ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfccb08d2f8998d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.400+0000 7f5d8a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:06.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.401+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c0b1070 con 0x55b4145eda70 2024-03-12T05:19:06.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.401+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5d9c096290 con 0x55b4145eda70 2024-03-12T05:19:06.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.401+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d940044f0 con 0x55b4145eda70 2024-03-12T05:19:06.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.402+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d9c09aea0 con 0x55b4145eda70 2024-03-12T05:19:06.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.402+0000 7f5dac00d740 10 client.17628 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.402+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b41454a6a0 con 0x55b4145eda70 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5d9c07c080 con 0x55b4145eda70 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5d9c008030 con 0x55b4145eda70 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 10 client.17628 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 10 client.17628 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 20 client.17628 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b414655a60 0x55b414657e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 4 client.17628 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f5d940044f0 con 0x55b414655a60 2024-03-12T05:19:06.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.403+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b414655a60 0x55b414657e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.404+0000 7f5da2ffd640 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b414655a60 0x55b414657e80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b10f46b4ff9bd221 server_cookie=1b77e7884c2424c9 in_seq=0 out_seq=0 2024-03-12T05:19:06.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.404+0000 7f5d8a7fc640 10 client.17628 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:06.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.405+0000 7f5d8a7fc640 1 -- 172.21.15.45:0/3287923865 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f5d940044f0 con 0x55b414655a60 2024-03-12T05:19:06.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.405+0000 7f5d8a7fc640 10 client.17628 handle_command_reply: tid=0 2024-03-12T05:19:06.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.405+0000 7f5dac00d740 1 client.17628 shutdown 2024-03-12T05:19:06.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.405+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 msgr2=0x55b4145f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.405+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c099920 tx=0x7f5d9c099ce0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 shutdown_connections 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b4145eda70 0x55b4145f3670 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55b414655a60 0x55b414657e80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b4145e4ca0 0x55b4145f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 --2- 172.21.15.45:0/3287923865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4145e4380 0x55b4145f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5d8a7fc640 0 client.17628 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 >> 172.21.15.45:0/3287923865 conn(0x55b4145deca0 msgr2=0x55b4145e00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 shutdown_connections 2024-03-12T05:19:06.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.406+0000 7f5dac00d740 1 -- 172.21.15.45:0/3287923865 wait complete. 2024-03-12T05:19:06.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.407+0000 7f5dac00d740 20 client.17628 trim_cache size 0 max 16384 2024-03-12T05:19:06.403 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 msgr2=0x7f5d900a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 0x7f5d900a4120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009460 tx=0x7f5d8c0093b0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 msgr2=0x7f5da41a6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 0x7f5da41a6ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00be20 tx=0x7f5d9c00bf10 comp rx=0 tx=0).stop 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 shutdown_connections 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5d900a1c60 0x7f5d900a4120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5da41a6610 0x7f5da41a6ac0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5da411a960 0x7f5da41ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 --2- 172.21.15.45:0/661032567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da410f9a0 0x7f5da41ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 >> 172.21.15.45:0/661032567 conn(0x7f5da406c260 msgr2=0x7f5da4071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.408+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 shutdown_connections 2024-03-12T05:19:06.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.409+0000 7f5d99ffb640 1 -- 172.21.15.45:0/661032567 wait complete. 2024-03-12T05:19:06.422 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:06.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.608+0000 7ff495c36640 1 -- 172.21.15.45:0/1650718228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff490108080 msgr2=0x7ff490114500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.608+0000 7ff495c36640 1 --2- 172.21.15.45:0/1650718228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff490108080 0x7ff490114500 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff484009900 tx=0x7ff48402bc30 comp rx=0 tx=0).stop 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 -- 172.21.15.45:0/1650718228 shutdown_connections 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 --2- 172.21.15.45:0/1650718228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff490108080 0x7ff490114500 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 --2- 172.21.15.45:0/1650718228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4901076c0 0x7ff490107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 --2- 172.21.15.45:0/1650718228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 0x7ff49010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 -- 172.21.15.45:0/1650718228 >> 172.21.15.45:0/1650718228 conn(0x7ff490076ed0 msgr2=0x7ff490105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 -- 172.21.15.45:0/1650718228 shutdown_connections 2024-03-12T05:19:06.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.609+0000 7ff495c36640 1 -- 172.21.15.45:0/1650718228 wait complete. 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.610+0000 7ff495c36640 1 Processor -- start 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff495c36640 1 -- start start 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff495c36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff495c36640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff490108080 0x7ff4901a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff495c36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 0x7ff49019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff495c36640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49019dff0 con 0x7ff49010d640 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.611+0000 7ff494c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41168/0 (socket says 172.21.15.45:41168) 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 -- 172.21.15.45:0/2024389488 learned_addr learned my addr 172.21.15.45:0/2024389488 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff495435640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 0x7ff49019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff48ffff640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff490108080 0x7ff4901a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49019e160 con 0x7ff4901076c0 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49019e320 con 0x7ff490108080 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 -- 172.21.15.45:0/2024389488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff490108080 msgr2=0x7ff4901a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff490108080 0x7ff4901a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 -- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 msgr2=0x7ff49019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 0x7ff49019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.612+0000 7ff494c34640 1 -- 172.21.15.45:0/2024389488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484009560 con 0x7ff4901076c0 2024-03-12T05:19:06.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.613+0000 7ff494c34640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff48000e9f0 tx=0x7ff48000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.613+0000 7ff48dffb640 1 -- 172.21.15.45:0/2024389488 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480009700 con 0x7ff4901076c0 2024-03-12T05:19:06.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.613+0000 7ff48dffb640 1 -- 172.21.15.45:0/2024389488 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff48000cd80 con 0x7ff4901076c0 2024-03-12T05:19:06.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.613+0000 7ff48dffb640 1 -- 172.21.15.45:0/2024389488 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480018840 con 0x7ff4901076c0 2024-03-12T05:19:06.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.613+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49019e600 con 0x7ff4901076c0 2024-03-12T05:19:06.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.614+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4901aa840 con 0x7ff4901076c0 2024-03-12T05:19:06.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.617+0000 7ff48dffb640 1 -- 172.21.15.45:0/2024389488 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff4800189a0 con 0x7ff4901076c0 2024-03-12T05:19:06.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 Processor -- start 2024-03-12T05:19:06.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 -- start start 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17460 0x558fa9f17840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f22d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9ed2960 con 0x558fa9f17460 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9e86310 con 0x558fa9f20960 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff499a5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9e83840 con 0x558fa9f17d80 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff494c34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff494c34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56230/0 (socket says 172.21.15.45:56230) 2024-03-12T05:19:06.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.618+0000 7ff494c34640 1 -- 172.21.15.45:0/3786893080 learned_addr learned my addr 172.21.15.45:0/3786893080 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 -- 172.21.15.45:0/3786893080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 msgr2=0x558fa9f22d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:19:06.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f22d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 -- 172.21.15.45:0/3786893080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17460 msgr2=0x558fa9f17840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17460 0x558fa9f17840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 -- 172.21.15.45:0/3786893080 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa9e41060 con 0x558fa9f17d80 2024-03-12T05:19:06.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.619+0000 7ff494c34640 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff4800cb1a0 tx=0x7ff4800d3e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cae1a3f31c221dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.620+0000 7ff476ffd640 1 -- 172.21.15.45:0/3786893080 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800d5da0 con 0x558fa9f17d80 2024-03-12T05:19:06.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.620+0000 7ff476ffd640 1 -- 172.21.15.45:0/3786893080 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4800d2030 con 0x558fa9f17d80 2024-03-12T05:19:06.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.620+0000 7ff476ffd640 1 -- 172.21.15.45:0/3786893080 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800d7760 con 0x558fa9f17d80 2024-03-12T05:19:06.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.620+0000 7ff48dffb640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 0x7ff4780a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.620+0000 7ff48dffb640 1 -- 172.21.15.45:0/2024389488 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff480014070 con 0x7ff4901076c0 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff495435640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 0x7ff4780a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 -- 172.21.15.45:0/3786893080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 msgr2=0x558fa9f20420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff4800cb1a0 tx=0x7ff4800d3e80 comp rx=0 tx=0).stop 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 -- 172.21.15.45:0/3786893080 shutdown_connections 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f22d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17d80 0x558fa9f20420 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3786893080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17460 0x558fa9f17840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 -- 172.21.15.45:0/3786893080 >> 172.21.15.45:0/3786893080 conn(0x558fa9f11b90 msgr2=0x558fa9f11f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.621+0000 7ff499a5f740 1 -- 172.21.15.45:0/3786893080 shutdown_connections 2024-03-12T05:19:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.622+0000 7ff499a5f740 1 -- 172.21.15.45:0/3786893080 wait complete. 2024-03-12T05:19:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.622+0000 7ff499a5f740 1 -- start start 2024-03-12T05:19:06.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.622+0000 7ff499a5f740 1 Processor -- start 2024-03-12T05:19:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f17460 0x558fa9f1d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 0x558fa9f1e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9e7f020 con 0x558fa9f17d80 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9b52430 con 0x558fa9f17460 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9b0ee00 con 0x558fa9f20960 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff499a5f740 10 client.0 _ll_register_callbacks cb 0x558fa9e26cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34062/0 (socket says 172.21.15.45:34062) 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.624+0000 7ff48ffff640 1 -- 172.21.15.45:0/687905796 learned_addr learned my addr 172.21.15.45:0/687905796 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff494c34640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f17460 0x558fa9f1d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff495435640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 0x7ff4780a40e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff484005ab0 tx=0x7ff48402c700 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff495435640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 0x558fa9f1e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff48ffff640 1 -- 172.21.15.45:0/687905796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 msgr2=0x558fa9f1e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff48ffff640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 0x558fa9f1e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff48ffff640 1 -- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f17460 msgr2=0x558fa9f1d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff48ffff640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f17460 0x558fa9f1d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff48ffff640 1 -- 172.21.15.45:0/687905796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4901aa840 con 0x558fa9f17d80 2024-03-12T05:19:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.625+0000 7ff495435640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 0x558fa9f1e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.626+0000 7ff48ffff640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff4880106c0 tx=0x7ff488010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815bba0454b62163 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.622 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.626+0000 7ff46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.627+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488003b70 con 0x558fa9f17d80 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.627+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff488003cd0 con 0x558fa9f17d80 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.627+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fa9b7e7e0 con 0x558fa9f17d80 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.627+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff488003e90 con 0x558fa9f17d80 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 10 client.17637 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fa9e7d6a0 con 0x558fa9f17d80 2024-03-12T05:19:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558fa9b7e7e0 con 0x558fa9f17d80 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff48801a030 con 0x558fa9f17d80 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 10 client.17637 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 10 client.17637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 20 client.17637 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.628+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f30ec0 0x558fa9f332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.629+0000 7ff499a5f740 4 client.17637 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.629+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fa9bbb720 con 0x558fa9f30ec0 2024-03-12T05:19:06.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.629+0000 7ff494c34640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f30ec0 0x558fa9f332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.630+0000 7ff46ffff640 10 client.17637 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:06.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.630+0000 7ff494c34640 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f30ec0 0x558fa9f332e0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2582f720dc8c75 server_cookie=82cc802474b209d in_seq=0 out_seq=0 2024-03-12T05:19:06.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.631+0000 7ff46ffff640 1 -- 172.21.15.45:0/687905796 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558fa9bbb720 con 0x558fa9f30ec0 2024-03-12T05:19:06.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.631+0000 7ff46ffff640 10 client.17637 handle_command_reply: tid=0 2024-03-12T05:19:06.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.631+0000 7ff499a5f740 1 client.17637 shutdown 2024-03-12T05:19:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 msgr2=0x558fa9f1df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff4880106c0 tx=0x7ff488010b80 comp rx=0 tx=0).stop 2024-03-12T05:19:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 shutdown_connections 2024-03-12T05:19:06.627 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f30ec0 0x558fa9f332e0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1df20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f20960 0x558fa9f1e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 --2- 172.21.15.45:0/687905796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f17460 0x558fa9f1d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 >> 172.21.15.45:0/687905796 conn(0x558fa9f11b90 msgr2=0x558fa9f138e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff46ffff640 0 client.17637 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 shutdown_connections 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 1 -- 172.21.15.45:0/687905796 wait complete. 2024-03-12T05:19:06.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.632+0000 7ff499a5f740 20 client.17637 trim_cache size 0 max 16384 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.654+0000 7ff499a5f740 1 Processor -- start 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.654+0000 7ff499a5f740 1 -- start start 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f20110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9e04130 con 0x558fa9f17d80 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9f2d8f0 con 0x558fa9f20960 2024-03-12T05:19:06.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff499a5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9f1d7d0 con 0x558fa9f17460 2024-03-12T05:19:06.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff495435640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff495435640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:41196/0 (socket says 172.21.15.45:41196) 2024-03-12T05:19:06.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff495435640 1 -- 172.21.15.45:0/151091757 learned_addr learned my addr 172.21.15.45:0/151091757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.655+0000 7ff494c34640 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f20110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.656+0000 7ff495435640 1 -- 172.21.15.45:0/151091757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 msgr2=0x558fa9f20110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.656+0000 7ff495435640 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f20110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.656+0000 7ff495435640 1 -- 172.21.15.45:0/151091757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 msgr2=0x558fa9f1c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.656+0000 7ff495435640 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.656+0000 7ff495435640 1 -- 172.21.15.45:0/151091757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa9e19c20 con 0x558fa9f20960 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.657+0000 7ff495435640 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff484009900 tx=0x7ff48403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc76952f28a38ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.657+0000 7ff4767fc640 1 -- 172.21.15.45:0/151091757 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48402c960 con 0x558fa9f20960 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.657+0000 7ff4767fc640 1 -- 172.21.15.45:0/151091757 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4840044e0 con 0x558fa9f20960 2024-03-12T05:19:06.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.657+0000 7ff4767fc640 1 -- 172.21.15.45:0/151091757 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484004dc0 con 0x558fa9f20960 2024-03-12T05:19:06.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 -- 172.21.15.45:0/151091757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 msgr2=0x558fa9f1c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff484009900 tx=0x7ff48403f000 comp rx=0 tx=0).stop 2024-03-12T05:19:06.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 -- 172.21.15.45:0/151091757 shutdown_connections 2024-03-12T05:19:06.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f1c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f1c7f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 --2- 172.21.15.45:0/151091757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f20110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 -- 172.21.15.45:0/151091757 >> 172.21.15.45:0/151091757 conn(0x558fa9f11b90 msgr2=0x558fa9f18d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 -- 172.21.15.45:0/151091757 shutdown_connections 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.659+0000 7ff499a5f740 1 -- 172.21.15.45:0/151091757 wait complete. 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.660+0000 7ff499a5f740 1 -- start start 2024-03-12T05:19:06.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.660+0000 7ff499a5f740 1 Processor -- start 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.660+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.660+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f25f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff499a5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff499a5f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9f26a80 con 0x558fa9f17d80 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff499a5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9f26bf0 con 0x558fa9f20960 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff499a5f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa9f26db0 con 0x558fa9f17460 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff494c34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f25f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff499a5f740 10 client.0 _ll_register_callbacks cb 0x7ff4800d3420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:06.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f25f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34076/0 (socket says 172.21.15.45:34076) 2024-03-12T05:19:06.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff48ffff640 1 -- 172.21.15.45:0/3532542765 learned_addr learned my addr 172.21.15.45:0/3532542765 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff495435640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.661+0000 7ff494c34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56246/0 (socket says 172.21.15.45:56246) 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 -- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 msgr2=0x558fa9f264c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f264c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 -- 172.21.15.45:0/3532542765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 msgr2=0x558fa9f25f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f25f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 -- 172.21.15.45:0/3532542765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff490114e00 con 0x558fa9f17460 2024-03-12T05:19:06.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff495435640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff494c34640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff48000e9c0 tx=0x7ff480000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0b1dd9c97d3e72c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff46ffff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48009a4e0 con 0x558fa9f17460 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4800d2030 con 0x558fa9f17460 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.662+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff48009ae10 con 0x558fa9f17460 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.663+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff48009a250 con 0x558fa9f17460 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.663+0000 7ff499a5f740 10 client.27103 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.663+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fa9e3f8c0 con 0x558fa9f17460 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.663+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff4800d2030 con 0x558fa9f17460 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.663+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff4800cb4b0 con 0x558fa9f17460 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 10 client.27103 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 20 client.27103 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f886a0 0x558fa9f8aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 4 client.27103 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558fa9c0a890 con 0x558fa9f886a0 2024-03-12T05:19:06.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.664+0000 7ff48ffff640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f886a0 0x558fa9f8aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:06.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.665+0000 7ff48ffff640 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f886a0 0x558fa9f8aac0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b693b6ef5d523833 server_cookie=f44e09290d426d95 in_seq=0 out_seq=0 2024-03-12T05:19:06.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.665+0000 7ff46ffff640 10 client.27103 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.665+0000 7ff46ffff640 1 -- 172.21.15.45:0/3532542765 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558fa9c0a890 con 0x558fa9f886a0 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.665+0000 7ff46ffff640 10 client.27103 handle_command_reply: tid=0 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.665+0000 7ff499a5f740 1 client.27103 shutdown 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 msgr2=0x558fa9f27ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff48000e9c0 tx=0x7ff480000bc0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 shutdown_connections 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558fa9f886a0 0x558fa9f8aac0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fa9f17d80 0x558fa9f25f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fa9f20960 0x558fa9f264c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 --2- 172.21.15.45:0/3532542765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fa9f17460 0x558fa9f27ae0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 >> 172.21.15.45:0/3532542765 conn(0x558fa9f11b90 msgr2=0x558fa9f1c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff46ffff640 0 client.27103 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:06.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.666+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 shutdown_connections 2024-03-12T05:19:06.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.667+0000 7ff499a5f740 1 -- 172.21.15.45:0/3532542765 wait complete. 2024-03-12T05:19:06.663 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.667+0000 7ff499a5f740 20 client.27103 trim_cache size 0 max 16384 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 msgr2=0x7ff4780a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 0x7ff4780a40e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff484005ab0 tx=0x7ff48402c700 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 msgr2=0x7ff4901a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff48000e9f0 tx=0x7ff48000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 shutdown_connections 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff4780a1c20 0x7ff4780a40e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff49010d640 0x7ff49019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff490108080 0x7ff4901a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 --2- 172.21.15.45:0/2024389488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4901076c0 0x7ff4901a37a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 >> 172.21.15.45:0/2024389488 conn(0x7ff490076ed0 msgr2=0x7ff490077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:06.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 shutdown_connections 2024-03-12T05:19:06.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:06.668+0000 7ff495c36640 1 -- 172.21.15.45:0/2024389488 wait complete. 2024-03-12T05:19:07.680 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:19:07.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:19:07.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:19:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.941+0000 7f733c87d640 1 -- 172.21.15.45:0/3197005382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 msgr2=0x7f7338116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:07.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.941+0000 7f733c87d640 1 --2- 172.21.15.45:0/3197005382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f7338116210 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7330009be0 tx=0x7f7330030090 comp rx=0 tx=0).stop 2024-03-12T05:19:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.946+0000 7f733c87d640 1 -- 172.21.15.45:0/3197005382 shutdown_connections 2024-03-12T05:19:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.946+0000 7f733c87d640 1 --2- 172.21.15.45:0/3197005382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f7338116210 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.946+0000 7f733c87d640 1 --2- 172.21.15.45:0/3197005382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73381086c0 0x7f7338108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.946+0000 7f733c87d640 1 --2- 172.21.15.45:0/3197005382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f7338108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.946+0000 7f733c87d640 1 -- 172.21.15.45:0/3197005382 >> 172.21.15.45:0/3197005382 conn(0x7f733806c030 msgr2=0x7f733806c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:07.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.947+0000 7f733c87d640 1 -- 172.21.15.45:0/3197005382 shutdown_connections 2024-03-12T05:19:07.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.947+0000 7f733c87d640 1 -- 172.21.15.45:0/3197005382 wait complete. 2024-03-12T05:19:07.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.948+0000 7f733c87d640 1 Processor -- start 2024-03-12T05:19:07.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.948+0000 7f733c87d640 1 -- start start 2024-03-12T05:19:07.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73381086c0 0x7f73381a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f733819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733819e010 con 0x7f733806d410 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733819e180 con 0x7f7338113e20 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f733c87d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733819e340 con 0x7f73381086c0 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f73377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f73377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40032/0 (socket says 172.21.15.45:40032) 2024-03-12T05:19:07.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.949+0000 7f73377fe640 1 -- 172.21.15.45:0/2607377242 learned_addr learned my addr 172.21.15.45:0/2607377242 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f7337fff640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f733819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 -- 172.21.15.45:0/2607377242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73381086c0 msgr2=0x7f73381a4c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73381086c0 0x7f73381a4c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 -- 172.21.15.45:0/2607377242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 msgr2=0x7f733819d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f733819d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 -- 172.21.15.45:0/2607377242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330009840 con 0x7f733806d410 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f7337fff640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f733819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73377fe640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f732800d950 tx=0x7f732800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.950+0000 7f73357fa640 1 -- 172.21.15.45:0/2607377242 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328009c30 con 0x7f733806d410 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.951+0000 7f73357fa640 1 -- 172.21.15.45:0/2607377242 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7328009d90 con 0x7f733806d410 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.951+0000 7f73357fa640 1 -- 172.21.15.45:0/2607377242 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732800f6b0 con 0x7f733806d410 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.951+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733819e620 con 0x7f733806d410 2024-03-12T05:19:07.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.951+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73381a2540 con 0x7f733806d410 2024-03-12T05:19:07.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.957+0000 7f73357fa640 1 -- 172.21.15.45:0/2607377242 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7328010a80 con 0x7f733806d410 2024-03-12T05:19:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.958+0000 7f73357fa640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 0x7f73240a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.958+0000 7f73357fa640 1 -- 172.21.15.45:0/2607377242 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f73280ccde0 con 0x7f733806d410 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.961+0000 7f734078d740 1 Processor -- start 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.961+0000 7f734078d740 1 -- start start 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8526e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85277530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae85279f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae851dcf90 con 0x55ae85277b00 2024-03-12T05:19:07.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae851da4c0 con 0x55ae8526ef80 2024-03-12T05:19:07.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f734078d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae851d5ca0 con 0x55ae8526e5d0 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f732effd640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 0x7f73240a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f7337fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae85279f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f7337fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae85279f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40042/0 (socket says 172.21.15.45:40042) 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.962+0000 7f7337fff640 1 -- 172.21.15.45:0/4231739521 learned_addr learned my addr 172.21.15.45:0/4231739521 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85277530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f73377fe640 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8526e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 -- 172.21.15.45:0/4231739521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 msgr2=0x55ae8526e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8526e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 -- 172.21.15.45:0/4231739521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 msgr2=0x55ae85279f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:07.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae85279f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.963+0000 7f732effd640 1 -- 172.21.15.45:0/4231739521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae851980b0 con 0x55ae8526ef80 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.964+0000 7f732effd640 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85277530 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f733819f020 tx=0x7f7320013c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f141323e99924b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.964+0000 7f732dffb640 1 -- 172.21.15.45:0/4231739521 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320018070 con 0x55ae8526ef80 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.964+0000 7f732dffb640 1 -- 172.21.15.45:0/4231739521 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7320005d50 con 0x55ae8526ef80 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.965+0000 7f734078d740 1 -- 172.21.15.45:0/4231739521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 msgr2=0x55ae85277530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:07.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.965+0000 7f734078d740 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85277530 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f733819f020 tx=0x7f7320013c70 comp rx=0 tx=0).stop 2024-03-12T05:19:07.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.966+0000 7f732effd640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 0x7f73240a4270 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7320010000 tx=0x7f732001a000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 -- 172.21.15.45:0/4231739521 shutdown_connections 2024-03-12T05:19:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae85279f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85277530 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 --2- 172.21.15.45:0/4231739521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8526e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 -- 172.21.15.45:0/4231739521 >> 172.21.15.45:0/4231739521 conn(0x55ae85268a90 msgr2=0x55ae85268e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 -- 172.21.15.45:0/4231739521 shutdown_connections 2024-03-12T05:19:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 -- 172.21.15.45:0/4231739521 wait complete. 2024-03-12T05:19:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.967+0000 7f734078d740 1 -- start start 2024-03-12T05:19:07.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 Processor -- start 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae85274a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae852754e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae8517af40 con 0x55ae85277b00 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae84ea94b0 con 0x55ae8526ef80 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae8515b670 con 0x55ae8526e5d0 2024-03-12T05:19:07.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.968+0000 7f734078d740 10 client.0 _ll_register_callbacks cb 0x55ae851c8f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49142/0 (socket says 172.21.15.45:49142) 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 -- 172.21.15.45:0/259792755 learned_addr learned my addr 172.21.15.45:0/259792755 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f73377fe640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae85274a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f7337fff640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae852754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 -- 172.21.15.45:0/259792755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 msgr2=0x55ae85274a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae85274a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 -- 172.21.15.45:0/259792755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 msgr2=0x55ae852754e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae852754e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 -- 172.21.15.45:0/259792755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73200134c0 con 0x55ae8526ef80 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.969+0000 7f732effd640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f732000e930 tx=0x7f7320013f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52285b86e681e4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f7316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320018070 con 0x55ae8526ef80 2024-03-12T05:19:07.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae851aff40 con 0x55ae8526ef80 2024-03-12T05:19:07.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f7320005d50 con 0x55ae8526ef80 2024-03-12T05:19:07.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f732000ebc0 con 0x55ae8526ef80 2024-03-12T05:19:07.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f734078d740 10 client.27125 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:07.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.970+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ae851d4320 con 0x55ae8526ef80 2024-03-12T05:19:07.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:07 smithi130 ceph-mon[35398]: pgmap v254: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.6 MiB/s wr, 1.12k op/s 2024-03-12T05:19:07.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:07 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:19:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ae851aff40 con 0x55ae8526ef80 2024-03-12T05:19:07.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7320026680 con 0x55ae8526ef80 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 10 client.27125 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 10 client.27125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 20 client.27125 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae85288050 0x55ae8528a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 4 client.27125 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae851cc310 con 0x55ae85288050 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.971+0000 7f73377fe640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae85288050 0x55ae8528a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.972+0000 7f73377fe640 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae85288050 0x55ae8528a470 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92b1fdf1fc689ff8 server_cookie=bd279ec173b3b7a6 in_seq=0 out_seq=0 2024-03-12T05:19:07.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.972+0000 7f7316ffd640 10 client.27125 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:07.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f7316ffd640 1 -- 172.21.15.45:0/259792755 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ae851cc310 con 0x55ae85288050 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f7316ffd640 10 client.27125 handle_command_reply: tid=0 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 client.27125 shutdown 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 msgr2=0x55ae85274fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f732000e930 tx=0x7f7320013f80 comp rx=0 tx=0).stop 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 shutdown_connections 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae85288050 0x55ae8528a470 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae85277b00 0x55ae852754e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526ef80 0x55ae85274fa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 --2- 172.21.15.45:0/259792755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae85274a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.974+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 >> 172.21.15.45:0/259792755 conn(0x55ae85268a90 msgr2=0x55ae8526a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.975+0000 7f7316ffd640 0 client.27125 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:07.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.975+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 shutdown_connections 2024-03-12T05:19:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.975+0000 7f734078d740 1 -- 172.21.15.45:0/259792755 wait complete. 2024-03-12T05:19:07.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:07.975+0000 7f734078d740 20 client.27125 trim_cache size 0 max 16384 2024-03-12T05:19:07.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:07 smithi116 ceph-mon[35486]: pgmap v254: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.6 MiB/s wr, 1.12k op/s 2024-03-12T05:19:07.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:07 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:19:08.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.006+0000 7f734078d740 1 Processor -- start 2024-03-12T05:19:08.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.007+0000 7f734078d740 1 -- start start 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.007+0000 7f734078d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.007+0000 7f734078d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae85271cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.007+0000 7f73377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.007+0000 7f73377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49148/0 (socket says 172.21.15.45:49148) 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f73377fe640 1 -- 172.21.15.45:0/1980635317 learned_addr learned my addr 172.21.15.45:0/1980635317 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f734078d740 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae85277b00 0x55ae85272200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae851cc310 con 0x55ae8526ef80 2024-03-12T05:19:08.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f732effd640 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae85271cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f7337fff640 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae85277b00 0x55ae85272200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae851aff40 con 0x55ae8526e5d0 2024-03-12T05:19:08.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.008+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85284b20 con 0x55ae85277b00 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 -- 172.21.15.45:0/1980635317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae85277b00 msgr2=0x55ae85272200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae85277b00 0x55ae85272200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 -- 172.21.15.45:0/1980635317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 msgr2=0x55ae85271cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae85271cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 -- 172.21.15.45:0/1980635317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae85170c00 con 0x55ae8526e5d0 2024-03-12T05:19:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.009+0000 7f73377fe640 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7328095e60 tx=0x7f732800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b31416990b84fba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.012+0000 7f732d7fa640 1 -- 172.21.15.45:0/1980635317 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328097d00 con 0x55ae8526e5d0 2024-03-12T05:19:08.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.012+0000 7f732d7fa640 1 -- 172.21.15.45:0/1980635317 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7328096070 con 0x55ae8526e5d0 2024-03-12T05:19:08.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.012+0000 7f732d7fa640 1 -- 172.21.15.45:0/1980635317 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280b2550 con 0x55ae8526e5d0 2024-03-12T05:19:08.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.013+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 msgr2=0x55ae8528edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.013+0000 7f734078d740 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7328095e60 tx=0x7f732800bf60 comp rx=0 tx=0).stop 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 shutdown_connections 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae85271cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae85277b00 0x55ae85272200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 --2- 172.21.15.45:0/1980635317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae8526e5d0 0x55ae8528edf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 >> 172.21.15.45:0/1980635317 conn(0x55ae85268a90 msgr2=0x55ae8526fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 shutdown_connections 2024-03-12T05:19:08.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 -- 172.21.15.45:0/1980635317 wait complete. 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.014+0000 7f734078d740 1 -- start start 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 Processor -- start 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae8527e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f73377fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f73377fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57386/0 (socket says 172.21.15.45:57386) 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 0x55ae852d85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f73377fe640 1 -- 172.21.15.45:0/3357396385 learned_addr learned my addr 172.21.15.45:0/3357396385 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae852d8b00 con 0x55ae8526ef80 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae852d8c70 con 0x55ae85277b00 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.015+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae852d8e30 con 0x55ae8526e5d0 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 0x55ae852d85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f734078d740 10 client.0 _ll_register_callbacks cb 0x7f732000edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:08.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 -- 172.21.15.45:0/3357396385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 msgr2=0x55ae8527ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 -- 172.21.15.45:0/3357396385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 msgr2=0x55ae8527e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae8527e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f7337fff640 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73381a2540 con 0x55ae85277b00 2024-03-12T05:19:08.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.016+0000 7f73377fe640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:08.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.017+0000 7f7337fff640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 0x55ae852d85c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73300096c0 tx=0x7f7330037d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb2f2a619e55e80f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.019+0000 7f7316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.019+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330037e90 con 0x55ae85277b00 2024-03-12T05:19:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.019+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f73300375f0 con 0x55ae85277b00 2024-03-12T05:19:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.020+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae851fb900 con 0x55ae85277b00 2024-03-12T05:19:08.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.020+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73300377b0 con 0x55ae85277b00 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.020+0000 7f734078d740 10 client.27131 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.020+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ae85196910 con 0x55ae85277b00 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ae851fb900 con 0x55ae85277b00 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7330004660 con 0x55ae85277b00 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 10 client.27131 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:08.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 10 client.27131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:08.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 20 client.27131 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:08.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae853026c0 0x55ae85304b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 4 client.27131 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:19:08.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.022+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ae84ed5860 con 0x55ae853026c0 2024-03-12T05:19:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.024+0000 7f73377fe640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae853026c0 0x55ae85304b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.026+0000 7f73377fe640 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae853026c0 0x55ae85304b00 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46bb050725a3582c server_cookie=174115b806edc8b1 in_seq=0 out_seq=0 2024-03-12T05:19:08.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.026+0000 7f7316ffd640 10 client.27131 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.027+0000 7f7316ffd640 1 -- 172.21.15.45:0/3357396385 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ae84ed5860 con 0x55ae853026c0 2024-03-12T05:19:08.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.027+0000 7f7316ffd640 10 client.27131 handle_command_reply: tid=0 2024-03-12T05:19:08.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.027+0000 7f734078d740 1 client.27131 shutdown 2024-03-12T05:19:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 msgr2=0x55ae852d85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 0x55ae852d85c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73300096c0 tx=0x7f7330037d20 comp rx=0 tx=0).stop 2024-03-12T05:19:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 shutdown_connections 2024-03-12T05:19:08.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ae853026c0 0x55ae85304b00 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ae8526ef80 0x55ae8527e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae85277b00 0x55ae852d85c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 --2- 172.21.15.45:0/3357396385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae8526e5d0 0x55ae8527ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 >> 172.21.15.45:0/3357396385 conn(0x55ae85268a90 msgr2=0x55ae85272f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.032+0000 7f7316ffd640 0 client.27131 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.033+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 shutdown_connections 2024-03-12T05:19:08.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.033+0000 7f734078d740 1 -- 172.21.15.45:0/3357396385 wait complete. 2024-03-12T05:19:08.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.033+0000 7f734078d740 20 client.27131 trim_cache size 0 max 16384 2024-03-12T05:19:08.031 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:08.031 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:19:08.032 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "666b29b9-8370-41ae-b621-72736898659e", 2024-03-12T05:19:08.032 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:19:08.032 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.035+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 msgr2=0x7f73240a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.035+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 0x7f73240a4270 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7320010000 tx=0x7f732001a000 comp rx=0 tx=0).stop 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.035+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 msgr2=0x7f73381a4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.035+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f732800d950 tx=0x7f732800de10 comp rx=0 tx=0).stop 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 shutdown_connections 2024-03-12T05:19:08.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f73240a1db0 0x7f73240a4270 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7338113e20 0x7f733819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73381086c0 0x7f73381a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 --2- 172.21.15.45:0/2607377242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f733806d410 0x7f73381a4750 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.036+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 >> 172.21.15.45:0/2607377242 conn(0x7f733806c030 msgr2=0x7f733806fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.038+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 shutdown_connections 2024-03-12T05:19:08.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.039+0000 7f733c87d640 1 -- 172.21.15.45:0/2607377242 wait complete. 2024-03-12T05:19:08.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:07 smithi045 ceph-mon[32174]: pgmap v254: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.6 MiB/s wr, 1.12k op/s 2024-03-12T05:19:08.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:07 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:19:08.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:19:08.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:19:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 -- 172.21.15.45:0/898518595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204108100 msgr2=0x7f42041145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 --2- 172.21.15.45:0/898518595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204108100 0x7f42041145c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009c10 tx=0x7f41fc02bf70 comp rx=0 tx=0).stop 2024-03-12T05:19:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 -- 172.21.15.45:0/898518595 shutdown_connections 2024-03-12T05:19:08.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 --2- 172.21.15.45:0/898518595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204108100 0x7f42041145c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 --2- 172.21.15.45:0/898518595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204107760 0x7f4204107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 --2- 172.21.15.45:0/898518595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 0x7f420410ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 -- 172.21.15.45:0/898518595 >> 172.21.15.45:0/898518595 conn(0x7f4204076f50 msgr2=0x7f4204105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 -- 172.21.15.45:0/898518595 shutdown_connections 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.269+0000 7f420bba1640 1 -- 172.21.15.45:0/898518595 wait complete. 2024-03-12T05:19:08.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.270+0000 7f420bba1640 1 Processor -- start 2024-03-12T05:19:08.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.274+0000 7f420bba1640 1 -- start start 2024-03-12T05:19:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.274+0000 7f420bba1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 0x7f42041a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420bba1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 0x7f42041a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420bba1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 0x7f420419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420bba1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420419e100 con 0x7f4204107760 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420bba1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420419e270 con 0x7f4204108100 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420bba1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420419e430 con 0x7f420410c640 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420913c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 0x7f42041a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420993d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 0x7f42041a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420913c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 0x7f42041a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49172/0 (socket says 172.21.15.45:49172) 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420913c640 1 -- 172.21.15.45:0/3592215453 learned_addr learned my addr 172.21.15.45:0/3592215453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.275+0000 7f420a13e640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 0x7f420419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 -- 172.21.15.45:0/3592215453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 msgr2=0x7f420419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 0x7f420419dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 -- 172.21.15.45:0/3592215453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 msgr2=0x7f42041a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 0x7f42041a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 -- 172.21.15.45:0/3592215453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc009840 con 0x7f4204107760 2024-03-12T05:19:08.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.276+0000 7f420993d640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 0x7f42041a3900 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f420000e9f0 tx=0x7f420000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.279+0000 7f41faffd640 1 -- 172.21.15.45:0/3592215453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420000ccc0 con 0x7f4204107760 2024-03-12T05:19:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.279+0000 7f41faffd640 1 -- 172.21.15.45:0/3592215453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f42000044d0 con 0x7f4204107760 2024-03-12T05:19:08.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.279+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420419e710 con 0x7f4204107760 2024-03-12T05:19:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.279+0000 7f41faffd640 1 -- 172.21.15.45:0/3592215453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200004a00 con 0x7f4204107760 2024-03-12T05:19:08.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.279+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42041aa9a0 con 0x7f4204107760 2024-03-12T05:19:08.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.284+0000 7f420e77a740 1 Processor -- start 2024-03-12T05:19:08.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 -- start start 2024-03-12T05:19:08.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e85332b0 0x5576e8533690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e853c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e84a2280 con 0x5576e8533bd0 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e849f7b0 con 0x5576e85332b0 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.285+0000 7f420e77a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e849af90 con 0x5576e853c7b0 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.286+0000 7f420913c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.286+0000 7f420913c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57418/0 (socket says 172.21.15.45:57418) 2024-03-12T05:19:08.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.286+0000 7f420913c640 1 -- 172.21.15.45:0/443967902 learned_addr learned my addr 172.21.15.45:0/443967902 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.286+0000 7f420a13e640 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e85332b0 0x5576e8533690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 -- 172.21.15.45:0/443967902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e85332b0 msgr2=0x5576e8533690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e85332b0 0x5576e8533690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 -- 172.21.15.45:0/443967902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 msgr2=0x5576e853c270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e853c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 -- 172.21.15.45:0/443967902 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576e8459570 con 0x5576e853c7b0 2024-03-12T05:19:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f420913c640 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f41f40045f0 tx=0x7f41f400bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258c15d709abfef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f41effff640 1 -- 172.21.15.45:0/443967902 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003dd0 con 0x5576e853c7b0 2024-03-12T05:19:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f41effff640 1 -- 172.21.15.45:0/443967902 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41f400bec0 con 0x5576e853c7b0 2024-03-12T05:19:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.287+0000 7f41effff640 1 -- 172.21.15.45:0/443967902 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4011900 con 0x5576e853c7b0 2024-03-12T05:19:08.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.288+0000 7f41faffd640 1 -- 172.21.15.45:0/3592215453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4200004b60 con 0x7f4204107760 2024-03-12T05:19:08.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.288+0000 7f41faffd640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 0x7f41f00a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.288+0000 7f420e77a740 1 -- 172.21.15.45:0/443967902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 msgr2=0x5576e853eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.288+0000 7f420e77a740 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f41f40045f0 tx=0x7f41f400bc70 comp rx=0 tx=0).stop 2024-03-12T05:19:08.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.289+0000 7f41faffd640 1 -- 172.21.15.45:0/3592215453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f4200014070 con 0x7f4204107760 2024-03-12T05:19:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420a13e640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 0x7f41f00a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420e77a740 1 -- 172.21.15.45:0/443967902 shutdown_connections 2024-03-12T05:19:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420e77a740 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e853c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420e77a740 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e853eba0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420e77a740 1 --2- 172.21.15.45:0/443967902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e85332b0 0x5576e8533690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.290+0000 7f420e77a740 1 -- 172.21.15.45:0/443967902 >> 172.21.15.45:0/443967902 conn(0x5576e852d990 msgr2=0x5576e852dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.292+0000 7f420e77a740 1 -- 172.21.15.45:0/443967902 shutdown_connections 2024-03-12T05:19:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.293+0000 7f420a13e640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 0x7f41f00a4230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41fc034fd0 tx=0x7f41fc005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.293+0000 7f420e77a740 1 -- 172.21.15.45:0/443967902 wait complete. 2024-03-12T05:19:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.293+0000 7f420e77a740 1 -- start start 2024-03-12T05:19:08.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.293+0000 7f420e77a740 1 Processor -- start 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 0x5576e853c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e8539570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e8539cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e816e430 con 0x5576e8533bd0 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e812ae00 con 0x5576e853c7b0 2024-03-12T05:19:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e841ffd0 con 0x5576e85332b0 2024-03-12T05:19:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420993d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 0x5576e853c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420993d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 0x5576e853c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57434/0 (socket says 172.21.15.45:57434) 2024-03-12T05:19:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420993d640 1 -- 172.21.15.45:0/1299889258 learned_addr learned my addr 172.21.15.45:0/1299889258 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420a13e640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e8539cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.294+0000 7f420e77a740 10 client.0 _ll_register_callbacks cb 0x5576e8475fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 -- 172.21.15.45:0/1299889258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 msgr2=0x5576e853c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 0x5576e853c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 -- 172.21.15.45:0/1299889258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 msgr2=0x5576e8539570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e8539570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 -- 172.21.15.45:0/1299889258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f4004220 con 0x5576e853c7b0 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f420a13e640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e8539cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036040 tx=0x7f41fc02bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77804fafb3dfdbb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.295+0000 7f41ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:08.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.296+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc004860 con 0x5576e853c7b0 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.296+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f41fc034430 con 0x5576e853c7b0 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.296+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576e81d7720 con 0x5576e853c7b0 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.296+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41fc0345f0 con 0x5576e853c7b0 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.297+0000 7f420e77a740 10 client.27140 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.297+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5576e8499610 con 0x5576e853c7b0 2024-03-12T05:19:08.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5576e81d7720 con 0x5576e853c7b0 2024-03-12T05:19:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f41fc047070 con 0x5576e853c7b0 2024-03-12T05:19:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 10 client.27140 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 10 client.27140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 20 client.27140 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:08.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e855d9b0 0x5576e855fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 4 client.27140 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.298+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576e8226890 con 0x5576e855d9b0 2024-03-12T05:19:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.299+0000 7f420993d640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e855d9b0 0x5576e855fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.300+0000 7f420993d640 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e855d9b0 0x5576e855fdd0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9becdbad471fb5e server_cookie=79390e1f3cb9f832 in_seq=0 out_seq=0 2024-03-12T05:19:08.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.300+0000 7f41ed7fa640 10 client.27140 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.301+0000 7f41ed7fa640 1 -- 172.21.15.45:0/1299889258 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5576e8226890 con 0x5576e855d9b0 2024-03-12T05:19:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.301+0000 7f41ed7fa640 10 client.27140 handle_command_reply: tid=0 2024-03-12T05:19:08.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.301+0000 7f420e77a740 1 client.27140 shutdown 2024-03-12T05:19:08.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.302+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 msgr2=0x5576e8539cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.302+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e8539cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036040 tx=0x7f41fc02bf70 comp rx=0 tx=0).stop 2024-03-12T05:19:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 shutdown_connections 2024-03-12T05:19:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e855d9b0 0x5576e855fdd0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e8533bd0 0x5576e8539570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e8539cb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 --2- 172.21.15.45:0/1299889258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e85332b0 0x5576e853c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 >> 172.21.15.45:0/1299889258 conn(0x5576e852d990 msgr2=0x5576e852f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f41ed7fa640 0 client.27140 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 shutdown_connections 2024-03-12T05:19:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.303+0000 7f420e77a740 1 -- 172.21.15.45:0/1299889258 wait complete. 2024-03-12T05:19:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.304+0000 7f420e77a740 20 client.27140 trim_cache size 0 max 16384 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 Processor -- start 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 -- start start 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e8559140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e8533bd0 0x5576e853bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e852e9e0 con 0x5576e85332b0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e8532cf0 con 0x5576e853c7b0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420e77a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e852ec10 con 0x5576e8533bd0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420a13e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420a13e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49204/0 (socket says 172.21.15.45:49204) 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420a13e640 1 -- 172.21.15.45:0/2737768590 learned_addr learned my addr 172.21.15.45:0/2737768590 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420993d640 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e8559140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.336+0000 7f420913c640 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e8533bd0 0x5576e853bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 -- 172.21.15.45:0/2737768590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e8533bd0 msgr2=0x5576e853bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e8533bd0 0x5576e853bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 -- 172.21.15.45:0/2737768590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 msgr2=0x5576e8559140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e8559140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 -- 172.21.15.45:0/2737768590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576e8435ac0 con 0x5576e853c7b0 2024-03-12T05:19:08.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f420a13e640 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036010 tx=0x7f41fc005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e75bc250d50fad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f41ef7fe640 1 -- 172.21.15.45:0/2737768590 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc02cb10 con 0x5576e853c7b0 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f41ef7fe640 1 -- 172.21.15.45:0/2737768590 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41fc004860 con 0x5576e853c7b0 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.337+0000 7f41ef7fe640 1 -- 172.21.15.45:0/2737768590 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc034c10 con 0x5576e853c7b0 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 -- 172.21.15.45:0/2737768590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 msgr2=0x5576e853c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f41fc036010 tx=0x7f41fc005be0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 -- 172.21.15.45:0/2737768590 shutdown_connections 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e853c7b0 0x5576e853c1e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e8533bd0 0x5576e853bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 --2- 172.21.15.45:0/2737768590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e8559140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 -- 172.21.15.45:0/2737768590 >> 172.21.15.45:0/2737768590 conn(0x5576e852d990 msgr2=0x5576e8534850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 -- 172.21.15.45:0/2737768590 shutdown_connections 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.339+0000 7f420e77a740 1 -- 172.21.15.45:0/2737768590 wait complete. 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.340+0000 7f420e77a740 1 -- start start 2024-03-12T05:19:08.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.340+0000 7f420e77a740 1 Processor -- start 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.340+0000 7f420e77a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e85516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e8533bd0 0x5576e8551bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e85b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420993d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e85516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420993d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e85516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40128/0 (socket says 172.21.15.45:40128) 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420993d640 1 -- 172.21.15.45:0/4217480316 learned_addr learned my addr 172.21.15.45:0/4217480316 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:08.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e85522a0 con 0x5576e85332b0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e85b4430 con 0x5576e8533bd0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576e85b4570 con 0x5576e853c7b0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e85b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420e77a740 10 client.0 _ll_register_callbacks cb 0x7f41fc043050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420913c640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e8533bd0 0x5576e8551bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 -- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e8533bd0 msgr2=0x5576e8551bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e8533bd0 0x5576e8551bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 -- 172.21.15.45:0/4217480316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 msgr2=0x5576e85516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e85516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.341+0000 7f420a13e640 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576e8499610 con 0x5576e853c7b0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f420a13e640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e85b3ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f41fc054980 tx=0x7f41fc054690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dd071fac895ca62 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f41ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:08.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc034840 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576e819a7e0 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f41fc009a60 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41fc0349a0 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f420e77a740 10 client.27133 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.342+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5576e845b6a0 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.343+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f41fc034b00 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.343+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f41fc062030 con 0x5576e853c7b0 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 10 client.27133 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 10 client.27133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 20 client.27133 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:08.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e85b5820 0x5576e85b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:08.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 4 client.27133 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:19:08.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5576e819a7e0 con 0x5576e85b5820 2024-03-12T05:19:08.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.344+0000 7f420993d640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e85b5820 0x5576e85b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:08.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.345+0000 7f420993d640 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e85b5820 0x5576e85b7c10 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c24629760f82b5e0 server_cookie=cc290b04048319d in_seq=0 out_seq=0 2024-03-12T05:19:08.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.345+0000 7f41ed7fa640 10 client.27133 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.345+0000 7f41ed7fa640 1 -- 172.21.15.45:0/4217480316 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5576e819a7e0 con 0x5576e85b5820 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.345+0000 7f41ed7fa640 10 client.27133 handle_command_reply: tid=0 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.345+0000 7f420e77a740 1 client.27133 shutdown 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 msgr2=0x5576e85b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e85b3ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f41fc054980 tx=0x7f41fc054690 comp rx=0 tx=0).stop 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 shutdown_connections 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5576e85b5820 0x5576e85b7c10 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576e853c7b0 0x5576e85b3ef0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.346+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5576e8533bd0 0x5576e8551bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.347+0000 7f420e77a740 1 --2- 172.21.15.45:0/4217480316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5576e85332b0 0x5576e85516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.347+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 >> 172.21.15.45:0/4217480316 conn(0x5576e852d990 msgr2=0x5576e8537de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.347+0000 7f41ed7fa640 0 client.27133 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.347+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 shutdown_connections 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.347+0000 7f420e77a740 1 -- 172.21.15.45:0/4217480316 wait complete. 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.348+0000 7f420e77a740 20 client.27133 trim_cache size 0 max 16384 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout: "666b29b9-8370-41ae-b621-72736898659e": { 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:19:08.345 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "666b29b9-8370-41ae-b621-72736898659e", 2024-03-12T05:19:08.346 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:19:08.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:19:08.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:19:08.346 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:08.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.349+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 msgr2=0x7f41f00a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.349+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 0x7f41f00a4230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41fc034fd0 tx=0x7f41fc005fd0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.349+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 msgr2=0x7f42041a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.349+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 0x7f42041a3900 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f420000e9f0 tx=0x7f420000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 shutdown_connections 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f41f00a1d70 0x7f41f00a4230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f420410c640 0x7f420419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4204108100 0x7f42041a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 --2- 172.21.15.45:0/3592215453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4204107760 0x7f42041a3900 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 >> 172.21.15.45:0/3592215453 conn(0x7f4204076f50 msgr2=0x7f4204077510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 shutdown_connections 2024-03-12T05:19:08.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:08.350+0000 7f420bba1640 1 -- 172.21.15.45:0/3592215453 wait complete. 2024-03-12T05:19:08.365 INFO:tasks.cephfs.filesystem:scrub status for tag:666b29b9-8370-41ae-b621-72736898659e - {'path': '/', 'tag': '666b29b9-8370-41ae-b621-72736898659e', 'options': 'recursive,force'} 2024-03-12T05:19:08.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T05:19:08.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:19:08.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:19:08.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:19:08.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:19:08.969 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:08 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:08.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T05:19:08.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:19:08.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:08.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:19:08.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:19:08.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:08 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:19:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:08 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 425.0 s, 1095.2 tps, lat 91.401 ms stddev 67.167 2024-03-12T05:19:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:09 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:09 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:09 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:09 smithi130 ceph-mon[35398]: pgmap v256: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:09.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:09 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:09 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:09 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:09.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:09 smithi116 ceph-mon[35486]: pgmap v256: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:10.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:09 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:10.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:09 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:10.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:09 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:10.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:09 smithi045 ceph-mon[32174]: pgmap v256: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:11.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:11 smithi130 ceph-mon[35398]: pgmap v257: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:11.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:11 smithi116 ceph-mon[35486]: pgmap v257: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:12.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:11 smithi045 ceph-mon[32174]: pgmap v257: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:13.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:13 smithi130 ceph-mon[35398]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:13.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:13 smithi116 ceph-mon[35486]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:14.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:13 smithi045 ceph-mon[32174]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:19:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 430.0 s, 1350.4 tps, lat 73.980 ms stddev 41.882 2024-03-12T05:19:15.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:15 smithi130 ceph-mon[35398]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T05:19:15.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:15 smithi116 ceph-mon[35486]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T05:19:15.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:15 smithi045 ceph-mon[32174]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T05:19:16.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:16 smithi130 ceph-mon[35398]: pgmap v260: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:19:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:16 smithi116 ceph-mon[35486]: pgmap v260: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:19:16.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:16 smithi045 ceph-mon[32174]: pgmap v260: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:19:18.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:18 smithi116 ceph-mon[35486]: pgmap v261: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:19:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:18 smithi045 ceph-mon[32174]: pgmap v261: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:19:18.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:18 smithi130 ceph-mon[35398]: pgmap v261: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:19:19.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 435.0 s, 1467.2 tps, lat 68.138 ms stddev 22.229 2024-03-12T05:19:20.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:20 smithi116 ceph-mon[35486]: pgmap v262: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:19:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:20 smithi045 ceph-mon[32174]: pgmap v262: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:19:20.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:20 smithi130 ceph-mon[35398]: pgmap v262: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-03-12T05:19:22.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:22 smithi116 ceph-mon[35486]: pgmap v263: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:22 smithi045 ceph-mon[32174]: pgmap v263: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:22.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:22 smithi130 ceph-mon[35398]: pgmap v263: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:23.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:23.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:23.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:23.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:23.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:23.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:23.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:23.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:23.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:23.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:23.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:23.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:24 smithi045 ceph-mon[32174]: pgmap v264: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 440.0 s, 1384.6 tps, lat 72.375 ms stddev 27.700 2024-03-12T05:19:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:24 smithi130 ceph-mon[35398]: pgmap v264: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:24 smithi116 ceph-mon[35486]: pgmap v264: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:26.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:26 smithi045 ceph-mon[32174]: pgmap v265: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:19:26.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:26 smithi130 ceph-mon[35398]: pgmap v265: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:19:26.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:26 smithi116 ceph-mon[35486]: pgmap v265: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:19:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:28 smithi045 ceph-mon[32174]: pgmap v266: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:19:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:28 smithi130 ceph-mon[35398]: pgmap v266: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:19:28.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:28 smithi116 ceph-mon[35486]: pgmap v266: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:19:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 445.0 s, 1552.0 tps, lat 64.295 ms stddev 20.555 2024-03-12T05:19:30.768 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:30 smithi130 ceph-mon[35398]: pgmap v267: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.72k op/s 2024-03-12T05:19:30.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:30 smithi045 ceph-mon[32174]: pgmap v267: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.72k op/s 2024-03-12T05:19:30.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:30 smithi116 ceph-mon[35486]: pgmap v267: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.72k op/s 2024-03-12T05:19:31.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:31 smithi045 ceph-mon[32174]: pgmap v268: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:19:31.892 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:31 smithi116 ceph-mon[35486]: pgmap v268: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:19:31.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:31 smithi130 ceph-mon[35398]: pgmap v268: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:19:33.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:33 smithi130 ceph-mon[35398]: pgmap v269: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-03-12T05:19:33.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:33 smithi116 ceph-mon[35486]: pgmap v269: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-03-12T05:19:34.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:33 smithi045 ceph-mon[32174]: pgmap v269: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 11 MiB/s wr, 1.19k op/s 2024-03-12T05:19:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 450.0 s, 1545.8 tps, lat 64.850 ms stddev 21.990 2024-03-12T05:19:36.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:36 smithi116 ceph-mon[35486]: pgmap v270: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:19:36.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:36 smithi045 ceph-mon[32174]: pgmap v270: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:19:36.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:36 smithi130 ceph-mon[35398]: pgmap v270: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 17 MiB/s wr, 1.91k op/s 2024-03-12T05:19:38.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:19:38.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:19:38.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f222affd640 1 -- 172.21.15.45:0/2822642817 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c032340 con 0x7f222c1166d0 2024-03-12T05:19:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 -- 172.21.15.45:0/2822642817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 msgr2=0x7f222c118ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 --2- 172.21.15.45:0/2822642817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c118ac0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f221c009860 tx=0x7f221c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 -- 172.21.15.45:0/2822642817 shutdown_connections 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 --2- 172.21.15.45:0/2822642817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c118ac0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 --2- 172.21.15.45:0/2822642817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 --2- 172.21.15.45:0/2822642817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 0x7f222c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.558+0000 7f22322e1640 1 -- 172.21.15.45:0/2822642817 >> 172.21.15.45:0/2822642817 conn(0x7f222c06c270 msgr2=0x7f222c06c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.559+0000 7f22322e1640 1 -- 172.21.15.45:0/2822642817 shutdown_connections 2024-03-12T05:19:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.559+0000 7f22322e1640 1 -- 172.21.15.45:0/2822642817 wait complete. 2024-03-12T05:19:38.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.559+0000 7f22322e1640 1 Processor -- start 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 -- start start 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 0x7f222c120f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c119680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c1217b0 con 0x7f222c108290 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c119bc0 con 0x7f222c108bb0 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.560+0000 7f22322e1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c119d50 con 0x7f222c1166d0 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35454/0 (socket says 172.21.15.45:35454) 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 -- 172.21.15.45:0/1866426722 learned_addr learned my addr 172.21.15.45:0/1866426722 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222bfff640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 0x7f222c120f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f223087e640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c119680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 -- 172.21.15.45:0/1866426722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 msgr2=0x7f222c119680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c119680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 -- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 msgr2=0x7f222c120f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 0x7f222c120f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 -- 172.21.15.45:0/1866426722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c009490 con 0x7f222c108bb0 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f223087e640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c119680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.561+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f222400efb0 tx=0x7f222400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.562+0000 7f22297fa640 1 -- 172.21.15.45:0/1866426722 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222400b570 con 0x7f222c108bb0 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.562+0000 7f22297fa640 1 -- 172.21.15.45:0/1866426722 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f222400bb80 con 0x7f222c108bb0 2024-03-12T05:19:38.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.563+0000 7f22297fa640 1 -- 172.21.15.45:0/1866426722 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224016cb0 con 0x7f222c108bb0 2024-03-12T05:19:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.563+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1bb690 con 0x7f222c108bb0 2024-03-12T05:19:38.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.563+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1bba00 con 0x7f222c108bb0 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.566+0000 7f2234f8b740 1 Processor -- start 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 -- start start 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f90ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936ef4280 con 0x55d936f8e7b0 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936ef17b0 con 0x55d936f85bd0 2024-03-12T05:19:38.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f2234f8b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936eecf90 con 0x55d936f852b0 2024-03-12T05:19:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f223087e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f223087e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35582/0 (socket says 172.21.15.45:35582) 2024-03-12T05:19:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f223087e640 1 -- 172.21.15.45:0/1851033714 learned_addr learned my addr 172.21.15.45:0/1851033714 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f222bfff640 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.567+0000 7f223087e640 1 -- 172.21.15.45:0/1851033714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 msgr2=0x55d936f8e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f223087e640 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f223087e640 1 -- 172.21.15.45:0/1851033714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 msgr2=0x55d936f90ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f223087e640 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f90ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f223087e640 1 -- 172.21.15.45:0/1851033714 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d936eab570 con 0x55d936f852b0 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f223087e640 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f221c02c0b0 tx=0x7f221c0324b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e71210b909dc6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.568+0000 7f22297fa640 1 -- 172.21.15.45:0/1866426722 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f222400a040 con 0x7f222c108bb0 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.569+0000 7f22127fc640 1 -- 172.21.15.45:0/1851033714 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c02dea0 con 0x55d936f852b0 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.569+0000 7f22127fc640 1 -- 172.21.15.45:0/1851033714 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f221c0044f0 con 0x55d936f852b0 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.569+0000 7f22127fc640 1 -- 172.21.15.45:0/1851033714 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0053c0 con 0x55d936f852b0 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.569+0000 7f22297fa640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 0x7f22140a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 -- 172.21.15.45:0/1851033714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 msgr2=0x55d936f85690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f221c02c0b0 tx=0x7f221c0324b0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 -- 172.21.15.45:0/1851033714 shutdown_connections 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f90ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1851033714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f852b0 0x55d936f85690 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 -- 172.21.15.45:0/1851033714 >> 172.21.15.45:0/1851033714 conn(0x55d936f7f990 msgr2=0x55d936f7fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f22297fa640 1 -- 172.21.15.45:0/1866426722 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f22240cda30 con 0x7f222c108bb0 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f222bfff640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 0x7f22140a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 -- 172.21.15.45:0/1851033714 shutdown_connections 2024-03-12T05:19:38.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.570+0000 7f2234f8b740 1 -- 172.21.15.45:0/1851033714 wait complete. 2024-03-12T05:19:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.571+0000 7f2234f8b740 1 -- start start 2024-03-12T05:19:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.571+0000 7f222bfff640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 0x7f22140a41b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55d936f84d80 tx=0x7f2220005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.571+0000 7f2234f8b740 1 Processor -- start 2024-03-12T05:19:38.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f852b0 0x55d936f8e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f8bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936bc0430 con 0x55d936f852b0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936b7ce00 con 0x55d936f85bd0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936e71fd0 con 0x55d936f8e7b0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f2234f8b740 10 client.0 _ll_register_callbacks cb 0x55d936ec7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f223087e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f8bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f222b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f222b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35484/0 (socket says 172.21.15.45:35484) 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f222b7fe640 1 -- 172.21.15.45:0/1131206679 learned_addr learned my addr 172.21.15.45:0/1131206679 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.572+0000 7f222bfff640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f852b0 0x55d936f8e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 -- 172.21.15.45:0/1131206679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 msgr2=0x55d936f8bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f8bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 -- 172.21.15.45:0/1131206679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f852b0 msgr2=0x55d936f8e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f852b0 0x55d936f8e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 -- 172.21.15.45:0/1131206679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c10fee0 con 0x55d936f85bd0 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f222b7fe640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f222409a2d0 tx=0x7f222409a020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be657357f35c2fed server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f220b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240a2d70 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2224096c40 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.573+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d936bec7e0 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2224096e20 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f2234f8b740 10 client.27158 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d936eeb610 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f22240a6020 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f222401d020 con 0x55d936f85bd0 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f2234f8b740 10 client.27158 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f2234f8b740 10 client.27158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.574+0000 7f2234f8b740 20 client.27158 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:38.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.575+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936fa0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.575+0000 7f2234f8b740 4 client.27158 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.575+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d936c29720 con 0x55d936f9e990 2024-03-12T05:19:38.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.575+0000 7f222bfff640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936fa0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.575+0000 7f220b7fe640 10 client.27158 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.576+0000 7f222bfff640 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936fa0db0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6ba183c4c3563bc server_cookie=71f2f78344e593a4 in_seq=0 out_seq=0 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.577+0000 7f220b7fe640 1 -- 172.21.15.45:0/1131206679 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d936c29720 con 0x55d936f9e990 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.577+0000 7f220b7fe640 10 client.27158 handle_command_reply: tid=0 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.577+0000 7f2234f8b740 1 client.27158 shutdown 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 msgr2=0x55d936f8b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f222409a2d0 tx=0x7f222409a020 comp rx=0 tx=0).stop 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 shutdown_connections 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936fa0db0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f8bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f220b7fe640 0 client.27158 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:38.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f85bd0 0x55d936f8b5e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 --2- 172.21.15.45:0/1131206679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f852b0 0x55d936f8e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.578+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 >> 172.21.15.45:0/1131206679 conn(0x55d936f7f990 msgr2=0x55d936f81730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.579+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 shutdown_connections 2024-03-12T05:19:38.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.579+0000 7f2234f8b740 1 -- 172.21.15.45:0/1131206679 wait complete. 2024-03-12T05:19:38.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.579+0000 7f2234f8b740 20 client.27158 trim_cache size 0 max 16384 2024-03-12T05:19:38.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 Processor -- start 2024-03-12T05:19:38.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 -- start start 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f85bd0 0x55d936f8e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f95660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936f84f10 con 0x55d936f8e7b0 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936f809e0 con 0x55d936f852b0 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.607+0000 7f2234f8b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936f86420 con 0x55d936f85bd0 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35488/0 (socket says 172.21.15.45:35488) 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 -- 172.21.15.45:0/3433644479 learned_addr learned my addr 172.21.15.45:0/3433644479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222b7fe640 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f85bd0 0x55d936f8e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f223087e640 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f95660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 -- 172.21.15.45:0/3433644479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f85bd0 msgr2=0x55d936f8e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f85bd0 0x55d936f8e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 -- 172.21.15.45:0/3433644479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 msgr2=0x55d936f95660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f95660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 -- 172.21.15.45:0/3433644479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d936e87ac0 con 0x55d936f852b0 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f222bfff640 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f22200042f0 tx=0x7f2220016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6497ef355354f3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.608+0000 7f2211ffb640 1 -- 172.21.15.45:0/3433644479 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220024040 con 0x55d936f852b0 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.609+0000 7f2211ffb640 1 -- 172.21.15.45:0/3433644479 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f22200044a0 con 0x55d936f852b0 2024-03-12T05:19:38.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.609+0000 7f2211ffb640 1 -- 172.21.15.45:0/3433644479 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200119e0 con 0x55d936f852b0 2024-03-12T05:19:38.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.609+0000 7f2234f8b740 1 -- 172.21.15.45:0/3433644479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 msgr2=0x55d936f8dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.609+0000 7f2234f8b740 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f22200042f0 tx=0x7f2220016000 comp rx=0 tx=0).stop 2024-03-12T05:19:38.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 -- 172.21.15.45:0/3433644479 shutdown_connections 2024-03-12T05:19:38.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f8e7b0 0x55d936f95660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f85bd0 0x55d936f8e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 --2- 172.21.15.45:0/3433644479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f8dd10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 -- 172.21.15.45:0/3433644479 >> 172.21.15.45:0/3433644479 conn(0x55d936f7f990 msgr2=0x55d936f868f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 -- 172.21.15.45:0/3433644479 shutdown_connections 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 -- 172.21.15.45:0/3433644479 wait complete. 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.610+0000 7f2234f8b740 1 -- start start 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 Processor -- start 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f85bd0 0x55d936f94680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f94bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936ff4ec0 con 0x55d936f85bd0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936ff5000 con 0x55d936f852b0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d936ff5170 con 0x55d936f8e7b0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f222bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f2234f8b740 10 client.0 _ll_register_callbacks cb 0x7f221c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f222bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35502/0 (socket says 172.21.15.45:35502) 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f222bfff640 1 -- 172.21.15.45:0/150837527 learned_addr learned my addr 172.21.15.45:0/150837527 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f222b7fe640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f85bd0 0x55d936f94680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.611+0000 7f223087e640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f94bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 -- 172.21.15.45:0/150837527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 msgr2=0x55d936f94bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f94bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 -- 172.21.15.45:0/150837527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f85bd0 msgr2=0x55d936f94680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f85bd0 0x55d936f94680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 -- 172.21.15.45:0/150837527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2220009d40 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f220b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f222bfff640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f22200095e0 tx=0x7f2220004160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbbbc5b7da78238a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220024070 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f221c0044f0 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.612+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2220029510 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22200296d0 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 10 client.27164 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d936ead6a0 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f221c0044f0 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2220032600 con 0x55d936f852b0 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 10 client.27164 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:38.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 10 client.27164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 20 client.27164 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936ff8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 4 client.27164 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d936c78890 con 0x55d936f9e990 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.613+0000 7f222b7fe640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936ff8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.614+0000 7f222b7fe640 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936ff8910 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e64f979f4daf2b0 server_cookie=dc259571de59fb42 in_seq=0 out_seq=0 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.614+0000 7f220b7fe640 10 client.27164 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.614+0000 7f220b7fe640 1 -- 172.21.15.45:0/150837527 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d936c78890 con 0x55d936f9e990 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.614+0000 7f220b7fe640 10 client.27164 handle_command_reply: tid=0 2024-03-12T05:19:38.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.614+0000 7f2234f8b740 1 client.27164 shutdown 2024-03-12T05:19:38.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 msgr2=0x55d936f94140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f22200095e0 tx=0x7f2220004160 comp rx=0 tx=0).stop 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 shutdown_connections 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d936f9e990 0x55d936ff8910 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f220b7fe640 0 client.27164 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d936f85bd0 0x55d936f94680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d936f8e7b0 0x55d936f94bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 --2- 172.21.15.45:0/150837527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d936f852b0 0x55d936f94140 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 >> 172.21.15.45:0/150837527 conn(0x55d936f7f990 msgr2=0x55d936f8ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 shutdown_connections 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 1 -- 172.21.15.45:0/150837527 wait complete. 2024-03-12T05:19:38.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.615+0000 7f2234f8b740 20 client.27164 trim_cache size 0 max 16384 2024-03-12T05:19:38.617 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:38.617 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:19:38.617 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:19:38.617 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.616+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 msgr2=0x7f22140a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.616+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 0x7f22140a41b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55d936f84d80 tx=0x7f2220005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.616+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 msgr2=0x7f222c119140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.616+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f222400efb0 tx=0x7f222400c800 comp rx=0 tx=0).stop 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.616+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 shutdown_connections 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22140a1cf0 0x7f22140a41b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f222c1166d0 0x7f222c119680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f222c108bb0 0x7f222c119140 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 --2- 172.21.15.45:0/1866426722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f222c108290 0x7f222c120f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 >> 172.21.15.45:0/1866426722 conn(0x7f222c06c270 msgr2=0x7f222c116d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 shutdown_connections 2024-03-12T05:19:38.619 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.617+0000 7f22322e1640 1 -- 172.21.15.45:0/1866426722 wait complete. 2024-03-12T05:19:38.632 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:19:38.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:19:38.666 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:38 smithi045 ceph-mon[32174]: pgmap v271: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:38.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:38 smithi116 ceph-mon[35486]: pgmap v271: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:38.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 -- 172.21.15.45:0/3336671204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c0ffc10 msgr2=0x7f646c10dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 --2- 172.21.15.45:0/3336671204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c0ffc10 0x7f646c10dfc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6464009860 tx=0x7f646402bb70 comp rx=0 tx=0).stop 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 -- 172.21.15.45:0/3336671204 shutdown_connections 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 --2- 172.21.15.45:0/3336671204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c0ffc10 0x7f646c10dfc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 --2- 172.21.15.45:0/3336671204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c0ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 --2- 172.21.15.45:0/3336671204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c107260 0x7f646c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 -- 172.21.15.45:0/3336671204 >> 172.21.15.45:0/3336671204 conn(0x7f646c0761e0 msgr2=0x7f646c0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 -- 172.21.15.45:0/3336671204 shutdown_connections 2024-03-12T05:19:38.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.803+0000 7f64747e2640 1 -- 172.21.15.45:0/3336671204 wait complete. 2024-03-12T05:19:38.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.804+0000 7f64747e2640 1 Processor -- start 2024-03-12T05:19:38.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.804+0000 7f64747e2640 1 -- start start 2024-03-12T05:19:38.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c107260 0x7f646c19bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c19c4a0 con 0x7f646c0ffc10 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c19c610 con 0x7f646c0ff250 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f64747e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c19c7d0 con 0x7f646c107260 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f6471d7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f6471d7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60644/0 (socket says 172.21.15.45:60644) 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f6471d7d640 1 -- 172.21.15.45:0/1912320189 learned_addr learned my addr 172.21.15.45:0/1912320189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f647257e640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.805+0000 7f6472d7f640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c107260 0x7f646c19bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 -- 172.21.15.45:0/1912320189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c107260 msgr2=0x7f646c19bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c107260 0x7f646c19bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 -- 172.21.15.45:0/1912320189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 msgr2=0x7f646c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 -- 172.21.15.45:0/1912320189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464009490 con 0x7f646c0ffc10 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f647257e640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.806+0000 7f6471d7d640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f645c00e9f0 tx=0x7f645c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.807+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c009720 con 0x7f646c0ffc10 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.807+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c19cab0 con 0x7f646c0ffc10 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.807+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c1a4330 con 0x7f646c0ffc10 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.807+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f645c00cd80 con 0x7f646c0ffc10 2024-03-12T05:19:38.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.807+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c010430 con 0x7f646c0ffc10 2024-03-12T05:19:38.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.809+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c0683c0 con 0x7f646c0ffc10 2024-03-12T05:19:38.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.811+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f645c005020 con 0x7f646c0ffc10 2024-03-12T05:19:38.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.813+0000 7f64637fe640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 0x7f64580a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:38.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.813+0000 7f647257e640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 0x7f64580a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:38.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.813+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f645c014070 con 0x7f646c0ffc10 2024-03-12T05:19:38.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.814+0000 7f647257e640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 0x7f64580a4060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6468005fd0 tx=0x7f6468010040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:38.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:38.820+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f645c0969f0 con 0x7f646c0ffc10 2024-03-12T05:19:38.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:38 smithi130 ceph-mon[35398]: pgmap v271: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:19:39.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.221+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f646c1096f0 con 0x7f646c0ffc10 2024-03-12T05:19:39.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.222+0000 7f64637fe640 1 -- 172.21.15.45:0/1912320189 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f646c1096f0 con 0x7f646c0ffc10 2024-03-12T05:19:39.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:19:39.219 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:19:39.219 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 msgr2=0x7f64580a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 0x7f64580a4060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6468005fd0 tx=0x7f6468010040 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 msgr2=0x7f646c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f645c00e9f0 tx=0x7f645c00edf0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 shutdown_connections 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64580a1ba0 0x7f64580a4060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c0ffc10 0x7f646c197b20 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f646c107260 0x7f646c19bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 --2- 172.21.15.45:0/1912320189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f646c0ff250 0x7f646c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 >> 172.21.15.45:0/1912320189 conn(0x7f646c0761e0 msgr2=0x7f646c0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.225+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 shutdown_connections 2024-03-12T05:19:39.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.226+0000 7f64747e2640 1 -- 172.21.15.45:0/1912320189 wait complete. 2024-03-12T05:19:39.241 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:39.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 -- 172.21.15.45:0/2749057436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400103d50 msgr2=0x7f9400107d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 --2- 172.21.15.45:0/2749057436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400103d50 0x7f9400107d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009930 tx=0x7f93f002bc20 comp rx=0 tx=0).stop 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 -- 172.21.15.45:0/2749057436 shutdown_connections 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 --2- 172.21.15.45:0/2749057436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9400108470 0x7f9400110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 --2- 172.21.15.45:0/2749057436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400103d50 0x7f9400107d90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 --2- 172.21.15.45:0/2749057436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 0x7f9400103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.457+0000 7f9407216640 1 -- 172.21.15.45:0/2749057436 >> 172.21.15.45:0/2749057436 conn(0x7f94000fec20 msgr2=0x7f9400101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.458+0000 7f9407216640 1 -- 172.21.15.45:0/2749057436 shutdown_connections 2024-03-12T05:19:39.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.458+0000 7f9407216640 1 -- 172.21.15.45:0/2749057436 wait complete. 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.458+0000 7f9407216640 1 Processor -- start 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 -- start start 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 0x7f9400199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f940019de50 0x7f940019e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940019e840 con 0x7f94001033a0 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940019e9b0 con 0x7f9400108470 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9407216640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940019eb20 con 0x7f940019de50 2024-03-12T05:19:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9406214640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 0x7f9400199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9405a13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9405a13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35536/0 (socket says 172.21.15.45:35536) 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9405a13640 1 -- 172.21.15.45:0/2850466937 learned_addr learned my addr 172.21.15.45:0/2850466937 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.459+0000 7f9406a15640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f940019de50 0x7f940019e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 -- 172.21.15.45:0/2850466937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f940019de50 msgr2=0x7f940019e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f940019de50 0x7f940019e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 -- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 msgr2=0x7f9400199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 0x7f9400199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 -- 172.21.15.45:0/2850466937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f0009560 con 0x7f9400108470 2024-03-12T05:19:39.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f9405a13640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f93f002c160 tx=0x7f93f00042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f93f77fe640 1 -- 172.21.15.45:0/2850466937 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00044e0 con 0x7f9400108470 2024-03-12T05:19:39.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f93f77fe640 1 -- 172.21.15.45:0/2850466937 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93f0005d50 con 0x7f9400108470 2024-03-12T05:19:39.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.460+0000 7f93f77fe640 1 -- 172.21.15.45:0/2850466937 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0005620 con 0x7f9400108470 2024-03-12T05:19:39.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.461+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94001a6100 con 0x7f9400108470 2024-03-12T05:19:39.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.461+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94001a64c0 con 0x7f9400108470 2024-03-12T05:19:39.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f93f77fe640 1 -- 172.21.15.45:0/2850466937 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f93f0004dc0 con 0x7f9400108470 2024-03-12T05:19:39.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 Processor -- start 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 -- start start 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482eed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55934829e960 con 0x5593482ec960 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559348252310 con 0x5593482e3d80 2024-03-12T05:19:39.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f940b045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55934824f840 con 0x5593482e3460 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9406a15640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9406214640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9406214640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35542/0 (socket says 172.21.15.45:35542) 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9406a15640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35646/0 (socket says 172.21.15.45:35646) 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9406214640 1 -- 172.21.15.45:0/372260375 learned_addr learned my addr 172.21.15.45:0/372260375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.464+0000 7f9405a13640 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f93f77fe640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 0x7f93ec0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 -- 172.21.15.45:0/372260375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 msgr2=0x5593482e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 -- 172.21.15.45:0/372260375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 msgr2=0x5593482eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482eed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 -- 172.21.15.45:0/372260375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55934820d060 con 0x5593482e3d80 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406a15640 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406a15640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 0x7f93ec0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f93f77fe640 1 -- 172.21.15.45:0/2850466937 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93f00efc70 con 0x7f9400108470 2024-03-12T05:19:39.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406214640 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00bcb0 tx=0x7f93fc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d67b3a1d0e6951d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f93f4bfe640 1 -- 172.21.15.45:0/372260375 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc01a460 con 0x5593482e3d80 2024-03-12T05:19:39.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f93f4bfe640 1 -- 172.21.15.45:0/372260375 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93fc01aa70 con 0x5593482e3d80 2024-03-12T05:19:39.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f93f4bfe640 1 -- 172.21.15.45:0/372260375 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011b10 con 0x5593482e3d80 2024-03-12T05:19:39.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.465+0000 7f9406a15640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 0x7f93ec0a40e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009fd0 tx=0x7f93f8004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 -- 172.21.15.45:0/372260375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 msgr2=0x5593482ec420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00bcb0 tx=0x7f93fc00f730 comp rx=0 tx=0).stop 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 -- 172.21.15.45:0/372260375 shutdown_connections 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482eed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482ec420 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 --2- 172.21.15.45:0/372260375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.466+0000 7f940b045740 1 -- 172.21.15.45:0/372260375 >> 172.21.15.45:0/372260375 conn(0x5593482ddb90 msgr2=0x5593482ddf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 -- 172.21.15.45:0/372260375 shutdown_connections 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 -- 172.21.15.45:0/372260375 wait complete. 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 -- start start 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 Processor -- start 2024-03-12T05:19:39.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482e9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.467+0000 7f940b045740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482ea430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f940b045740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55934824b020 con 0x5593482ec960 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f940b045740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559347f1e430 con 0x5593482e3d80 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f940b045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559347edae00 con 0x5593482e3460 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9406214640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9406214640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35658/0 (socket says 172.21.15.45:35658) 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9406214640 1 -- 172.21.15.45:0/29730597 learned_addr learned my addr 172.21.15.45:0/29730597 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482e9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9406a15640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f940b045740 10 client.0 _ll_register_callbacks cb 0x5593481f2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 -- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 msgr2=0x5593482e99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 -- 172.21.15.45:0/29730597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 msgr2=0x5593482ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482ea430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 -- 172.21.15.45:0/29730597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc00b8e0 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9406214640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f9405a13640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482e9ef0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f93f00ed440 tx=0x7f93f0005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245fa1a0f35ac426 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.468+0000 7f93e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00bd020 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93f00bc920 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559347f4a7e0 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f00c0a70 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 10 client.27182 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593482496a0 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93f00d9080 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93f003c070 con 0x5593482e3d80 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 10 client.27182 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 10 client.27182 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:19:39.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 20 client.27182 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5593482fc8c0 0x5593482fed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 4 client.27182 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.469+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559347f87720 con 0x5593482fc8c0 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.470+0000 7f9406214640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5593482fc8c0 0x5593482fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.470+0000 7f9406214640 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5593482fc8c0 0x5593482fed20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9f411e6fdb688b46 server_cookie=9684eef388b612d3 in_seq=0 out_seq=0 2024-03-12T05:19:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.470+0000 7f93e9ffb640 10 client.27182 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f93e9ffb640 1 -- 172.21.15.45:0/29730597 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559347f87720 con 0x5593482fc8c0 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f93e9ffb640 10 client.27182 handle_command_reply: tid=0 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 client.27182 shutdown 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 msgr2=0x5593482e9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482e9ef0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f93f00ed440 tx=0x7f93f0005bf0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 shutdown_connections 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5593482fc8c0 0x5593482fed20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482ec960 0x5593482ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x5593482e9ef0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f93e9ffb640 0 client.27182 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:39.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 --2- 172.21.15.45:0/29730597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482e99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.471+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 >> 172.21.15.45:0/29730597 conn(0x5593482ddb90 msgr2=0x5593482df8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.472+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 shutdown_connections 2024-03-12T05:19:39.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.472+0000 7f940b045740 1 -- 172.21.15.45:0/29730597 wait complete. 2024-03-12T05:19:39.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.472+0000 7f940b045740 20 client.27182 trim_cache size 0 max 16384 2024-03-12T05:19:39.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 Processor -- start 2024-03-12T05:19:39.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 -- start start 2024-03-12T05:19:39.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482ec960 0x5593482f3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593482d9340 con 0x5593482e3d80 2024-03-12T05:19:39.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593481d0130 con 0x5593482ec960 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.496+0000 7f940b045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593482debc0 con 0x5593482e3460 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9405a13640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9405a13640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60696/0 (socket says 172.21.15.45:60696) 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35670/0 (socket says 172.21.15.45:35670) 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9405a13640 1 -- 172.21.15.45:0/3751141079 learned_addr learned my addr 172.21.15.45:0/3751141079 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406a15640 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482ec960 0x5593482f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 -- 172.21.15.45:0/3751141079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482ec960 msgr2=0x5593482f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482ec960 0x5593482f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 -- 172.21.15.45:0/3751141079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 msgr2=0x5593482dcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9406214640 1 -- 172.21.15.45:0/3751141079 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593481e5c20 con 0x5593482e3460 2024-03-12T05:19:39.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.497+0000 7f9405a13640 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.498+0000 7f9406214640 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0117f0 tx=0x7f93fc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a61eb25bba2c7b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.498+0000 7f93ebfff640 1 -- 172.21.15.45:0/3751141079 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc02c070 con 0x5593482e3460 2024-03-12T05:19:39.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.498+0000 7f93ebfff640 1 -- 172.21.15.45:0/3751141079 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93fc01f040 con 0x5593482e3460 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.498+0000 7f93ebfff640 1 -- 172.21.15.45:0/3751141079 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc01e040 con 0x5593482e3460 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- 172.21.15.45:0/3751141079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 msgr2=0x5593482ea250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0117f0 tx=0x7f93fc005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- 172.21.15.45:0/3751141079 shutdown_connections 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3d80 0x5593482dcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482ec960 0x5593482f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 --2- 172.21.15.45:0/3751141079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482e3460 0x5593482ea250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- 172.21.15.45:0/3751141079 >> 172.21.15.45:0/3751141079 conn(0x5593482ddb90 msgr2=0x5593482dc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- 172.21.15.45:0/3751141079 shutdown_connections 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- 172.21.15.45:0/3751141079 wait complete. 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 -- start start 2024-03-12T05:19:39.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.499+0000 7f940b045740 1 Processor -- start 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482ec960 0x5593482f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593482f3230 con 0x5593482e3460 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593482f33a0 con 0x5593482e3d80 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593482f3560 con 0x5593482ec960 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9406214640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f940b045740 10 client.0 _ll_register_callbacks cb 0x7f93fc00b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9406214640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60704/0 (socket says 172.21.15.45:60704) 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35564/0 (socket says 172.21.15.45:35564) 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9406214640 1 -- 172.21.15.45:0/2711712989 learned_addr learned my addr 172.21.15.45:0/2711712989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9406a15640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482ec960 0x5593482f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 -- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482ec960 msgr2=0x5593482f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482ec960 0x5593482f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 -- 172.21.15.45:0/2711712989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 msgr2=0x559348354af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.500+0000 7f9405a13640 1 -- 172.21.15.45:0/2711712989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f00f3020 con 0x5593482e3d80 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f9406214640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f9405a13640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f93f002c200 tx=0x7f93f00ed8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=452f66f7c165b79b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f93e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00c4e30 con 0x5593482e3d80 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93f00bd020 con 0x5593482e3d80 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93fc01aa70 con 0x5593482e3d80 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f00bc920 con 0x5593482e3d80 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f940b045740 10 client.27188 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.501+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55934820d060 con 0x5593482e3d80 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93f00d9070 con 0x5593482e3d80 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93f003c040 con 0x5593482e3d80 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 10 client.27188 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 10 client.27188 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 20 client.27188 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55934835d580 0x55934835f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 4 client.27188 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f93fc01aa70 con 0x55934835d580 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.502+0000 7f9406214640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55934835d580 0x55934835f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f93e9ffb640 10 client.27188 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f9406214640 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55934835d580 0x55934835f970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fd6c542bb066aece server_cookie=4b8933e60b390690 in_seq=0 out_seq=0 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f93e9ffb640 1 -- 172.21.15.45:0/2711712989 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f93fc01aa70 con 0x55934835d580 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f93e9ffb640 10 client.27188 handle_command_reply: tid=0 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f940b045740 1 client.27188 shutdown 2024-03-12T05:19:39.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 msgr2=0x559348355030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f93f002c200 tx=0x7f93f00ed8b0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 shutdown_connections 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55934835d580 0x55934835f970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.504+0000 7f93e9ffb640 0 client.27188 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593482ec960 0x5593482f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593482e3d80 0x559348355030 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 --2- 172.21.15.45:0/2711712989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5593482e3460 0x559348354af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 >> 172.21.15.45:0/2711712989 conn(0x5593482ddb90 msgr2=0x5593482dccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 shutdown_connections 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 1 -- 172.21.15.45:0/2711712989 wait complete. 2024-03-12T05:19:39.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.505+0000 7f940b045740 20 client.27188 trim_cache size 0 max 16384 2024-03-12T05:19:39.505 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:39.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.506+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 msgr2=0x7f93ec0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.506+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 0x7f93ec0a40e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009fd0 tx=0x7f93f8004ab0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 msgr2=0x7f9400199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f93f002c160 tx=0x7f93f00042f0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 shutdown_connections 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93ec0a1c20 0x7f93ec0a40e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f940019de50 0x7f940019e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9400108470 0x7f9400199ad0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 --2- 172.21.15.45:0/2850466937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94001033a0 0x7f9400199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.507+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 >> 172.21.15.45:0/2850466937 conn(0x7f94000fec20 msgr2=0x7f94000ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.508+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 shutdown_connections 2024-03-12T05:19:39.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.508+0000 7f9407216640 1 -- 172.21.15.45:0/2850466937 wait complete. 2024-03-12T05:19:39.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:39 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1912320189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:39.524 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:39.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:19:39.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.695+0000 7fe3db9ac640 1 -- 172.21.15.45:0/3099480321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 msgr2=0x7fe3d41143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/3099480321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d41143b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009860 tx=0x7fe3c402bb70 comp rx=0 tx=0).stop 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 -- 172.21.15.45:0/3099480321 shutdown_connections 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/3099480321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d41143b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/3099480321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d4075fd0 0x7fe3d4111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/3099480321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d40756b0 0x7fe3d4075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 -- 172.21.15.45:0/3099480321 >> 172.21.15.45:0/3099480321 conn(0x7fe3d40fedd0 msgr2=0x7fe3d41011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 -- 172.21.15.45:0/3099480321 shutdown_connections 2024-03-12T05:19:39.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.696+0000 7fe3db9ac640 1 -- 172.21.15.45:0/3099480321 wait complete. 2024-03-12T05:19:39.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 Processor -- start 2024-03-12T05:19:39.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 -- start start 2024-03-12T05:19:39.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 0x7fe3d41a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d419e140 con 0x7fe3d4111fc0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d419e2b0 con 0x7fe3d40756b0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3db9ac640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d419e470 con 0x7fe3d4075fd0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.697+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35584/0 (socket says 172.21.15.45:35584) 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 -- 172.21.15.45:0/686237093 learned_addr learned my addr 172.21.15.45:0/686237093 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 -- 172.21.15.45:0/686237093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 msgr2=0x7fe3d41a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 0x7fe3d41a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 0x7fe3d41a3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 -- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 msgr2=0x7fe3d419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 -- 172.21.15.45:0/686237093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4009490 con 0x7fe3d40756b0 2024-03-12T05:19:39.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 0x7fe3d41a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:39.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.698+0000 7fe3d9748640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c420 tx=0x7fe3cc00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.699+0000 7fe3ca7fc640 1 -- 172.21.15.45:0/686237093 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc009b50 con 0x7fe3d40756b0 2024-03-12T05:19:39.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.699+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d419e750 con 0x7fe3d40756b0 2024-03-12T05:19:39.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.699+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d41aa990 con 0x7fe3d40756b0 2024-03-12T05:19:39.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.699+0000 7fe3ca7fc640 1 -- 172.21.15.45:0/686237093 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3cc003970 con 0x7fe3d40756b0 2024-03-12T05:19:39.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.699+0000 7fe3ca7fc640 1 -- 172.21.15.45:0/686237093 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0047c0 con 0x7fe3d40756b0 2024-03-12T05:19:39.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.702+0000 7fe3ca7fc640 1 -- 172.21.15.45:0/686237093 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe3cc00a040 con 0x7fe3d40756b0 2024-03-12T05:19:39.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.703+0000 7fe3ca7fc640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 0x7fe3c00a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.704+0000 7fe3ca7fc640 1 -- 172.21.15.45:0/686237093 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe3cc0cd830 con 0x7fe3d40756b0 2024-03-12T05:19:39.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.704+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 0x7fe3c00a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.705+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 0x7fe3c00a4160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4078240 tx=0x7fe3d0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.705+0000 7fe3de580740 1 Processor -- start 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.705+0000 7fe3de580740 1 -- start start 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cb7c960 con 0x562b0cbc1250 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cb30310 con 0x562b0cbca750 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3de580740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cb2d840 con 0x562b0cbc1b70 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d9f49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d9f49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35586/0 (socket says 172.21.15.45:35586) 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d9f49640 1 -- 172.21.15.45:0/3168239131 learned_addr learned my addr 172.21.15.45:0/3168239131 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d9748640 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 -- 172.21.15.45:0/3168239131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 msgr2=0x562b0cbccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 -- 172.21.15.45:0/3168239131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 msgr2=0x562b0cbc1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.706+0000 7fe3d8f47640 1 -- 172.21.15.45:0/3168239131 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b0caeb060 con 0x562b0cbc1b70 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.707+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.707+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbca210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0010da0 tx=0x7fe3d0013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e95964e6327ff37 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.708+0000 7fe3bf7fe640 1 -- 172.21.15.45:0/3168239131 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0004030 con 0x562b0cbc1b70 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.708+0000 7fe3bf7fe640 1 -- 172.21.15.45:0/3168239131 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3d0003610 con 0x562b0cbc1b70 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.708+0000 7fe3bf7fe640 1 -- 172.21.15.45:0/3168239131 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d001f690 con 0x562b0cbc1b70 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 -- 172.21.15.45:0/3168239131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 msgr2=0x562b0cbca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbca210 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0010da0 tx=0x7fe3d0013000 comp rx=0 tx=0).stop 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 -- 172.21.15.45:0/3168239131 shutdown_connections 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbca210 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 --2- 172.21.15.45:0/3168239131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 -- 172.21.15.45:0/3168239131 >> 172.21.15.45:0/3168239131 conn(0x562b0cbbb980 msgr2=0x562b0cbbbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 -- 172.21.15.45:0/3168239131 shutdown_connections 2024-03-12T05:19:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.709+0000 7fe3de580740 1 -- 172.21.15.45:0/3168239131 wait complete. 2024-03-12T05:19:39.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.710+0000 7fe3de580740 1 -- start start 2024-03-12T05:19:39.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.710+0000 7fe3de580740 1 Processor -- start 2024-03-12T05:19:39.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.710+0000 7fe3de580740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.710+0000 7fe3de580740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.710+0000 7fe3de580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbc81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3de580740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cb29020 con 0x562b0cbc1250 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3de580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0c7fc430 con 0x562b0cbca750 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3de580740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0c7b8e00 con 0x562b0cbc1b70 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3de580740 10 client.0 _ll_register_callbacks cb 0x562b0cad0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d8f47640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d8f47640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35708/0 (socket says 172.21.15.45:35708) 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d8f47640 1 -- 172.21.15.45:0/77496792 learned_addr learned my addr 172.21.15.45:0/77496792 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9748640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbc81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 -- 172.21.15.45:0/77496792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 msgr2=0x562b0cbc7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 -- 172.21.15.45:0/77496792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 msgr2=0x562b0cbc7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9f49640 1 -- 172.21.15.45:0/77496792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d0010a80 con 0x562b0cbca750 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.711+0000 7fe3d9748640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbc81c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009830 tx=0x7fe3c4004300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58c1f796e9fb128 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4034850 con 0x562b0cbca750 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe3c402c510 con 0x562b0cbca750 2024-03-12T05:19:39.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b0c8287e0 con 0x562b0cbca750 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.712+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3c402c6d0 con 0x562b0cbca750 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3de580740 10 client.27200 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562b0cb276a0 con 0x562b0cbca750 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3c404e030 con 0x562b0cbca750 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe3c4045020 con 0x562b0cbca750 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3de580740 10 client.27200 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3de580740 10 client.27200 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.713+0000 7fe3de580740 20 client.27200 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.714+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cbdac30 0x562b0cbdd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.714+0000 7fe3de580740 4 client.27200 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.714+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b0c865720 con 0x562b0cbdac30 2024-03-12T05:19:39.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.714+0000 7fe3d9748640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cbdac30 0x562b0cbdd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.715+0000 7fe3bcff9640 10 client.27200 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:39.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.715+0000 7fe3d9748640 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cbdac30 0x562b0cbdd050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d2225cdc5397236 server_cookie=2d7b0b4c9c5a1b90 in_seq=0 out_seq=0 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.716+0000 7fe3bcff9640 1 -- 172.21.15.45:0/77496792 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b0c865720 con 0x562b0cbdac30 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.716+0000 7fe3bcff9640 10 client.27200 handle_command_reply: tid=0 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.716+0000 7fe3de580740 1 client.27200 shutdown 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.716+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 msgr2=0x562b0cbc81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.716+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbc81c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009830 tx=0x7fe3c4004300 comp rx=0 tx=0).stop 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 shutdown_connections 2024-03-12T05:19:39.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cbdac30 0x562b0cbdd050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbca750 0x562b0cbc81c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 --2- 172.21.15.45:0/77496792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbc1250 0x562b0cbc7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 >> 172.21.15.45:0/77496792 conn(0x562b0cbbb980 msgr2=0x562b0cbbd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3bcff9640 0 client.27200 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 shutdown_connections 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 1 -- 172.21.15.45:0/77496792 wait complete. 2024-03-12T05:19:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.717+0000 7fe3de580740 20 client.27200 trim_cache size 0 max 16384 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.737+0000 7fe3de580740 1 Processor -- start 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.737+0000 7fe3de580740 1 -- start start 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cbc4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0caae130 con 0x562b0cbca750 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cbc0c90 con 0x562b0cbc1250 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3de580740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cbcd420 con 0x562b0cbc1b70 2024-03-12T05:19:39.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35602/0 (socket says 172.21.15.45:35602) 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 -- 172.21.15.45:0/2001859036 learned_addr learned my addr 172.21.15.45:0/2001859036 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cbc4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 -- 172.21.15.45:0/2001859036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 msgr2=0x562b0cbc49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 -- 172.21.15.45:0/2001859036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 msgr2=0x562b0cbc4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cbc4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.738+0000 7fe3d9748640 1 -- 172.21.15.45:0/2001859036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b0cac3c20 con 0x562b0cbc1250 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.739+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc096da0 tx=0x7fe3cc0cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ae8c5d0c4af068 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.739+0000 7fe3beffd640 1 -- 172.21.15.45:0/2001859036 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0a2b70 con 0x562b0cbc1250 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.739+0000 7fe3beffd640 1 -- 172.21.15.45:0/2001859036 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3cc0986d0 con 0x562b0cbc1250 2024-03-12T05:19:39.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.739+0000 7fe3beffd640 1 -- 172.21.15.45:0/2001859036 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0a26e0 con 0x562b0cbc1250 2024-03-12T05:19:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.740+0000 7fe3de580740 1 -- 172.21.15.45:0/2001859036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 msgr2=0x562b0cbc9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.740+0000 7fe3de580740 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc096da0 tx=0x7fe3cc0cd040 comp rx=0 tx=0).stop 2024-03-12T05:19:39.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 -- 172.21.15.45:0/2001859036 shutdown_connections 2024-03-12T05:19:39.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cbc4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1b70 0x562b0cbc49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 --2- 172.21.15.45:0/2001859036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1250 0x562b0cbc9e70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 -- 172.21.15.45:0/2001859036 >> 172.21.15.45:0/2001859036 conn(0x562b0cbbb980 msgr2=0x562b0cbc2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 -- 172.21.15.45:0/2001859036 shutdown_connections 2024-03-12T05:19:39.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 -- 172.21.15.45:0/2001859036 wait complete. 2024-03-12T05:19:39.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 -- start start 2024-03-12T05:19:39.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.744+0000 7fe3de580740 1 Processor -- start 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cc2b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3d9748640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35718/0 (socket says 172.21.15.45:35718) 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3d8f47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3d8f47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35604/0 (socket says 172.21.15.45:35604) 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cc2b6c0 con 0x562b0cbca750 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cc2b830 con 0x562b0cbc1b70 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b0cc2b9f0 con 0x562b0cbc1250 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3de580740 10 client.0 _ll_register_callbacks cb 0x7fe3d0016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.745+0000 7fe3d9748640 1 -- 172.21.15.45:0/2855649455 learned_addr learned my addr 172.21.15.45:0/2855649455 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cc2b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 -- 172.21.15.45:0/2855649455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 msgr2=0x562b0cbd09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 -- 172.21.15.45:0/2855649455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 msgr2=0x562b0cc2b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cc2b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d41aa990 con 0x562b0cbc1b70 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d9f49640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cc2b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3d8f47640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0012cb0 tx=0x7fe3d0005050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c38273ea450998ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0030070 con 0x562b0cbc1b70 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe3d0005490 con 0x562b0cbc1b70 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3d0003610 con 0x562b0cbc1b70 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.746+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3d0003780 con 0x562b0cbc1b70 2024-03-12T05:19:39.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 10 client.27206 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562b0cae98c0 con 0x562b0cbc1b70 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe3d0003d90 con 0x562b0cbc1b70 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe3d0028b00 con 0x562b0cbc1b70 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 10 client.27206 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 10 client.27206 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 20 client.27206 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cc33230 0x562b0cc35650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 4 client.27206 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.747+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fe3d0003610 con 0x562b0cc33230 2024-03-12T05:19:39.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cc33230 0x562b0cc35650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.747 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:39 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1912320189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3d9748640 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cc33230 0x562b0cc35650 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19be217ebc1a07c5 server_cookie=3fa8b27eba1b5a4f in_seq=0 out_seq=0 2024-03-12T05:19:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3bcff9640 10 client.27206 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3bcff9640 1 -- 172.21.15.45:0/2855649455 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fe3d0003610 con 0x562b0cc33230 2024-03-12T05:19:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3bcff9640 10 client.27206 handle_command_reply: tid=0 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3de580740 1 client.27206 shutdown 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 msgr2=0x562b0cbd0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.748+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0012cb0 tx=0x7fe3d0005050 comp rx=0 tx=0).stop 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 shutdown_connections 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x562b0cc33230 0x562b0cc35650 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3bcff9640 0 client.27206 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562b0cbca750 0x562b0cc2b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562b0cbc1b70 0x562b0cbd0f30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 --2- 172.21.15.45:0/2855649455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b0cbc1250 0x562b0cbd09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 >> 172.21.15.45:0/2855649455 conn(0x562b0cbbb980 msgr2=0x562b0cbc76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 shutdown_connections 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 1 -- 172.21.15.45:0/2855649455 wait complete. 2024-03-12T05:19:39.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.749+0000 7fe3de580740 20 client.27206 trim_cache size 0 max 16384 2024-03-12T05:19:39.750 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:39.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 msgr2=0x7fe3c00a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 0x7fe3c00a4160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4078240 tx=0x7fe3d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:19:39.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 msgr2=0x7fe3d41a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c420 tx=0x7fe3cc00c8e0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 shutdown_connections 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe3c00a1ca0 0x7fe3c00a4160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3d4111fc0 0x7fe3d419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3d4075fd0 0x7fe3d41a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 --2- 172.21.15.45:0/686237093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3d40756b0 0x7fe3d41a38f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 >> 172.21.15.45:0/686237093 conn(0x7fe3d40fedd0 msgr2=0x7fe3d41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.750+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 shutdown_connections 2024-03-12T05:19:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.751+0000 7fe3db9ac640 1 -- 172.21.15.45:0/686237093 wait complete. 2024-03-12T05:19:39.763 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:39.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:19:39.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 -- 172.21.15.45:0/3828911100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab60101d30 msgr2=0x7fab60105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 --2- 172.21.15.45:0/3828911100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab60101d30 0x7fab60105d70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fab50009930 tx=0x7fab5002bc30 comp rx=0 tx=0).stop 2024-03-12T05:19:39.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 -- 172.21.15.45:0/3828911100 shutdown_connections 2024-03-12T05:19:39.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 --2- 172.21.15.45:0/3828911100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab60101d30 0x7fab60105d70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 --2- 172.21.15.45:0/3828911100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60106450 0x7fab6010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 --2- 172.21.15.45:0/3828911100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60101380 0x7fab60101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 -- 172.21.15.45:0/3828911100 >> 172.21.15.45:0/3828911100 conn(0x7fab60078580 msgr2=0x7fab60078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 -- 172.21.15.45:0/3828911100 shutdown_connections 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.944+0000 7fab658b5640 1 -- 172.21.15.45:0/3828911100 wait complete. 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 Processor -- start 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 -- start start 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60101380 0x7fab6010dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60106450 0x7fab6010af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601aa290 con 0x7fab6010b4a0 2024-03-12T05:19:39.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601aa400 con 0x7fab60106450 2024-03-12T05:19:39.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab658b5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601aa5c0 con 0x7fab60101380 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab5effd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60101380 0x7fab6010dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60770/0 (socket says 172.21.15.45:60770) 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.945+0000 7fab5f7fe640 1 -- 172.21.15.45:0/605491462 learned_addr learned my addr 172.21.15.45:0/605491462 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab57fff640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60106450 0x7fab6010af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 -- 172.21.15.45:0/605491462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60101380 msgr2=0x7fab6010dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60101380 0x7fab6010dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 -- 172.21.15.45:0/605491462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60106450 msgr2=0x7fab6010af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60106450 0x7fab6010af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 -- 172.21.15.45:0/605491462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab50009560 con 0x7fab6010b4a0 2024-03-12T05:19:39.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab5f7fe640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fab5800eb10 tx=0x7fab5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.946+0000 7fab648b3640 1 -- 172.21.15.45:0/605491462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58004b20 con 0x7fab6010b4a0 2024-03-12T05:19:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.947+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab601aa8a0 con 0x7fab6010b4a0 2024-03-12T05:19:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.947+0000 7fab648b3640 1 -- 172.21.15.45:0/605491462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab58004c80 con 0x7fab6010b4a0 2024-03-12T05:19:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.947+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab601aad80 con 0x7fab6010b4a0 2024-03-12T05:19:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.947+0000 7fab648b3640 1 -- 172.21.15.45:0/605491462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58010690 con 0x7fab6010b4a0 2024-03-12T05:19:39.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab648b3640 1 -- 172.21.15.45:0/605491462 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fab580107f0 con 0x7fab6010b4a0 2024-03-12T05:19:39.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 Processor -- start 2024-03-12T05:19:39.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 -- start start 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d787c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787cb70 0x5592d7885210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d7887b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d7837960 con 0x5592d787c250 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d77eb310 con 0x5592d7885750 2024-03-12T05:19:39.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.950+0000 7fab68563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d77e8840 con 0x5592d787cb70 2024-03-12T05:19:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.951+0000 7fab648b3640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 0x7fab4c0a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.951+0000 7fab57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d7887b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.951+0000 7fab57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d787c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.951+0000 7fab57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d787c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60778/0 (socket says 172.21.15.45:60778) 2024-03-12T05:19:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.951+0000 7fab57fff640 1 -- 172.21.15.45:0/1622428512 learned_addr learned my addr 172.21.15.45:0/1622428512 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab648b3640 1 -- 172.21.15.45:0/605491462 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab58014070 con 0x7fab6010b4a0 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab5effd640 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787cb70 0x5592d7885210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab57fff640 1 -- 172.21.15.45:0/1622428512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787cb70 msgr2=0x5592d7885210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab57fff640 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787cb70 0x5592d7885210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab57fff640 1 -- 172.21.15.45:0/1622428512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 msgr2=0x5592d787c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab57fff640 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d787c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab57fff640 1 -- 172.21.15.45:0/1622428512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592d77a6060 con 0x5592d7885750 2024-03-12T05:19:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.952+0000 7fab5f7fe640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 0x7fab4c0a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 455.0 s, 1567.9 tps, lat 63.455 ms stddev 23.361 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.953+0000 7fab57fff640 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d7887b40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fab6006aa20 tx=0x7fab50005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c528c750dc3513 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.953+0000 7fab5f7fe640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 0x7fab4c0a4110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab580970e0 tx=0x7fab580cdc10 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.953+0000 7fab55ffb640 1 -- 172.21.15.45:0/1622428512 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50005d50 con 0x5592d7885750 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.953+0000 7fab55ffb640 1 -- 172.21.15.45:0/1622428512 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab50004040 con 0x5592d7885750 2024-03-12T05:19:39.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.953+0000 7fab55ffb640 1 -- 172.21.15.45:0/1622428512 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50003b80 con 0x5592d7885750 2024-03-12T05:19:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.954+0000 7fab68563740 1 -- 172.21.15.45:0/1622428512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 msgr2=0x5592d7887b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.954+0000 7fab68563740 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d7887b40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fab6006aa20 tx=0x7fab50005fd0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 -- 172.21.15.45:0/1622428512 shutdown_connections 2024-03-12T05:19:39.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d7887b40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787cb70 0x5592d7885210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 --2- 172.21.15.45:0/1622428512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d787c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 -- 172.21.15.45:0/1622428512 >> 172.21.15.45:0/1622428512 conn(0x5592d7876980 msgr2=0x5592d7876d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 -- 172.21.15.45:0/1622428512 shutdown_connections 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.955+0000 7fab68563740 1 -- 172.21.15.45:0/1622428512 wait complete. 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 -- start start 2024-03-12T05:19:39.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 Processor -- start 2024-03-12T05:19:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d7882d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d7883250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d77e4020 con 0x5592d787cb70 2024-03-12T05:19:39.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d74b7430 con 0x5592d787c250 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d7473e00 con 0x5592d7885750 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.956+0000 7fab68563740 10 client.0 _ll_register_callbacks cb 0x5592d778bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35622/0 (socket says 172.21.15.45:35622) 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 -- 172.21.15.45:0/4278034630 learned_addr learned my addr 172.21.15.45:0/4278034630 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d7883250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab5effd640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d7882d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 -- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 msgr2=0x5592d7883250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d7883250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 -- 172.21.15.45:0/4278034630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 msgr2=0x5592d7882d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d7882d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.957+0000 7fab57fff640 1 -- 172.21.15.45:0/4278034630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6010e830 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab57fff640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fab5002c240 tx=0x7fab50004b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae84f78c7d66c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab3effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab5effd640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d7882d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50003ba0 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab5002de40 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.958+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592d74e37e0 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab5002cae0 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 10 client.27218 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592d77e26a0 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5592d74e37e0 con 0x5592d787c250 2024-03-12T05:19:39.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab5004b020 con 0x5592d787c250 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 10 client.27218 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 10 client.27218 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 20 client.27218 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d7895860 0x5592d7897c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 4 client.27218 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:39.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.959+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592d7520720 con 0x5592d7895860 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.960+0000 7fab5effd640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d7895860 0x5592d7897c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.960+0000 7fab5effd640 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d7895860 0x5592d7897c80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b33b903fca0b378a server_cookie=6474345980629da8 in_seq=0 out_seq=0 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.960+0000 7fab3effd640 10 client.27218 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.961+0000 7fab3effd640 1 -- 172.21.15.45:0/4278034630 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5592d7520720 con 0x5592d7895860 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.961+0000 7fab3effd640 10 client.27218 handle_command_reply: tid=0 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.961+0000 7fab68563740 1 client.27218 shutdown 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 msgr2=0x5592d78827d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fab5002c240 tx=0x7fab50004b80 comp rx=0 tx=0).stop 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 shutdown_connections 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d7895860 0x5592d7897c80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d7882d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d7883250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 --2- 172.21.15.45:0/4278034630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787c250 0x5592d78827d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 >> 172.21.15.45:0/4278034630 conn(0x5592d7876980 msgr2=0x5592d78786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab3effd640 0 client.27218 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:39.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 shutdown_connections 2024-03-12T05:19:39.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 1 -- 172.21.15.45:0/4278034630 wait complete. 2024-03-12T05:19:39.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.962+0000 7fab68563740 20 client.27218 trim_cache size 0 max 16384 2024-03-12T05:19:39.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:39 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1912320189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:19:39.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.981+0000 7fab68563740 1 Processor -- start 2024-03-12T05:19:39.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.981+0000 7fab68563740 1 -- start start 2024-03-12T05:19:39.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787cb70 0x5592d7883070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60794/0 (socket says 172.21.15.45:60794) 2024-03-12T05:19:39.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d787fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d7769130 con 0x5592d787c250 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d78725c0 con 0x5592d787cb70 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab68563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d7888390 con 0x5592d7885750 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab57fff640 1 -- 172.21.15.45:0/1446163817 learned_addr learned my addr 172.21.15.45:0/1446163817 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.982+0000 7fab57fff640 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d787fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 -- 172.21.15.45:0/1446163817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 msgr2=0x5592d787fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d787fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 -- 172.21.15.45:0/1446163817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787cb70 msgr2=0x5592d7883070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787cb70 0x5592d7883070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 -- 172.21.15.45:0/1446163817 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592d777ec20 con 0x5592d787c250 2024-03-12T05:19:39.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.983+0000 7fab57fff640 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fab50043d50 tx=0x7fab50005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18399a047aa9f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.984+0000 7fab557fa640 1 -- 172.21.15.45:0/1446163817 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5002cae0 con 0x5592d787c250 2024-03-12T05:19:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.984+0000 7fab557fa640 1 -- 172.21.15.45:0/1446163817 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab50052070 con 0x5592d787c250 2024-03-12T05:19:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.984+0000 7fab557fa640 1 -- 172.21.15.45:0/1446163817 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50043440 con 0x5592d787c250 2024-03-12T05:19:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.985+0000 7fab68563740 1 -- 172.21.15.45:0/1446163817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 msgr2=0x5592d7884f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.985+0000 7fab68563740 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fab50043d50 tx=0x7fab50005fd0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 -- 172.21.15.45:0/1446163817 shutdown_connections 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d7885750 0x5592d787fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d787cb70 0x5592d7883070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 --2- 172.21.15.45:0/1446163817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787c250 0x5592d7884f00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 -- 172.21.15.45:0/1446163817 >> 172.21.15.45:0/1446163817 conn(0x5592d7876980 msgr2=0x5592d787dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 -- 172.21.15.45:0/1446163817 shutdown_connections 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.986+0000 7fab68563740 1 -- 172.21.15.45:0/1446163817 wait complete. 2024-03-12T05:19:39.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.987+0000 7fab68563740 1 -- start start 2024-03-12T05:19:39.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.987+0000 7fab68563740 1 Processor -- start 2024-03-12T05:19:39.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.987+0000 7fab68563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d788ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d78ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d788b450 con 0x5592d787cb70 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d78eb280 con 0x5592d7885750 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592d78eb3c0 con 0x5592d787c250 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab68563740 10 client.0 _ll_register_callbacks cb 0x7fab50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab5effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d788ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35780/0 (socket says 172.21.15.45:35780) 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 -- 172.21.15.45:0/2438422045 learned_addr learned my addr 172.21.15.45:0/2438422045 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d78ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 -- 172.21.15.45:0/2438422045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 msgr2=0x5592d78ead40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d78ead40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 -- 172.21.15.45:0/2438422045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 msgr2=0x5592d788ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d788ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 -- 172.21.15.45:0/2438422045 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592d77e26a0 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab57fff640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab5002c170 tx=0x7fab50000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc1bc177341cb09 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.988+0000 7fab3effd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500437f0 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab50043950 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab50004040 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab500046c0 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab68563740 10 client.27205 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592d77a48c0 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.989+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab50076020 con 0x5592d787c250 2024-03-12T05:19:39.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab5004b070 con 0x5592d787c250 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 10 client.27205 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 10 client.27205 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 20 client.27205 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d78ec5e0 0x5592d78ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 4 client.27205 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592d756f890 con 0x5592d78ec5e0 2024-03-12T05:19:39.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.990+0000 7fab5effd640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d78ec5e0 0x5592d78ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab5effd640 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d78ec5e0 0x5592d78ee9d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8393dce0b2f11b4 server_cookie=699d04b560cdf18b in_seq=0 out_seq=0 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab3effd640 10 client.27205 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab3effd640 1 -- 172.21.15.45:0/2438422045 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592d756f890 con 0x5592d78ec5e0 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab3effd640 10 client.27205 handle_command_reply: tid=0 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab68563740 1 client.27205 shutdown 2024-03-12T05:19:39.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 msgr2=0x5592d788a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.991+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab5002c170 tx=0x7fab50000bc0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 shutdown_connections 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5592d78ec5e0 0x5592d78ee9d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592d787cb70 0x5592d788ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab3effd640 0 client.27205 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592d7885750 0x5592d78ead40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 --2- 172.21.15.45:0/2438422045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592d787c250 0x5592d788a610 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 >> 172.21.15.45:0/2438422045 conn(0x5592d7876980 msgr2=0x5592d78810a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 shutdown_connections 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 1 -- 172.21.15.45:0/2438422045 wait complete. 2024-03-12T05:19:39.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.992+0000 7fab68563740 20 client.27205 trim_cache size 0 max 16384 2024-03-12T05:19:39.992 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 msgr2=0x7fab4c0a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 0x7fab4c0a4110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab580970e0 tx=0x7fab580cdc10 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 msgr2=0x7fab6010b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fab5800eb10 tx=0x7fab5800c2d0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 shutdown_connections 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab4c0a1c50 0x7fab4c0a4110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6010b4a0 0x7fab6010b950 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab60106450 0x7fab6010af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 --2- 172.21.15.45:0/605491462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab60101380 0x7fab6010dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:39.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 >> 172.21.15.45:0/605491462 conn(0x7fab60078580 msgr2=0x7fab600ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 shutdown_connections 2024-03-12T05:19:39.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:39.993+0000 7fab658b5640 1 -- 172.21.15.45:0/605491462 wait complete. 2024-03-12T05:19:40.007 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:40.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:19:40.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.183+0000 7f9f0e720640 1 -- 172.21.15.45:0/3141774654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f0810d730 msgr2=0x7f9f0810db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.183+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3141774654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f0810d730 0x7f9f0810db10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009890 tx=0x7f9ef002bb70 comp rx=0 tx=0).stop 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 -- 172.21.15.45:0/3141774654 shutdown_connections 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3141774654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f08107720 0x7f9f08107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3141774654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f081080e0 0x7f9f081145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3141774654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f0810d730 0x7f9f0810db10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 -- 172.21.15.45:0/3141774654 >> 172.21.15.45:0/3141774654 conn(0x7f9f080756c0 msgr2=0x7f9f08077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 -- 172.21.15.45:0/3141774654 shutdown_connections 2024-03-12T05:19:40.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.184+0000 7f9f0e720640 1 -- 172.21.15.45:0/3141774654 wait complete. 2024-03-12T05:19:40.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 Processor -- start 2024-03-12T05:19:40.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 -- start start 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 0x7f9f081a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0810d730 0x7f9f0819daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0819e140 con 0x7f9f0810d730 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0819e280 con 0x7f9f08107720 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.185+0000 7f9f0e720640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0819e440 con 0x7f9f081080e0 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.186+0000 7f9f077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.186+0000 7f9f077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35804/0 (socket says 172.21.15.45:35804) 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.186+0000 7f9f077fe640 1 -- 172.21.15.45:0/3554120799 learned_addr learned my addr 172.21.15.45:0/3554120799 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.186+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0810d730 0x7f9f0819daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f07fff640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 0x7f9f081a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f077fe640 1 -- 172.21.15.45:0/3554120799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 msgr2=0x7f9f081a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f077fe640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 0x7f9f081a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f077fe640 1 -- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0810d730 msgr2=0x7f9f0819daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f077fe640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0810d730 0x7f9f0819daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f077fe640 1 -- 172.21.15.45:0/3554120799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef0009490 con 0x7f9f081080e0 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.187+0000 7f9f07fff640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 0x7f9f081a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.188+0000 7f9f077fe640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800ea20 tx=0x7f9ef800eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.188+0000 7f9f057fa640 1 -- 172.21.15.45:0/3554120799 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8009700 con 0x7f9f081080e0 2024-03-12T05:19:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.188+0000 7f9f057fa640 1 -- 172.21.15.45:0/3554120799 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ef800cd80 con 0x7f9f081080e0 2024-03-12T05:19:40.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.189+0000 7f9f057fa640 1 -- 172.21.15.45:0/3554120799 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8018710 con 0x7f9f081080e0 2024-03-12T05:19:40.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.189+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0819e720 con 0x7f9f081080e0 2024-03-12T05:19:40.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.189+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08079bb0 con 0x7f9f081080e0 2024-03-12T05:19:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 Processor -- start 2024-03-12T05:19:40.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 -- start start 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e046630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e046b70 0x55559e04f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 0x55559e051b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e001960 con 0x55559e046250 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559dfb5310 con 0x55559e04f750 2024-03-12T05:19:40.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.193+0000 7f9f113cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559dfb2840 con 0x55559e046b70 2024-03-12T05:19:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f0ccbd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e046630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f0ccbd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e046630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60838/0 (socket says 172.21.15.45:60838) 2024-03-12T05:19:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f0ccbd640 1 -- 172.21.15.45:0/1631033385 learned_addr learned my addr 172.21.15.45:0/1631033385 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f057fa640 1 -- 172.21.15.45:0/3554120799 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ef8018870 con 0x7f9f081080e0 2024-03-12T05:19:40.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 0x55559e051b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.194+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e046b70 0x55559e04f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 -- 172.21.15.45:0/1631033385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e046b70 msgr2=0x55559e04f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e046b70 0x55559e04f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 -- 172.21.15.45:0/1631033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 msgr2=0x55559e046630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e046630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 -- 172.21.15.45:0/1631033385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55559df70060 con 0x55559e04f750 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 0x55559e051b40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cb330 tx=0x7f9ef80d3b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdcd8105869a7da server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f057fa640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 0x7f9ef40a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.195+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 0x7f9ef40a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.196+0000 7f9efe7fc640 1 -- 172.21.15.45:0/1631033385 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80cdea0 con 0x55559e04f750 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.196+0000 7f9efe7fc640 1 -- 172.21.15.45:0/1631033385 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ef80d36b0 con 0x55559e04f750 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.196+0000 7f9efe7fc640 1 -- 172.21.15.45:0/1631033385 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef809af10 con 0x55559e04f750 2024-03-12T05:19:40.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.196+0000 7f9f057fa640 1 -- 172.21.15.45:0/3554120799 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9ef8014070 con 0x7f9f081080e0 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 0x7f9ef40a4040 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000e5e0 tx=0x7f9f00010040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 -- 172.21.15.45:0/1631033385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 msgr2=0x55559e051b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 0x55559e051b40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80cb330 tx=0x7f9ef80d3b00 comp rx=0 tx=0).stop 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 -- 172.21.15.45:0/1631033385 shutdown_connections 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e04f750 0x55559e051b40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e046b70 0x55559e04f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1631033385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e046630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 -- 172.21.15.45:0/1631033385 >> 172.21.15.45:0/1631033385 conn(0x55559e040980 msgr2=0x55559e040d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 -- 172.21.15.45:0/1631033385 shutdown_connections 2024-03-12T05:19:40.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.197+0000 7f9f113cc740 1 -- 172.21.15.45:0/1631033385 wait complete. 2024-03-12T05:19:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.198+0000 7f9f113cc740 1 -- start start 2024-03-12T05:19:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.198+0000 7f9f113cc740 1 Processor -- start 2024-03-12T05:19:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e04d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60852/0 (socket says 172.21.15.45:60852) 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f07fff640 1 -- 172.21.15.45:0/1300846960 learned_addr learned my addr 172.21.15.45:0/1300846960 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559dfae020 con 0x55559e046250 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559dc81430 con 0x55559e046b70 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559dc3de00 con 0x55559e04f750 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e04d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.199+0000 7f9f113cc740 10 client.0 _ll_register_callbacks cb 0x55559df55cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 -- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 msgr2=0x55559e04d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e04d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 -- 172.21.15.45:0/1300846960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 msgr2=0x55559e04cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e04cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08079bb0 con 0x55559e046250 2024-03-12T05:19:40.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005e00 tx=0x7f9ef0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e348a5d5e269bd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.200+0000 7f9ee77fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0034850 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9ef00044a0 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55559dcea720 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ef0005030 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9f113cc740 10 client.17751 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.201+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55559dfac6a0 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55559dcea720 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ef00046c0 con 0x55559e046250 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 10 client.17751 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 10 client.17751 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 20 client.17751 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:40.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e05fab0 0x55559e061ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 4 client.17751 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55559dd39890 con 0x55559e05fab0 2024-03-12T05:19:40.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.202+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e05fab0 0x55559e061ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.203+0000 7f9ee77fe640 10 client.17751 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:40.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.203+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e05fab0 0x55559e061ee0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6986ff3695a05dbe server_cookie=cfe3a746166a91f7 in_seq=0 out_seq=0 2024-03-12T05:19:40.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.203+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1300846960 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55559dd39890 con 0x55559e05fab0 2024-03-12T05:19:40.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9ee77fe640 10 client.17751 handle_command_reply: tid=0 2024-03-12T05:19:40.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 client.17751 shutdown 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 msgr2=0x55559e04c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0005e00 tx=0x7f9ef0004270 comp rx=0 tx=0).stop 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 shutdown_connections 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e05fab0 0x55559e061ee0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e04d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9ee77fe640 0 client.17751 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e04cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1300846960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04c740 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 >> 172.21.15.45:0/1300846960 conn(0x55559e040980 msgr2=0x55559e0426d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 shutdown_connections 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 1 -- 172.21.15.45:0/1300846960 wait complete. 2024-03-12T05:19:40.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.204+0000 7f9f113cc740 20 client.17751 trim_cache size 0 max 16384 2024-03-12T05:19:40.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.225+0000 7f9f113cc740 1 Processor -- start 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 -- start start 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e03fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60864/0 (socket says 172.21.15.45:60864) 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e03fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e03fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35692/0 (socket says 172.21.15.45:35692) 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f0ccbd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559df33130 con 0x55559e046250 2024-03-12T05:19:40.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e05b510 con 0x55559e046b70 2024-03-12T05:19:40.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f113cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e067130 con 0x55559e04f750 2024-03-12T05:19:40.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f0ccbd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35828/0 (socket says 172.21.15.45:35828) 2024-03-12T05:19:40.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.226+0000 7f9f0ccbd640 1 -- 172.21.15.45:0/3776897595 learned_addr learned my addr 172.21.15.45:0/3776897595 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 -- 172.21.15.45:0/3776897595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 msgr2=0x55559e03fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e03fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 -- 172.21.15.45:0/3776897595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 msgr2=0x55559e04cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 -- 172.21.15.45:0/3776897595 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55559df48c20 con 0x55559e04f750 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000d820 tx=0x7f9f00016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ca4e5a3e945235 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.227+0000 7f9efdffb640 1 -- 172.21.15.45:0/3776897595 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00013040 con 0x55559e04f750 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.228+0000 7f9efdffb640 1 -- 172.21.15.45:0/3776897595 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9f0000de60 con 0x55559e04f750 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.228+0000 7f9efdffb640 1 -- 172.21.15.45:0/3776897595 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00025430 con 0x55559e04f750 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.228+0000 7f9f113cc740 1 -- 172.21.15.45:0/3776897595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 msgr2=0x55559e056600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000d820 tx=0x7f9f00016000 comp rx=0 tx=0).stop 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 -- 172.21.15.45:0/3776897595 shutdown_connections 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e056600 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e03fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 --2- 172.21.15.45:0/3776897595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e04cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 -- 172.21.15.45:0/3776897595 >> 172.21.15.45:0/3776897595 conn(0x55559e040980 msgr2=0x55559e03f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 -- 172.21.15.45:0/3776897595 shutdown_connections 2024-03-12T05:19:40.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.229+0000 7f9f113cc740 1 -- 172.21.15.45:0/3776897595 wait complete. 2024-03-12T05:19:40.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.230+0000 7f9f113cc740 1 -- start start 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.230+0000 7f9f113cc740 1 Processor -- start 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.230+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e0558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e055eb0 con 0x55559e046250 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60872/0 (socket says 172.21.15.45:60872) 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e056020 con 0x55559e046b70 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 -- 172.21.15.45:0/1790016282 learned_addr learned my addr 172.21.15.45:0/1790016282 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559e0561e0 con 0x55559e04f750 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f113cc740 10 client.0 _ll_register_callbacks cb 0x7f9ef80d33e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f0ccbd640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e0558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 -- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 msgr2=0x55559e0558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e0558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 -- 172.21.15.45:0/1790016282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 msgr2=0x55559e0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.231+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9f07fff640 1 -- 172.21.15.45:0/1790016282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0001f040 con 0x55559e046250 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9f07fff640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0004a50 tx=0x7f9ef0004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128b6daab56c4775 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9ee77fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0032400 con 0x55559e046250 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9ef0000e40 con 0x55559e046250 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.232+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55559dcad7e0 con 0x55559e046250 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ef00044a0 con 0x55559e046250 2024-03-12T05:19:40.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 10 client.17757 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55559df6e8c0 con 0x55559e046250 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55559dcad7e0 con 0x55559e046250 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ef00050f0 con 0x55559e046250 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 10 client.17757 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 10 client.17757 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 20 client.17757 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:40.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e0b7fa0 0x55559e0ba3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 4 client.17757 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.233+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55559dc59700 con 0x55559e0b7fa0 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.234+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e0b7fa0 0x55559e0ba3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.234+0000 7f9ee77fe640 10 client.17757 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f077fe640 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e0b7fa0 0x55559e0ba3c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7fa739e9792f558 server_cookie=c0114039d8dd529f in_seq=0 out_seq=0 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9ee77fe640 1 -- 172.21.15.45:0/1790016282 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55559dc59700 con 0x55559e0b7fa0 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9ee77fe640 10 client.17757 handle_command_reply: tid=0 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 client.17757 shutdown 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 msgr2=0x55559e0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0004a50 tx=0x7f9ef0004e70 comp rx=0 tx=0).stop 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 shutdown_connections 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55559e0b7fa0 0x55559e0ba3c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9ee77fe640 0 client.17757 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:19:40.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55559e04f750 0x55559e0558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55559e046b70 0x55559e0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 --2- 172.21.15.45:0/1790016282 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55559e046250 0x55559e0b0440 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.235+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 >> 172.21.15.45:0/1790016282 conn(0x55559e040980 msgr2=0x55559e03fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.236+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 shutdown_connections 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.236+0000 7f9f113cc740 1 -- 172.21.15.45:0/1790016282 wait complete. 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.236+0000 7f9f113cc740 20 client.17757 trim_cache size 0 max 16384 2024-03-12T05:19:40.234 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:40.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 msgr2=0x7f9ef40a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 0x7f9ef40a4040 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000e5e0 tx=0x7f9f00010040 comp rx=0 tx=0).stop 2024-03-12T05:19:40.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 msgr2=0x7f9f081a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800ea20 tx=0x7f9ef800eee0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 shutdown_connections 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9ef40a1b80 0x7f9ef40a4040 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0810d730 0x7f9f0819daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f081080e0 0x7f9f081a3e80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 --2- 172.21.15.45:0/3554120799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f08107720 0x7f9f081a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 >> 172.21.15.45:0/3554120799 conn(0x7f9f080756c0 msgr2=0x7f9f08076020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 shutdown_connections 2024-03-12T05:19:40.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.237+0000 7f9f0e720640 1 -- 172.21.15.45:0/3554120799 wait complete. 2024-03-12T05:19:40.250 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:19:40.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 -- 172.21.15.45:0/2301374550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd6810d730 msgr2=0x7efd6810db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 --2- 172.21.15.45:0/2301374550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd6810d730 0x7efd6810db10 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7efd5c009930 tx=0x7efd5c02bc20 comp rx=0 tx=0).stop 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 -- 172.21.15.45:0/2301374550 shutdown_connections 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 --2- 172.21.15.45:0/2301374550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd681080e0 0x7efd681145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 --2- 172.21.15.45:0/2301374550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd68107720 0x7efd68107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 --2- 172.21.15.45:0/2301374550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd6810d730 0x7efd6810db10 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.426+0000 7efd6dc22640 1 -- 172.21.15.45:0/2301374550 >> 172.21.15.45:0/2301374550 conn(0x7efd680756c0 msgr2=0x7efd68077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.427+0000 7efd6dc22640 1 -- 172.21.15.45:0/2301374550 shutdown_connections 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.427+0000 7efd6dc22640 1 -- 172.21.15.45:0/2301374550 wait complete. 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.427+0000 7efd6dc22640 1 Processor -- start 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd6dc22640 1 -- start start 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd6dc22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd6dc22640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd681080e0 0x7efd68081c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd6dc22640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd6810d730 0x7efd6807a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.428+0000 7efd677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60894/0 (socket says 172.21.15.45:60894) 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 -- 172.21.15.45:0/21732051 learned_addr learned my addr 172.21.15.45:0/21732051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6807b0f0 con 0x7efd68107720 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6807b260 con 0x7efd681080e0 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd67fff640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd6810d730 0x7efd6807a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6807b420 con 0x7efd6810d730 2024-03-12T05:19:40.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 -- 172.21.15.45:0/21732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd6810d730 msgr2=0x7efd6807a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd6810d730 0x7efd6807a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 -- 172.21.15.45:0/21732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd681080e0 msgr2=0x7efd68081c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:19:40.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd681080e0 0x7efd68081c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.429+0000 7efd677fe640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c009560 con 0x7efd68107720 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.430+0000 7efd677fe640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7efd5c005ab0 tx=0x7efd5c005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.430+0000 7efd64ff9640 1 -- 172.21.15.45:0/21732051 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0043d0 con 0x7efd68107720 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.430+0000 7efd64ff9640 1 -- 172.21.15.45:0/21732051 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd5c003aa0 con 0x7efd68107720 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.430+0000 7efd64ff9640 1 -- 172.21.15.45:0/21732051 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0053d0 con 0x7efd68107720 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.430+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6807f220 con 0x7efd68107720 2024-03-12T05:19:40.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.431+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6807f620 con 0x7efd68107720 2024-03-12T05:19:40.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.434+0000 7efd64ff9640 1 -- 172.21.15.45:0/21732051 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efd5c005530 con 0x7efd68107720 2024-03-12T05:19:40.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.434+0000 7efd707f0740 1 Processor -- start 2024-03-12T05:19:40.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.434+0000 7efd707f0740 1 -- start start 2024-03-12T05:19:40.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e707630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e712b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e6c2960 con 0x55dd0e707250 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e676310 con 0x55dd0e707b70 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd707f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e673840 con 0x55dd0e710750 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd64ff9640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 0x7efd500a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e707630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e712b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e712b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35854/0 (socket says 172.21.15.45:35854) 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35718/0 (socket says 172.21.15.45:35718) 2024-03-12T05:19:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd66ffd640 1 -- 172.21.15.45:0/2735329178 learned_addr learned my addr 172.21.15.45:0/2735329178 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd64ff9640 1 -- 172.21.15.45:0/21732051 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd5c0efe20 con 0x7efd68107720 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd67fff640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 0x7efd500a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 -- 172.21.15.45:0/2735329178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 msgr2=0x55dd0e712b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e712b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 -- 172.21.15.45:0/2735329178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 msgr2=0x55dd0e707630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e707630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.435+0000 7efd677fe640 1 -- 172.21.15.45:0/2735329178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd0e631060 con 0x55dd0e707b70 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.436+0000 7efd677fe640 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0ed650 tx=0x7efd5c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8112614fb748b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.436+0000 7efd4dffb640 1 -- 172.21.15.45:0/2735329178 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0bcb30 con 0x55dd0e707b70 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.436+0000 7efd67fff640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 0x7efd500a41b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efd60009fd0 tx=0x7efd60004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.436+0000 7efd4dffb640 1 -- 172.21.15.45:0/2735329178 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd5c0bd230 con 0x55dd0e707b70 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.436+0000 7efd4dffb640 1 -- 172.21.15.45:0/2735329178 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0bcd90 con 0x55dd0e707b70 2024-03-12T05:19:40.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- 172.21.15.45:0/2735329178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 msgr2=0x55dd0e710210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0ed650 tx=0x7efd5c009260 comp rx=0 tx=0).stop 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- 172.21.15.45:0/2735329178 shutdown_connections 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e712b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e710210 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 --2- 172.21.15.45:0/2735329178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e707630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- 172.21.15.45:0/2735329178 >> 172.21.15.45:0/2735329178 conn(0x55dd0e701980 msgr2=0x55dd0e701d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- 172.21.15.45:0/2735329178 shutdown_connections 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- 172.21.15.45:0/2735329178 wait complete. 2024-03-12T05:19:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.437+0000 7efd707f0740 1 -- start start 2024-03-12T05:19:40.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 Processor -- start 2024-03-12T05:19:40.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e70d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e66f020 con 0x55dd0e707250 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e342430 con 0x55dd0e707b70 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e2fee00 con 0x55dd0e710750 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.438+0000 7efd707f0740 10 client.0 _ll_register_callbacks cb 0x55dd0e616cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35860/0 (socket says 172.21.15.45:35860) 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35728/0 (socket says 172.21.15.45:35728) 2024-03-12T05:19:40.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 -- 172.21.15.45:0/1208878979 learned_addr learned my addr 172.21.15.45:0/1208878979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 -- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 msgr2=0x55dd0e70dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 -- 172.21.15.45:0/1208878979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 msgr2=0x55dd0e70d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e70d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 -- 172.21.15.45:0/1208878979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6807f620 con 0x55dd0e710750 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd66ffd640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.439+0000 7efd67fff640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efd6000eff0 tx=0x7efd60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b9936112130d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6001f600 con 0x55dd0e710750 2024-03-12T05:19:40.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd6001f760 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd0e36e7e0 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd60013020 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd707f0740 10 client.27235 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.440+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dd0e66d6a0 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dd0e36e7e0 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd60012790 con 0x55dd0e710750 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 10 client.27235 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 10 client.27235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 20 client.27235 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e720930 0x55dd0e722d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 4 client.27235 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd0e3ab720 con 0x55dd0e720930 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.441+0000 7efd677fe640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e720930 0x55dd0e722d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.442+0000 7efd677fe640 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e720930 0x55dd0e722d50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31614d1110cc2e5 server_cookie=3ab0bcfb67cc12b3 in_seq=0 out_seq=0 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.442+0000 7efd46ffd640 10 client.27235 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.442+0000 7efd46ffd640 1 -- 172.21.15.45:0/1208878979 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dd0e3ab720 con 0x55dd0e720930 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.442+0000 7efd46ffd640 10 client.27235 handle_command_reply: tid=0 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.442+0000 7efd707f0740 1 client.27235 shutdown 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 msgr2=0x55dd0e70e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efd6000eff0 tx=0x7efd60004a40 comp rx=0 tx=0).stop 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 shutdown_connections 2024-03-12T05:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e720930 0x55dd0e722d50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70e200 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707b70 0x55dd0e70dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd46ffd640 0 client.27235 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 --2- 172.21.15.45:0/1208878979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707250 0x55dd0e70d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 >> 172.21.15.45:0/1208878979 conn(0x55dd0e701980 msgr2=0x55dd0e7036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.443+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 shutdown_connections 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.444+0000 7efd707f0740 1 -- 172.21.15.45:0/1208878979 wait complete. 2024-03-12T05:19:40.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.444+0000 7efd707f0740 20 client.27235 trim_cache size 0 max 16384 2024-03-12T05:19:40.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.466+0000 7efd707f0740 1 Processor -- start 2024-03-12T05:19:40.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.466+0000 7efd707f0740 1 -- start start 2024-03-12T05:19:40.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7276f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e5f4130 con 0x55dd0e707b70 2024-03-12T05:19:40.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e6fd2b0 con 0x55dd0e707250 2024-03-12T05:19:40.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd707f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e706cc0 con 0x55dd0e710750 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35736/0 (socket says 172.21.15.45:35736) 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 -- 172.21.15.45:0/1334983022 learned_addr learned my addr 172.21.15.45:0/1334983022 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd67fff640 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 -- 172.21.15.45:0/1334983022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 msgr2=0x55dd0e70aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 -- 172.21.15.45:0/1334983022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 msgr2=0x55dd0e7276f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7276f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.467+0000 7efd677fe640 1 -- 172.21.15.45:0/1334983022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd0e609c20 con 0x55dd0e707250 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.468+0000 7efd677fe640 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd5c02c200 tx=0x7efd5c0bc770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df3fc9a2aca2279b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.468+0000 7efd4d7fa640 1 -- 172.21.15.45:0/1334983022 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0bcbb0 con 0x55dd0e707250 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.468+0000 7efd4d7fa640 1 -- 172.21.15.45:0/1334983022 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd5c0c0c80 con 0x55dd0e707250 2024-03-12T05:19:40.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.468+0000 7efd4d7fa640 1 -- 172.21.15.45:0/1334983022 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0bcff0 con 0x55dd0e707250 2024-03-12T05:19:40.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- 172.21.15.45:0/1334983022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 msgr2=0x55dd0e70feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd5c02c200 tx=0x7efd5c0bc770 comp rx=0 tx=0).stop 2024-03-12T05:19:40.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- 172.21.15.45:0/1334983022 shutdown_connections 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7276f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e710750 0x55dd0e70aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 --2- 172.21.15.45:0/1334983022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e707250 0x55dd0e70feb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- 172.21.15.45:0/1334983022 >> 172.21.15.45:0/1334983022 conn(0x55dd0e701980 msgr2=0x55dd0e707fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- 172.21.15.45:0/1334983022 shutdown_connections 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- 172.21.15.45:0/1334983022 wait complete. 2024-03-12T05:19:40.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.470+0000 7efd707f0740 1 -- start start 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 Processor -- start 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e707250 0x55dd0e715ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 0x55dd0e716920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e776e80 con 0x55dd0e707b70 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e776ff0 con 0x55dd0e710750 2024-03-12T05:19:40.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.471+0000 7efd707f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd0e7771b0 con 0x55dd0e707250 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd707f0740 10 client.0 _ll_register_callbacks cb 0x7efd5c0b8730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e707250 0x55dd0e715ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 0x55dd0e716920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 0x55dd0e716920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35738/0 (socket says 172.21.15.45:35738) 2024-03-12T05:19:40.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd67fff640 1 -- 172.21.15.45:0/1375119569 learned_addr learned my addr 172.21.15.45:0/1375119569 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:40.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 -- 172.21.15.45:0/1375119569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e707250 msgr2=0x55dd0e715ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e707250 0x55dd0e715ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 -- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 msgr2=0x55dd0e716920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:40.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 0x55dd0e716920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.472+0000 7efd66ffd640 1 -- 172.21.15.45:0/1375119569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd0e631060 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.473+0000 7efd66ffd640 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7163e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55dd0e70e910 tx=0x7efd5800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f26484f18140446 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.473+0000 7efd46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.473+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58016070 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.473+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd5c0bd230 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.473+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd58011440 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.474+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd58011620 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.474+0000 7efd707f0740 10 client.17769 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.474+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dd0e62f8c0 con 0x55dd0e707b70 2024-03-12T05:19:40.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.474+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd5c0bd230 con 0x55dd0e707b70 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd58024800 con 0x55dd0e707b70 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 10 client.17769 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 10 client.17769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 20 client.17769 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e778ab0 0x55dd0e77aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 4 client.17769 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd0e3fa890 con 0x55dd0e778ab0 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.475+0000 7efd677fe640 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e778ab0 0x55dd0e77aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.476+0000 7efd677fe640 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e778ab0 0x55dd0e77aed0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c66083e9bb69f9f8 server_cookie=73a21db1e2106a4f in_seq=0 out_seq=0 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.476+0000 7efd46ffd640 10 client.17769 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.476+0000 7efd46ffd640 1 -- 172.21.15.45:0/1375119569 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd0e3fa890 con 0x55dd0e778ab0 2024-03-12T05:19:40.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.476+0000 7efd46ffd640 10 client.17769 handle_command_reply: tid=0 2024-03-12T05:19:40.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.476+0000 7efd707f0740 1 client.17769 shutdown 2024-03-12T05:19:40.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.477+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 msgr2=0x55dd0e7163e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.477+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7163e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55dd0e70e910 tx=0x7efd5800b940 comp rx=0 tx=0).stop 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 shutdown_connections 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55dd0e778ab0 0x55dd0e77aed0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd46ffd640 0 client.17769 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dd0e707b70 0x55dd0e7163e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd0e710750 0x55dd0e716920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 --2- 172.21.15.45:0/1375119569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dd0e707250 0x55dd0e715ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.478+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 >> 172.21.15.45:0/1375119569 conn(0x55dd0e701980 msgr2=0x55dd0e7276f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.479+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 shutdown_connections 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.479+0000 7efd707f0740 1 -- 172.21.15.45:0/1375119569 wait complete. 2024-03-12T05:19:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.479+0000 7efd707f0740 20 client.17769 trim_cache size 0 max 16384 2024-03-12T05:19:40.477 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 msgr2=0x7efd500a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 0x7efd500a41b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efd60009fd0 tx=0x7efd60004ab0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 msgr2=0x7efd68081700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7efd5c005ab0 tx=0x7efd5c005e60 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 shutdown_connections 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd500a1cf0 0x7efd500a41b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd6810d730 0x7efd6807a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd681080e0 0x7efd68081c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 --2- 172.21.15.45:0/21732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd68107720 0x7efd68081700 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:40.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 >> 172.21.15.45:0/21732051 conn(0x7efd680756c0 msgr2=0x7efd68076020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:40.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.480+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 shutdown_connections 2024-03-12T05:19:40.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:40.481+0000 7efd6dc22640 1 -- 172.21.15.45:0/21732051 wait complete. 2024-03-12T05:19:40.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:40 smithi045 ceph-mon[32174]: pgmap v272: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:19:40.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:40 smithi116 ceph-mon[35486]: pgmap v272: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:19:40.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:40 smithi130 ceph-mon[35398]: pgmap v272: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 17 MiB/s wr, 1.90k op/s 2024-03-12T05:19:41.498 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:19:41.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:19:41.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:19:41.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.687+0000 7f6284ed4640 1 -- 172.21.15.45:0/4071573299 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c032340 con 0x7f628010d9d0 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 -- 172.21.15.45:0/4071573299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d9d0 msgr2=0x7f628010de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 --2- 172.21.15.45:0/4071573299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d9d0 0x7f628010de50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f627c009860 tx=0x7f627c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 -- 172.21.15.45:0/4071573299 shutdown_connections 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 --2- 172.21.15.45:0/4071573299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628011a960 0x7f628011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 --2- 172.21.15.45:0/4071573299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010d9d0 0x7f628010de50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 --2- 172.21.15.45:0/4071573299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f628010f9a0 0x7f628010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 -- 172.21.15.45:0/4071573299 >> 172.21.15.45:0/4071573299 conn(0x7f628006c260 msgr2=0x7f628006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 -- 172.21.15.45:0/4071573299 shutdown_connections 2024-03-12T05:19:41.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.688+0000 7f628813a640 1 -- 172.21.15.45:0/4071573299 wait complete. 2024-03-12T05:19:41.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.689+0000 7f628813a640 1 Processor -- start 2024-03-12T05:19:41.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 -- start start 2024-03-12T05:19:41.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010f9a0 0x7f62801a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f628011a960 0x7f62801aad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801ab380 con 0x7f628010d9d0 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801ab4f0 con 0x7f628010f9a0 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f628813a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801ab6b0 con 0x7f628011a960 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f6285ed6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f6285ed6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60948/0 (socket says 172.21.15.45:60948) 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.690+0000 7f6285ed6640 1 -- 172.21.15.45:0/305698827 learned_addr learned my addr 172.21.15.45:0/305698827 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f62856d5640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010f9a0 0x7f62801a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 -- 172.21.15.45:0/305698827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f628011a960 msgr2=0x7f62801aad80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:19:41.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f628011a960 0x7f62801aad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 -- 172.21.15.45:0/305698827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010f9a0 msgr2=0x7f62801a6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010f9a0 0x7f62801a6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 -- 172.21.15.45:0/305698827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c009490 con 0x7f628010d9d0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.691+0000 7f6285ed6640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f627000e9c0 tx=0x7f627000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.692+0000 7f6276ffd640 1 -- 172.21.15.45:0/305698827 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000ccc0 con 0x7f628010d9d0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.692+0000 7f6276ffd640 1 -- 172.21.15.45:0/305698827 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f627000ce20 con 0x7f628010d9d0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.692+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62801ab990 con 0x7f628010d9d0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.692+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62801b3210 con 0x7f628010d9d0 2024-03-12T05:19:41.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.692+0000 7f6276ffd640 1 -- 172.21.15.45:0/305698827 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270010430 con 0x7f628010d9d0 2024-03-12T05:19:41.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 Processor -- start 2024-03-12T05:19:41.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 -- start start 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.696+0000 7f628ade1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f2245f90 con 0x5566f22d7d30 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f628ade1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22434c0 con 0x5566f22d7380 2024-03-12T05:19:41.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f628ade1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f223eca0 con 0x5566f22e08b0 2024-03-12T05:19:41.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35764/0 (socket says 172.21.15.45:35764) 2024-03-12T05:19:41.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 -- 172.21.15.45:0/2637802573 learned_addr learned my addr 172.21.15.45:0/2637802573 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62856d5640 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 -- 172.21.15.45:0/2637802573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 msgr2=0x5566f22e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 -- 172.21.15.45:0/2637802573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 msgr2=0x5566f22e02e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22e02e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.697+0000 7f62866d7640 1 -- 172.21.15.45:0/2637802573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566f22010b0 con 0x5566f22d7380 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.698+0000 7f62866d7640 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6278009c80 tx=0x7f627800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a26ded553715a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.698+0000 7f626bfff640 1 -- 172.21.15.45:0/2637802573 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278004b20 con 0x5566f22d7380 2024-03-12T05:19:41.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.698+0000 7f6276ffd640 1 -- 172.21.15.45:0/305698827 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6270004f20 con 0x7f628010d9d0 2024-03-12T05:19:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.698+0000 7f626bfff640 1 -- 172.21.15.45:0/2637802573 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6278004c80 con 0x5566f22d7380 2024-03-12T05:19:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.698+0000 7f626bfff640 1 -- 172.21.15.45:0/2637802573 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62780116d0 con 0x5566f22d7380 2024-03-12T05:19:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.699+0000 7f6276ffd640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 0x7f626c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.699+0000 7f6276ffd640 1 -- 172.21.15.45:0/305698827 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6270014070 con 0x7f628010d9d0 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f6285ed6640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 0x7f626c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 -- 172.21.15.45:0/2637802573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 msgr2=0x5566f22d7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6278009c80 tx=0x7f627800e6c0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 -- 172.21.15.45:0/2637802573 shutdown_connections 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 --2- 172.21.15.45:0/2637802573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22d7760 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.700+0000 7f628ade1740 1 -- 172.21.15.45:0/2637802573 >> 172.21.15.45:0/2637802573 conn(0x5566f22d1840 msgr2=0x5566f22d1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.701+0000 7f6285ed6640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 0x7f626c0a40e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5566f22d6e50 tx=0x7f6270009f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.701+0000 7f628ade1740 1 -- 172.21.15.45:0/2637802573 shutdown_connections 2024-03-12T05:19:41.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.701+0000 7f628ade1740 1 -- 172.21.15.45:0/2637802573 wait complete. 2024-03-12T05:19:41.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.701+0000 7f628ade1740 1 -- start start 2024-03-12T05:19:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.702+0000 7f628ade1740 1 Processor -- start 2024-03-12T05:19:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.702+0000 7f628ade1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 0x5566f22dd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.702+0000 7f628ade1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62866d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 0x5566f22dd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f628ade1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22e08b0 0x5566f22de340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35772/0 (socket says 172.21.15.45:35772) 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62866d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 0x5566f22dd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35914/0 (socket says 172.21.15.45:35914) 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f628ade1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f21e3f40 con 0x5566f22e08b0 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 -- 172.21.15.45:0/123738269 learned_addr learned my addr 172.21.15.45:0/123738269 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f1f124b0 con 0x5566f22d7d30 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f21c4670 con 0x5566f22d7380 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f628ade1740 10 client.0 _ll_register_callbacks cb 0x5566f2231f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62866d7640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22e08b0 0x5566f22de340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 -- 172.21.15.45:0/123738269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 msgr2=0x5566f22dd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 0x5566f22dd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 -- 172.21.15.45:0/123738269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22e08b0 msgr2=0x5566f22de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22e08b0 0x5566f22de340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.703+0000 7f62856d5640 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62780098e0 con 0x5566f22d7d30 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.704+0000 7f62856d5640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f627c009780 tx=0x7f627c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d4e0982a76d8b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.704+0000 7f62697fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.704+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c032400 con 0x5566f22d7d30 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.704+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f627c034640 con 0x5566f22d7d30 2024-03-12T05:19:41.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.704+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566f2218f40 con 0x5566f22d7d30 2024-03-12T05:19:41.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627c034800 con 0x5566f22d7d30 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 10 client.27266 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5566f223d320 con 0x5566f22d7d30 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5566f2218f40 con 0x5566f22d7d30 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f627c02c8d0 con 0x5566f22d7d30 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 10 client.27266 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 10 client.27266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 20 client.27266 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f22f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 4 client.27266 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.706+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566f2235310 con 0x5566f22f0e30 2024-03-12T05:19:41.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.707+0000 7f62866d7640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f22f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.707+0000 7f62866d7640 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f22f3250 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5bbc00e5b905b6b server_cookie=33c778338a2bd1cc in_seq=0 out_seq=0 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.707+0000 7f62697fa640 10 client.27266 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f62697fa640 1 -- 172.21.15.45:0/123738269 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566f2235310 con 0x5566f22f0e30 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f62697fa640 10 client.27266 handle_command_reply: tid=0 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 client.27266 shutdown 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 msgr2=0x5566f22dde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f627c009780 tx=0x7f627c004270 comp rx=0 tx=0).stop 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 shutdown_connections 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f22f3250 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.708+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22e08b0 0x5566f22de340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f22dde00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f62697fa640 0 client.27266 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 1 --2- 172.21.15.45:0/123738269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22d7380 0x5566f22dd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 >> 172.21.15.45:0/123738269 conn(0x5566f22d1840 msgr2=0x5566f22d37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 shutdown_connections 2024-03-12T05:19:41.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 1 -- 172.21.15.45:0/123738269 wait complete. 2024-03-12T05:19:41.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.709+0000 7f628ade1740 20 client.27266 trim_cache size 0 max 16384 2024-03-12T05:19:41.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 Processor -- start 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 -- start start 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22dab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22db060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22d2640 con 0x5566f22d7d30 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22d3db0 con 0x5566f22d7380 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f628ade1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22ed9a0 con 0x5566f22e08b0 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f62866d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f62866d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35788/0 (socket says 172.21.15.45:35788) 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.738+0000 7f62866d7640 1 -- 172.21.15.45:0/1798676645 learned_addr learned my addr 172.21.15.45:0/1798676645 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22db060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62856d5640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 -- 172.21.15.45:0/1798676645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 msgr2=0x5566f22db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22db060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 -- 172.21.15.45:0/1798676645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 msgr2=0x5566f22dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22dab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 -- 172.21.15.45:0/1798676645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566f22d71f0 con 0x5566f22d7380 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62856d5640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f62866d7640 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6278009bd0 tx=0x7f6278009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59198fff5614c059 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f626b7fe640 1 -- 172.21.15.45:0/1798676645 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802b560 con 0x5566f22d7380 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f626b7fe640 1 -- 172.21.15.45:0/1798676645 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6278003e20 con 0x5566f22d7380 2024-03-12T05:19:41.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.739+0000 7f626b7fe640 1 -- 172.21.15.45:0/1798676645 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278005270 con 0x5566f22d7380 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.740+0000 7f628ade1740 1 -- 172.21.15.45:0/1798676645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 msgr2=0x5566f22de160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.740+0000 7f628ade1740 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6278009bd0 tx=0x7f6278009580 comp rx=0 tx=0).stop 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 -- 172.21.15.45:0/1798676645 shutdown_connections 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7d30 0x5566f22dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22db060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 --2- 172.21.15.45:0/1798676645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7380 0x5566f22de160 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 -- 172.21.15.45:0/1798676645 >> 172.21.15.45:0/1798676645 conn(0x5566f22d1840 msgr2=0x5566f22dc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 -- 172.21.15.45:0/1798676645 shutdown_connections 2024-03-12T05:19:41.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 -- 172.21.15.45:0/1798676645 wait complete. 2024-03-12T05:19:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 -- start start 2024-03-12T05:19:41.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.741+0000 7f628ade1740 1 Processor -- start 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7380 0x5566f2341810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f2341d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22e7280 con 0x5566f22d7380 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22e73f0 con 0x5566f22d7d30 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f22e75b0 con 0x5566f22e08b0 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f628ade1740 10 client.0 _ll_register_callbacks cb 0x7f62780094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f62856d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f2341d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f62856d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f2341d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35804/0 (socket says 172.21.15.45:35804) 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f62866d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7380 0x5566f2341810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f62856d5640 1 -- 172.21.15.45:0/1295126856 learned_addr learned my addr 172.21.15.45:0/1295126856 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.743+0000 7f62866d7640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 -- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 msgr2=0x5566f2341d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f2341d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 -- 172.21.15.45:0/1295126856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7380 msgr2=0x5566f2341810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7380 0x5566f2341810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 -- 172.21.15.45:0/1295126856 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6278026040 con 0x5566f22e08b0 2024-03-12T05:19:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62866d7640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e6cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f627801c040 tx=0x7f6278009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b213eb0e2b0cdfe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.744+0000 7f62697fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:41.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802b8f0 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6278005170 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566f2264900 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6278005330 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f628ade1740 10 client.27256 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.745+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5566f21d9c00 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6278045080 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6278037070 con 0x5566f22e08b0 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 10 client.27256 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 10 client.27256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 20 client.27256 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f234b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 4 client.27256 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:19:41.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5566f1f3e860 con 0x5566f22f0e30 2024-03-12T05:19:41.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.746+0000 7f62856d5640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f234b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.747+0000 7f62856d5640 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f234b2d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf96319f5cd25c07 server_cookie=71db26998e1b71e7 in_seq=0 out_seq=0 2024-03-12T05:19:41.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.747+0000 7f62697fa640 10 client.27256 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.748+0000 7f62697fa640 1 -- 172.21.15.45:0/1295126856 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5566f1f3e860 con 0x5566f22f0e30 2024-03-12T05:19:41.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.748+0000 7f62697fa640 10 client.27256 handle_command_reply: tid=0 2024-03-12T05:19:41.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.748+0000 7f628ade1740 1 client.27256 shutdown 2024-03-12T05:19:41.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.748+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 msgr2=0x5566f22e6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e6cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f627801c040 tx=0x7f6278009f30 comp rx=0 tx=0).stop 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 shutdown_connections 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5566f22f0e30 0x5566f234b2d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f62697fa640 0 client.27256 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5566f22e08b0 0x5566f22e6cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566f22d7d30 0x5566f2341d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 --2- 172.21.15.45:0/1295126856 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5566f22d7380 0x5566f2341810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 >> 172.21.15.45:0/1295126856 conn(0x5566f22d1840 msgr2=0x5566f22deec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 shutdown_connections 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 1 -- 172.21.15.45:0/1295126856 wait complete. 2024-03-12T05:19:41.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.749+0000 7f628ade1740 20 client.27256 trim_cache size 0 max 16384 2024-03-12T05:19:41.746 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:41.746 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:19:41.746 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "186f8cd4-6f8a-48e1-a615-81a596014065", 2024-03-12T05:19:41.746 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:19:41.746 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 msgr2=0x7f626c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 0x7f626c0a40e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5566f22d6e50 tx=0x7f6270009f90 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 msgr2=0x7f62801a64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f627000e9c0 tx=0x7f627000ee80 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 shutdown_connections 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f626c0a1c20 0x7f626c0a40e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f628011a960 0x7f62801aad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f628010f9a0 0x7f62801a6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 --2- 172.21.15.45:0/305698827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628010d9d0 0x7f62801a64c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 >> 172.21.15.45:0/305698827 conn(0x7f628006c260 msgr2=0x7f6280071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 shutdown_connections 2024-03-12T05:19:41.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.750+0000 7f628813a640 1 -- 172.21.15.45:0/305698827 wait complete. 2024-03-12T05:19:41.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:19:41.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 -- 172.21.15.45:0/105345687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073520 msgr2=0x7f3d24073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 --2- 172.21.15.45:0/105345687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073520 0x7f3d24073900 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009900 tx=0x7f3d1402bcc0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 -- 172.21.15.45:0/105345687 shutdown_connections 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 --2- 172.21.15.45:0/105345687 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d24114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 --2- 172.21.15.45:0/105345687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073e40 0x7f3d24111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 --2- 172.21.15.45:0/105345687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073520 0x7f3d24073900 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.950+0000 7f3d2a440640 1 -- 172.21.15.45:0/105345687 >> 172.21.15.45:0/105345687 conn(0x7f3d240fcc00 msgr2=0x7f3d240ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.951+0000 7f3d2a440640 1 -- 172.21.15.45:0/105345687 shutdown_connections 2024-03-12T05:19:41.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.951+0000 7f3d2a440640 1 -- 172.21.15.45:0/105345687 wait complete. 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.951+0000 7f3d2a440640 1 Processor -- start 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.951+0000 7f3d2a440640 1 -- start start 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073e40 0x7f3d2419e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d241a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241a2c10 con 0x7f3d24112140 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241a2d80 con 0x7f3d24073e40 2024-03-12T05:19:41.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d2a440640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241a2f40 con 0x7f3d24073520 2024-03-12T05:19:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35954/0 (socket says 172.21.15.45:35954) 2024-03-12T05:19:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 -- 172.21.15.45:0/2173010373 learned_addr learned my addr 172.21.15.45:0/2173010373 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073e40 0x7f3d2419e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d289dd640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d241a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 -- 172.21.15.45:0/2173010373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073e40 msgr2=0x7f3d2419e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073e40 0x7f3d2419e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 -- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 msgr2=0x7f3d241a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.952+0000 7f3d23fff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d241a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.953+0000 7f3d23fff640 1 -- 172.21.15.45:0/2173010373 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14009560 con 0x7f3d24073520 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.953+0000 7f3d289dd640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d241a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.953+0000 7f3d23fff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000c00 tx=0x7f3d1402d890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.954+0000 7f3d21ffb640 1 -- 172.21.15.45:0/2173010373 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1402dd50 con 0x7f3d24073520 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.954+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d241a31c0 con 0x7f3d24073520 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.954+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2406ad80 con 0x7f3d24073520 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.954+0000 7f3d21ffb640 1 -- 172.21.15.45:0/2173010373 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d14003aa0 con 0x7f3d24073520 2024-03-12T05:19:41.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.954+0000 7f3d21ffb640 1 -- 172.21.15.45:0/2173010373 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d14005390 con 0x7f3d24073520 2024-03-12T05:19:41.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.958+0000 7f3d21ffb640 1 -- 172.21.15.45:0/2173010373 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3d140054f0 con 0x7f3d24073520 2024-03-12T05:19:41.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d21ffb640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 0x7f3d100a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 Processor -- start 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 0x7f3d100a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 -- start start 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe4942b0 0x5617fe494690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe403280 con 0x5617fe4942b0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe4007b0 con 0x5617fe494bd0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.960+0000 7f3d2d013740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe3fbf90 con 0x5617fe49d7b0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35970/0 (socket says 172.21.15.45:35970) 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 -- 172.21.15.45:0/2154672265 learned_addr learned my addr 172.21.15.45:0/2154672265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d23fff640 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe4942b0 0x5617fe494690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 -- 172.21.15.45:0/2154672265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 msgr2=0x5617fe49d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 -- 172.21.15.45:0/2154672265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe4942b0 msgr2=0x5617fe494690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe4942b0 0x5617fe494690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 -- 172.21.15.45:0/2154672265 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617fe3ba570 con 0x5617fe49d7b0 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d21ffb640 1 -- 172.21.15.45:0/2173010373 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d140efdc0 con 0x7f3d24073520 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.961+0000 7f3d289dd640 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009f80 tx=0x7f3d1c020800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b44fe80782d7aed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.962+0000 7f3d1a7fc640 1 -- 172.21.15.45:0/2154672265 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c022710 con 0x5617fe49d7b0 2024-03-12T05:19:41.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.962+0000 7f3d1a7fc640 1 -- 172.21.15.45:0/2154672265 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d1c022d20 con 0x5617fe49d7b0 2024-03-12T05:19:41.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- 172.21.15.45:0/2154672265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 msgr2=0x5617fe49fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009f80 tx=0x7f3d1c020800 comp rx=0 tx=0).stop 2024-03-12T05:19:41.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 0x7f3d100a4230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3d24104ef0 tx=0x7f3d0c014840 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- 172.21.15.45:0/2154672265 shutdown_connections 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe49d7b0 0x5617fe49fba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 --2- 172.21.15.45:0/2154672265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe4942b0 0x5617fe494690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- 172.21.15.45:0/2154672265 >> 172.21.15.45:0/2154672265 conn(0x5617fe48e990 msgr2=0x5617fe48ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- 172.21.15.45:0/2154672265 shutdown_connections 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- 172.21.15.45:0/2154672265 wait complete. 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.963+0000 7f3d2d013740 1 -- start start 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 Processor -- start 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe49d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe0cf430 con 0x5617fe49d7b0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe08be00 con 0x5617fe494bd0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe380fd0 con 0x5617fe4942b0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d2d013740 10 client.0 _ll_register_callbacks cb 0x5617fe3d6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe49d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32788/0 (socket says 172.21.15.45:32788) 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d289dd640 1 -- 172.21.15.45:0/3716842709 learned_addr learned my addr 172.21.15.45:0/3716842709 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.964+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 -- 172.21.15.45:0/3716842709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 msgr2=0x5617fe49d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe49d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 -- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 msgr2=0x5617fe49a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 -- 172.21.15.45:0/3716842709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c009be0 con 0x5617fe49d7b0 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:19:41.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d289dd640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c02ba70 tx=0x7f3d1c02b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c736af0891a11d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.965+0000 7f3d037fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c004e10 con 0x5617fe49d7b0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617fe0fb7e0 con 0x5617fe49d7b0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d1c003970 con 0x5617fe49d7b0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d1c003b30 con 0x5617fe49d7b0 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d2d013740 10 client.17796 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:41.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5617fe3fa610 con 0x5617fe49d7b0 2024-03-12T05:19:41.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.966+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d1c042020 con 0x5617fe49d7b0 2024-03-12T05:19:41.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3d1c034070 con 0x5617fe49d7b0 2024-03-12T05:19:41.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 10 client.17796 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:41.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 10 client.17796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 20 client.17796 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe4ad9f0 0x5617fe4afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 4 client.17796 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617fe138720 con 0x5617fe4ad9f0 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.967+0000 7f3d23fff640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe4ad9f0 0x5617fe4afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.968+0000 7f3d23fff640 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe4ad9f0 0x5617fe4afe10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15ff3470f39b53e1 server_cookie=f040483ad78edaa1 in_seq=0 out_seq=0 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.968+0000 7f3d037fe640 10 client.17796 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d037fe640 1 -- 172.21.15.45:0/3716842709 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5617fe138720 con 0x5617fe4ad9f0 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d037fe640 10 client.17796 handle_command_reply: tid=0 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 client.17796 shutdown 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 msgr2=0x5617fe49a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c02ba70 tx=0x7f3d1c02b6c0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 shutdown_connections 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe4ad9f0 0x5617fe4afe10 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe49a990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe49a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3716842709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe49d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 >> 172.21.15.45:0/3716842709 conn(0x5617fe48e990 msgr2=0x5617fe490730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.969+0000 7f3d037fe640 0 client.17796 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.970+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 shutdown_connections 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.970+0000 7f3d2d013740 1 -- 172.21.15.45:0/3716842709 wait complete. 2024-03-12T05:19:41.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.970+0000 7f3d2d013740 20 client.17796 trim_cache size 0 max 16384 2024-03-12T05:19:41.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.997+0000 7f3d2d013740 1 Processor -- start 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 -- start start 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe494bd0 0x5617fe49d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe495710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe493be0 con 0x5617fe49d7b0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe49b7a0 con 0x5617fe4942b0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d2d013740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe48a3f0 con 0x5617fe494bd0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35848/0 (socket says 172.21.15.45:35848) 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d23fff640 1 -- 172.21.15.45:0/892041430 learned_addr learned my addr 172.21.15.45:0/892041430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe494bd0 0x5617fe49d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d289dd640 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe495710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.998+0000 7f3d23fff640 1 -- 172.21.15.45:0/892041430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe494bd0 msgr2=0x5617fe49d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d23fff640 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe494bd0 0x5617fe49d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d23fff640 1 -- 172.21.15.45:0/892041430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 msgr2=0x5617fe495710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d23fff640 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe495710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d23fff640 1 -- 172.21.15.45:0/892041430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617fe396ac0 con 0x5617fe4942b0 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d23fff640 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3d14005ec0 tx=0x7f3d14009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658459a192213d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d19ffb640 1 -- 172.21.15.45:0/892041430 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d140bd170 con 0x5617fe4942b0 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d19ffb640 1 -- 172.21.15.45:0/892041430 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d140bca70 con 0x5617fe4942b0 2024-03-12T05:19:41.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:41.999+0000 7f3d19ffb640 1 -- 172.21.15.45:0/892041430 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1407d340 con 0x5617fe4942b0 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.000+0000 7f3d2d013740 1 -- 172.21.15.45:0/892041430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 msgr2=0x5617fe49cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.000+0000 7f3d2d013740 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3d14005ec0 tx=0x7f3d14009260 comp rx=0 tx=0).stop 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 -- 172.21.15.45:0/892041430 shutdown_connections 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe495710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe494bd0 0x5617fe49d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 --2- 172.21.15.45:0/892041430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe4942b0 0x5617fe49cb80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 -- 172.21.15.45:0/892041430 >> 172.21.15.45:0/892041430 conn(0x5617fe48e990 msgr2=0x5617fe49a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 -- 172.21.15.45:0/892041430 shutdown_connections 2024-03-12T05:19:41.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.001+0000 7f3d2d013740 1 -- 172.21.15.45:0/892041430 wait complete. 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 -- start start 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 Processor -- start 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe4a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe4a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe514f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe4a2400 con 0x5617fe49d7b0 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe515440 con 0x5617fe494bd0 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617fe515580 con 0x5617fe4942b0 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d2d013740 10 client.0 _ll_register_callbacks cb 0x7f3d1c009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:19:42.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.002+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe514f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d289dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe514f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32796/0 (socket says 172.21.15.45:32796) 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe4a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d289dd640 1 -- 172.21.15.45:0/3468739152 learned_addr learned my addr 172.21.15.45:0/3468739152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe4a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 -- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 msgr2=0x5617fe4a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe4a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 -- 172.21.15.45:0/3468739152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 msgr2=0x5617fe514f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe514f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 -- 172.21.15.45:0/3468739152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2406ad80 con 0x5617fe4942b0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d23fff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe4a15e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x5617fe493d50 tx=0x7f3d140b8df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ce13714d862048 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d037fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.003+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d140c5210 con 0x5617fe4942b0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d140bd170 con 0x5617fe4942b0 2024-03-12T05:19:42.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d1c022d20 con 0x5617fe4942b0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d140b90e0 con 0x5617fe4942b0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d2d013740 10 client.27274 fetch_fsmap learned FSMap version 33 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5617fe3bc6a0 con 0x5617fe4942b0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d1c022d20 con 0x5617fe4942b0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.004+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3d1403c070 con 0x5617fe4942b0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 10 client.27274 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 10 client.27274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 20 client.27274 populate_metadata read hostname 'smithi045' 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe516870 0x5617fe518c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 4 client.27274 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5617fe187890 con 0x5617fe516870 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe516870 0x5617fe518c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d1bfff640 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe516870 0x5617fe518c90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd8611ba23d70321 server_cookie=22ad0b3212c07952 in_seq=0 out_seq=0 2024-03-12T05:19:42.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.005+0000 7f3d037fe640 10 client.27274 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d037fe640 1 -- 172.21.15.45:0/3468739152 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5617fe187890 con 0x5617fe516870 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d037fe640 10 client.27274 handle_command_reply: tid=0 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 client.27274 shutdown 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 msgr2=0x5617fe4a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe4a15e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x5617fe493d50 tx=0x7f3d140b8df0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 shutdown_connections 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5617fe516870 0x5617fe518c90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617fe49d7b0 0x5617fe514f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d037fe640 0 client.27274 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617fe494bd0 0x5617fe4a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 --2- 172.21.15.45:0/3468739152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617fe4942b0 0x5617fe4a15e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 >> 172.21.15.45:0/3468739152 conn(0x5617fe48e990 msgr2=0x5617fe495710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 shutdown_connections 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.006+0000 7f3d2d013740 1 -- 172.21.15.45:0/3468739152 wait complete. 2024-03-12T05:19:42.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.007+0000 7f3d2d013740 20 client.27274 trim_cache size 0 max 16384 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "186f8cd4-6f8a-48e1-a615-81a596014065": { 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "186f8cd4-6f8a-48e1-a615-81a596014065", 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:19:42.007 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:19:42.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 msgr2=0x7f3d100a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:42.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 0x7f3d100a4230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3d24104ef0 tx=0x7f3d0c014840 comp rx=0 tx=0).stop 2024-03-12T05:19:42.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 msgr2=0x7f3d2419dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3d14000c00 tx=0x7f3d1402d890 comp rx=0 tx=0).stop 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 shutdown_connections 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d100a1d70 0x7f3d100a4230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d24112140 0x7f3d241a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d24073e40 0x7f3d2419e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 --2- 172.21.15.45:0/2173010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d24073520 0x7f3d2419dda0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 >> 172.21.15.45:0/2173010373 conn(0x7f3d240fcc00 msgr2=0x7f3d240fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 shutdown_connections 2024-03-12T05:19:42.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:19:42.008+0000 7f3d2a440640 1 -- 172.21.15.45:0/2173010373 wait complete. 2024-03-12T05:19:42.022 INFO:tasks.cephfs.filesystem:scrub status for tag:186f8cd4-6f8a-48e1-a615-81a596014065 - {'path': '/', 'tag': '186f8cd4-6f8a-48e1-a615-81a596014065', 'options': 'recursive,force'} 2024-03-12T05:19:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:42 smithi116 ceph-mon[35486]: pgmap v273: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:19:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:42 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:42 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:42 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:42.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:42 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:42.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:42 smithi045 ceph-mon[32174]: pgmap v273: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:19:42.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:42 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:42.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:42 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:42.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:42 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:42.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:42 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:42 smithi130 ceph-mon[35398]: pgmap v273: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:19:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:42 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:42 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:42 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:42.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:42 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:43 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:43 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:43 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:19:44.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:44 smithi045 ceph-mon[32174]: pgmap v274: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-03-12T05:19:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 460.0 s, 1427.9 tps, lat 70.173 ms stddev 26.939 2024-03-12T05:19:44.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:44 smithi130 ceph-mon[35398]: pgmap v274: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-03-12T05:19:44.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:44 smithi116 ceph-mon[35486]: pgmap v274: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-03-12T05:19:46.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:46 smithi045 ceph-mon[32174]: pgmap v275: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 16 MiB/s wr, 1.87k op/s 2024-03-12T05:19:46.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:46 smithi130 ceph-mon[35398]: pgmap v275: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 16 MiB/s wr, 1.87k op/s 2024-03-12T05:19:46.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:46 smithi116 ceph-mon[35486]: pgmap v275: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 16 MiB/s wr, 1.87k op/s 2024-03-12T05:19:48.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:48 smithi045 ceph-mon[32174]: pgmap v276: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 1.16k op/s 2024-03-12T05:19:48.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:48 smithi130 ceph-mon[35398]: pgmap v276: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 1.16k op/s 2024-03-12T05:19:48.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:48 smithi116 ceph-mon[35486]: pgmap v276: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.9 MiB/s wr, 1.16k op/s 2024-03-12T05:19:49.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:49 smithi045 ceph-mon[32174]: pgmap v277: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 465.0 s, 1494.6 tps, lat 66.977 ms stddev 30.060 2024-03-12T05:19:49.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:49 smithi130 ceph-mon[35398]: pgmap v277: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:49.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:49 smithi116 ceph-mon[35486]: pgmap v277: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:19:51.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:51 smithi130 ceph-mon[35398]: pgmap v278: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:19:51.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:51 smithi116 ceph-mon[35486]: pgmap v278: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:19:52.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:51 smithi045 ceph-mon[32174]: pgmap v278: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:19:52.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:52 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:52.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:52 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:53.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:52 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:53.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:53.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:53.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:19:53.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:53 smithi130 ceph-mon[35398]: pgmap v279: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.9 MiB/s wr, 1.15k op/s 2024-03-12T05:19:53.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:53.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:53.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:19:53.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:53 smithi116 ceph-mon[35486]: pgmap v279: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.9 MiB/s wr, 1.15k op/s 2024-03-12T05:19:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:19:54.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:53 smithi045 ceph-mon[32174]: pgmap v279: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.9 MiB/s wr, 1.15k op/s 2024-03-12T05:19:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 470.0 s, 1370.0 tps, lat 72.790 ms stddev 37.740 2024-03-12T05:19:55.970 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:55 smithi130 ceph-mon[35398]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:19:55.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:55 smithi116 ceph-mon[35486]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:19:56.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:55 smithi045 ceph-mon[32174]: pgmap v280: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:19:57.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:57 smithi130 ceph-mon[35398]: pgmap v281: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-03-12T05:19:57.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:57 smithi116 ceph-mon[35486]: pgmap v281: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-03-12T05:19:58.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:57 smithi045 ceph-mon[32174]: pgmap v281: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.16k op/s 2024-03-12T05:19:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 475.0 s, 1314.8 tps, lat 76.259 ms stddev 59.405 2024-03-12T05:19:59.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:19:59 smithi130 ceph-mon[35398]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:19:59.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:19:59 smithi116 ceph-mon[35486]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:20:00.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:19:59 smithi045 ceph-mon[32174]: pgmap v282: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:20:00.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:00 smithi130 ceph-mon[35398]: overall HEALTH_OK 2024-03-12T05:20:00.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:00 smithi116 ceph-mon[35486]: overall HEALTH_OK 2024-03-12T05:20:01.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:00 smithi045 ceph-mon[32174]: overall HEALTH_OK 2024-03-12T05:20:01.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:01 smithi116 ceph-mon[35486]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:20:01.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:01 smithi130 ceph-mon[35398]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:20:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:01 smithi045 ceph-mon[32174]: pgmap v283: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:20:03.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:03 smithi130 ceph-mon[35398]: pgmap v284: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:20:03.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:03 smithi116 ceph-mon[35486]: pgmap v284: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:20:04.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:03 smithi045 ceph-mon[32174]: pgmap v284: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:20:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 480.0 s, 1442.8 tps, lat 69.344 ms stddev 30.604 2024-03-12T05:20:05.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:05 smithi130 ceph-mon[35398]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:20:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:05 smithi116 ceph-mon[35486]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:20:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:05 smithi045 ceph-mon[32174]: pgmap v285: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:20:07.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:07 smithi130 ceph-mon[35398]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 1.11k op/s 2024-03-12T05:20:07.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:07 smithi116 ceph-mon[35486]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 1.11k op/s 2024-03-12T05:20:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:07 smithi045 ceph-mon[32174]: pgmap v286: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.7 MiB/s wr, 1.11k op/s 2024-03-12T05:20:08.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:08 smithi130 ceph-mon[35398]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T05:20:08.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:08 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:20:08.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:08 smithi116 ceph-mon[35486]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T05:20:08.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:08 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:20:09.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:08 smithi045 ceph-mon[32174]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T05:20:09.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:08 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:20:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 485.0 s, 1476.6 tps, lat 67.594 ms stddev 37.259 2024-03-12T05:20:09.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:09 smithi130 ceph-mon[35398]: pgmap v288: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:09.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:09 smithi116 ceph-mon[35486]: pgmap v288: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:10.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:09 smithi045 ceph-mon[32174]: pgmap v288: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:11.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:11 smithi130 ceph-mon[35398]: pgmap v289: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:11.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:11 smithi116 ceph-mon[35486]: pgmap v289: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:12.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:20:12.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:20:12.070 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:11 smithi045 ceph-mon[32174]: pgmap v289: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:12.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 -- 172.21.15.45:0/2310716047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc112100 msgr2=0x7fa6cc1144f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 --2- 172.21.15.45:0/2310716047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc112100 0x7fa6cc1144f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009860 tx=0x7fa6b402bc10 comp rx=0 tx=0).stop 2024-03-12T05:20:12.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 -- 172.21.15.45:0/2310716047 shutdown_connections 2024-03-12T05:20:12.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 --2- 172.21.15.45:0/2310716047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 0x7fa6cc111bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 --2- 172.21.15.45:0/2310716047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc112100 0x7fa6cc1144f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 --2- 172.21.15.45:0/2310716047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc073520 0x7fa6cc073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 -- 172.21.15.45:0/2310716047 >> 172.21.15.45:0/2310716047 conn(0x7fa6cc0fcc20 msgr2=0x7fa6cc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 -- 172.21.15.45:0/2310716047 shutdown_connections 2024-03-12T05:20:12.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.237+0000 7fa6d2502640 1 -- 172.21.15.45:0/2310716047 wait complete. 2024-03-12T05:20:12.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 Processor -- start 2024-03-12T05:20:12.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 -- start start 2024-03-12T05:20:12.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 0x7fa6cc1a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc112100 0x7fa6cc19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc19e100 con 0x7fa6cc073e40 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc19e270 con 0x7fa6cc073520 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.238+0000 7fa6d2502640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc19e430 con 0x7fa6cc112100 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60396/0 (socket says 172.21.15.45:60396) 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 -- 172.21.15.45:0/1944398888 learned_addr learned my addr 172.21.15.45:0/1944398888 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc112100 0x7fa6cc19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 0x7fa6cc1a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 -- 172.21.15.45:0/1944398888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc112100 msgr2=0x7fa6cc19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc112100 0x7fa6cc19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 -- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 msgr2=0x7fa6cc1a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 0x7fa6cc1a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 -- 172.21.15.45:0/1944398888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc009840 con 0x7fa6cc073520 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.239+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400e9c0 tx=0x7fa6c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.240+0000 7fa6c97fa640 1 -- 172.21.15.45:0/1944398888 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400ccc0 con 0x7fa6cc073520 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.240+0000 7fa6c97fa640 1 -- 172.21.15.45:0/1944398888 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa6c400ce20 con 0x7fa6cc073520 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.240+0000 7fa6c97fa640 1 -- 172.21.15.45:0/1944398888 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4010430 con 0x7fa6cc073520 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.240+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b4009490 con 0x7fa6cc073520 2024-03-12T05:20:12.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.240+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc19e900 con 0x7fa6cc073520 2024-03-12T05:20:12.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.244+0000 7fa6c97fa640 1 -- 172.21.15.45:0/1944398888 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa6c4004ee0 con 0x7fa6cc073520 2024-03-12T05:20:12.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.245+0000 7fa6c97fa640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 0x7fa6b80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.245+0000 7fa6c97fa640 1 -- 172.21.15.45:0/1944398888 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa6c4014070 con 0x7fa6cc073520 2024-03-12T05:20:12.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.245+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 0x7fa6b80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.246+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 0x7fa6b80a40e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ff320 tx=0x7fa6bc009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.247+0000 7fa6d51b4740 1 Processor -- start 2024-03-12T05:20:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.247+0000 7fa6d51b4740 1 -- start start 2024-03-12T05:20:12.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.247+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4ba9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4ba9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4ba9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41242/0 (socket says 172.21.15.45:41242) 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6cbfff640 1 -- 172.21.15.45:0/3139072695 learned_addr learned my addr 172.21.15.45:0/3139072695 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 0x5608b4bb2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb27b0 0x5608b4bb4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4b18280 con 0x5608b4ba92b0 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4b157b0 con 0x5608b4bb27b0 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4b10f90 con 0x5608b4ba9bd0 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 0x5608b4bb2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.248+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb27b0 0x5608b4bb4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/3139072695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb27b0 msgr2=0x5608b4bb4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb27b0 0x5608b4bb4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/3139072695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 msgr2=0x5608b4ba9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4ba9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/3139072695 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608b4acf570 con 0x5608b4ba9bd0 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 0x5608b4bb2270 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005fd0 tx=0x7fa6bc013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d22fcedb38af860a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.249+0000 7fa6c27fc640 1 -- 172.21.15.45:0/3139072695 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc004d20 con 0x5608b4ba9bd0 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.250+0000 7fa6c27fc640 1 -- 172.21.15.45:0/3139072695 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa6bc015070 con 0x5608b4ba9bd0 2024-03-12T05:20:12.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.250+0000 7fa6c27fc640 1 -- 172.21.15.45:0/3139072695 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0054c0 con 0x5608b4ba9bd0 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 msgr2=0x5608b4bb2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 0x5608b4bb2270 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005fd0 tx=0x7fa6bc013000 comp rx=0 tx=0).stop 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 shutdown_connections 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb27b0 0x5608b4bb4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4ba9bd0 0x5608b4bb2270 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3139072695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4ba9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.251+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 >> 172.21.15.45:0/3139072695 conn(0x5608b4ba3990 msgr2=0x5608b4ba3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.252+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 shutdown_connections 2024-03-12T05:20:12.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.252+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3139072695 wait complete. 2024-03-12T05:20:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.253+0000 7fa6d51b4740 1 -- start start 2024-03-12T05:20:12.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.253+0000 7fa6d51b4740 1 Processor -- start 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4baf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb27b0 0x5608b4bafbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b47e4430 con 0x5608b4bb27b0 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b47a0e00 con 0x5608b4ba92b0 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4a95fd0 con 0x5608b4bb0110 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d51b4740 10 client.0 _ll_register_callbacks cb 0x5608b4aebfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35200/0 (socket says 172.21.15.45:35200) 2024-03-12T05:20:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 -- 172.21.15.45:0/4029399492 learned_addr learned my addr 172.21.15.45:0/4029399492 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 -- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 msgr2=0x5608b4baf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4baf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 -- 172.21.15.45:0/4029399492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb27b0 msgr2=0x5608b4bafbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb27b0 0x5608b4bafbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.254+0000 7fa6d0a9f640 1 -- 172.21.15.45:0/4029399492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc011500 con 0x5608b4bb0110 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc06a870 tx=0x7fa6b4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9d30ff2183454f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4034640 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa6b40347a0 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608b48107e0 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6b4034960 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6d51b4740 10 client.27286 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5608b4b0f610 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.255+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5608b48107e0 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa6b4003b20 con 0x5608b4bb0110 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 10 client.27286 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 10 client.27286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 20 client.27286 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4bc2ec0 0x5608b4bc5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 4 client.27286 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608b484d720 con 0x5608b4bc2ec0 2024-03-12T05:20:12.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.256+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4bc2ec0 0x5608b4bc5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.257+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4bc2ec0 0x5608b4bc5320 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43437184f4c3c9e server_cookie=e3dd4cd39c4c2b81 in_seq=0 out_seq=0 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.257+0000 7fa6ab7fe640 10 client.27286 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.257+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/4029399492 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5608b484d720 con 0x5608b4bc2ec0 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.257+0000 7fa6ab7fe640 10 client.27286 handle_command_reply: tid=0 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.257+0000 7fa6d51b4740 1 client.27286 shutdown 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 msgr2=0x5608b4bbe710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc06a870 tx=0x7fa6b4004270 comp rx=0 tx=0).stop 2024-03-12T05:20:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 shutdown_connections 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4bc2ec0 0x5608b4bc5320 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb27b0 0x5608b4bafbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb0110 0x5608b4bbe710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/4029399492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4baf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 >> 172.21.15.45:0/4029399492 conn(0x5608b4ba3990 msgr2=0x5608b4ba5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6ab7fe640 0 client.27286 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.258+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 shutdown_connections 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.259+0000 7fa6d51b4740 1 -- 172.21.15.45:0/4029399492 wait complete. 2024-03-12T05:20:12.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.259+0000 7fa6d51b4740 20 client.27286 trim_cache size 0 max 16384 2024-03-12T05:20:12.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.291+0000 7fa6d51b4740 1 Processor -- start 2024-03-12T05:20:12.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.291+0000 7fa6d51b4740 1 -- start start 2024-03-12T05:20:12.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.291+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.291+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 0x5608b4bb9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.291+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bbe110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41266/0 (socket says 172.21.15.45:41266) 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cbfff640 1 -- 172.21.15.45:0/2824967626 learned_addr learned my addr 172.21.15.45:0/2824967626 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 0x5608b4bb9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4bb0650 con 0x5608b4ba92b0 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4b9f0e0 con 0x5608b4bb0110 2024-03-12T05:20:12.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4b9de60 con 0x5608b4bb27b0 2024-03-12T05:20:12.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bbe110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/2824967626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 msgr2=0x5608b4bbe110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bbe110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/2824967626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 msgr2=0x5608b4bb9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cb7fe640 1 -- 172.21.15.45:0/2824967626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608b4aabac0 con 0x5608b4bb0110 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.292+0000 7fa6d0a9f640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bbe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.293+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 0x5608b4bb9ba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5608b4baa0f0 tx=0x7fa6bc0093a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40cd8322e583d88a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.293+0000 7fa6c1ffb640 1 -- 172.21.15.45:0/2824967626 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00e8f0 con 0x5608b4bb0110 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.293+0000 7fa6c1ffb640 1 -- 172.21.15.45:0/2824967626 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa6bc005000 con 0x5608b4bb0110 2024-03-12T05:20:12.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.293+0000 7fa6c1ffb640 1 -- 172.21.15.45:0/2824967626 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc01e680 con 0x5608b4bb0110 2024-03-12T05:20:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 msgr2=0x5608b4bb9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 0x5608b4bb9ba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5608b4baa0f0 tx=0x7fa6bc0093a0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 shutdown_connections 2024-03-12T05:20:12.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bbe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4bb0110 0x5608b4bb9ba0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/2824967626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4ba92b0 0x5608b4bb9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 >> 172.21.15.45:0/2824967626 conn(0x5608b4ba3990 msgr2=0x5608b4baa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 shutdown_connections 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- 172.21.15.45:0/2824967626 wait complete. 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.295+0000 7fa6d51b4740 1 -- start start 2024-03-12T05:20:12.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 Processor -- start 2024-03-12T05:20:12.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb0110 0x5608b4bb7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bb8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4bb8a70 con 0x5608b4bb0110 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4bb8be0 con 0x5608b4ba92b0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b4c19450 con 0x5608b4bb27b0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d51b4740 10 client.0 _ll_register_callbacks cb 0x7fa6bc009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60430/0 (socket says 172.21.15.45:60430) 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6d0a9f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bb8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6cbfff640 1 -- 172.21.15.45:0/3555789719 learned_addr learned my addr 172.21.15.45:0/3555789719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.296+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb0110 0x5608b4bb7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 -- 172.21.15.45:0/3555789719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 msgr2=0x5608b4bb8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bb8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 -- 172.21.15.45:0/3555789719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb0110 msgr2=0x5608b4bb7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb0110 0x5608b4bb7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 -- 172.21.15.45:0/3555789719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc19e900 con 0x5608b4ba92b0 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6cbfff640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa6c409a410 tx=0x7fa6c409aed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23be7a5acde70f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40a2eb0 con 0x5608b4ba92b0 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.297+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa6c4098a10 con 0x5608b4ba92b0 2024-03-12T05:20:12.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6bc015070 con 0x5608b4ba92b0 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c40cb4b0 con 0x5608b4ba92b0 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 10 client.27299 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5608b4b0f610 con 0x5608b4ba92b0 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa6bc015070 con 0x5608b4ba92b0 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa6c401f060 con 0x5608b4ba92b0 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 10 client.27299 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 10 client.27299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:12.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 20 client.27299 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4c1ac10 0x5608b4c1d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 4 client.27299 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.298+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5608b489c890 con 0x5608b4c1ac10 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4c1ac10 0x5608b4c1d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6ab7fe640 10 client.27299 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6cb7fe640 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4c1ac10 0x5608b4c1d030 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8428ef8b266bb249 server_cookie=5183e3aa0da1275a in_seq=0 out_seq=0 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6ab7fe640 1 -- 172.21.15.45:0/3555789719 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5608b489c890 con 0x5608b4c1ac10 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6ab7fe640 10 client.27299 handle_command_reply: tid=0 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.299+0000 7fa6d51b4740 1 client.27299 shutdown 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 msgr2=0x5608b4bb99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa6c409a410 tx=0x7fa6c409aed0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 shutdown_connections 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5608b4c1ac10 0x5608b4c1d030 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5608b4bb0110 0x5608b4bb7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5608b4bb27b0 0x5608b4bb8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6ab7fe640 0 client.27299 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 --2- 172.21.15.45:0/3555789719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5608b4ba92b0 0x5608b4bb99f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 >> 172.21.15.45:0/3555789719 conn(0x5608b4ba3990 msgr2=0x5608b4bbe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 shutdown_connections 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 1 -- 172.21.15.45:0/3555789719 wait complete. 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.300+0000 7fa6d51b4740 20 client.27299 trim_cache size 0 max 16384 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:20:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:20:12.299 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 msgr2=0x7fa6b80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 0x7fa6b80a40e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0ff320 tx=0x7fa6bc009500 comp rx=0 tx=0).stop 2024-03-12T05:20:12.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 msgr2=0x7fa6cc1a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400e9c0 tx=0x7fa6c400ee80 comp rx=0 tx=0).stop 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 shutdown_connections 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa6b80a1c20 0x7fa6b80a40e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.301+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6cc073e40 0x7fa6cc1a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.302+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6cc112100 0x7fa6cc19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.302+0000 7fa6d2502640 1 --2- 172.21.15.45:0/1944398888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc073520 0x7fa6cc1a3900 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.302+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 >> 172.21.15.45:0/1944398888 conn(0x7fa6cc0fcc20 msgr2=0x7fa6cc0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.302+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 shutdown_connections 2024-03-12T05:20:12.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.302+0000 7fa6d2502640 1 -- 172.21.15.45:0/1944398888 wait complete. 2024-03-12T05:20:12.317 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:20:12.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 -- 172.21.15.45:0/1637861234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c10a330 msgr2=0x7f8e3c1167f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/1637861234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1167f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009cd0 tx=0x7f8e3402c0e0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 -- 172.21.15.45:0/1637861234 shutdown_connections 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/1637861234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1167f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/1637861234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c109970 0x7f8e3c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/1637861234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10f930 0x7f8e3c10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 -- 172.21.15.45:0/1637861234 >> 172.21.15.45:0/1637861234 conn(0x7f8e3c072720 msgr2=0x7f8e3c072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 -- 172.21.15.45:0/1637861234 shutdown_connections 2024-03-12T05:20:12.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.492+0000 7f8e42b7a640 1 -- 172.21.15.45:0/1637861234 wait complete. 2024-03-12T05:20:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.493+0000 7f8e42b7a640 1 Processor -- start 2024-03-12T05:20:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.493+0000 7f8e42b7a640 1 -- start start 2024-03-12T05:20:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c109970 0x7f8e3c19ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1a4e30 con 0x7f8e3c109970 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1a4fa0 con 0x7f8e3c10a330 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e42b7a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c1a5160 con 0x7f8e3c10f930 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e41117640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60448/0 (socket says 172.21.15.45:60448) 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e41117640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35236/0 (socket says 172.21.15.45:35236) 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 -- 172.21.15.45:0/2492173560 learned_addr learned my addr 172.21.15.45:0/2492173560 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e40916640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c109970 0x7f8e3c19ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 -- 172.21.15.45:0/2492173560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 msgr2=0x7f8e3c1a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 -- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c109970 msgr2=0x7f8e3c19ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.494+0000 7f8e3bfff640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c109970 0x7f8e3c19ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.495+0000 7f8e3bfff640 1 -- 172.21.15.45:0/2492173560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34009900 con 0x7f8e3c10a330 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.495+0000 7f8e41117640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.495+0000 7f8e3bfff640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e9f0 tx=0x7f8e2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.495+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e28009700 con 0x7f8e3c10a330 2024-03-12T05:20:12.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.496+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8e2800cd00 con 0x7f8e3c10a330 2024-03-12T05:20:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.496+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e280187a0 con 0x7f8e3c10a330 2024-03-12T05:20:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.496+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c1a5440 con 0x7f8e3c10a330 2024-03-12T05:20:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.496+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c1accc0 con 0x7f8e3c10a330 2024-03-12T05:20:12.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.499+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c067f70 con 0x7f8e3c10a330 2024-03-12T05:20:12.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.502+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8e28010430 con 0x7f8e3c10a330 2024-03-12T05:20:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.503+0000 7f8e39ffb640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 0x7f8e2c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:12.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.503+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f8e28014070 con 0x7f8e3c10a330 2024-03-12T05:20:12.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.504+0000 7f8e40916640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 0x7f8e2c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:12.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.505+0000 7f8e40916640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 0x7f8e2c0a4270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009ae0 tx=0x7f8e30005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:12.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.508+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e28052d20 con 0x7f8e3c10a330 2024-03-12T05:20:12.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.921+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e3c111eb0 con 0x7f8e3c10a330 2024-03-12T05:20:12.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.922+0000 7f8e39ffb640 1 -- 172.21.15.45:0/2492173560 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f8e280969f0 con 0x7f8e3c10a330 2024-03-12T05:20:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:20:12.918 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:20:12.918 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 msgr2=0x7f8e2c0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 0x7f8e2c0a4270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009ae0 tx=0x7f8e30005f90 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 msgr2=0x7f8e3c1a04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e9f0 tx=0x7f8e2800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 shutdown_connections 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8e2c0a1db0 0x7f8e2c0a4270 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e3c10f930 0x7f8e3c1a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e3c10a330 0x7f8e3c1a04b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 --2- 172.21.15.45:0/2492173560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e3c109970 0x7f8e3c19ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 >> 172.21.15.45:0/2492173560 conn(0x7f8e3c072720 msgr2=0x7f8e3c06df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:12.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.926+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 shutdown_connections 2024-03-12T05:20:12.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:12.927+0000 7f8e42b7a640 1 -- 172.21.15.45:0/2492173560 wait complete. 2024-03-12T05:20:12.941 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:12.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:20:12.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:12 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:20:12.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:12 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:20:12.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:12 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:20:12.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:12 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:20:12.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:12 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:20:12.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:12 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:20:12.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:12 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:20:12.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:12 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:20:12.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:12 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.118+0000 7feb2794b640 1 -- 172.21.15.45:0/2155146791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 msgr2=0x7feb20100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.118+0000 7feb2794b640 1 --2- 172.21.15.45:0/2155146791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20100640 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009930 tx=0x7feb1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.118+0000 7feb2794b640 1 -- 172.21.15.45:0/2155146791 shutdown_connections 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.118+0000 7feb2794b640 1 --2- 172.21.15.45:0/2155146791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20105330 0x7feb2010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.118+0000 7feb2794b640 1 --2- 172.21.15.45:0/2155146791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20100c10 0x7feb20104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.119+0000 7feb2794b640 1 --2- 172.21.15.45:0/2155146791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20100640 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.119+0000 7feb2794b640 1 -- 172.21.15.45:0/2155146791 >> 172.21.15.45:0/2155146791 conn(0x7feb20077180 msgr2=0x7feb20077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.119+0000 7feb2794b640 1 -- 172.21.15.45:0/2155146791 shutdown_connections 2024-03-12T05:20:13.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.119+0000 7feb2794b640 1 -- 172.21.15.45:0/2155146791 wait complete. 2024-03-12T05:20:13.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.120+0000 7feb2794b640 1 Processor -- start 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 -- start start 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20105330 0x7feb20102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 0x7feb201a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201a5ed0 con 0x7feb20100260 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201a6040 con 0x7feb20102770 2024-03-12T05:20:13.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb2794b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201a6200 con 0x7feb20105330 2024-03-12T05:20:13.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb256e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb256e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41308/0 (socket says 172.21.15.45:41308) 2024-03-12T05:20:13.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb256e7640 1 -- 172.21.15.45:0/728349207 learned_addr learned my addr 172.21.15.45:0/728349207 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 0x7feb201a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.121+0000 7feb24ee6640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20105330 0x7feb20102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 -- 172.21.15.45:0/728349207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20105330 msgr2=0x7feb20102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20105330 0x7feb20102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 -- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 msgr2=0x7feb20101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 -- 172.21.15.45:0/728349207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10009490 con 0x7feb20102770 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 0x7feb201a5900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7feb18009fd0 tx=0x7feb1800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.122+0000 7feb167fc640 1 -- 172.21.15.45:0/728349207 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1800ce20 con 0x7feb20102770 2024-03-12T05:20:13.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.123+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1c009560 con 0x7feb20102770 2024-03-12T05:20:13.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.123+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201a6860 con 0x7feb20102770 2024-03-12T05:20:13.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.123+0000 7feb167fc640 1 -- 172.21.15.45:0/728349207 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb18005020 con 0x7feb20102770 2024-03-12T05:20:13.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.123+0000 7feb167fc640 1 -- 172.21.15.45:0/728349207 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18010690 con 0x7feb20102770 2024-03-12T05:20:13.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.126+0000 7feb2a5f4740 1 Processor -- start 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb167fc640 1 -- 172.21.15.45:0/728349207 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7feb18004b20 con 0x7feb20102770 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 -- start start 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47a9b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 0x561f47aa4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47aa4750 0x561f47aa6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47a56960 con 0x561f47a9bb70 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47a0a310 con 0x561f47a9b250 2024-03-12T05:20:13.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.127+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47a07840 con 0x561f47aa4750 2024-03-12T05:20:13.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb256e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 0x561f47aa4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb256e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 0x561f47aa4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41322/0 (socket says 172.21.15.45:41322) 2024-03-12T05:20:13.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb256e7640 1 -- 172.21.15.45:0/232371758 learned_addr learned my addr 172.21.15.45:0/232371758 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47a9b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47aa4750 0x561f47aa6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb167fc640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 0x7feb0c0a4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb167fc640 1 -- 172.21.15.45:0/728349207 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7feb18014070 con 0x7feb20102770 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 -- 172.21.15.45:0/232371758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47aa4750 msgr2=0x561f47aa6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47aa4750 0x561f47aa6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 -- 172.21.15.45:0/232371758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 msgr2=0x561f47aa4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 0x561f47aa4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.128+0000 7feb24ee6640 1 -- 172.21.15.45:0/232371758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f479c5060 con 0x561f47a9b250 2024-03-12T05:20:13.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.129+0000 7feb24ee6640 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47a9b630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7feb100093e0 tx=0x7feb10009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f52f638c1fc2253 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.129+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 0x7feb0c0a4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.129+0000 7feb0b7fe640 1 -- 172.21.15.45:0/232371758 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10018070 con 0x561f47a9b250 2024-03-12T05:20:13.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.129+0000 7feb0b7fe640 1 -- 172.21.15.45:0/232371758 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb10004220 con 0x561f47a9b250 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.130+0000 7feb2a5f4740 1 -- 172.21.15.45:0/232371758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 msgr2=0x561f47a9b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.130+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47a9b630 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7feb100093e0 tx=0x7feb10009370 comp rx=0 tx=0).stop 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.130+0000 7feb25ee8640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 0x7feb0c0a4190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feb180059e0 tx=0x7feb18004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 -- 172.21.15.45:0/232371758 shutdown_connections 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9bb70 0x561f47aa4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47aa4750 0x561f47aa6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/232371758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47a9b630 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 -- 172.21.15.45:0/232371758 >> 172.21.15.45:0/232371758 conn(0x561f47a95980 msgr2=0x561f47a95d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 -- 172.21.15.45:0/232371758 shutdown_connections 2024-03-12T05:20:13.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.131+0000 7feb2a5f4740 1 -- 172.21.15.45:0/232371758 wait complete. 2024-03-12T05:20:13.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.132+0000 7feb2a5f4740 1 -- start start 2024-03-12T05:20:13.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.132+0000 7feb2a5f4740 1 Processor -- start 2024-03-12T05:20:13.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.132+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47aa1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 0x561f47aa2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47a03020 con 0x561f47a9b250 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f476d6430 con 0x561f47aa4750 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47692e00 con 0x561f47a9bb70 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb2a5f4740 10 client.0 _ll_register_callbacks cb 0x561f479aacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41324/0 (socket says 172.21.15.45:41324) 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 -- 172.21.15.45:0/3355691102 learned_addr learned my addr 172.21.15.45:0/3355691102 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 0x561f47aa2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb256e7640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47aa1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 -- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 msgr2=0x561f47aa1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47aa1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 -- 172.21.15.45:0/3355691102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 msgr2=0x561f47aa1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.133+0000 7feb24ee6640 1 -- 172.21.15.45:0/3355691102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb10013400 con 0x561f47aa4750 2024-03-12T05:20:13.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.134+0000 7feb24ee6640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 0x561f47aa2090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feb10003e70 tx=0x7feb10003d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be38202fb4c82b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.134+0000 7feb08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10018070 con 0x561f47aa4750 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7feb10005030 con 0x561f47aa4750 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f477027e0 con 0x561f47aa4750 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb10010ec0 con 0x561f47aa4750 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb2a5f4740 10 client.27317 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561f47a016a0 con 0x561f47aa4750 2024-03-12T05:20:13.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.135+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561f477027e0 con 0x561f47aa4750 2024-03-12T05:20:13.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feb10004680 con 0x561f47aa4750 2024-03-12T05:20:13.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 10 client.27317 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 10 client.27317 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 20 client.27317 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47abd1c0 0x561f47abf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 4 client.27317 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.136+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f4773f720 con 0x561f47abd1c0 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.137+0000 7feb256e7640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47abd1c0 0x561f47abf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.139+0000 7feb256e7640 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47abd1c0 0x561f47abf5e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a34968f5c3f0ac25 server_cookie=8a182abac8dc229a in_seq=0 out_seq=0 2024-03-12T05:20:13.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.139+0000 7feb08ff9640 10 client.27317 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:13.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.140+0000 7feb08ff9640 1 -- 172.21.15.45:0/3355691102 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561f4773f720 con 0x561f47abd1c0 2024-03-12T05:20:13.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.140+0000 7feb08ff9640 10 client.27317 handle_command_reply: tid=0 2024-03-12T05:20:13.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 client.27317 shutdown 2024-03-12T05:20:13.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 msgr2=0x561f47aa2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 0x561f47aa2090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feb10003e70 tx=0x7feb10003d70 comp rx=0 tx=0).stop 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 shutdown_connections 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47abd1c0 0x561f47abf5e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47aa4750 0x561f47aa2090 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feb10003e70 tx=0x7feb10003d70 comp rx=0 tx=0).stop 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47aa1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/3355691102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 >> 172.21.15.45:0/3355691102 conn(0x561f47a95980 msgr2=0x561f47a976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.141+0000 7feb08ff9640 0 client.27317 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:13.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.142+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 shutdown_connections 2024-03-12T05:20:13.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.142+0000 7feb2a5f4740 1 -- 172.21.15.45:0/3355691102 wait complete. 2024-03-12T05:20:13.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.142+0000 7feb2a5f4740 20 client.27317 trim_cache size 0 max 16384 2024-03-12T05:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.165+0000 7feb2a5f4740 1 Processor -- start 2024-03-12T05:20:13.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.165+0000 7feb2a5f4740 1 -- start start 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.165+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.165+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9e870 0x561f47a9ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47988130 con 0x561f47a9b250 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60504/0 (socket says 172.21.15.45:60504) 2024-03-12T05:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41328/0 (socket says 172.21.15.45:41328) 2024-03-12T05:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 -- 172.21.15.45:0/577401521 learned_addr learned my addr 172.21.15.45:0/577401521 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47a96bb0 con 0x561f47a9bb70 2024-03-12T05:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47ac47b0 con 0x561f47a9e870 2024-03-12T05:20:13.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb24ee6640 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9e870 0x561f47a9ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 -- 172.21.15.45:0/577401521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9e870 msgr2=0x561f47a9ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9e870 0x561f47a9ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 -- 172.21.15.45:0/577401521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 msgr2=0x561f47aa3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.166+0000 7feb256e7640 1 -- 172.21.15.45:0/577401521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f4799dc20 con 0x561f47a9bb70 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.167+0000 7feb256e7640 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x561f47a9ad20 tx=0x7feb1c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b85847667610e169 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.167+0000 7feb0affd640 1 -- 172.21.15.45:0/577401521 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c045040 con 0x561f47a9bb70 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.167+0000 7feb0affd640 1 -- 172.21.15.45:0/577401521 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb1c047040 con 0x561f47a9bb70 2024-03-12T05:20:13.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.167+0000 7feb0affd640 1 -- 172.21.15.45:0/577401521 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c04c690 con 0x561f47a9bb70 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.168+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 msgr2=0x561f47aa1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.168+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x561f47a9ad20 tx=0x7feb1c005a20 comp rx=0 tx=0).stop 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.168+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 shutdown_connections 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.168+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9e870 0x561f47a9ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.169+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9bb70 0x561f47aa1e80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.169+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/577401521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9b250 0x561f47aa3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.169+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 >> 172.21.15.45:0/577401521 conn(0x561f47a95980 msgr2=0x561f47a9c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.169+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 shutdown_connections 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.169+0000 7feb2a5f4740 1 -- 172.21.15.45:0/577401521 wait complete. 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.170+0000 7feb2a5f4740 1 -- start start 2024-03-12T05:20:13.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.170+0000 7feb2a5f4740 1 Processor -- start 2024-03-12T05:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.170+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47ab36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.170+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47ab3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb2a5f4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 0x561f47b0c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47b0ce20 con 0x561f47a9e870 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47b0cf90 con 0x561f47a9b250 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb2a5f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f47b0d150 con 0x561f47a9bb70 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb2a5f4740 10 client.0 _ll_register_callbacks cb 0x7feb10009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47ab36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb24ee6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47ab36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60510/0 (socket says 172.21.15.45:60510) 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb24ee6640 1 -- 172.21.15.45:0/914504794 learned_addr learned my addr 172.21.15.45:0/914504794 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.171+0000 7feb24ee6640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 0x561f47b0c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.172+0000 7feb256e7640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47ab3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 -- 172.21.15.45:0/914504794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 msgr2=0x561f47ab36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47ab36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 -- 172.21.15.45:0/914504794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 msgr2=0x561f47b0c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 0x561f47b0c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 -- 172.21.15.45:0/914504794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2010d710 con 0x561f47a9bb70 2024-03-12T05:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb24ee6640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 0x561f47b0c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb256e7640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47ab3c00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x561f47a96230 tx=0x7feb1c0047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8834673e3037a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.173+0000 7feb08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c045040 con 0x561f47a9bb70 2024-03-12T05:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7feb1c02de40 con 0x561f47a9bb70 2024-03-12T05:20:13.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb10004220 con 0x561f47a9bb70 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb1c02c970 con 0x561f47a9bb70 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb2a5f4740 10 client.27316 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.174+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561f479c38c0 con 0x561f47a9bb70 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7feb10004220 con 0x561f47a9bb70 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feb1c03c040 con 0x561f47a9bb70 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 10 client.27316 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 10 client.27316 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 20 client.27316 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47b15920 0x561f47b17d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 4 client.27316 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561f4778e890 con 0x561f47b15920 2024-03-12T05:20:13.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.175+0000 7feb24ee6640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47b15920 0x561f47b17d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb08ff9640 10 client.27316 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb24ee6640 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47b15920 0x561f47b17d40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99b949e84063bfef server_cookie=c15e2c1e95cfb327 in_seq=0 out_seq=0 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb08ff9640 1 -- 172.21.15.45:0/914504794 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561f4778e890 con 0x561f47b15920 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb08ff9640 10 client.27316 handle_command_reply: tid=0 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 client.27316 shutdown 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 msgr2=0x561f47ab3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47ab3c00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x561f47a96230 tx=0x7feb1c0047e0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 shutdown_connections 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561f47b15920 0x561f47b17d40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb08ff9640 0 client.27316 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561f47a9e870 0x561f47b0c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561f47a9bb70 0x561f47ab3c00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 --2- 172.21.15.45:0/914504794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f47a9b250 0x561f47ab36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 >> 172.21.15.45:0/914504794 conn(0x561f47a95980 msgr2=0x561f47a9ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 shutdown_connections 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.176+0000 7feb2a5f4740 1 -- 172.21.15.45:0/914504794 wait complete. 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.177+0000 7feb2a5f4740 20 client.27316 trim_cache size 0 max 16384 2024-03-12T05:20:13.174 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.177+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 msgr2=0x7feb0c0a4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.177+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 0x7feb0c0a4190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feb180059e0 tx=0x7feb18004a40 comp rx=0 tx=0).stop 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.177+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 msgr2=0x7feb201a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.177+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 0x7feb201a5900 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7feb18009fd0 tx=0x7feb1800ef10 comp rx=0 tx=0).stop 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 shutdown_connections 2024-03-12T05:20:13.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7feb0c0a1cd0 0x7feb0c0a4190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb20102770 0x7feb201a5900 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb20105330 0x7feb20102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 --2- 172.21.15.45:0/728349207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb20100260 0x7feb20101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 >> 172.21.15.45:0/728349207 conn(0x7feb20077180 msgr2=0x7feb200686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 shutdown_connections 2024-03-12T05:20:13.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.178+0000 7feb2794b640 1 -- 172.21.15.45:0/728349207 wait complete. 2024-03-12T05:20:13.191 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:13.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 -- 172.21.15.45:0/3607841405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc100c10 msgr2=0x7f70fc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 --2- 172.21.15.45:0/3607841405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc100c10 0x7f70fc104c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009930 tx=0x7f70ec02bc20 comp rx=0 tx=0).stop 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 -- 172.21.15.45:0/3607841405 shutdown_connections 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 --2- 172.21.15.45:0/3607841405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc105330 0x7f70fc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 --2- 172.21.15.45:0/3607841405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc100c10 0x7f70fc104c50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 --2- 172.21.15.45:0/3607841405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc100260 0x7f70fc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.423+0000 7f7104316640 1 -- 172.21.15.45:0/3607841405 >> 172.21.15.45:0/3607841405 conn(0x7f70fc077180 msgr2=0x7f70fc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.424+0000 7f7104316640 1 -- 172.21.15.45:0/3607841405 shutdown_connections 2024-03-12T05:20:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.424+0000 7f7104316640 1 -- 172.21.15.45:0/3607841405 wait complete. 2024-03-12T05:20:13.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.425+0000 7f7104316640 1 Processor -- start 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.425+0000 7f7104316640 1 -- start start 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.425+0000 7f7104316640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc100260 0x7f70fc10cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.425+0000 7f7104316640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f7104316640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 0x7f70fc10b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f7104316640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc1aa2e0 con 0x7f70fc100260 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f7104316640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc112c20 con 0x7f70fc10b440 2024-03-12T05:20:13.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f7104316640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc112de0 con 0x7f70fc105330 2024-03-12T05:20:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35286/0 (socket says 172.21.15.45:35286) 2024-03-12T05:20:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 -- 172.21.15.45:0/2022817547 learned_addr learned my addr 172.21.15.45:0/2022817547 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71020b2640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc100260 0x7f70fc10cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71028b3640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 0x7f70fc10b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 -- 172.21.15.45:0/2022817547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 msgr2=0x7f70fc10b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 0x7f70fc10b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 -- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc100260 msgr2=0x7f70fc10cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc100260 0x7f70fc10cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.426+0000 7f71018b1640 1 -- 172.21.15.45:0/2022817547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec009560 con 0x7f70fc105330 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f71028b3640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 0x7f70fc10b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f71018b1640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009230 tx=0x7f70ec004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f70f37fe640 1 -- 172.21.15.45:0/2022817547 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec004440 con 0x7f70fc105330 2024-03-12T05:20:13.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f70f37fe640 1 -- 172.21.15.45:0/2022817547 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70ec003aa0 con 0x7f70fc105330 2024-03-12T05:20:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f70f37fe640 1 -- 172.21.15.45:0/2022817547 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec005540 con 0x7f70fc105330 2024-03-12T05:20:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.427+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc113060 con 0x7f70fc105330 2024-03-12T05:20:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.428+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc113520 con 0x7f70fc105330 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.431+0000 7f7106fbd740 1 Processor -- start 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.432+0000 7f7106fbd740 1 -- start start 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.432+0000 7f70f37fe640 1 -- 172.21.15.45:0/2022817547 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f70ec003c10 con 0x7f70fc105330 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0319f00 0x5611b031a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0334220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b02d5960 con 0x5611b0334760 2024-03-12T05:20:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b0289310 con 0x5611b031a820 2024-03-12T05:20:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f7106fbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b0286840 con 0x5611b0319f00 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f71018b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f71018b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41374/0 (socket says 172.21.15.45:41374) 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.433+0000 7f71018b1640 1 -- 172.21.15.45:0/2756949781 learned_addr learned my addr 172.21.15.45:0/2756949781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71020b2640 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0334220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f70f37fe640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 0x7f70e80a4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71028b3640 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0319f00 0x5611b031a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71028b3640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 0x7f70e80a4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71018b1640 1 -- 172.21.15.45:0/2756949781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0319f00 msgr2=0x5611b031a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71018b1640 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0319f00 0x5611b031a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71018b1640 1 -- 172.21.15.45:0/2756949781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 msgr2=0x5611b0334220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71018b1640 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0334220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f71018b1640 1 -- 172.21.15.45:0/2756949781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611b0244060 con 0x5611b0334760 2024-03-12T05:20:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.434+0000 7f70f37fe640 1 -- 172.21.15.45:0/2022817547 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f70ec0ef8c0 con 0x7f70fc105330 2024-03-12T05:20:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.435+0000 7f71018b1640 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0b89d0 tx=0x7f70ec0b8940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351e1f7cc4d2caeb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.435+0000 7f71028b3640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 0x7f70e80a4190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f70f4011c10 tx=0x7f70f4004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.435+0000 7f70f0bfe640 1 -- 172.21.15.45:0/2756949781 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0bcc70 con 0x5611b0334760 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.435+0000 7f70f0bfe640 1 -- 172.21.15.45:0/2756949781 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70ec0bc570 con 0x5611b0334760 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.436+0000 7f70f0bfe640 1 -- 172.21.15.45:0/2756949781 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec07ffd0 con 0x5611b0334760 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.436+0000 7f7106fbd740 1 -- 172.21.15.45:0/2756949781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 msgr2=0x5611b0336b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.436+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0b89d0 tx=0x7f70ec0b8940 comp rx=0 tx=0).stop 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 -- 172.21.15.45:0/2756949781 shutdown_connections 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b0336b50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0334220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2756949781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0319f00 0x5611b031a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 -- 172.21.15.45:0/2756949781 >> 172.21.15.45:0/2756949781 conn(0x5611b0314980 msgr2=0x5611b0314d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 -- 172.21.15.45:0/2756949781 shutdown_connections 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 -- 172.21.15.45:0/2756949781 wait complete. 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 -- start start 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.437+0000 7f7106fbd740 1 Processor -- start 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0331630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b03320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b0282020 con 0x5611b0319f00 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aff55430 con 0x5611b031a820 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611aff11e00 con 0x5611b0334760 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f7106fbd740 10 client.0 _ll_register_callbacks cb 0x5611b0229cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f71018b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f71018b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60554/0 (socket says 172.21.15.45:60554) 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f71018b1640 1 -- 172.21.15.45:0/2703239625 learned_addr learned my addr 172.21.15.45:0/2703239625 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f71028b3640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b03320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.438+0000 7f71020b2640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0331630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 -- 172.21.15.45:0/2703239625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 msgr2=0x5611b03320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b03320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 -- 172.21.15.45:0/2703239625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 msgr2=0x5611b0331630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0331630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 -- 172.21.15.45:0/2703239625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc113520 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f71018b1640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f70ec004c20 tx=0x7f70ec0efb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ef5581d9f3066a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f70e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0ed400 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f70ec0bcc70 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611aff817e0 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.439+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70ec004dc0 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 10 client.27332 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5611b02806a0 con 0x5611b031a820 2024-03-12T05:20:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5611aff817e0 con 0x5611b031a820 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f70ec03c070 con 0x5611b031a820 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 10 client.27332 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 10 client.27332 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 20 client.27332 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b03443e0 0x5611b03467d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 4 client.27332 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611affbe720 con 0x5611b03443e0 2024-03-12T05:20:13.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.440+0000 7f71020b2640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b03443e0 0x5611b03467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.441+0000 7f70e5ffb640 10 client.27332 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.441+0000 7f71020b2640 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b03443e0 0x5611b03467d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=36ab73d509d635f2 server_cookie=97872d81795ed2e8 in_seq=0 out_seq=0 2024-03-12T05:20:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.442+0000 7f70e5ffb640 1 -- 172.21.15.45:0/2703239625 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5611affbe720 con 0x5611b03443e0 2024-03-12T05:20:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.442+0000 7f70e5ffb640 10 client.27332 handle_command_reply: tid=0 2024-03-12T05:20:13.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.442+0000 7f7106fbd740 1 client.27332 shutdown 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.442+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 msgr2=0x5611b0331b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.442+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f70ec004c20 tx=0x7f70ec0efb00 comp rx=0 tx=0).stop 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 shutdown_connections 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b03443e0 0x5611b03467d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b03320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b0331b70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2703239625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0331630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 >> 172.21.15.45:0/2703239625 conn(0x5611b0314980 msgr2=0x5611b03166d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f70e5ffb640 0 client.27332 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 shutdown_connections 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 1 -- 172.21.15.45:0/2703239625 wait complete. 2024-03-12T05:20:13.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.443+0000 7f7106fbd740 20 client.27332 trim_cache size 0 max 16384 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 Processor -- start 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 -- start start 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b032ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b034b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b0207130 con 0x5611b0319f00 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b03345e0 con 0x5611b031a820 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.463+0000 7f7106fbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b03105c0 con 0x5611b0334760 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71028b3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b034b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41378/0 (socket says 172.21.15.45:41378) 2024-03-12T05:20:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 -- 172.21.15.45:0/2711096276 learned_addr learned my addr 172.21.15.45:0/2711096276 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71018b1640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b032ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 -- 172.21.15.45:0/2711096276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 msgr2=0x5611b034b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b034b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 -- 172.21.15.45:0/2711096276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 msgr2=0x5611b032ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b032ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71020b2640 1 -- 172.21.15.45:0/2711096276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611b021cc20 con 0x5611b0319f00 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71028b3640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b034b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.464+0000 7f71018b1640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b032ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.465+0000 7f71020b2640 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009fd0 tx=0x7f70f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cba792859e77063 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.465+0000 7f70e7fff640 1 -- 172.21.15.45:0/2711096276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8003d50 con 0x5611b0319f00 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.465+0000 7f70e7fff640 1 -- 172.21.15.45:0/2711096276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f70f8018040 con 0x5611b0319f00 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.465+0000 7f70e7fff640 1 -- 172.21.15.45:0/2711096276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8010890 con 0x5611b0319f00 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 -- 172.21.15.45:0/2711096276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 msgr2=0x5611b0333d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009fd0 tx=0x7f70f80046c0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 -- 172.21.15.45:0/2711096276 shutdown_connections 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b0334760 0x5611b034b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b031a820 0x5611b032ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 --2- 172.21.15.45:0/2711096276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0319f00 0x5611b0333d60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 -- 172.21.15.45:0/2711096276 >> 172.21.15.45:0/2711096276 conn(0x5611b0314980 msgr2=0x5611b030fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 -- 172.21.15.45:0/2711096276 shutdown_connections 2024-03-12T05:20:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.466+0000 7f7106fbd740 1 -- 172.21.15.45:0/2711096276 wait complete. 2024-03-12T05:20:13.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.467+0000 7f7106fbd740 1 -- start start 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.467+0000 7f7106fbd740 1 Processor -- start 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 0x5611b0339c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b033a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60578/0 (socket says 172.21.15.45:60578) 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 -- 172.21.15.45:0/1181674831 learned_addr learned my addr 172.21.15.45:0/1181674831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b033a910 con 0x5611b0334760 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b033aa80 con 0x5611b0319f00 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611b039a8e0 con 0x5611b031a820 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f7106fbd740 10 client.0 _ll_register_callbacks cb 0x7f70ec0b8890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71018b1640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 0x5611b0339c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71028b3640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b033a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 -- 172.21.15.45:0/1181674831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 msgr2=0x5611b0339c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 0x5611b0339c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.468+0000 7f71020b2640 1 -- 172.21.15.45:0/1181674831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 msgr2=0x5611b033a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f71020b2640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b033a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f71020b2640 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611b02806a0 con 0x5611b0319f00 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f71018b1640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 0x5611b0339c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f71020b2640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009560 tx=0x7f70f8003d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a02f1aec6a2158 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f70e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f801f450 con 0x5611b0319f00 2024-03-12T05:20:13.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.469+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f70f8005050 con 0x5611b0319f00 2024-03-12T05:20:13.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.470+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611b0331460 con 0x5611b0319f00 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.470+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70f801f890 con 0x5611b0319f00 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.470+0000 7f7106fbd740 10 client.27338 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.470+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f70f40117d0 con 0x5611b0319f00 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5611b0331460 con 0x5611b0319f00 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f70f8004b50 con 0x5611b0319f00 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 10 client.27338 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 10 client.27338 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 20 client.27338 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b039bff0 0x5611b039e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 4 client.27338 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.471+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f70ec0bc570 con 0x5611b039bff0 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f71018b1640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b039bff0 0x5611b039e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f71018b1640 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b039bff0 0x5611b039e410 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ffcd5769051a1986 server_cookie=2f5edbb4bbd99390 in_seq=0 out_seq=0 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f70e5ffb640 10 client.27338 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f70e5ffb640 1 -- 172.21.15.45:0/1181674831 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f70ec0bc570 con 0x5611b039bff0 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f70e5ffb640 10 client.27338 handle_command_reply: tid=0 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.472+0000 7f7106fbd740 1 client.27338 shutdown 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 msgr2=0x5611b033b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009560 tx=0x7f70f8003d80 comp rx=0 tx=0).stop 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 shutdown_connections 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5611b039bff0 0x5611b039e410 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611b0334760 0x5611b033a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f70e5ffb640 0 client.27338 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611b031a820 0x5611b0339c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 --2- 172.21.15.45:0/1181674831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611b0319f00 0x5611b033b820 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 >> 172.21.15.45:0/1181674831 conn(0x5611b0314980 msgr2=0x5611b032ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 shutdown_connections 2024-03-12T05:20:13.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 1 -- 172.21.15.45:0/1181674831 wait complete. 2024-03-12T05:20:13.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.473+0000 7f7106fbd740 20 client.27338 trim_cache size 0 max 16384 2024-03-12T05:20:13.471 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 msgr2=0x7f70e80a4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 0x7f70e80a4190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f70f4011c10 tx=0x7f70f4004a40 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 msgr2=0x7f70fc10aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009230 tx=0x7f70ec004040 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 shutdown_connections 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f70e80a1cd0 0x7f70e80a4190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70fc10b440 0x7f70fc10b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70fc105330 0x7f70fc10aeb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 --2- 172.21.15.45:0/2022817547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70fc100260 0x7f70fc10cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 >> 172.21.15.45:0/2022817547 conn(0x7f70fc077180 msgr2=0x7f70fc0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.475+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 shutdown_connections 2024-03-12T05:20:13.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.476+0000 7f7104316640 1 -- 172.21.15.45:0/2022817547 wait complete. 2024-03-12T05:20:13.489 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:13.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3651060820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc103c40 msgr2=0x7fa0cc1040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3651060820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1040c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009ca0 tx=0x7fa0c402c100 comp rx=0 tx=0).stop 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3651060820 shutdown_connections 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3651060820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc068d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3651060820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1040c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3651060820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 0x7fa0cc068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.672+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3651060820 >> 172.21.15.45:0/3651060820 conn(0x7fa0cc077180 msgr2=0x7fa0cc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.673+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3651060820 shutdown_connections 2024-03-12T05:20:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.673+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3651060820 wait complete. 2024-03-12T05:20:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.674+0000 7fa0d2f0c640 1 Processor -- start 2024-03-12T05:20:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.674+0000 7fa0d2f0c640 1 -- start start 2024-03-12T05:20:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.674+0000 7fa0d2f0c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.674+0000 7fa0d2f0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d2f0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 0x7fa0cc19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d2f0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc19e060 con 0x7fa0cc103c40 2024-03-12T05:20:13.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d2f0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc19e1d0 con 0x7fa0cc1060d0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d2f0c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc19e390 con 0x7fa0cc0688c0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35344/0 (socket says 172.21.15.45:35344) 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3077497256 learned_addr learned my addr 172.21.15.45:0/3077497256 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d14a9640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 0x7fa0cc19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3077497256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 msgr2=0x7fa0cc19da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 0x7fa0cc19da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc103c40 msgr2=0x7fa0cc1a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.675+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3077497256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c4009900 con 0x7fa0cc0688c0 2024-03-12T05:20:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00d950 tx=0x7fa0bc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0c9ffb640 1 -- 172.21.15.45:0/3077497256 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc009a30 con 0x7fa0cc0688c0 2024-03-12T05:20:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc19e670 con 0x7fa0cc0688c0 2024-03-12T05:20:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0c9ffb640 1 -- 172.21.15.45:0/3077497256 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0bc0044d0 con 0x7fa0cc0688c0 2024-03-12T05:20:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0c9ffb640 1 -- 172.21.15.45:0/3077497256 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0047a0 con 0x7fa0cc0688c0 2024-03-12T05:20:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.676+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc1aa900 con 0x7fa0cc0688c0 2024-03-12T05:20:13.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.679+0000 7fa0c9ffb640 1 -- 172.21.15.45:0/3077497256 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa0bc004900 con 0x7fa0cc0688c0 2024-03-12T05:20:13.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.680+0000 7fa0c9ffb640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 0x7fa0b40a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.681+0000 7fa0c9ffb640 1 -- 172.21.15.45:0/3077497256 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa0bc0cd120 con 0x7fa0cc0688c0 2024-03-12T05:20:13.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.681+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 0x7fa0b40a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0d5bbe740 1 Processor -- start 2024-03-12T05:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 0x7fa0b40a4120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0779b0 tx=0x7fa0c0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0d5bbe740 1 -- start start 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 0x55680daad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 0x55680dab6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.682+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680da68960 con 0x55680daadb70 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680da1c310 con 0x55680daad250 2024-03-12T05:20:13.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680da19840 con 0x55680dab6750 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 0x55680dab6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 0x55680dab6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41414/0 (socket says 172.21.15.45:41414) 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0cbfff640 1 -- 172.21.15.45:0/2111351578 learned_addr learned my addr 172.21.15.45:0/2111351578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 0x55680daad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d14a9640 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/2111351578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 msgr2=0x55680dab8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/2111351578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 msgr2=0x55680dab6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 0x55680dab6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.683+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/2111351578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55680d9d7060 con 0x55680daad250 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.684+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 0x55680daad630 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0ca8c0 tx=0x7fa0bc099a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf248fb2761cb25a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.684+0000 7fa0b2ffd640 1 -- 172.21.15.45:0/2111351578 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc09d070 con 0x55680daad250 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.684+0000 7fa0b2ffd640 1 -- 172.21.15.45:0/2111351578 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0bc09def0 con 0x55680daad250 2024-03-12T05:20:13.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.684+0000 7fa0b2ffd640 1 -- 172.21.15.45:0/2111351578 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0991f0 con 0x55680daad250 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/2111351578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 msgr2=0x55680daad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 0x55680daad630 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0ca8c0 tx=0x7fa0bc099a00 comp rx=0 tx=0).stop 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/2111351578 shutdown_connections 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680daadb70 0x55680dab6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/2111351578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daad250 0x55680daad630 secure :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0ca8c0 tx=0x7fa0bc099a00 comp rx=0 tx=0).stop 2024-03-12T05:20:13.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.685+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/2111351578 >> 172.21.15.45:0/2111351578 conn(0x55680daa7980 msgr2=0x55680daa7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.686+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/2111351578 shutdown_connections 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.686+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/2111351578 wait complete. 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.686+0000 7fa0d5bbe740 1 -- start start 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 Processor -- start 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680da15020 con 0x55680dab4230 2024-03-12T05:20:13.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680d6e8430 con 0x55680daadb70 2024-03-12T05:20:13.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680d6a4e00 con 0x55680dab6750 2024-03-12T05:20:13.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.687+0000 7fa0d5bbe740 10 client.0 _ll_register_callbacks cb 0x55680d9bccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60622/0 (socket says 172.21.15.45:60622) 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3877557907 learned_addr learned my addr 172.21.15.45:0/3877557907 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d14a9640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 msgr2=0x55680dab3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3877557907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 msgr2=0x55680dac26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.688+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/3877557907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc1aa900 con 0x55680daadb70 2024-03-12T05:20:13.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.689+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0ca920 tx=0x7fa0bc00b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7512eb6c85f25332 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.689+0000 7fa0abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.691+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0b6070 con 0x55680daadb70 2024-03-12T05:20:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.691+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa0bc0b82b0 con 0x55680daadb70 2024-03-12T05:20:13.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55680d7147e0 con 0x55680daadb70 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0bc09d070 con 0x55680daadb70 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0d5bbe740 10 client.27347 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55680da136a0 con 0x55680daadb70 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55680d7147e0 con 0x55680daadb70 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.692+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0bc0991f0 con 0x55680daadb70 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 10 client.27347 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 10 client.27347 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 20 client.27347 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680dac6d70 0x55680dac9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 4 client.27347 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55680d751720 con 0x55680dac6d70 2024-03-12T05:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.693+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680dac6d70 0x55680dac9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.694+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680dac6d70 0x55680dac9190 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fd859984a63417d server_cookie=df985d7099df4b82 in_seq=0 out_seq=0 2024-03-12T05:20:13.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.694+0000 7fa0abfff640 10 client.27347 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.695+0000 7fa0abfff640 1 -- 172.21.15.45:0/3877557907 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55680d751720 con 0x55680dac6d70 2024-03-12T05:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.695+0000 7fa0abfff640 10 client.27347 handle_command_reply: tid=0 2024-03-12T05:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.695+0000 7fa0d5bbe740 1 client.27347 shutdown 2024-03-12T05:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 msgr2=0x55680dab37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0ca920 tx=0x7fa0bc00b660 comp rx=0 tx=0).stop 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 shutdown_connections 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680dac6d70 0x55680dac9190 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680dab6750 0x55680dab3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/3877557907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680daadb70 0x55680dab37b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 >> 172.21.15.45:0/3877557907 conn(0x55680daa7980 msgr2=0x55680daa96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0abfff640 0 client.27347 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.696+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 shutdown_connections 2024-03-12T05:20:13.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.697+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/3877557907 wait complete. 2024-03-12T05:20:13.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.699+0000 7fa0d5bbe740 20 client.27347 trim_cache size 0 max 16384 2024-03-12T05:20:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.724+0000 7fa0d5bbe740 1 Processor -- start 2024-03-12T05:20:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.724+0000 7fa0d5bbe740 1 -- start start 2024-03-12T05:20:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab6750 0x55680dab0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680d99a130 con 0x55680dab4230 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680daacee0 con 0x55680dab6750 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d0ca8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35374/0 (socket says 172.21.15.45:35374) 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/819947318 learned_addr learned my addr 172.21.15.45:0/819947318 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680daa35c0 con 0x55680daadb70 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.725+0000 7fa0d14a9640 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab6750 0x55680dab0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/819947318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab6750 msgr2=0x55680dab0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab6750 0x55680dab0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/819947318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 msgr2=0x55680dac2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 -- 172.21.15.45:0/819947318 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55680d9afc20 con 0x55680daadb70 2024-03-12T05:20:13.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.726+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc015000 tx=0x7fa0bc005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1580dd362872eb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.727+0000 7fa0b27fc640 1 -- 172.21.15.45:0/819947318 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0b6030 con 0x55680daadb70 2024-03-12T05:20:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.727+0000 7fa0b27fc640 1 -- 172.21.15.45:0/819947318 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0bc09a4a0 con 0x55680daadb70 2024-03-12T05:20:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.727+0000 7fa0b27fc640 1 -- 172.21.15.45:0/819947318 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc09d070 con 0x55680daadb70 2024-03-12T05:20:13.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 msgr2=0x55680dac20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc015000 tx=0x7fa0bc005770 comp rx=0 tx=0).stop 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 shutdown_connections 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab4230 0x55680dac2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab6750 0x55680dab0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/819947318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dac20f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 >> 172.21.15.45:0/819947318 conn(0x55680daa7980 msgr2=0x55680dacdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 shutdown_connections 2024-03-12T05:20:13.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.728+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/819947318 wait complete. 2024-03-12T05:20:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.729+0000 7fa0d5bbe740 1 -- start start 2024-03-12T05:20:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.729+0000 7fa0d5bbe740 1 Processor -- start 2024-03-12T05:20:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dabd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab6750 0x55680dabc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680dabc8a0 con 0x55680dab6750 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680dabca10 con 0x55680dab4230 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55680db1d280 con 0x55680daadb70 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d5bbe740 10 client.0 _ll_register_callbacks cb 0x7fa0bc009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60646/0 (socket says 172.21.15.45:60646) 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0cbfff640 1 -- 172.21.15.45:0/1288265709 learned_addr learned my addr 172.21.15.45:0/1288265709 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d14a9640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab6750 0x55680dabc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.730+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dabd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 -- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 msgr2=0x55680dabd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dabd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 -- 172.21.15.45:0/1288265709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab6750 msgr2=0x55680dabc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab6750 0x55680dabc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 -- 172.21.15.45:0/1288265709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55680da136a0 con 0x55680dab4230 2024-03-12T05:20:13.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dabd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0cbfff640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009730 tx=0x7fa0c0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1100cb06558ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.731+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0022070 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa0c001d620 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55680d7a0890 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0c001d7e0 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 10 client.27353 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55680d9d58c0 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa0c003f020 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0c003ba90 con 0x55680dab4230 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 10 client.27353 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 10 client.27353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.732+0000 7fa0d5bbe740 20 client.27353 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.733+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680db1eaf0 0x55680db20f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.733+0000 7fa0d5bbe740 4 client.27353 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:13.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.733+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55680d7a0890 con 0x55680db1eaf0 2024-03-12T05:20:13.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.733+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680db1eaf0 0x55680db20f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d0ca8640 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680db1eaf0 0x55680db20f10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3111225ff38bc561 server_cookie=9f3fa0e017b9d14a in_seq=0 out_seq=0 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0abfff640 10 client.27353 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0abfff640 1 -- 172.21.15.45:0/1288265709 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55680d7a0890 con 0x55680db1eaf0 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0abfff640 10 client.27353 handle_command_reply: tid=0 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 client.27353 shutdown 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 msgr2=0x55680dabbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009730 tx=0x7fa0c0004730 comp rx=0 tx=0).stop 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 shutdown_connections 2024-03-12T05:20:13.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55680db1eaf0 0x55680db20f10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55680dab6750 0x55680dabc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680dab4230 0x55680dabbce0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 --2- 172.21.15.45:0/1288265709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55680daadb70 0x55680dabd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 >> 172.21.15.45:0/1288265709 conn(0x55680daa7980 msgr2=0x55680dac20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0abfff640 0 client.27353 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 shutdown_connections 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.734+0000 7fa0d5bbe740 1 -- 172.21.15.45:0/1288265709 wait complete. 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.735+0000 7fa0d5bbe740 20 client.27353 trim_cache size 0 max 16384 2024-03-12T05:20:13.732 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 msgr2=0x7fa0b40a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 0x7fa0b40a4120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0779b0 tx=0x7fa0c0005f90 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 msgr2=0x7fa0cc1a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00d950 tx=0x7fa0bc00de10 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 shutdown_connections 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa0b40a1c60 0x7fa0b40a4120 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0cc103c40 0x7fa0cc1a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0cc1060d0 0x7fa0cc19da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 --2- 172.21.15.45:0/3077497256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0cc0688c0 0x7fa0cc1a3860 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.736+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 >> 172.21.15.45:0/3077497256 conn(0x7fa0cc077180 msgr2=0x7fa0cc0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.737+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 shutdown_connections 2024-03-12T05:20:13.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.737+0000 7fa0d2f0c640 1 -- 172.21.15.45:0/3077497256 wait complete. 2024-03-12T05:20:13.757 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:13.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:20:13.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.942+0000 7fc72dced640 1 -- 172.21.15.45:0/4144740069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 msgr2=0x7fc7200a6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.942+0000 7fc72dced640 1 --2- 172.21.15.45:0/4144740069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc7200a6d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009900 tx=0x7fc71c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.942+0000 7fc72dced640 1 -- 172.21.15.45:0/4144740069 shutdown_connections 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 --2- 172.21.15.45:0/4144740069 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7200b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 --2- 172.21.15.45:0/4144740069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200a7310 0x7fc7200aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 --2- 172.21.15.45:0/4144740069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc7200a6d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 -- 172.21.15.45:0/4144740069 >> 172.21.15.45:0/4144740069 conn(0x7fc72001a640 msgr2=0x7fc72001aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 -- 172.21.15.45:0/4144740069 shutdown_connections 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 -- 172.21.15.45:0/4144740069 wait complete. 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.943+0000 7fc72dced640 1 Processor -- start 2024-03-12T05:20:13.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 -- start start 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc720149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7201498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200b2250 con 0x7fc7200aaf90 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200b23c0 con 0x7fc7200b1860 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72dced640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7200b2580 con 0x7fc7200a6960 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc720149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60666/0 (socket says 172.21.15.45:60666) 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc720149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35396/0 (socket says 172.21.15.45:35396) 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 -- 172.21.15.45:0/2639231564 learned_addr learned my addr 172.21.15.45:0/2639231564 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc727fff640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7201498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 -- 172.21.15.45:0/2639231564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 msgr2=0x7fc720149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc720149380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 -- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 msgr2=0x7fc7201498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7201498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc72d4ec640 1 -- 172.21.15.45:0/2639231564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c009560 con 0x7fc7200b1860 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.944+0000 7fc727fff640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7201498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc72804fc90 tx=0x7fc728072b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc725ffb640 1 -- 172.21.15.45:0/2639231564 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728068220 con 0x7fc7200b1860 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc725ffb640 1 -- 172.21.15.45:0/2639231564 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc728068380 con 0x7fc7200b1860 2024-03-12T05:20:13.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc725ffb640 1 -- 172.21.15.45:0/2639231564 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728067c50 con 0x7fc7200b1860 2024-03-12T05:20:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc720150240 con 0x7fc7200b1860 2024-03-12T05:20:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.945+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc720150670 con 0x7fc7200b1860 2024-03-12T05:20:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.948+0000 7fc725ffb640 1 -- 172.21.15.45:0/2639231564 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc728070820 con 0x7fc7200b1860 2024-03-12T05:20:13.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.949+0000 7fc731c04740 1 Processor -- start 2024-03-12T05:20:13.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.949+0000 7fc731c04740 1 -- start start 2024-03-12T05:20:13.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.949+0000 7fc725ffb640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 0x7fc7180a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1d80 0x562cd6bca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6b7c960 con 0x562cd6bca960 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6b30310 con 0x562cd6bc1460 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc731c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6b2d840 con 0x562cd6bc1d80 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc725ffb640 1 -- 172.21.15.45:0/2639231564 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fc728130690 con 0x7fc7200b1860 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60674/0 (socket says 172.21.15.45:60674) 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 -- 172.21.15.45:0/3116743269 learned_addr learned my addr 172.21.15.45:0/3116743269 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 -- 172.21.15.45:0/3116743269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1d80 msgr2=0x562cd6bca420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:20:13.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1d80 0x562cd6bca420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 -- 172.21.15.45:0/3116743269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 msgr2=0x562cd6bccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72cceb640 1 -- 172.21.15.45:0/3116743269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562cd6aeb060 con 0x562cd6bc1460 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.950+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.951+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc7200a55b0 tx=0x7fc71c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b65bddc5b98b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.951+0000 7fc727fff640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 0x7fc7180a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.952+0000 7fc727fff640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 0x7fc7180a41b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x562cd6bc0f30 tx=0x7fc714005ed0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.953+0000 7fc712ffd640 1 -- 172.21.15.45:0/3116743269 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c02c7a0 con 0x562cd6bc1460 2024-03-12T05:20:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.953+0000 7fc712ffd640 1 -- 172.21.15.45:0/3116743269 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc71c0044f0 con 0x562cd6bc1460 2024-03-12T05:20:13.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.953+0000 7fc712ffd640 1 -- 172.21.15.45:0/3116743269 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c003c00 con 0x562cd6bc1460 2024-03-12T05:20:13.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.954+0000 7fc731c04740 1 -- 172.21.15.45:0/3116743269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 msgr2=0x562cd6bc1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.954+0000 7fc731c04740 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc7200a55b0 tx=0x7fc71c0046c0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 -- 172.21.15.45:0/3116743269 shutdown_connections 2024-03-12T05:20:13.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1d80 0x562cd6bca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 --2- 172.21.15.45:0/3116743269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1460 0x562cd6bc1840 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 -- 172.21.15.45:0/3116743269 >> 172.21.15.45:0/3116743269 conn(0x562cd6bbbb90 msgr2=0x562cd6bbbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 -- 172.21.15.45:0/3116743269 shutdown_connections 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 -- 172.21.15.45:0/3116743269 wait complete. 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.955+0000 7fc731c04740 1 -- start start 2024-03-12T05:20:13.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 Processor -- start 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bc7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6bc7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6b29020 con 0x562cd6bca960 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd67fc430 con 0x562cd6bc1d80 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd67b8e00 con 0x562cd6bc1460 2024-03-12T05:20:13.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc731c04740 10 client.0 _ll_register_callbacks cb 0x562cd6ad0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc72d4ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc72d4ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41502/0 (socket says 172.21.15.45:41502) 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc72d4ec640 1 -- 172.21.15.45:0/2043330642 learned_addr learned my addr 172.21.15.45:0/2043330642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.956+0000 7fc72cceb640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bc7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6bc7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 -- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 msgr2=0x562cd6bc7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bc7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 -- 172.21.15.45:0/2043330642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 msgr2=0x562cd6bc8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 -- 172.21.15.45:0/2043330642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7200a55e0 con 0x562cd6bc1d80 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc727fff640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6bc7ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc714005d00 tx=0x7fc7140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ab739be7ba0424 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc707fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.957+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714005020 con 0x562cd6bc1d80 2024-03-12T05:20:13.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.958+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fc714005180 con 0x562cd6bc1d80 2024-03-12T05:20:13.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.958+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562cd68287e0 con 0x562cd6bc1d80 2024-03-12T05:20:13.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.958+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc714005340 con 0x562cd6bc1d80 2024-03-12T05:20:13.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.958+0000 7fc731c04740 10 client.27365 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.958+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562cd6b276a0 con 0x562cd6bc1d80 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc71402b030 con 0x562cd6bc1d80 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc7140106e0 con 0x562cd6bc1d80 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 10 client.27365 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 10 client.27365 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 20 client.27365 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6bee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 4 client.27365 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562cd6865720 con 0x562cd6bebda0 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.959+0000 7fc72cceb640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6bee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.960+0000 7fc72cceb640 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6bee1c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=216787c9a26a024f server_cookie=d3977f2aafe0eb50 in_seq=0 out_seq=0 2024-03-12T05:20:13.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.960+0000 7fc707fff640 10 client.27365 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.961+0000 7fc707fff640 1 -- 172.21.15.45:0/2043330642 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562cd6865720 con 0x562cd6bebda0 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.961+0000 7fc707fff640 10 client.27365 handle_command_reply: tid=0 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.961+0000 7fc731c04740 1 client.27365 shutdown 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.961+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 msgr2=0x562cd6bc7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.961+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6bc7ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc714005d00 tx=0x7fc7140046c0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 shutdown_connections 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bca960 0x562cd6bc8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6bee1c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6bc7ed0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc707fff640 0 client.27365 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 --2- 172.21.15.45:0/2043330642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bc7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 >> 172.21.15.45:0/2043330642 conn(0x562cd6bbbb90 msgr2=0x562cd6bbd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.962+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 shutdown_connections 2024-03-12T05:20:13.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.963+0000 7fc731c04740 1 -- 172.21.15.45:0/2043330642 wait complete. 2024-03-12T05:20:13.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.963+0000 7fc731c04740 20 client.27365 trim_cache size 0 max 16384 2024-03-12T05:20:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 Processor -- start 2024-03-12T05:20:13.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 -- start start 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1d80 0x562cd6bc8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 0x562cd6bc4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6aae130 con 0x562cd6bc1d80 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6be7800 con 0x562cd6bca960 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.985+0000 7fc731c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6bc0550 con 0x562cd6bc1460 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35418/0 (socket says 172.21.15.45:35418) 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 -- 172.21.15.45:0/3839132864 learned_addr learned my addr 172.21.15.45:0/3839132864 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 0x562cd6bc4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc727fff640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1d80 0x562cd6bc8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 -- 172.21.15.45:0/3839132864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 msgr2=0x562cd6bc4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 0x562cd6bc4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 -- 172.21.15.45:0/3839132864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1d80 msgr2=0x562cd6bc8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1d80 0x562cd6bc8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72cceb640 1 -- 172.21.15.45:0/3839132864 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562cd6ac3c20 con 0x562cd6bc1460 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.986+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 0x562cd6bc4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.987+0000 7fc72cceb640 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009900 tx=0x7fc71c041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e93f6d80e178bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.987+0000 7fc7127fc640 1 -- 172.21.15.45:0/3839132864 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c04f040 con 0x562cd6bc1460 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.987+0000 7fc7127fc640 1 -- 172.21.15.45:0/3839132864 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc71c0558d0 con 0x562cd6bc1460 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.987+0000 7fc7127fc640 1 -- 172.21.15.45:0/3839132864 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c005480 con 0x562cd6bc1460 2024-03-12T05:20:13.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.988+0000 7fc731c04740 1 -- 172.21.15.45:0/3839132864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 msgr2=0x562cd6bca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.988+0000 7fc731c04740 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009900 tx=0x7fc71c041040 comp rx=0 tx=0).stop 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 -- 172.21.15.45:0/3839132864 shutdown_connections 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1d80 0x562cd6bc8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bca960 0x562cd6bc4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 --2- 172.21.15.45:0/3839132864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bc1460 0x562cd6bca0c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 -- 172.21.15.45:0/3839132864 >> 172.21.15.45:0/3839132864 conn(0x562cd6bbbb90 msgr2=0x562cd6bc2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 -- 172.21.15.45:0/3839132864 shutdown_connections 2024-03-12T05:20:13.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 -- 172.21.15.45:0/3839132864 wait complete. 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 -- start start 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.989+0000 7fc731c04740 1 Processor -- start 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 0x562cd6be2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6be0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bca960 0x562cd6be1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6be19a0 con 0x562cd6bc1460 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6be1b10 con 0x562cd6bc1d80 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cd6be1cd0 con 0x562cd6bca960 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc731c04740 10 client.0 _ll_register_callbacks cb 0x7fc71c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 0x562cd6be2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc72cceb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 0x562cd6be2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41532/0 (socket says 172.21.15.45:41532) 2024-03-12T05:20:13.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc72cceb640 1 -- 172.21.15.45:0/2927070531 learned_addr learned my addr 172.21.15.45:0/2927070531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:13.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6be0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc72d4ec640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bca960 0x562cd6be1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 -- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bca960 msgr2=0x562cd6be1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bca960 0x562cd6be1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 -- 172.21.15.45:0/2927070531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 msgr2=0x562cd6be2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 0x562cd6be2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.990+0000 7fc727fff640 1 -- 172.21.15.45:0/2927070531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc720150670 con 0x562cd6bc1d80 2024-03-12T05:20:13.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc727fff640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6be0e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc7140092f0 tx=0x7fc71400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c2521ab0314cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc707fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714005030 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc71c0044f0 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fc714005190 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc714005350 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc731c04740 10 client.27371 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.991+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562cd6aeb060 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc71c0044f0 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc71401e5c0 con 0x562cd6bc1d80 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 10 client.27371 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 10 client.27371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 20 client.27371 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6c45920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 4 client.27371 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:13.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562cd68b4890 con 0x562cd6bebda0 2024-03-12T05:20:13.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.992+0000 7fc72cceb640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6c45920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc72cceb640 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6c45920 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27a35a4a0c6ca359 server_cookie=ffb478f2c41327ab in_seq=0 out_seq=0 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc707fff640 10 client.27371 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc707fff640 1 -- 172.21.15.45:0/2927070531 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562cd68b4890 con 0x562cd6bebda0 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc707fff640 10 client.27371 handle_command_reply: tid=0 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 client.27371 shutdown 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 msgr2=0x562cd6be0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6be0e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc7140092f0 tx=0x7fc71400eef0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 shutdown_connections 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562cd6bebda0 0x562cd6c45920 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562cd6bca960 0x562cd6be1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562cd6bc1d80 0x562cd6be0e50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.993+0000 7fc731c04740 1 --2- 172.21.15.45:0/2927070531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562cd6bc1460 0x562cd6be2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.994+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 >> 172.21.15.45:0/2927070531 conn(0x562cd6bbbb90 msgr2=0x562cd6bba440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.994+0000 7fc707fff640 0 client.27371 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.994+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 shutdown_connections 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.994+0000 7fc731c04740 1 -- 172.21.15.45:0/2927070531 wait complete. 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.994+0000 7fc731c04740 20 client.27371 trim_cache size 0 max 16384 2024-03-12T05:20:13.991 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 msgr2=0x7fc7180a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 0x7fc7180a41b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x562cd6bc0f30 tx=0x7fc714005ed0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 msgr2=0x7fc7200b1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc72804fc90 tx=0x7fc728072b50 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 shutdown_connections 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7180a1cf0 0x7fc7180a41b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7200aaf90 0x7fc7201498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7200b1860 0x7fc7200b1d10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 --2- 172.21.15.45:0/2639231564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7200a6960 0x7fc720149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:13.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 >> 172.21.15.45:0/2639231564 conn(0x7fc72001a640 msgr2=0x7fc7200a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:13.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 shutdown_connections 2024-03-12T05:20:13.993 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:13.995+0000 7fc72dced640 1 -- 172.21.15.45:0/2639231564 wait complete. 2024-03-12T05:20:14.011 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:14.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:20:14.048 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:13 smithi045 ceph-mon[32174]: pgmap v290: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:14.049 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:13 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2492173560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:14.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.191+0000 7f1b1b59e640 1 -- 172.21.15.45:0/3062160626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b140957f0 msgr2=0x7f1b140a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.191+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/3062160626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b140957f0 0x7f1b140a1ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c066f90 tx=0x7f1b1c096db0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 -- 172.21.15.45:0/3062160626 shutdown_connections 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/3062160626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b140957f0 0x7f1b140a1ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/3062160626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 0x7f1b140952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/3062160626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b14099bc0 0x7f1b14099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 -- 172.21.15.45:0/3062160626 >> 172.21.15.45:0/3062160626 conn(0x7f1b1400a740 msgr2=0x7f1b1400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.192+0000 7f1b1b59e640 1 -- 172.21.15.45:0/3062160626 shutdown_connections 2024-03-12T05:20:14.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.193+0000 7f1b1b59e640 1 -- 172.21.15.45:0/3062160626 wait complete. 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.194+0000 7f1b1b59e640 1 Processor -- start 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.194+0000 7f1b1b59e640 1 -- start start 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 0x7f1b14138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 0x7f1b14138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 0x7f1b14132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14132fa0 con 0x7f1b140957f0 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14133110 con 0x7f1b14099bc0 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1b59e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b141332d0 con 0x7f1b14094e50 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 0x7f1b14138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 0x7f1b14138940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41550/0 (socket says 172.21.15.45:41550) 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b19d9b640 1 -- 172.21.15.45:0/315130996 learned_addr learned my addr 172.21.15.45:0/315130996 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 0x7f1b14132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 0x7f1b14138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 -- 172.21.15.45:0/315130996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 msgr2=0x7f1b14138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 0x7f1b14138400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 -- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 msgr2=0x7f1b14138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 0x7f1b14138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.195+0000 7f1b1ad9d640 1 -- 172.21.15.45:0/315130996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c04f750 con 0x7f1b14099bc0 2024-03-12T05:20:14.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 0x7f1b14132950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c04f1b0 tx=0x7f1b1c069290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b0b7fe640 1 -- 172.21.15.45:0/315130996 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c067710 con 0x7f1b14099bc0 2024-03-12T05:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b0b7fe640 1 -- 172.21.15.45:0/315130996 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b1c067870 con 0x7f1b14099bc0 2024-03-12T05:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b0b7fe640 1 -- 172.21.15.45:0/315130996 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c09f670 con 0x7f1b14099bc0 2024-03-12T05:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b14133550 con 0x7f1b14099bc0 2024-03-12T05:20:14.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.196+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1413f2c0 con 0x7f1b14099bc0 2024-03-12T05:20:14.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.199+0000 7f1b0b7fe640 1 -- 172.21.15.45:0/315130996 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1b1c0a7070 con 0x7f1b14099bc0 2024-03-12T05:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 Processor -- start 2024-03-12T05:20:14.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 -- start start 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef65840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef70d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ef20960 con 0x55ef9ef65460 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9eed4310 con 0x55ef9ef6e960 2024-03-12T05:20:14.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.200+0000 7f1b23c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9eed1840 con 0x55ef9ef65d80 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b0b7fe640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 0x7f1b040a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef65840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef65840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41558/0 (socket says 172.21.15.45:41558) 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 -- 172.21.15.45:0/2932262931 learned_addr learned my addr 172.21.15.45:0/2932262931 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef70d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b0b7fe640 1 -- 172.21.15.45:0/315130996 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f1b1c1598a0 con 0x7f1b14099bc0 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 -- 172.21.15.45:0/2932262931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 msgr2=0x55ef9ef6e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 -- 172.21.15.45:0/2932262931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 msgr2=0x55ef9ef65840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef65840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.201+0000 7f1b19d9b640 1 -- 172.21.15.45:0/2932262931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef9ee8f060 con 0x55ef9ef6e960 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.202+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef70d50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009fb0 tx=0x7f1b10009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a609038a1b6d5b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.202+0000 7f1b08ff9640 1 -- 172.21.15.45:0/2932262931 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10004030 con 0x55ef9ef6e960 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.202+0000 7f1b08ff9640 1 -- 172.21.15.45:0/2932262931 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b10005020 con 0x55ef9ef6e960 2024-03-12T05:20:14.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.202+0000 7f1b08ff9640 1 -- 172.21.15.45:0/2932262931 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10004bc0 con 0x55ef9ef6e960 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 -- 172.21.15.45:0/2932262931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 msgr2=0x55ef9ef70d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef70d50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009fb0 tx=0x7f1b10009260 comp rx=0 tx=0).stop 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 -- 172.21.15.45:0/2932262931 shutdown_connections 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef70d50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 --2- 172.21.15.45:0/2932262931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef65840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.203+0000 7f1b23c04740 1 -- 172.21.15.45:0/2932262931 >> 172.21.15.45:0/2932262931 conn(0x55ef9ef5fb90 msgr2=0x55ef9ef5ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.204+0000 7f1b23c04740 1 -- 172.21.15.45:0/2932262931 shutdown_connections 2024-03-12T05:20:14.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.204+0000 7f1b23c04740 1 -- 172.21.15.45:0/2932262931 wait complete. 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.204+0000 7f1b23c04740 1 -- start start 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.205+0000 7f1b23c04740 1 Processor -- start 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.205+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 0x7f1b040a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.205+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9eecd020 con 0x55ef9ef65460 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9eba0430 con 0x55ef9ef65d80 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9eb5ce00 con 0x55ef9ef6e960 2024-03-12T05:20:14.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b23c04740 10 client.0 _ll_register_callbacks cb 0x55ef9ee74cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60702/0 (socket says 172.21.15.45:60702) 2024-03-12T05:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 -- 172.21.15.45:0/1490278291 learned_addr learned my addr 172.21.15.45:0/1490278291 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 -- 172.21.15.45:0/1490278291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 msgr2=0x55ef9ef6c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 -- 172.21.15.45:0/1490278291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 msgr2=0x55ef9ef6ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.206+0000 7f1b1a59c640 1 -- 172.21.15.45:0/1490278291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b10019040 con 0x55ef9ef65d80 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.207+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.207+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c023750 tx=0x7f1b0c023c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c691d5558655c763 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.207+0000 7f1b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.207+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c028070 con 0x55ef9ef65d80 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.207+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef9ebcc7e0 con 0x55ef9ef65d80 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f1b0c020930 con 0x55ef9ef65d80 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b1ad9d640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 0x7f1b040a4120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c126290 tx=0x7f1b1c04ff00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b0c020af0 con 0x55ef9ef65d80 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b23c04740 10 client.27383 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef9eecb6a0 con 0x55ef9ef65d80 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.208+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ef9ebcc7e0 con 0x55ef9ef65d80 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1b0c021cc0 con 0x55ef9ef65d80 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 10 client.27383 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 10 client.27383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 20 client.27383 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9ef7ef80 0x55ef9ef813a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 4 client.27383 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef9ec09720 con 0x55ef9ef7ef80 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.209+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9ef7ef80 0x55ef9ef813a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.210+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9ef7ef80 0x55ef9ef813a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13fe6f217a6a9a58 server_cookie=9c134ba17d49ecd2 in_seq=0 out_seq=0 2024-03-12T05:20:14.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.210+0000 7f1b01ffb640 10 client.27383 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.210+0000 7f1b01ffb640 1 -- 172.21.15.45:0/1490278291 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ef9ec09720 con 0x55ef9ef7ef80 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.210+0000 7f1b01ffb640 10 client.27383 handle_command_reply: tid=0 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.210+0000 7f1b23c04740 1 client.27383 shutdown 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 msgr2=0x55ef9ef6bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c023750 tx=0x7f1b0c023c10 comp rx=0 tx=0).stop 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 shutdown_connections 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9ef7ef80 0x55ef9ef813a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6bf50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 --2- 172.21.15.45:0/1490278291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 >> 172.21.15.45:0/1490278291 conn(0x55ef9ef5fb90 msgr2=0x55ef9ef618e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b01ffb640 0 client.27383 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:14.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 shutdown_connections 2024-03-12T05:20:14.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 1 -- 172.21.15.45:0/1490278291 wait complete. 2024-03-12T05:20:14.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.211+0000 7f1b23c04740 20 client.27383 trim_cache size 0 max 16384 2024-03-12T05:20:14.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:13 smithi130 ceph-mon[35398]: pgmap v290: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:14.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:13 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2492173560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:14.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 Processor -- start 2024-03-12T05:20:14.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 -- start start 2024-03-12T05:20:14.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ee52130 con 0x55ef9ef65460 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ef65be0 con 0x55ef9ef6e960 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.232+0000 7f1b23c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ef647e0 con 0x55ef9ef65d80 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35464/0 (socket says 172.21.15.45:35464) 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3693903469 learned_addr learned my addr 172.21.15.45:0/3693903469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:14.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3693903469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 msgr2=0x55ef9ef6a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3693903469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 msgr2=0x55ef9ef6e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.233+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3693903469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef9ee67c20 con 0x55ef9ef65d80 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.234+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.234+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000c00 tx=0x7f1b0c009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3374897782336e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.234+0000 7f1b03fff640 1 -- 172.21.15.45:0/3693903469 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c028070 con 0x55ef9ef65d80 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.234+0000 7f1b03fff640 1 -- 172.21.15.45:0/3693903469 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1b0c02ceb0 con 0x55ef9ef65d80 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.234+0000 7f1b03fff640 1 -- 172.21.15.45:0/3693903469 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c0050c0 con 0x55ef9ef65d80 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 -- 172.21.15.45:0/3693903469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 msgr2=0x55ef9ef6c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c000c00 tx=0x7f1b0c009f80 comp rx=0 tx=0).stop 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 -- 172.21.15.45:0/3693903469 shutdown_connections 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef6e960 0x55ef9ef6a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65d80 0x55ef9ef6c2b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3693903469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef65460 0x55ef9ef6e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 -- 172.21.15.45:0/3693903469 >> 172.21.15.45:0/3693903469 conn(0x55ef9ef5fb90 msgr2=0x55ef9ef85d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.235+0000 7f1b23c04740 1 -- 172.21.15.45:0/3693903469 shutdown_connections 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.236+0000 7f1b23c04740 1 -- 172.21.15.45:0/3693903469 wait complete. 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.236+0000 7f1b23c04740 1 -- start start 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.236+0000 7f1b23c04740 1 Processor -- start 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65460 0x55ef9ef73880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef73dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ef74520 con 0x55ef9ef6e960 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9ef74690 con 0x55ef9ef65d80 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef9efd49e0 con 0x55ef9ef65460 2024-03-12T05:20:14.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b23c04740 10 client.0 _ll_register_callbacks cb 0x7f1b10009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:14.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b19d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41580/0 (socket says 172.21.15.45:41580) 2024-03-12T05:20:14.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b19d9b640 1 -- 172.21.15.45:0/3354065975 learned_addr learned my addr 172.21.15.45:0/3354065975 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:14.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef73dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65460 0x55ef9ef73880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3354065975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65460 msgr2=0x55ef9ef73880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65460 0x55ef9ef73880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3354065975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 msgr2=0x55ef9efd44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b1a59c640 1 -- 172.21.15.45:0/3354065975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b10032040 con 0x55ef9ef65d80 2024-03-12T05:20:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.237+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b1a59c640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef73dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55ef9ef7ed10 tx=0x7f1b0c0055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f066f9e7afa97c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c028040 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f1b0c0037f0 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef9ec58890 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b0c0039b0 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b23c04740 10 client.27389 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.238+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef9eecb6a0 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ef9ec58890 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1b0c02c970 con 0x55ef9ef65d80 2024-03-12T05:20:14.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 10 client.27389 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 10 client.27389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 20 client.27389 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9efd5e30 0x55ef9efd8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 4 client.27389 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef9eb78700 con 0x55ef9efd5e30 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.239+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9efd5e30 0x55ef9efd8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.240+0000 7f1b01ffb640 10 client.27389 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.240+0000 7f1b19d9b640 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9efd5e30 0x55ef9efd8220 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80d5c9eaa421acfb server_cookie=ec0f69eb1faba334 in_seq=0 out_seq=0 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.240+0000 7f1b01ffb640 1 -- 172.21.15.45:0/3354065975 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef9eb78700 con 0x55ef9efd5e30 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.240+0000 7f1b01ffb640 10 client.27389 handle_command_reply: tid=0 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.240+0000 7f1b23c04740 1 client.27389 shutdown 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 msgr2=0x55ef9ef73dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef73dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55ef9ef7ed10 tx=0x7f1b0c0055e0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 shutdown_connections 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ef9efd5e30 0x55ef9efd8220 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ef9ef6e960 0x55ef9efd44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ef9ef65d80 0x55ef9ef73dc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 --2- 172.21.15.45:0/3354065975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef9ef65460 0x55ef9ef73880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 >> 172.21.15.45:0/3354065975 conn(0x55ef9ef5fb90 msgr2=0x55ef9ef6a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b01ffb640 0 client.27389 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 shutdown_connections 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 1 -- 172.21.15.45:0/3354065975 wait complete. 2024-03-12T05:20:14.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.241+0000 7f1b23c04740 20 client.27389 trim_cache size 0 max 16384 2024-03-12T05:20:14.241 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.243+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 msgr2=0x7f1b040a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.243+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 0x7f1b040a4120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c126290 tx=0x7f1b1c04ff00 comp rx=0 tx=0).stop 2024-03-12T05:20:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.243+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 msgr2=0x7f1b14132950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.243+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 0x7f1b14132950 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c04f1b0 tx=0x7f1b1c069290 comp rx=0 tx=0).stop 2024-03-12T05:20:14.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.244+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 shutdown_connections 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.245+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1b040a1c60 0x7f1b040a4120 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.245+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b140957f0 0x7f1b14138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.245+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b14099bc0 0x7f1b14132950 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.245+0000 7f1b1b59e640 1 --2- 172.21.15.45:0/315130996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b14094e50 0x7f1b14138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.245+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 >> 172.21.15.45:0/315130996 conn(0x7f1b1400a740 msgr2=0x7f1b140987f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.246+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 shutdown_connections 2024-03-12T05:20:14.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:14.246+0000 7f1b1b59e640 1 -- 172.21.15.45:0/315130996 wait complete. 2024-03-12T05:20:14.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:13 smithi116 ceph-mon[35486]: pgmap v290: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:20:14.244 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:13 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2492173560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 490.0 s, 1285.4 tps, lat 77.778 ms stddev 29.210 2024-03-12T05:20:15.262 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:20:15.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:20:15.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:20:15.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3887398996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4106450 msgr2=0x7fb4d410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3887398996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4106450 0x7fb4d410e020 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009930 tx=0x7fb4c402bc30 comp rx=0 tx=0).stop 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3887398996 shutdown_connections 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3887398996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d4101d30 0x7fb4d4105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3887398996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4106450 0x7fb4d410e020 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3887398996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4d4101380 0x7fb4d4101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3887398996 >> 172.21.15.45:0/3887398996 conn(0x7fb4d4078580 msgr2=0x7fb4d4078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3887398996 shutdown_connections 2024-03-12T05:20:15.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.447+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3887398996 wait complete. 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 Processor -- start 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 -- start start 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4101380 0x7fb4d41976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4d4106450 0x7fb4d4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d419c960 con 0x7fb4d419bf70 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d419cad0 con 0x7fb4d4101380 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.448+0000 7fb4da1e9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d419cc40 con 0x7fb4d4106450 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41592/0 (socket says 172.21.15.45:41592) 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 -- 172.21.15.45:0/3830130564 learned_addr learned my addr 172.21.15.45:0/3830130564 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4101380 0x7fb4d41976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 -- 172.21.15.45:0/3830130564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4d4106450 msgr2=0x7fb4d4197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4d4106450 0x7fb4d4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 -- 172.21.15.45:0/3830130564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4101380 msgr2=0x7fb4d41976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4101380 0x7fb4d41976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 -- 172.21.15.45:0/3830130564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c4009560 con 0x7fb4d419bf70 2024-03-12T05:20:15.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.449+0000 7fb4d3fff640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c1c0 tx=0x7fb4c4004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.450+0000 7fb4d0ff9640 1 -- 172.21.15.45:0/3830130564 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40042d0 con 0x7fb4d419bf70 2024-03-12T05:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.450+0000 7fb4d0ff9640 1 -- 172.21.15.45:0/3830130564 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb4c4004430 con 0x7fb4d419bf70 2024-03-12T05:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.450+0000 7fb4d0ff9640 1 -- 172.21.15.45:0/3830130564 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40052e0 con 0x7fb4d419bf70 2024-03-12T05:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.450+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d40733b0 con 0x7fb4d419bf70 2024-03-12T05:20:15.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.450+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d4073870 con 0x7fb4d419bf70 2024-03-12T05:20:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.454+0000 7fb4d0ff9640 1 -- 172.21.15.45:0/3830130564 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb4c4005440 con 0x7fb4d419bf70 2024-03-12T05:20:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 Processor -- start 2024-03-12T05:20:15.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 -- start start 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378557170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937856faa0 0x559378571e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593784c3f90 con 0x55937856faa0 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593784c14c0 con 0x55937855e3e0 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593784bcca0 con 0x55937855da30 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4d0ff9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 0x7fb4bc0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35500/0 (socket says 172.21.15.45:35500) 2024-03-12T05:20:15.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/1410362055 learned_addr learned my addr 172.21.15.45:0/1410362055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.455+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937856faa0 0x559378571e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d0ff9640 1 -- 172.21.15.45:0/3830130564 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb4c402de40 con 0x7fb4d419bf70 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378557170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d3fff640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 0x7fb4bc0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/1410362055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 msgr2=0x559378557170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378557170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/1410362055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937856faa0 msgr2=0x559378571e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937856faa0 0x559378571e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/1410362055 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55937847f0b0 con 0x55937855da30 2024-03-12T05:20:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.456+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378557170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:15.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.457+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8030360 tx=0x7fb4c80095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96052525ff819469 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.457+0000 7fb4b9ffb640 1 -- 172.21.15.45:0/1410362055 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8004230 con 0x55937855da30 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.457+0000 7fb4b9ffb640 1 -- 172.21.15.45:0/1410362055 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb4c8030e20 con 0x55937855da30 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.457+0000 7fb4d3fff640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 0x7fb4bc0a4270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40c4dd0 tx=0x7fb4c4009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.457+0000 7fb4b9ffb640 1 -- 172.21.15.45:0/1410362055 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8004bc0 con 0x55937855da30 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/1410362055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 msgr2=0x55937855de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8030360 tx=0x7fb4c80095b0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/1410362055 shutdown_connections 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937856faa0 0x559378571e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378557170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/1410362055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937855da30 0x55937855de10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/1410362055 >> 172.21.15.45:0/1410362055 conn(0x559378558050 msgr2=0x559378558430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.458+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/1410362055 shutdown_connections 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.459+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/1410362055 wait complete. 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.459+0000 7fb4dcdc0740 1 -- start start 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.459+0000 7fb4dcdc0740 1 Processor -- start 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378554610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378554b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x559378555090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559378461f40 con 0x55937855da30 2024-03-12T05:20:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593781904b0 con 0x55937855e3e0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559378442670 con 0x55937856faa0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4dcdc0740 10 client.0 _ll_register_callbacks cb 0x5593784aff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378554610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378554b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378554610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41622/0 (socket says 172.21.15.45:41622) 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/2532906024 learned_addr learned my addr 172.21.15.45:0/2532906024 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.460+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x559378555090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 msgr2=0x559378554b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378554b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/2532906024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 msgr2=0x559378554610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378554610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/2532906024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c8009cd0 con 0x55937856faa0 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x559378555090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80048a0 tx=0x7fb4c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9278527968fe6568 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8005460 con 0x55937856faa0 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.461+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb4c8004b50 con 0x55937856faa0 2024-03-12T05:20:15.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.462+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559378496f40 con 0x55937856faa0 2024-03-12T05:20:15.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.462+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4c8004d10 con 0x55937856faa0 2024-03-12T05:20:15.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.462+0000 7fb4dcdc0740 10 client.27391 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:15.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.462+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593784bb320 con 0x55937856faa0 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.463+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559378496f40 con 0x55937856faa0 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.464+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb4c803d410 con 0x55937856faa0 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 10 client.27391 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 10 client.27391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 20 client.27391 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55937856c2a0 0x55937856e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 4 client.27391 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593784b3310 con 0x55937856c2a0 2024-03-12T05:20:15.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.465+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55937856c2a0 0x55937856e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.467+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55937856c2a0 0x55937856e6c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a4fd82b16ea91 server_cookie=d1882f19c17b52ef in_seq=0 out_seq=0 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.467+0000 7fb4b2ffd640 10 client.27391 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.467+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/2532906024 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5593784b3310 con 0x55937856c2a0 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.467+0000 7fb4b2ffd640 10 client.27391 handle_command_reply: tid=0 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.467+0000 7fb4dcdc0740 1 client.27391 shutdown 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 msgr2=0x559378555090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x559378555090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80048a0 tx=0x7fb4c80046c0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 shutdown_connections 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55937856c2a0 0x55937856e6c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x559378555090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855e3e0 0x559378554b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2532906024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378554610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 >> 172.21.15.45:0/2532906024 conn(0x559378558050 msgr2=0x559378559fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4b2ffd640 0 client.27391 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 shutdown_connections 2024-03-12T05:20:15.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.468+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2532906024 wait complete. 2024-03-12T05:20:15.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.469+0000 7fb4dcdc0740 20 client.27391 trim_cache size 0 max 16384 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.499+0000 7fb4dcdc0740 1 Processor -- start 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.499+0000 7fb4dcdc0740 1 -- start start 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.499+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 0x559378556d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.499+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 0x559378562e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x5593785633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593784b3310 con 0x55937855e3e0 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559378496f40 con 0x55937855da30 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4dcdc0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559378568df0 con 0x55937856faa0 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 0x559378556d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 0x559378556d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60742/0 (socket says 172.21.15.45:60742) 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/2190821448 learned_addr learned my addr 172.21.15.45:0/2190821448 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.500+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 0x559378562e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x5593785633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d37fe640 1 -- 172.21.15.45:0/2190821448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 msgr2=0x5593785633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x5593785633a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d37fe640 1 -- 172.21.15.45:0/2190821448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 msgr2=0x559378556d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 0x559378556d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.501+0000 7fb4d37fe640 1 -- 172.21.15.45:0/2190821448 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559378457c00 con 0x55937855e3e0 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.502+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x5593785633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:15.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.502+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 0x559378562e60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x55937856ef90 tx=0x7fb4cc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb68794882740e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.503+0000 7fb4b97fa640 1 -- 172.21.15.45:0/2190821448 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc018070 con 0x55937855e3e0 2024-03-12T05:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.503+0000 7fb4b97fa640 1 -- 172.21.15.45:0/2190821448 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb4cc019040 con 0x55937855e3e0 2024-03-12T05:20:15.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.503+0000 7fb4b97fa640 1 -- 172.21.15.45:0/2190821448 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc01aea0 con 0x55937855e3e0 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2190821448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 msgr2=0x559378562e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 0x559378562e60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x55937856ef90 tx=0x7fb4cc009500 comp rx=0 tx=0).stop 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2190821448 shutdown_connections 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855e3e0 0x559378562e60 secure :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0x55937856ef90 tx=0x7fb4cc009500 comp rx=0 tx=0).stop 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55937856faa0 0x5593785633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/2190821448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937855da30 0x559378556d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.504+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2190821448 >> 172.21.15.45:0/2190821448 conn(0x559378558050 msgr2=0x55937856efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.505+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2190821448 shutdown_connections 2024-03-12T05:20:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.505+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/2190821448 wait complete. 2024-03-12T05:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.505+0000 7fb4dcdc0740 1 -- start start 2024-03-12T05:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.506+0000 7fb4dcdc0740 1 Processor -- start 2024-03-12T05:20:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.506+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378562f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 0x559378561250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559378561790 0x559378561c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378562f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378562f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41642/0 (socket says 172.21.15.45:41642) 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4d2ffd640 1 -- 172.21.15.45:0/3814297512 learned_addr learned my addr 172.21.15.45:0/3814297512 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593785635f0 con 0x55937855da30 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559378563760 con 0x55937856faa0 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593785c5da0 con 0x559378561790 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.507+0000 7fb4dcdc0740 10 client.0 _ll_register_callbacks cb 0x7fb4c80098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 0x559378561250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559378561790 0x559378561c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 -- 172.21.15.45:0/3814297512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559378561790 msgr2=0x559378561c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559378561790 0x559378561c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 -- 172.21.15.45:0/3814297512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 msgr2=0x559378562f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378562f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.508+0000 7fb4d37fe640 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc003b10 con 0x55937856faa0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4d37fe640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 0x559378561250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5593785559c0 tx=0x7fb4cc009d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416a5db6d6b4c3f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc018040 con 0x55937856faa0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb4cc01a630 con 0x55937856faa0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593784e2900 con 0x55937856faa0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4cc009590 con 0x55937856faa0 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4dcdc0740 10 client.27401 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:15.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.509+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55937847d910 con 0x55937856faa0 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5593784e2900 con 0x55937856faa0 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb4cc012450 con 0x55937856faa0 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 10 client.27401 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 10 client.27401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 20 client.27401 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5593785c74f0 0x5593785c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 4 client.27401 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.510+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5593781bc860 con 0x5593785c74f0 2024-03-12T05:20:15.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.511+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5593785c74f0 0x5593785c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.512+0000 7fb4d2ffd640 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5593785c74f0 0x5593785c9910 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4551d100e7d52f9 server_cookie=9949444ccd234c7 in_seq=0 out_seq=0 2024-03-12T05:20:15.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.512+0000 7fb4b2ffd640 10 client.27401 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.513+0000 7fb4b2ffd640 1 -- 172.21.15.45:0/3814297512 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5593781bc860 con 0x5593785c74f0 2024-03-12T05:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.513+0000 7fb4b2ffd640 10 client.27401 handle_command_reply: tid=0 2024-03-12T05:20:15.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.513+0000 7fb4dcdc0740 1 client.27401 shutdown 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 msgr2=0x559378561250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 0x559378561250 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5593785559c0 tx=0x7fb4cc009d80 comp rx=0 tx=0).stop 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 shutdown_connections 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5593785c74f0 0x5593785c9910 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4b2ffd640 0 client.27401 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559378561790 0x559378561c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55937856faa0 0x559378561250 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 --2- 172.21.15.45:0/3814297512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55937855da30 0x559378562f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 >> 172.21.15.45:0/3814297512 conn(0x559378558050 msgr2=0x55937854e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 shutdown_connections 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 1 -- 172.21.15.45:0/3814297512 wait complete. 2024-03-12T05:20:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.514+0000 7fb4dcdc0740 20 client.27401 trim_cache size 0 max 16384 2024-03-12T05:20:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:15.511 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:20:15.511 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "bfbaa9a3-320d-4467-8124-e2893219c243", 2024-03-12T05:20:15.511 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:20:15.511 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 msgr2=0x7fb4bc0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 0x7fb4bc0a4270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40c4dd0 tx=0x7fb4c4009260 comp rx=0 tx=0).stop 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 msgr2=0x7fb4d419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb4c402c1c0 tx=0x7fb4c4004bd0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 shutdown_connections 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4bc0a1db0 0x7fb4bc0a4270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4d419bf70 0x7fb4d419c420 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4d4106450 0x7fb4d4197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.515+0000 7fb4da1e9640 1 --2- 172.21.15.45:0/3830130564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4d4101380 0x7fb4d41976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.516+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 >> 172.21.15.45:0/3830130564 conn(0x7fb4d4078580 msgr2=0x7fb4d40ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.516+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 shutdown_connections 2024-03-12T05:20:15.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.516+0000 7fb4da1e9640 1 -- 172.21.15.45:0/3830130564 wait complete. 2024-03-12T05:20:15.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:20:15.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:20:15.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:15 smithi045 ceph-mon[32174]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T05:20:15.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:15 smithi116 ceph-mon[35486]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T05:20:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.751+0000 7f49fd081640 1 -- 172.21.15.45:0/858397367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108390 msgr2=0x7f49f8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.751+0000 7f49fd081640 1 --2- 172.21.15.45:0/858397367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108390 0x7f49f8108770 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f49f000dfa0 tx=0x7f49f002fee0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 -- 172.21.15.45:0/858397367 shutdown_connections 2024-03-12T05:20:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 --2- 172.21.15.45:0/858397367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f81166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 --2- 172.21.15.45:0/858397367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108cb0 0x7f49f8113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 --2- 172.21.15.45:0/858397367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108390 0x7f49f8108770 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 -- 172.21.15.45:0/858397367 >> 172.21.15.45:0/858397367 conn(0x7f49f806c270 msgr2=0x7f49f806c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 -- 172.21.15.45:0/858397367 shutdown_connections 2024-03-12T05:20:15.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.752+0000 7f49fd081640 1 -- 172.21.15.45:0/858397367 wait complete. 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 Processor -- start 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 -- start start 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108390 0x7f49f81a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819e0b0 con 0x7f49f81142d0 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819e220 con 0x7f49f8108cb0 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.753+0000 7f49fd081640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819e3e0 con 0x7f49f8108390 2024-03-12T05:20:15.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60760/0 (socket says 172.21.15.45:60760) 2024-03-12T05:20:15.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 -- 172.21.15.45:0/1185824459 learned_addr learned my addr 172.21.15.45:0/1185824459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108390 0x7f49f81a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 -- 172.21.15.45:0/1185824459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108390 msgr2=0x7f49f81a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108390 0x7f49f81a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 -- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 msgr2=0x7f49f819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f659c640 1 -- 172.21.15.45:0/1185824459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f000dc50 con 0x7f49f8108cb0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.754+0000 7f49f759e640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.755+0000 7f49f659c640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f49e800d950 tx=0x7f49e800dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.755+0000 7f49dffff640 1 -- 172.21.15.45:0/1185824459 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8009c30 con 0x7f49f8108cb0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.755+0000 7f49dffff640 1 -- 172.21.15.45:0/1185824459 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49e800de80 con 0x7f49f8108cb0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.755+0000 7f49dffff640 1 -- 172.21.15.45:0/1185824459 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8009c30 con 0x7f49f8108cb0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.756+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f819e6c0 con 0x7f49f8108cb0 2024-03-12T05:20:15.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.756+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f81a4920 con 0x7f49f8108cb0 2024-03-12T05:20:15.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.759+0000 7f49ffc51740 1 Processor -- start 2024-03-12T05:20:15.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.759+0000 7f49ffc51740 1 -- start start 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc4b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc56ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fbba280 con 0x56382fc547b0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fbb77b0 con 0x56382fc4bbd0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49ffc51740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fbb2f90 con 0x56382fc4b2b0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc4b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc56ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60776/0 (socket says 172.21.15.45:60776) 2024-03-12T05:20:15.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2649143376 learned_addr learned my addr 172.21.15.45:0/2649143376 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2649143376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 msgr2=0x56382fc4b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc4b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2649143376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 msgr2=0x56382fc56ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc56ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.760+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2649143376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56382fb71570 con 0x56382fc4bbd0 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.761+0000 7f49dffff640 1 -- 172.21.15.45:0/1185824459 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f49e8010460 con 0x7f49f8108cb0 2024-03-12T05:20:15.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.761+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004af0 tx=0x7f49f0030a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb4e0729923b1bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.761+0000 7f49dd7fa640 1 -- 172.21.15.45:0/2649143376 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0030ea0 con 0x56382fc4bbd0 2024-03-12T05:20:15.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.762+0000 7f49dd7fa640 1 -- 172.21.15.45:0/2649143376 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49f000b3b0 con 0x56382fc4bbd0 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.762+0000 7f49dffff640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 0x7f49e00a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.762+0000 7f49dffff640 1 -- 172.21.15.45:0/1185824459 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f49e80cd060 con 0x7f49f8108cb0 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.763+0000 7f49dd7fa640 1 -- 172.21.15.45:0/2649143376 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0042630 con 0x56382fc4bbd0 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.763+0000 7f49f759e640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 0x7f49e00a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.763+0000 7f49ffc51740 1 -- 172.21.15.45:0/2649143376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 msgr2=0x56382fc54270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.763+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004af0 tx=0x7f49f0030a00 comp rx=0 tx=0).stop 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49f759e640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 0x7f49e00a4140 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49ec005fd0 tx=0x7f49ec009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49ffc51740 1 -- 172.21.15.45:0/2649143376 shutdown_connections 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc56ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4bbd0 0x56382fc54270 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2649143376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc4b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.764+0000 7f49ffc51740 1 -- 172.21.15.45:0/2649143376 >> 172.21.15.45:0/2649143376 conn(0x56382fc45990 msgr2=0x56382fc45d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.765+0000 7f49ffc51740 1 -- 172.21.15.45:0/2649143376 shutdown_connections 2024-03-12T05:20:15.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.765+0000 7f49ffc51740 1 -- 172.21.15.45:0/2649143376 wait complete. 2024-03-12T05:20:15.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.766+0000 7f49ffc51740 1 -- start start 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.766+0000 7f49ffc51740 1 Processor -- start 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc51470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc547b0 0x56382fc519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382f886430 con 0x56382fc4bbd0 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382f842e00 con 0x56382fc547b0 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fb37fd0 con 0x56382fc4b2b0 2024-03-12T05:20:15.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35574/0 (socket says 172.21.15.45:35574) 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2353758452 learned_addr learned my addr 172.21.15.45:0/2353758452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f659c640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc51470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49ffc51740 10 client.0 _ll_register_callbacks cb 0x56382fb8dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc547b0 msgr2=0x56382fc519b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc547b0 0x56382fc519b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2353758452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 msgr2=0x56382fc51470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc51470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.767+0000 7f49f6d9d640 1 -- 172.21.15.45:0/2353758452 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f8117010 con 0x56382fc4b2b0 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.768+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49f003a040 tx=0x7f49f0030bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4986cdae3c38d050 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.768+0000 7f49daff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.768+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0075070 con 0x56382fc4b2b0 2024-03-12T05:20:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.768+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f49f006f460 con 0x56382fc4b2b0 2024-03-12T05:20:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.768+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56382f8b27e0 con 0x56382fc4b2b0 2024-03-12T05:20:15.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.769+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49f0030ea0 con 0x56382fc4b2b0 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.769+0000 7f49ffc51740 10 client.27406 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.769+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56382fbb1610 con 0x56382fc4b2b0 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49f006fad0 con 0x56382fc4b2b0 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f49f0078020 con 0x56382fc4b2b0 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 10 client.27406 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 10 client.27406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 20 client.27406 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fc64380 0x56382fc66810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 4 client.27406 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:15.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.770+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56382f8b27e0 con 0x56382fc64380 2024-03-12T05:20:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.771+0000 7f49f659c640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fc64380 0x56382fc66810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.771+0000 7f49daff5640 10 client.27406 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.772+0000 7f49f659c640 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fc64380 0x56382fc66810 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ba346dbaa985e4 server_cookie=d7203905ac81fae4 in_seq=0 out_seq=0 2024-03-12T05:20:15.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.774+0000 7f49daff5640 1 -- 172.21.15.45:0/2353758452 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56382f8b27e0 con 0x56382fc64380 2024-03-12T05:20:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49daff5640 10 client.27406 handle_command_reply: tid=0 2024-03-12T05:20:15.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 client.27406 shutdown 2024-03-12T05:20:15.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 msgr2=0x56382fc540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49f003a040 tx=0x7f49f0030bf0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 shutdown_connections 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fc64380 0x56382fc66810 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc51470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc547b0 0x56382fc519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.775+0000 7f49ffc51740 1 --2- 172.21.15.45:0/2353758452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4b2b0 0x56382fc540e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.776+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 >> 172.21.15.45:0/2353758452 conn(0x56382fc45990 msgr2=0x56382fc47730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.776+0000 7f49daff5640 0 client.27406 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.776+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 shutdown_connections 2024-03-12T05:20:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.776+0000 7f49ffc51740 1 -- 172.21.15.45:0/2353758452 wait complete. 2024-03-12T05:20:15.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.776+0000 7f49ffc51740 20 client.27406 trim_cache size 0 max 16384 2024-03-12T05:20:15.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.806+0000 7f49ffc51740 1 Processor -- start 2024-03-12T05:20:15.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.806+0000 7f49ffc51740 1 -- start start 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc53ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc547b0 0x56382fc540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc4af10 con 0x56382fc4bbd0 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc57480 con 0x56382fc4b2b0 2024-03-12T05:20:15.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49ffc51740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc46c10 con 0x56382fc547b0 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60794/0 (socket says 172.21.15.45:60794) 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.807+0000 7f49f6d9d640 1 -- 172.21.15.45:0/961858093 learned_addr learned my addr 172.21.15.45:0/961858093 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f759e640 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc547b0 0x56382fc540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 -- 172.21.15.45:0/961858093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc547b0 msgr2=0x56382fc540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc547b0 0x56382fc540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 -- 172.21.15.45:0/961858093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 msgr2=0x56382fc53ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc53ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 -- 172.21.15.45:0/961858093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56382fb4dac0 con 0x56382fc4b2b0 2024-03-12T05:20:15.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.808+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004af0 tx=0x7f49f000df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523c28578e09cabe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.809+0000 7f49dcff9640 1 -- 172.21.15.45:0/961858093 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0030ea0 con 0x56382fc4b2b0 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.809+0000 7f49dcff9640 1 -- 172.21.15.45:0/961858093 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49f0030950 con 0x56382fc4b2b0 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.809+0000 7f49dcff9640 1 -- 172.21.15.45:0/961858093 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f00043c0 con 0x56382fc4b2b0 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.810+0000 7f49ffc51740 1 -- 172.21.15.45:0/961858093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 msgr2=0x56382fc6b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.810+0000 7f49ffc51740 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004af0 tx=0x7f49f000df20 comp rx=0 tx=0).stop 2024-03-12T05:20:15.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.811+0000 7f49ffc51740 1 -- 172.21.15.45:0/961858093 shutdown_connections 2024-03-12T05:20:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.811+0000 7f49ffc51740 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc4bbd0 0x56382fc53ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.811+0000 7f49ffc51740 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc547b0 0x56382fc540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.811+0000 7f49ffc51740 1 --2- 172.21.15.45:0/961858093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4b2b0 0x56382fc6b170 secure :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f49f0004af0 tx=0x7f49f000df20 comp rx=0 tx=0).stop 2024-03-12T05:20:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.811+0000 7f49ffc51740 1 -- 172.21.15.45:0/961858093 >> 172.21.15.45:0/961858093 conn(0x56382fc45990 msgr2=0x56382fc4c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.812+0000 7f49ffc51740 1 -- 172.21.15.45:0/961858093 shutdown_connections 2024-03-12T05:20:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.812+0000 7f49ffc51740 1 -- 172.21.15.45:0/961858093 wait complete. 2024-03-12T05:20:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.812+0000 7f49ffc51740 1 -- start start 2024-03-12T05:20:15.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.812+0000 7f49ffc51740 1 Processor -- start 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc5bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 0x56382fc4db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc4e0a0 con 0x56382fc547b0 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc4e210 con 0x56382fc4d6e0 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382fc4e3d0 con 0x56382fc4bbd0 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49ffc51740 10 client.0 _ll_register_callbacks cb 0x7f49f000de70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49f6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:35590/0 (socket says 172.21.15.45:35590) 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49f6d9d640 1 -- 172.21.15.45:0/1422654174 learned_addr learned my addr 172.21.15.45:0/1422654174 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.813+0000 7f49f759e640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 0x56382fc4db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f659c640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc5bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 -- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 msgr2=0x56382fc4db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 0x56382fc4db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 -- 172.21.15.45:0/1422654174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 msgr2=0x56382fc5bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc5bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 -- 172.21.15.45:0/1422654174 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56382fbb1610 con 0x56382fc4bbd0 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f759e640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 0x56382fc4db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:15.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49f6d9d640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49f0044040 tx=0x7f49f006d660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7764d1a31a9f49a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49daff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f0075070 con 0x56382fc4bbd0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.814+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f49f000d5f0 con 0x56382fc4bbd0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.815+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56382fc512a0 con 0x56382fc4bbd0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.815+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49f0003b80 con 0x56382fc4bbd0 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.815+0000 7f49ffc51740 10 client.27412 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:15.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.815+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56382fb736a0 con 0x56382fc4bbd0 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.817+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f49f0003ce0 con 0x56382fc4bbd0 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.817+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f49f0078070 con 0x56382fc4bbd0 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.818+0000 7f49ffc51740 10 client.27412 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.818+0000 7f49ffc51740 10 client.27412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.819+0000 7f49ffc51740 20 client.27412 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.819+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fcbb880 0x56382fcbdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.819+0000 7f49ffc51740 4 client.27412 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.819+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56382fc512a0 con 0x56382fcbb880 2024-03-12T05:20:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.819+0000 7f49f659c640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fcbb880 0x56382fcbdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.820+0000 7f49f659c640 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fcbb880 0x56382fcbdc70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b73a8ac9ebfa6d2 server_cookie=4a0d9bab5e9914a2 in_seq=0 out_seq=0 2024-03-12T05:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.820+0000 7f49daff5640 10 client.27412 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.821+0000 7f49daff5640 1 -- 172.21.15.45:0/1422654174 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56382fc512a0 con 0x56382fcbb880 2024-03-12T05:20:15.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.821+0000 7f49daff5640 10 client.27412 handle_command_reply: tid=0 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.821+0000 7f49ffc51740 1 client.27412 shutdown 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.822+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 msgr2=0x56382fc5b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.822+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49f0044040 tx=0x7f49f006d660 comp rx=0 tx=0).stop 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.822+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 shutdown_connections 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.822+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56382fcbb880 0x56382fcbdc70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56382fc547b0 0x56382fc5bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56382fc4d6e0 0x56382fc4db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49daff5640 0 client.27412 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 --2- 172.21.15.45:0/1422654174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56382fc4bbd0 0x56382fc5b6f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 >> 172.21.15.45:0/1422654174 conn(0x56382fc45990 msgr2=0x56382fc6b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 shutdown_connections 2024-03-12T05:20:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.823+0000 7f49ffc51740 1 -- 172.21.15.45:0/1422654174 wait complete. 2024-03-12T05:20:15.821 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:15.821 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:20:15.821 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:20:15.821 INFO:teuthology.orchestra.run.smithi045.stdout: "bfbaa9a3-320d-4467-8124-e2893219c243": { 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "bfbaa9a3-320d-4467-8124-e2893219c243", 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:20:15.822 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.825+0000 7f49ffc51740 20 client.27412 trim_cache size 0 max 16384 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 msgr2=0x7f49e00a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 0x7f49e00a4140 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49ec005fd0 tx=0x7f49ec009260 comp rx=0 tx=0).stop 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 msgr2=0x7f49f81a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f49e800d950 tx=0x7f49e800dd50 comp rx=0 tx=0).stop 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 shutdown_connections 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f49e00a1c80 0x7f49e00a4140 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49f81142d0 0x7f49f819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49f8108cb0 0x7f49f81a3df0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 --2- 172.21.15.45:0/1185824459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49f8108390 0x7f49f81a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:15.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.827+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 >> 172.21.15.45:0/1185824459 conn(0x7f49f806c270 msgr2=0x7f49f806ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:15.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.828+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 shutdown_connections 2024-03-12T05:20:15.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:15.829+0000 7f49fd081640 1 -- 172.21.15.45:0/1185824459 wait complete. 2024-03-12T05:20:15.846 INFO:tasks.cephfs.filesystem:scrub status for tag:bfbaa9a3-320d-4467-8124-e2893219c243 - {'path': '/', 'tag': 'bfbaa9a3-320d-4467-8124-e2893219c243', 'options': 'recursive,force'} 2024-03-12T05:20:15.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:15 smithi130 ceph-mon[35398]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: pgmap v292: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 1.67k op/s 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:16.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.743 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:16 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: pgmap v292: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 1.67k op/s 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:16 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: pgmap v292: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 14 MiB/s wr, 1.67k op/s 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:16.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:16 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:18.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:18 smithi116 ceph-mon[35486]: pgmap v293: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:20:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:18 smithi045 ceph-mon[32174]: pgmap v293: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:20:18.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:18 smithi130 ceph-mon[35398]: pgmap v293: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.50k op/s 2024-03-12T05:20:19.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 495.0 s, 1341.0 tps, lat 74.430 ms stddev 28.567 2024-03-12T05:20:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:20 smithi045 ceph-mon[32174]: pgmap v294: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:20 smithi130 ceph-mon[35398]: pgmap v294: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:20 smithi116 ceph-mon[35486]: pgmap v294: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:22.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:22 smithi116 ceph-mon[35486]: pgmap v295: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:22 smithi045 ceph-mon[32174]: pgmap v295: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:22.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:22 smithi130 ceph-mon[35398]: pgmap v295: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:24 smithi045 ceph-mon[32174]: pgmap v296: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 500.0 s, 1460.4 tps, lat 68.715 ms stddev 20.055 2024-03-12T05:20:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:24 smithi130 ceph-mon[35398]: pgmap v296: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:24 smithi116 ceph-mon[35486]: pgmap v296: 97 pgs: 97 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-03-12T05:20:26.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:26 smithi045 ceph-mon[32174]: pgmap v297: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:20:26.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:26 smithi130 ceph-mon[35398]: pgmap v297: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:20:26.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:26 smithi116 ceph-mon[35486]: pgmap v297: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:20:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:28 smithi045 ceph-mon[32174]: pgmap v298: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 9.8 MiB/s wr, 1.14k op/s 2024-03-12T05:20:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:20:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:20:28.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:20:28.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:28 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:20:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:28 smithi130 ceph-mon[35398]: pgmap v298: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 9.8 MiB/s wr, 1.14k op/s 2024-03-12T05:20:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:20:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:20:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:20:28.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:28 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:20:28.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:28 smithi116 ceph-mon[35486]: pgmap v298: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 9.8 MiB/s wr, 1.14k op/s 2024-03-12T05:20:28.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:20:28.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:20:28.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:20:28.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:28 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:20:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 505.0 s, 1540.0 tps, lat 64.980 ms stddev 21.143 2024-03-12T05:20:30.768 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:30 smithi130 ceph-mon[35398]: pgmap v299: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:20:30.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:30 smithi045 ceph-mon[32174]: pgmap v299: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:20:30.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:30 smithi116 ceph-mon[35486]: pgmap v299: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 15 MiB/s wr, 1.73k op/s 2024-03-12T05:20:32.793 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:32 smithi116 ceph-mon[35486]: pgmap v300: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:20:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:32 smithi045 ceph-mon[32174]: pgmap v300: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:20:32.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:32 smithi130 ceph-mon[35398]: pgmap v300: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 12 MiB/s wr, 1.34k op/s 2024-03-12T05:20:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:34 smithi045 ceph-mon[32174]: pgmap v301: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:20:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 510.0 s, 1552.2 tps, lat 64.235 ms stddev 21.896 2024-03-12T05:20:34.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:34 smithi130 ceph-mon[35398]: pgmap v301: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:20:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:34 smithi116 ceph-mon[35486]: pgmap v301: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:20:35.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:35 smithi045 ceph-mon[32174]: pgmap v302: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:20:35.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:35 smithi130 ceph-mon[35398]: pgmap v302: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:20:35.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:35 smithi116 ceph-mon[35486]: pgmap v302: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 17 MiB/s wr, 1.98k op/s 2024-03-12T05:20:37.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:37 smithi130 ceph-mon[35398]: pgmap v303: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:20:37.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:37 smithi116 ceph-mon[35486]: pgmap v303: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:20:38.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:37 smithi045 ceph-mon[32174]: pgmap v303: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:20:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 515.0 s, 1389.8 tps, lat 71.603 ms stddev 49.093 2024-03-12T05:20:39.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:39 smithi130 ceph-mon[35398]: pgmap v304: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.85k op/s 2024-03-12T05:20:39.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:39 smithi116 ceph-mon[35486]: pgmap v304: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.85k op/s 2024-03-12T05:20:40.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:39 smithi045 ceph-mon[32174]: pgmap v304: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 16 MiB/s wr, 1.85k op/s 2024-03-12T05:20:41.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:41 smithi130 ceph-mon[35398]: pgmap v305: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:20:41.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:41 smithi116 ceph-mon[35486]: pgmap v305: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:20:42.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:41 smithi045 ceph-mon[32174]: pgmap v305: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.34k op/s 2024-03-12T05:20:43.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:43 smithi130 ceph-mon[35398]: pgmap v306: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:20:43.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:43 smithi116 ceph-mon[35486]: pgmap v306: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:20:44.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:43 smithi045 ceph-mon[32174]: pgmap v306: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:20:44.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 520.0 s, 1402.2 tps, lat 71.624 ms stddev 48.168 2024-03-12T05:20:45.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:20:45.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:20:45.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:45 smithi130 ceph-mon[35398]: pgmap v307: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.80k op/s 2024-03-12T05:20:45.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:45 smithi116 ceph-mon[35486]: pgmap v307: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.80k op/s 2024-03-12T05:20:46.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:45 smithi045 ceph-mon[32174]: pgmap v307: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 15 MiB/s wr, 1.80k op/s 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 -- 172.21.15.45:0/207939320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 msgr2=0x7f908c0682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 --2- 172.21.15.45:0/207939320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c0682e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9084009860 tx=0x7f908402bb70 comp rx=0 tx=0).stop 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 -- 172.21.15.45:0/207939320 shutdown_connections 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 --2- 172.21.15.45:0/207939320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c0682e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 --2- 172.21.15.45:0/207939320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100c20 0x7f908c104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.255+0000 7f9093cdf640 1 --2- 172.21.15.45:0/207939320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c100270 0x7f908c100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.256+0000 7f9093cdf640 1 -- 172.21.15.45:0/207939320 >> 172.21.15.45:0/207939320 conn(0x7f908c077230 msgr2=0x7f908c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.256+0000 7f9093cdf640 1 -- 172.21.15.45:0/207939320 shutdown_connections 2024-03-12T05:20:46.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.256+0000 7f9093cdf640 1 -- 172.21.15.45:0/207939320 wait complete. 2024-03-12T05:20:46.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.258+0000 7f9093cdf640 1 Processor -- start 2024-03-12T05:20:46.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.258+0000 7f9093cdf640 1 -- start start 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c10ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c110490 0x7f908c110910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1aa280 con 0x7f908c110490 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45038/0 (socket says 172.21.15.45:45038) 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 -- 172.21.15.45:0/659581578 learned_addr learned my addr 172.21.15.45:0/659581578 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1aa3f0 con 0x7f908c105340 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1aa5b0 con 0x7f908c100270 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f909127a640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c10ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f909227c640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c110490 0x7f908c110910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 -- 172.21.15.45:0/659581578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 msgr2=0x7f908c10ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c10ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 -- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c110490 msgr2=0x7f908c110910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c110490 0x7f908c110910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.259+0000 7f9091a7b640 1 -- 172.21.15.45:0/659581578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088009900 con 0x7f908c100270 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9091a7b640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f908800c420 tx=0x7f9088009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9082ffd640 1 -- 172.21.15.45:0/659581578 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800b630 con 0x7f908c100270 2024-03-12T05:20:46.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9084009490 con 0x7f908c100270 2024-03-12T05:20:46.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c1aac60 con 0x7f908c100270 2024-03-12T05:20:46.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9082ffd640 1 -- 172.21.15.45:0/659581578 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f908800bc40 con 0x7f908c100270 2024-03-12T05:20:46.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.260+0000 7f9082ffd640 1 -- 172.21.15.45:0/659581578 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088016d60 con 0x7f908c100270 2024-03-12T05:20:46.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.264+0000 7f909698c740 1 Processor -- start 2024-03-12T05:20:46.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 -- start start 2024-03-12T05:20:46.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c864690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c7d3280 con 0x55aa3c864bd0 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c7d07b0 con 0x55aa3c8642b0 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909698c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c7cbf90 con 0x55aa3c86d7b0 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f9082ffd640 1 -- 172.21.15.45:0/659581578 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f908800a040 con 0x7f908c100270 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909127a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909127a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45054/0 (socket says 172.21.15.45:45054) 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909127a640 1 -- 172.21.15.45:0/4139459877 learned_addr learned my addr 172.21.15.45:0/4139459877 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.265+0000 7f909227c640 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c864690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 -- 172.21.15.45:0/4139459877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 msgr2=0x55aa3c864690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c864690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 -- 172.21.15.45:0/4139459877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c864bd0 msgr2=0x55aa3c86d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 -- 172.21.15.45:0/4139459877 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa3c78a570 con 0x55aa3c86d7b0 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909127a640 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f907c009ba0 tx=0x7f907c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d56dc2cadc6b9b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f9082ffd640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 0x7f90780a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f9082ffd640 1 -- 172.21.15.45:0/659581578 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f90880cd820 con 0x7f908c100270 2024-03-12T05:20:46.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f909227c640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 0x7f90780a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.266+0000 7f9077fff640 1 -- 172.21.15.45:0/4139459877 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c004230 con 0x55aa3c86d7b0 2024-03-12T05:20:46.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.267+0000 7f9077fff640 1 -- 172.21.15.45:0/4139459877 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f907c009e10 con 0x55aa3c86d7b0 2024-03-12T05:20:46.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.267+0000 7f9077fff640 1 -- 172.21.15.45:0/4139459877 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c004c20 con 0x55aa3c86d7b0 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- 172.21.15.45:0/4139459877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 msgr2=0x55aa3c86fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f907c009ba0 tx=0x7f907c011810 comp rx=0 tx=0).stop 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909227c640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 0x7f90780a41a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9084009830 tx=0x7f908402c090 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- 172.21.15.45:0/4139459877 shutdown_connections 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86fba0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 --2- 172.21.15.45:0/4139459877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c864690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- 172.21.15.45:0/4139459877 >> 172.21.15.45:0/4139459877 conn(0x55aa3c85e990 msgr2=0x55aa3c85ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- 172.21.15.45:0/4139459877 shutdown_connections 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- 172.21.15.45:0/4139459877 wait complete. 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.268+0000 7f909698c740 1 -- start start 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 Processor -- start 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c49f430 con 0x55aa3c86d7b0 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c45be00 con 0x55aa3c8642b0 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c750fd0 con 0x55aa3c864bd0 2024-03-12T05:20:46.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.269+0000 7f909698c740 10 client.0 _ll_register_callbacks cb 0x55aa3c7a6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45066/0 (socket says 172.21.15.45:45066) 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 -- 172.21.15.45:0/1124276419 learned_addr learned my addr 172.21.15.45:0/1124276419 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f9091a7b640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 -- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 msgr2=0x55aa3c86d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 -- 172.21.15.45:0/1124276419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 msgr2=0x55aa3c86ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.270+0000 7f909127a640 1 -- 172.21.15.45:0/1124276419 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c009850 con 0x55aa3c864bd0 2024-03-12T05:20:46.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.271+0000 7f909127a640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f907c019c20 tx=0x7f907c019c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7540c30c088945fc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.271+0000 7f90757fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:46.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.271+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c005680 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa3c4cb7e0 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f907c024470 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f907c024630 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f909698c740 10 client.27424 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55aa3c7ca610 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.272+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55aa3c4cb7e0 con 0x55aa3c864bd0 2024-03-12T05:20:46.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f907c004cf0 con 0x55aa3c864bd0 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 10 client.27424 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 10 client.27424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 20 client.27424 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c87da60 0x55aa3c87fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 4 client.27424 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.273+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa3c508720 con 0x55aa3c87da60 2024-03-12T05:20:46.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.274+0000 7f9091a7b640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c87da60 0x55aa3c87fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.275+0000 7f9091a7b640 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c87da60 0x55aa3c87fe80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=238304f1cf67aa08 server_cookie=7b1c8044abe22979 in_seq=0 out_seq=0 2024-03-12T05:20:46.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.275+0000 7f90757fa640 10 client.27424 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f90757fa640 1 -- 172.21.15.45:0/1124276419 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aa3c508720 con 0x55aa3c87da60 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f90757fa640 10 client.27424 handle_command_reply: tid=0 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 client.27424 shutdown 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 msgr2=0x55aa3c86a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f907c019c20 tx=0x7f907c019c50 comp rx=0 tx=0).stop 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 shutdown_connections 2024-03-12T05:20:46.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c87da60 0x55aa3c87fe80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c86ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86a560 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 --2- 172.21.15.45:0/1124276419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.276+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 >> 172.21.15.45:0/1124276419 conn(0x55aa3c85e990 msgr2=0x55aa3c860730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.279+0000 7f90757fa640 0 client.27424 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:46.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.279+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 shutdown_connections 2024-03-12T05:20:46.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.280+0000 7f909698c740 1 -- 172.21.15.45:0/1124276419 wait complete. 2024-03-12T05:20:46.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.280+0000 7f909698c740 20 client.27424 trim_cache size 0 max 16384 2024-03-12T05:20:46.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.311+0000 7f909698c740 1 Processor -- start 2024-03-12T05:20:46.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.311+0000 7f909698c740 1 -- start start 2024-03-12T05:20:46.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c85e7f0 con 0x55aa3c86d7b0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c879600 con 0x55aa3c8642b0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909698c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c86b8b0 con 0x55aa3c864bd0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909227c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909227c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58388/0 (socket says 172.21.15.45:58388) 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55972/0 (socket says 172.21.15.45:55972) 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909227c640 1 -- 172.21.15.45:0/3276063454 learned_addr learned my addr 172.21.15.45:0/3276063454 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.312+0000 7f909127a640 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 -- 172.21.15.45:0/3276063454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 msgr2=0x55aa3c86d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 -- 172.21.15.45:0/3276063454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 msgr2=0x55aa3c86cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 -- 172.21.15.45:0/3276063454 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa3c766ac0 con 0x55aa3c86d7b0 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f9091a7b640 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f909227c640 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9084009860 tx=0x7f9084005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6eeaee544816bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f90777fe640 1 -- 172.21.15.45:0/3276063454 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084034640 con 0x55aa3c86d7b0 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f90777fe640 1 -- 172.21.15.45:0/3276063454 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f908402ce90 con 0x55aa3c86d7b0 2024-03-12T05:20:46.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.313+0000 7f90777fe640 1 -- 172.21.15.45:0/3276063454 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084003850 con 0x55aa3c86d7b0 2024-03-12T05:20:46.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.314+0000 7f909698c740 1 -- 172.21.15.45:0/3276063454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 msgr2=0x55aa3c865850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.314+0000 7f909698c740 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9084009860 tx=0x7f9084005f90 comp rx=0 tx=0).stop 2024-03-12T05:20:46.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 -- 172.21.15.45:0/3276063454 shutdown_connections 2024-03-12T05:20:46.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c865850 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c86d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 --2- 172.21.15.45:0/3276063454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c86cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 -- 172.21.15.45:0/3276063454 >> 172.21.15.45:0/3276063454 conn(0x55aa3c85e990 msgr2=0x55aa3c8847a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 -- 172.21.15.45:0/3276063454 shutdown_connections 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 -- 172.21.15.45:0/3276063454 wait complete. 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 -- start start 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.315+0000 7f909698c740 1 Processor -- start 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c8cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c8cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c8cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c8cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f9091a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c8cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55980/0 (socket says 172.21.15.45:55980) 2024-03-12T05:20:46.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c8cc490 con 0x55aa3c86d7b0 2024-03-12T05:20:46.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c8cc600 con 0x55aa3c8642b0 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909127a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c8cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa3c8cc7c0 con 0x55aa3c864bd0 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f9091a7b640 1 -- 172.21.15.45:0/2962206457 learned_addr learned my addr 172.21.15.45:0/2962206457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909698c740 10 client.0 _ll_register_callbacks cb 0x7f907c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.316+0000 7f909227c640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c8cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 -- 172.21.15.45:0/2962206457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 msgr2=0x55aa3c8cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c8cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 -- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 msgr2=0x55aa3c8cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:46.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c8cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 -- 172.21.15.45:0/2962206457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c1aac60 con 0x55aa3c86d7b0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f909227c640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c8cbed0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9084009780 tx=0x7f908402c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a5407691c513dc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.317+0000 7f90757fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084032800 con 0x55aa3c86d7b0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9084004490 con 0x55aa3c86d7b0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f907c009e10 con 0x55aa3c86d7b0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9084032960 con 0x55aa3c86d7b0 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f909698c740 10 client.17949 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:46.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.318+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f908c068ba0 con 0x55aa3c86d7b0 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9084057070 con 0x55aa3c86d7b0 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9084003c10 con 0x55aa3c86d7b0 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 10 client.17949 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 20 client.17949 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c8d4d60 0x55aa3c8d7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 4 client.17949 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.319+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f907c009e10 con 0x55aa3c8d4d60 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.320+0000 7f9091a7b640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c8d4d60 0x55aa3c8d7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.320+0000 7f90757fa640 10 client.17949 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.320+0000 7f9091a7b640 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c8d4d60 0x55aa3c8d7150 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a81d01f9707ecb0 server_cookie=a17b4fcb477ca482 in_seq=0 out_seq=0 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.320+0000 7f90757fa640 1 -- 172.21.15.45:0/2962206457 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f907c009e10 con 0x55aa3c8d4d60 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.320+0000 7f90757fa640 10 client.17949 handle_command_reply: tid=0 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 client.17949 shutdown 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 msgr2=0x55aa3c8cbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c8cbed0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9084009780 tx=0x7f908402c740 comp rx=0 tx=0).stop 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 shutdown_connections 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55aa3c8d4d60 0x55aa3c8d7150 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55aa3c86d7b0 0x55aa3c8cbed0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aa3c864bd0 0x55aa3c8cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f90757fa640 0 client.17949 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 --2- 172.21.15.45:0/2962206457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa3c8642b0 0x55aa3c8cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 >> 172.21.15.45:0/2962206457 conn(0x55aa3c85e990 msgr2=0x55aa3c865850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 shutdown_connections 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 1 -- 172.21.15.45:0/2962206457 wait complete. 2024-03-12T05:20:46.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.321+0000 7f909698c740 20 client.17949 trim_cache size 0 max 16384 2024-03-12T05:20:46.322 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:46.322 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:20:46.322 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:20:46.322 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 msgr2=0x7f90780a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 0x7f90780a41a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9084009830 tx=0x7f908402c090 comp rx=0 tx=0).stop 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 msgr2=0x7f908c112e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f908800c420 tx=0x7f9088009f90 comp rx=0 tx=0).stop 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 shutdown_connections 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f90780a1ce0 0x7f90780a41a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f908c110490 0x7f908c110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f908c105340 0x7f908c10ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 --2- 172.21.15.45:0/659581578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f908c100270 0x7f908c112e60 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f908800c420 tx=0x7f9088009f90 comp rx=0 tx=0).stop 2024-03-12T05:20:46.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.323+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 >> 172.21.15.45:0/659581578 conn(0x7f908c077230 msgr2=0x7f908c103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.324+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 shutdown_connections 2024-03-12T05:20:46.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.325+0000 7f9093cdf640 1 -- 172.21.15.45:0/659581578 wait complete. 2024-03-12T05:20:46.340 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:20:46.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:20:46.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 -- 172.21.15.45:0/3294398834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070103f50 msgr2=0x7ff070107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 --2- 172.21.15.45:0/3294398834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070103f50 0x7ff070107f90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff064009860 tx=0x7ff06402bb70 comp rx=0 tx=0).stop 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 -- 172.21.15.45:0/3294398834 shutdown_connections 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 --2- 172.21.15.45:0/3294398834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff070110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 --2- 172.21.15.45:0/3294398834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070103f50 0x7ff070107f90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 --2- 172.21.15.45:0/3294398834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff070103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 -- 172.21.15.45:0/3294398834 >> 172.21.15.45:0/3294398834 conn(0x7ff0700fedc0 msgr2=0x7ff0701011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 -- 172.21.15.45:0/3294398834 shutdown_connections 2024-03-12T05:20:46.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.549+0000 7ff075c0e640 1 -- 172.21.15.45:0/3294398834 wait complete. 2024-03-12T05:20:46.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 Processor -- start 2024-03-12T05:20:46.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 -- start start 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff07019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07019a250 con 0x7ff0701035a0 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07019a3c0 con 0x7ff070108670 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.550+0000 7ff075c0e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07019a580 con 0x7ff070199780 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55998/0 (socket says 172.21.15.45:55998) 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 -- 172.21.15.45:0/2136060757 learned_addr learned my addr 172.21.15.45:0/2136060757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:46.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45100/0 (socket says 172.21.15.45:45100) 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06f7fe640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff07019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 -- 172.21.15.45:0/2136060757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 msgr2=0x7ff070199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 -- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 msgr2=0x7ff07019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff07019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06effd640 1 -- 172.21.15.45:0/2136060757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009490 con 0x7ff070108670 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06ffff640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.551+0000 7ff06f7fe640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff07019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff06effd640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff064037040 tx=0x7ff0640346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06402c920 con 0x7ff070108670 2024-03-12T05:20:46.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff06402ce90 con 0x7ff070108670 2024-03-12T05:20:46.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06402c920 con 0x7ff070108670 2024-03-12T05:20:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0701a6470 con 0x7ff070108670 2024-03-12T05:20:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.552+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701a68a0 con 0x7ff070108670 2024-03-12T05:20:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.554+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070104a70 con 0x7ff070108670 2024-03-12T05:20:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.555+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff06403b040 con 0x7ff070108670 2024-03-12T05:20:46.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.556+0000 7ff06cff9640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 0x7ff0580a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:46.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.556+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff0640eeff0 con 0x7ff070108670 2024-03-12T05:20:46.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.556+0000 7ff06f7fe640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 0x7ff0580a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:46.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.558+0000 7ff06f7fe640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 0x7ff0580a40c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff060005fd0 tx=0x7ff060009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:46.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.561+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0640b7f80 con 0x7ff070108670 2024-03-12T05:20:46.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.978+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff07010b8a0 con 0x7ff070108670 2024-03-12T05:20:46.975 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:20:46.976 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:20:46.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.979+0000 7ff06cff9640 1 -- 172.21.15.45:0/2136060757 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7ff064041090 con 0x7ff070108670 2024-03-12T05:20:46.976 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:20:46.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 msgr2=0x7ff0580a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 0x7ff0580a40c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff060005fd0 tx=0x7ff060009260 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 msgr2=0x7ff07019fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff064037040 tx=0x7ff0640346f0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 shutdown_connections 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0580a1c00 0x7ff0580a40c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff070199780 0x7ff070199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff070108670 0x7ff07019fbb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 --2- 172.21.15.45:0/2136060757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0701035a0 0x7ff07019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 >> 172.21.15.45:0/2136060757 conn(0x7ff0700fedc0 msgr2=0x7ff070100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 shutdown_connections 2024-03-12T05:20:46.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:46.983+0000 7ff075c0e640 1 -- 172.21.15.45:0/2136060757 wait complete. 2024-03-12T05:20:46.999 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:46.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.169+0000 7f482c16c640 1 -- 172.21.15.45:0/3068289129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824108660 msgr2=0x7f4824110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.169+0000 7f482c16c640 1 --2- 172.21.15.45:0/3068289129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824108660 0x7f4824110230 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f481c009cd0 tx=0x7f481c02c0e0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.171+0000 7f482c16c640 1 -- 172.21.15.45:0/3068289129 shutdown_connections 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.171+0000 7f482c16c640 1 --2- 172.21.15.45:0/3068289129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103f40 0x7f4824107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.171+0000 7f482c16c640 1 --2- 172.21.15.45:0/3068289129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824108660 0x7f4824110230 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.171+0000 7f482c16c640 1 --2- 172.21.15.45:0/3068289129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824103590 0x7f4824103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.171+0000 7f482c16c640 1 -- 172.21.15.45:0/3068289129 >> 172.21.15.45:0/3068289129 conn(0x7f48240fedf0 msgr2=0x7f4824101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.172+0000 7f482c16c640 1 -- 172.21.15.45:0/3068289129 shutdown_connections 2024-03-12T05:20:47.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.172+0000 7f482c16c640 1 -- 172.21.15.45:0/3068289129 wait complete. 2024-03-12T05:20:47.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 Processor -- start 2024-03-12T05:20:47.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 -- start start 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103590 0x7f482419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824199780 0x7f4824199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482419a230 con 0x7f4824103590 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482419a3a0 con 0x7f4824199780 2024-03-12T05:20:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482c16c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482419a510 con 0x7f4824108660 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f4829707640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f482a709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824199780 0x7f4824199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f4829707640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45124/0 (socket says 172.21.15.45:45124) 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.173+0000 7f4829707640 1 -- 172.21.15.45:0/3995866413 learned_addr learned my addr 172.21.15.45:0/3995866413 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829f08640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103590 0x7f482419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 -- 172.21.15.45:0/3995866413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824199780 msgr2=0x7f4824199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824199780 0x7f4824199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 -- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103590 msgr2=0x7f482419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103590 0x7f482419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 -- 172.21.15.45:0/3995866413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c009900 con 0x7f4824108660 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f4829707640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f481400e9c0 tx=0x7f481400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.174+0000 7f481affd640 1 -- 172.21.15.45:0/3995866413 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814009700 con 0x7f4824108660 2024-03-12T05:20:47.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.175+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48241a64c0 con 0x7f4824108660 2024-03-12T05:20:47.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.175+0000 7f481affd640 1 -- 172.21.15.45:0/3995866413 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f481400cd80 con 0x7f4824108660 2024-03-12T05:20:47.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.175+0000 7f481affd640 1 -- 172.21.15.45:0/3995866413 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814018470 con 0x7f4824108660 2024-03-12T05:20:47.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.175+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48241a69d0 con 0x7f4824108660 2024-03-12T05:20:47.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.178+0000 7f482ee1c740 1 Processor -- start 2024-03-12T05:20:47.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 -- start start 2024-03-12T05:20:47.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97a10 0x55dc3fea8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3feab350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fe52960 con 0x55dc3fe97a10 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fe06310 con 0x55dc3fe97060 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482ee1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fe03840 con 0x55dc3fea8f60 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482a709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482a709640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56020/0 (socket says 172.21.15.45:56020) 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f482a709640 1 -- 172.21.15.45:0/4053941073 learned_addr learned my addr 172.21.15.45:0/4053941073 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.179+0000 7f4829707640 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3feab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f482a709640 1 -- 172.21.15.45:0/4053941073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 msgr2=0x55dc3feab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f482a709640 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3feab350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f482a709640 1 -- 172.21.15.45:0/4053941073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97a10 msgr2=0x55dc3fea8a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f482a709640 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97a10 0x55dc3fea8a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f482a709640 1 -- 172.21.15.45:0/4053941073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc3fdc1060 con 0x55dc3fe97060 2024-03-12T05:20:47.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.180+0000 7f481affd640 1 -- 172.21.15.45:0/3995866413 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f48140185d0 con 0x7f4824108660 2024-03-12T05:20:47.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.181+0000 7f481affd640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 0x7f48100a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.181+0000 7f481affd640 1 -- 172.21.15.45:0/3995866413 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4814014070 con 0x7f4824108660 2024-03-12T05:20:47.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.184+0000 7f4829f08640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 0x7f48100a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.184+0000 7f482a709640 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f481c009cd0 tx=0x7f481c005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27221db445105d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.184+0000 7f480ffff640 1 -- 172.21.15.45:0/4053941073 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c032e90 con 0x55dc3fe97060 2024-03-12T05:20:47.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.184+0000 7f480ffff640 1 -- 172.21.15.45:0/4053941073 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f481c034a60 con 0x55dc3fe97060 2024-03-12T05:20:47.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.185+0000 7f480ffff640 1 -- 172.21.15.45:0/4053941073 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c003bd0 con 0x55dc3fe97060 2024-03-12T05:20:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.185+0000 7f482ee1c740 1 -- 172.21.15.45:0/4053941073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 msgr2=0x55dc3fe97440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.185+0000 7f482ee1c740 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f481c009cd0 tx=0x7f481c005f80 comp rx=0 tx=0).stop 2024-03-12T05:20:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.185+0000 7f4829f08640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 0x7f48100a41a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55dc3fe92c10 tx=0x7f4820005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 -- 172.21.15.45:0/4053941073 shutdown_connections 2024-03-12T05:20:47.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97a10 0x55dc3fea8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3feab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 --2- 172.21.15.45:0/4053941073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97060 0x55dc3fe97440 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 -- 172.21.15.45:0/4053941073 >> 172.21.15.45:0/4053941073 conn(0x55dc3fe91980 msgr2=0x55dc3fe91d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 -- 172.21.15.45:0/4053941073 shutdown_connections 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 -- 172.21.15.45:0/4053941073 wait complete. 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 -- start start 2024-03-12T05:20:47.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.186+0000 7f482ee1c740 1 Processor -- start 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fe97060 0x55dc3fea5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fea8f60 0x55dc3fea6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fdff020 con 0x55dc3fea8f60 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fad2430 con 0x55dc3fe97a10 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fa8ee00 con 0x55dc3fe97060 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482ee1c740 10 client.0 _ll_register_callbacks cb 0x55dc3fda6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f4829707640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f4829707640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56030/0 (socket says 172.21.15.45:56030) 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f4829707640 1 -- 172.21.15.45:0/3390879136 learned_addr learned my addr 172.21.15.45:0/3390879136 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f482a709640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fea8f60 0x55dc3fea6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.187+0000 7f4829f08640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fe97060 0x55dc3fea5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 -- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fe97060 msgr2=0x55dc3fea5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fe97060 0x55dc3fea5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 -- 172.21.15.45:0/3390879136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fea8f60 msgr2=0x55dc3fea6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fea8f60 0x55dc3fea6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 -- 172.21.15.45:0/3390879136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4824110a40 con 0x55dc3fe97a10 2024-03-12T05:20:47.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f4829707640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f481400c6a0 tx=0x7f481409a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2828c53c7b53ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.188+0000 7f480d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48140cfea0 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48140b8e40 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc3fafe7e0 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48140cb670 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 10 client.27452 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dc3fdfd6a0 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55dc3fafe7e0 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48140850b0 con 0x55dc3fe97a10 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 10 client.27452 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 10 client.27452 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:20:47.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.189+0000 7f482ee1c740 20 client.27452 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.190+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3feb9410 0x55dc3febb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.190+0000 7f482ee1c740 4 client.27452 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:47.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.190+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc3fb3b720 con 0x55dc3feb9410 2024-03-12T05:20:47.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.190+0000 7f4829f08640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3feb9410 0x55dc3febb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.191+0000 7f480d7fa640 10 client.27452 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:47.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.191+0000 7f4829f08640 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3feb9410 0x55dc3febb830 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8452032ab755fd0e server_cookie=5df7def200bdaef0 in_seq=0 out_seq=0 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.191+0000 7f480d7fa640 1 -- 172.21.15.45:0/3390879136 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dc3fb3b720 con 0x55dc3feb9410 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.191+0000 7f480d7fa640 10 client.27452 handle_command_reply: tid=0 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 client.27452 shutdown 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 msgr2=0x55dc3fea6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f481400c6a0 tx=0x7f481409a210 comp rx=0 tx=0).stop 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 shutdown_connections 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3feb9410 0x55dc3febb830 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fea8f60 0x55dc3fea6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea6450 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3390879136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fe97060 0x55dc3fea5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f480d7fa640 0 client.27452 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:47.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 >> 172.21.15.45:0/3390879136 conn(0x55dc3fe91980 msgr2=0x55dc3fe936d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 shutdown_connections 2024-03-12T05:20:47.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.192+0000 7f482ee1c740 1 -- 172.21.15.45:0/3390879136 wait complete. 2024-03-12T05:20:47.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.193+0000 7f482ee1c740 20 client.27452 trim_cache size 0 max 16384 2024-03-12T05:20:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 Processor -- start 2024-03-12T05:20:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 -- start start 2024-03-12T05:20:47.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3fea1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3fea3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fd84130 con 0x55dc3fe97060 2024-03-12T05:20:47.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3feab350 con 0x55dc3fe97a10 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f4829707640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f4829707640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56042/0 (socket says 172.21.15.45:56042) 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f4829707640 1 -- 172.21.15.45:0/627229289 learned_addr learned my addr 172.21.15.45:0/627229289 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482ee1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3fea89e0 con 0x55dc3fea8f60 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.215+0000 7f482a709640 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3fea3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829f08640 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3fea1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 -- 172.21.15.45:0/627229289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 msgr2=0x55dc3fea3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3fea3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 -- 172.21.15.45:0/627229289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 msgr2=0x55dc3fea1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3fea1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 -- 172.21.15.45:0/627229289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc3fd99c20 con 0x55dc3fe97a10 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.216+0000 7f4829707640 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4814098980 tx=0x7f481409a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f812eca4e494a74b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.217+0000 7f480f7fe640 1 -- 172.21.15.45:0/627229289 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814099410 con 0x55dc3fe97a10 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.217+0000 7f480f7fe640 1 -- 172.21.15.45:0/627229289 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f481409a4a0 con 0x55dc3fe97a10 2024-03-12T05:20:47.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.217+0000 7f480f7fe640 1 -- 172.21.15.45:0/627229289 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481409a6d0 con 0x55dc3fe97a10 2024-03-12T05:20:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 -- 172.21.15.45:0/627229289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 msgr2=0x55dc3fea1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4814098980 tx=0x7f481409a340 comp rx=0 tx=0).stop 2024-03-12T05:20:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 -- 172.21.15.45:0/627229289 shutdown_connections 2024-03-12T05:20:47.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3fea3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3fea1780 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 --2- 172.21.15.45:0/627229289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3fea1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 -- 172.21.15.45:0/627229289 >> 172.21.15.45:0/627229289 conn(0x55dc3fe91980 msgr2=0x55dc3fe8ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 -- 172.21.15.45:0/627229289 shutdown_connections 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.218+0000 7f482ee1c740 1 -- 172.21.15.45:0/627229289 wait complete. 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.219+0000 7f482ee1c740 1 -- start start 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.219+0000 7f482ee1c740 1 Processor -- start 2024-03-12T05:20:47.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.219+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3feaef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f482ee1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3feaf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f4829f08640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3feaef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f4829f08640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3feaef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58454/0 (socket says 172.21.15.45:58454) 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f4829f08640 1 -- 172.21.15.45:0/3111603717 learned_addr learned my addr 172.21.15.45:0/3111603717 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3ff0f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f4829707640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3feaf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3feafbb0 con 0x55dc3fe97060 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3ff0fe60 con 0x55dc3fe97a10 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.220+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc3ff0fff0 con 0x55dc3fea8f60 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f482ee1c740 10 client.0 _ll_register_callbacks cb 0x7f4814005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 -- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 msgr2=0x55dc3ff0f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3ff0f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 -- 172.21.15.45:0/3111603717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 msgr2=0x55dc3feaef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3feaef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc3fdfd6a0 con 0x55dc3fe97a10 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f4829707640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3feaf470 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55dc3feb5760 tx=0x7f48140852c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a282dc1ff2cb99 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f480d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814096970 con 0x55dc3fe97a10 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.221+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48140b8e40 con 0x55dc3fe97a10 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.222+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc3fea5d40 con 0x55dc3fe97a10 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.222+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f481409e820 con 0x55dc3fe97a10 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.222+0000 7f482ee1c740 10 client.27458 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.222+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55dc3fdbf8c0 con 0x55dc3fe97a10 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48140a5020 con 0x55dc3fe97a10 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4814099590 con 0x55dc3fe97a10 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 10 client.27458 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 10 client.27458 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 20 client.27458 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3ff112c0 0x55dc3ff136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 4 client.27458 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.223+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dc3fea5d40 con 0x55dc3ff112c0 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.224+0000 7f4829f08640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3ff112c0 0x55dc3ff136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.225+0000 7f4829f08640 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3ff112c0 0x55dc3ff136b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd15c9cdee942267 server_cookie=1854ab4fdc23cfbc in_seq=0 out_seq=0 2024-03-12T05:20:47.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.225+0000 7f480d7fa640 10 client.27458 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f480d7fa640 1 -- 172.21.15.45:0/3111603717 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dc3fea5d40 con 0x55dc3ff112c0 2024-03-12T05:20:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f480d7fa640 10 client.27458 handle_command_reply: tid=0 2024-03-12T05:20:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 client.27458 shutdown 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 msgr2=0x55dc3feaf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3feaf470 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55dc3feb5760 tx=0x7f48140852c0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 shutdown_connections 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55dc3ff112c0 0x55dc3ff136b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc3fea8f60 0x55dc3ff0f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dc3fe97a10 0x55dc3feaf470 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 --2- 172.21.15.45:0/3111603717 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55dc3fe97060 0x55dc3feaef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.226+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 >> 172.21.15.45:0/3111603717 conn(0x55dc3fe91980 msgr2=0x55dc3fea1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.227+0000 7f480d7fa640 0 client.27458 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:20:47.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.227+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 shutdown_connections 2024-03-12T05:20:47.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.227+0000 7f482ee1c740 1 -- 172.21.15.45:0/3111603717 wait complete. 2024-03-12T05:20:47.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.227+0000 7f482ee1c740 20 client.27458 trim_cache size 0 max 16384 2024-03-12T05:20:47.225 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 msgr2=0x7f48100a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 0x7f48100a41a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55dc3fe92c10 tx=0x7f4820005f90 comp rx=0 tx=0).stop 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 msgr2=0x7f482419fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f481400e9c0 tx=0x7f481400ee80 comp rx=0 tx=0).stop 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 shutdown_connections 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48100a1ce0 0x7f48100a41a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4824199780 0x7f4824199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4824108660 0x7f482419fb50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 --2- 172.21.15.45:0/3995866413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4824103590 0x7f482419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.228+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 >> 172.21.15.45:0/3995866413 conn(0x7f48240fedf0 msgr2=0x7f4824100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.229+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 shutdown_connections 2024-03-12T05:20:47.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.229+0000 7f482c16c640 1 -- 172.21.15.45:0/3995866413 wait complete. 2024-03-12T05:20:47.244 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:47.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3393082004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba78109b20 msgr2=0x7fba78109f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3393082004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba78109b20 0x7fba78109f00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fba68009ca0 tx=0x7fba6802c190 comp rx=0 tx=0).stop 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3393082004 shutdown_connections 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3393082004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba7810a440 0x7fba78112b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3393082004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba781130d0 0x7fba781154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3393082004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba78109b20 0x7fba78109f00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.424+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3393082004 >> 172.21.15.45:0/3393082004 conn(0x7fba7806c030 msgr2=0x7fba7806c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.425+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3393082004 shutdown_connections 2024-03-12T05:20:47.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.425+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3393082004 wait complete. 2024-03-12T05:20:47.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.426+0000 7fba7ce7d640 1 Processor -- start 2024-03-12T05:20:47.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.426+0000 7fba7ce7d640 1 -- start start 2024-03-12T05:20:47.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba7ce7d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba7ce7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 0x7fba7810f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba7ce7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba781130d0 0x7fba7810f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43052/0 (socket says 172.21.15.45:43052) 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.427+0000 7fba76ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 0x7fba7810f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba76ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 0x7fba7810f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33458/0 (socket says 172.21.15.45:33458) 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 -- 172.21.15.45:0/3982843976 learned_addr learned my addr 172.21.15.45:0/3982843976 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba781b2450 con 0x7fba78109b20 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba781b2590 con 0x7fba781130d0 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba781b2700 con 0x7fba7810a440 2024-03-12T05:20:47.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba77fff640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba781130d0 0x7fba7810f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 -- 172.21.15.45:0/3982843976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 msgr2=0x7fba7810f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 0x7fba7810f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 -- 172.21.15.45:0/3982843976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba781130d0 msgr2=0x7fba7810f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba781130d0 0x7fba7810f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.428+0000 7fba777fe640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70009490 con 0x7fba78109b20 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba777fe640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fba68005e30 tx=0x7fba680349e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba74ff9640 1 -- 172.21.15.45:0/3982843976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68032680 con 0x7fba78109b20 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba74ff9640 1 -- 172.21.15.45:0/3982843976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba68032c90 con 0x7fba78109b20 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba74ff9640 1 -- 172.21.15.45:0/3982843976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68004720 con 0x7fba78109b20 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68009900 con 0x7fba78109b20 2024-03-12T05:20:47.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.429+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba781b2b60 con 0x7fba78109b20 2024-03-12T05:20:47.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba74ff9640 1 -- 172.21.15.45:0/3982843976 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fba6803b040 con 0x7fba78109b20 2024-03-12T05:20:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 Processor -- start 2024-03-12T05:20:47.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 -- start start 2024-03-12T05:20:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518460 0x55d449518840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d449523d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4494d3960 con 0x55d449521960 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449487310 con 0x55d449518d80 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.433+0000 7fba80d8c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449484840 con 0x55d449518460 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50308/0 (socket says 172.21.15.45:50308) 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 -- 172.21.15.45:0/1073445018 learned_addr learned my addr 172.21.15.45:0/1073445018 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba74ff9640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 0x7fba640a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba77fff640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518460 0x55d449518840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 -- 172.21.15.45:0/1073445018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518460 msgr2=0x55d449518840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518460 0x55d449518840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba76ffd640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d449523d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 -- 172.21.15.45:0/1073445018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 msgr2=0x55d449523d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d449523d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 -- 172.21.15.45:0/1073445018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d449442060 con 0x55d449518d80 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba76ffd640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d449523d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba77fff640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 0x7fba640a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba777fe640 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fba680ed7e0 tx=0x7fba68088fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a563c260db50483b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba74ff9640 1 -- 172.21.15.45:0/3982843976 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fba680f00a0 con 0x7fba78109b20 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.434+0000 7fba6dffb640 1 -- 172.21.15.45:0/1073445018 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68004880 con 0x55d449518d80 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.435+0000 7fba6dffb640 1 -- 172.21.15.45:0/1073445018 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba680894b0 con 0x55d449518d80 2024-03-12T05:20:47.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.435+0000 7fba6dffb640 1 -- 172.21.15.45:0/1073445018 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6808cf00 con 0x55d449518d80 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba77fff640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 0x7fba640a4230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba60005e00 tx=0x7fba600046c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 -- 172.21.15.45:0/1073445018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 msgr2=0x55d449521420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fba680ed7e0 tx=0x7fba68088fa0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 -- 172.21.15.45:0/1073445018 shutdown_connections 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d449523d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518d80 0x55d449521420 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1073445018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518460 0x55d449518840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 -- 172.21.15.45:0/1073445018 >> 172.21.15.45:0/1073445018 conn(0x55d449512b90 msgr2=0x55d449512f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 -- 172.21.15.45:0/1073445018 shutdown_connections 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.436+0000 7fba80d8c740 1 -- 172.21.15.45:0/1073445018 wait complete. 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 -- start start 2024-03-12T05:20:47.435 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 Processor -- start 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d44951f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449480020 con 0x55d449521960 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449153430 con 0x55d449518460 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44910fe00 con 0x55d449518d80 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.437+0000 7fba80d8c740 10 client.0 _ll_register_callbacks cb 0x55d449427cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50322/0 (socket says 172.21.15.45:50322) 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 -- 172.21.15.45:0/3876364591 learned_addr learned my addr 172.21.15.45:0/3876364591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba77fff640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d44951f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 -- 172.21.15.45:0/3876364591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 msgr2=0x55d44951ef60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.436 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 -- 172.21.15.45:0/3876364591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 msgr2=0x55d44951f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d44951f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 -- 172.21.15.45:0/3876364591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba781b2b60 con 0x55d449518460 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba77fff640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d44951f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba777fe640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba6808f090 tx=0x7fba680edc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a0b2166b94fb924 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.438+0000 7fba56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68091300 con 0x55d449518460 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d44917f7e0 con 0x55d449518460 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fba680914c0 con 0x55d449518460 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba680c54f0 con 0x55d449518460 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 10 client.27470 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d44947e6a0 con 0x55d449518460 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d44917f7e0 con 0x55d449518460 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fba68088c10 con 0x55d449518460 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 10 client.27470 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 10 client.27470 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 20 client.27470 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d449531710 0x55d449533b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 4 client.27470 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:47.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.439+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4491bc720 con 0x55d449531710 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.440+0000 7fba76ffd640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d449531710 0x55d449533b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.440+0000 7fba76ffd640 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d449531710 0x55d449533b30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fbdee0bc5b71407b server_cookie=a58fcd327194df1d in_seq=0 out_seq=0 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.440+0000 7fba56ffd640 10 client.27470 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.441+0000 7fba56ffd640 1 -- 172.21.15.45:0/3876364591 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d4491bc720 con 0x55d449531710 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.441+0000 7fba56ffd640 10 client.27470 handle_command_reply: tid=0 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.441+0000 7fba80d8c740 1 client.27470 shutdown 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 msgr2=0x55d44951ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba6808f090 tx=0x7fba680edc90 comp rx=0 tx=0).stop 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 shutdown_connections 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d449531710 0x55d449533b30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449521960 0x55d44951f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3876364591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d44951ea20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 >> 172.21.15.45:0/3876364591 conn(0x55d449512b90 msgr2=0x55d4495148e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba56ffd640 0 client.27470 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.442+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 shutdown_connections 2024-03-12T05:20:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.443+0000 7fba80d8c740 1 -- 172.21.15.45:0/3876364591 wait complete. 2024-03-12T05:20:47.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.443+0000 7fba80d8c740 20 client.27470 trim_cache size 0 max 16384 2024-03-12T05:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 Processor -- start 2024-03-12T05:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 -- start start 2024-03-12T05:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449521960 0x55d4495384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449517ce0 con 0x55d449518460 2024-03-12T05:20:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449405130 con 0x55d449521960 2024-03-12T05:20:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba80d8c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449513b90 con 0x55d449518d80 2024-03-12T05:20:47.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.465+0000 7fba76ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43074/0 (socket says 172.21.15.45:43074) 2024-03-12T05:20:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 -- 172.21.15.45:0/1771990641 learned_addr learned my addr 172.21.15.45:0/1771990641 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba77fff640 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449521960 0x55d4495384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 -- 172.21.15.45:0/1771990641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 msgr2=0x55d44951f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 -- 172.21.15.45:0/1771990641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449521960 msgr2=0x55d4495384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449521960 0x55d4495384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba777fe640 1 -- 172.21.15.45:0/1771990641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d44941ac20 con 0x55d449518460 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.466+0000 7fba76ffd640 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.467+0000 7fba777fe640 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fba68037040 tx=0x7fba680c52d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b345102ea9413e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.467+0000 7fba6d7fa640 1 -- 172.21.15.45:0/1771990641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68091300 con 0x55d449518460 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.467+0000 7fba6d7fa640 1 -- 172.21.15.45:0/1771990641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba680c54f0 con 0x55d449518460 2024-03-12T05:20:47.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.467+0000 7fba6d7fa640 1 -- 172.21.15.45:0/1771990641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba680db290 con 0x55d449518460 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 -- 172.21.15.45:0/1771990641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 msgr2=0x55d449521150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fba68037040 tx=0x7fba680c52d0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 -- 172.21.15.45:0/1771990641 shutdown_connections 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449521960 0x55d4495384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449518d80 0x55d44951f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 --2- 172.21.15.45:0/1771990641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518460 0x55d449521150 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 -- 172.21.15.45:0/1771990641 >> 172.21.15.45:0/1771990641 conn(0x55d449512b90 msgr2=0x55d449519de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.468+0000 7fba80d8c740 1 -- 172.21.15.45:0/1771990641 shutdown_connections 2024-03-12T05:20:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.469+0000 7fba80d8c740 1 -- 172.21.15.45:0/1771990641 wait complete. 2024-03-12T05:20:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.469+0000 7fba80d8c740 1 -- start start 2024-03-12T05:20:47.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.469+0000 7fba80d8c740 1 Processor -- start 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d449528600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518d80 0x55d449526bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449528eb0 con 0x55d449518d80 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449527650 con 0x55d449518460 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d449527810 con 0x55d449521960 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba80d8c740 10 client.0 _ll_register_callbacks cb 0x7fba680b8ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33482/0 (socket says 172.21.15.45:33482) 2024-03-12T05:20:47.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 -- 172.21.15.45:0/3620134579 learned_addr learned my addr 172.21.15.45:0/3620134579 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba777fe640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d449528600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba76ffd640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518d80 0x55d449526bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 -- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 msgr2=0x55d449528600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d449528600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 -- 172.21.15.45:0/3620134579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518d80 msgr2=0x55d449526bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518d80 0x55d449526bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba77fff640 1 -- 172.21.15.45:0/3620134579 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba70009dd0 con 0x55d449521960 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.470+0000 7fba777fe640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d449528600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba77fff640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba60005fd0 tx=0x7fba60004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29f9313ca63adf57 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6002c620 con 0x55d449521960 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fba6002c780 con 0x55d449521960 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba680894b0 con 0x55d449521960 2024-03-12T05:20:47.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba6002c940 con 0x55d449521960 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba80d8c740 10 client.27463 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.471+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d44947e6a0 con 0x55d449521960 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fba60045020 con 0x55d449521960 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fba60023c90 con 0x55d449521960 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 10 client.27463 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 10 client.27463 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 20 client.27463 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d4495890d0 0x55d44958b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 4 client.27463 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fba680894b0 con 0x55d4495890d0 2024-03-12T05:20:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.472+0000 7fba777fe640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d4495890d0 0x55d44958b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba777fe640 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d4495890d0 0x55d44958b4f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ad3252f2d122687 server_cookie=b85b5e958bd2a202 in_seq=0 out_seq=0 2024-03-12T05:20:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba56ffd640 10 client.27463 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba56ffd640 1 -- 172.21.15.45:0/3620134579 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fba680894b0 con 0x55d4495890d0 2024-03-12T05:20:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba56ffd640 10 client.27463 handle_command_reply: tid=0 2024-03-12T05:20:47.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba80d8c740 1 client.27463 shutdown 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 msgr2=0x55d449527110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.473+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fba60005fd0 tx=0x7fba60004730 comp rx=0 tx=0).stop 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 shutdown_connections 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d4495890d0 0x55d44958b4f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d449518d80 0x55d449526bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d449521960 0x55d449527110 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 --2- 172.21.15.45:0/3620134579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d449518460 0x55d449528600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 >> 172.21.15.45:0/3620134579 conn(0x55d449512b90 msgr2=0x55d4495384d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.476+0000 7fba56ffd640 0 client.27463 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:20:47.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.477+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 shutdown_connections 2024-03-12T05:20:47.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.480+0000 7fba80d8c740 1 -- 172.21.15.45:0/3620134579 wait complete. 2024-03-12T05:20:47.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.482+0000 7fba80d8c740 20 client.27463 trim_cache size 0 max 16384 2024-03-12T05:20:47.479 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 msgr2=0x7fba640a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 0x7fba640a4230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba60005e00 tx=0x7fba600046c0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 msgr2=0x7fba7810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fba68005e30 tx=0x7fba680349e0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 shutdown_connections 2024-03-12T05:20:47.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fba640a1d70 0x7fba640a4230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba781130d0 0x7fba7810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba7810a440 0x7fba7810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 --2- 172.21.15.45:0/3982843976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fba78109b20 0x7fba7810ee30 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 >> 172.21.15.45:0/3982843976 conn(0x7fba7806c030 msgr2=0x7fba7806fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.485+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 shutdown_connections 2024-03-12T05:20:47.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.486+0000 7fba7ce7d640 1 -- 172.21.15.45:0/3982843976 wait complete. 2024-03-12T05:20:47.501 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:47.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08c911640 1 -- 172.21.15.45:0/2994575757 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084005220 con 0x7ff0881083f0 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 -- 172.21.15.45:0/2994575757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0881083f0 msgr2=0x7ff0881087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 --2- 172.21.15.45:0/2994575757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0881083f0 0x7ff0881087d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff084009370 tx=0x7ff08402bc30 comp rx=0 tx=0).stop 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 -- 172.21.15.45:0/2994575757 shutdown_connections 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 --2- 172.21.15.45:0/2994575757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088108da0 0x7ff08810c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 --2- 172.21.15.45:0/2994575757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff08810cf00 0x7ff088114a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 --2- 172.21.15.45:0/2994575757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0881083f0 0x7ff0881087d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.682+0000 7ff08fb77640 1 -- 172.21.15.45:0/2994575757 >> 172.21.15.45:0/2994575757 conn(0x7ff08806c6c0 msgr2=0x7ff08806cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.683+0000 7ff08fb77640 1 -- 172.21.15.45:0/2994575757 shutdown_connections 2024-03-12T05:20:47.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.683+0000 7ff08fb77640 1 -- 172.21.15.45:0/2994575757 wait complete. 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.683+0000 7ff08fb77640 1 Processor -- start 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 -- start start 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 0x7ff08819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 0x7ff08819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08819f760 0x7ff08819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881b2ab0 con 0x7ff08819f760 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881b2c20 con 0x7ff0881083f0 2024-03-12T05:20:47.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08fb77640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881b2de0 con 0x7ff08810cf00 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08d112640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 0x7ff08819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 0x7ff08819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 0x7ff08819dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50346/0 (socket says 172.21.15.45:50346) 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.684+0000 7ff08d913640 1 -- 172.21.15.45:0/3566063996 learned_addr learned my addr 172.21.15.45:0/3566063996 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08e114640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08819f760 0x7ff08819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 -- 172.21.15.45:0/3566063996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 msgr2=0x7ff08819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 0x7ff08819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 -- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08819f760 msgr2=0x7ff08819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08819f760 0x7ff08819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 -- 172.21.15.45:0/3566063996 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084009df0 con 0x7ff08810cf00 2024-03-12T05:20:47.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.685+0000 7ff08d112640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 0x7ff08819e000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff08806a830 tx=0x7ff07800df70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.688+0000 7ff07effd640 1 -- 172.21.15.45:0/3566063996 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800adf0 con 0x7ff08810cf00 2024-03-12T05:20:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.688+0000 7ff07effd640 1 -- 172.21.15.45:0/3566063996 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0780054d0 con 0x7ff08810cf00 2024-03-12T05:20:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.688+0000 7ff07effd640 1 -- 172.21.15.45:0/3566063996 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07800e6b0 con 0x7ff08810cf00 2024-03-12T05:20:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.688+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0881b30c0 con 0x7ff08810cf00 2024-03-12T05:20:47.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.688+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0881b3580 con 0x7ff08810cf00 2024-03-12T05:20:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.691+0000 7ff07effd640 1 -- 172.21.15.45:0/3566063996 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff078004f20 con 0x7ff08810cf00 2024-03-12T05:20:47.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.691+0000 7ff092822740 1 Processor -- start 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 -- start start 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062250 0x5649b6062630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b606b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b601d960 con 0x5649b6062250 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5fd1310 con 0x5649b606b750 2024-03-12T05:20:47.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff092822740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5fce840 con 0x5649b6062b70 2024-03-12T05:20:47.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff08e114640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062250 0x5649b6062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff08d112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff08d112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50362/0 (socket says 172.21.15.45:50362) 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.692+0000 7ff08d112640 1 -- 172.21.15.45:0/3115196272 learned_addr learned my addr 172.21.15.45:0/3115196272 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d913640 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b606b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 -- 172.21.15.45:0/3115196272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 msgr2=0x5649b606b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b606b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 -- 172.21.15.45:0/3115196272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062250 msgr2=0x5649b6062630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062250 0x5649b6062630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 -- 172.21.15.45:0/3115196272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649b5f8c060 con 0x5649b606b750 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff08d112640 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff0780cad00 tx=0x7ff0780d3600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a055e80a4eb6bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.693+0000 7ff073fff640 1 -- 172.21.15.45:0/3115196272 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780cdb80 con 0x5649b606b750 2024-03-12T05:20:47.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.694+0000 7ff073fff640 1 -- 172.21.15.45:0/3115196272 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0780cdce0 con 0x5649b606b750 2024-03-12T05:20:47.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.694+0000 7ff073fff640 1 -- 172.21.15.45:0/3115196272 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780d9920 con 0x5649b606b750 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.694+0000 7ff092822740 1 -- 172.21.15.45:0/3115196272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 msgr2=0x5649b606db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.694+0000 7ff092822740 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff0780cad00 tx=0x7ff0780d3600 comp rx=0 tx=0).stop 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 -- 172.21.15.45:0/3115196272 shutdown_connections 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b606b750 0x5649b606db40 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b606b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 --2- 172.21.15.45:0/3115196272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062250 0x5649b6062630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 -- 172.21.15.45:0/3115196272 >> 172.21.15.45:0/3115196272 conn(0x5649b605c980 msgr2=0x5649b605cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 -- 172.21.15.45:0/3115196272 shutdown_connections 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff092822740 1 -- 172.21.15.45:0/3115196272 wait complete. 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.695+0000 7ff07effd640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 0x7ff0740a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.696+0000 7ff092822740 1 -- start start 2024-03-12T05:20:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.696+0000 7ff07effd640 1 -- 172.21.15.45:0/3566063996 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff0780cd5c0 con 0x7ff08810cf00 2024-03-12T05:20:47.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.696+0000 7ff092822740 1 Processor -- start 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.696+0000 7ff092822740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 0x5649b6068790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.696+0000 7ff092822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08e114640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 0x7ff0740a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff092822740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6069210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff092822740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5fca020 con 0x5649b606b750 2024-03-12T05:20:47.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff092822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5c9d430 con 0x5649b6062b70 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff092822740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5c59e00 con 0x5649b6062250 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 0x5649b6068790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50368/0 (socket says 172.21.15.45:50368) 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 0x5649b6068790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33526/0 (socket says 172.21.15.45:33526) 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 -- 172.21.15.45:0/535310011 learned_addr learned my addr 172.21.15.45:0/535310011 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff092822740 10 client.0 _ll_register_callbacks cb 0x5649b5f71cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08e114640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6069210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 -- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 msgr2=0x5649b6068790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 0x5649b6068790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 -- 172.21.15.45:0/535310011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 msgr2=0x5649b6069210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6069210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 -- 172.21.15.45:0/535310011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0881b3580 con 0x5649b6062b70 2024-03-12T05:20:47.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08d112640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff0780a24e0 tx=0x7ff078000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36cfed583eb9f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff0717fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.697+0000 7ff08e114640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 0x7ff0740a3ff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff080009fd0 tx=0x7ff080004410 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780cdb80 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff0780048c0 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649b5cc97e0 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff078005640 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 10 client.27488 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5649b5fc86a0 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff078096510 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0780caf00 con 0x5649b6062b70 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 10 client.27488 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 10 client.27488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 20 client.27488 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.698+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b607b550 0x5649b607d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.699+0000 7ff092822740 4 client.27488 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.699+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649b5cc97e0 con 0x5649b607b550 2024-03-12T05:20:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.699+0000 7ff08d913640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b607b550 0x5649b607d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.699+0000 7ff08d913640 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b607b550 0x5649b607d970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f2f169edc31b1aa server_cookie=f858cdebbb0f1b19 in_seq=0 out_seq=0 2024-03-12T05:20:47.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.699+0000 7ff0717fa640 10 client.27488 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.700+0000 7ff0717fa640 1 -- 172.21.15.45:0/535310011 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5649b5cc97e0 con 0x5649b607b550 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.700+0000 7ff0717fa640 10 client.27488 handle_command_reply: tid=0 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.701+0000 7ff092822740 1 client.27488 shutdown 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.701+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 msgr2=0x5649b6068cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.701+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff0780a24e0 tx=0x7ff078000bc0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 shutdown_connections 2024-03-12T05:20:47.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b607b550 0x5649b607d970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062b70 0x5649b6068cd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 --2- 172.21.15.45:0/535310011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062250 0x5649b6068790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 >> 172.21.15.45:0/535310011 conn(0x5649b605c980 msgr2=0x5649b605e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.702+0000 7ff0717fa640 0 client.27488 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.703+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 shutdown_connections 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.703+0000 7ff092822740 1 -- 172.21.15.45:0/535310011 wait complete. 2024-03-12T05:20:47.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.703+0000 7ff092822740 20 client.27488 trim_cache size 0 max 16384 2024-03-12T05:20:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.724+0000 7ff092822740 1 Processor -- start 2024-03-12T05:20:47.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 -- start start 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b6063a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6063fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b5f4f130 con 0x5649b606b750 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b60585c0 con 0x5649b6062250 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff092822740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b606e110 con 0x5649b6062b70 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50374/0 (socket says 172.21.15.45:50374) 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff08d913640 1 -- 172.21.15.45:0/3822152729 learned_addr learned my addr 172.21.15.45:0/3822152729 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff08d112640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b6063a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.725+0000 7ff08e114640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6063fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 -- 172.21.15.45:0/3822152729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 msgr2=0x5649b6063a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b6063a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 -- 172.21.15.45:0/3822152729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 msgr2=0x5649b6063fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6063fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 -- 172.21.15.45:0/3822152729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649b5f64c20 con 0x5649b6062250 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d112640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b6063a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff08d913640 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff084000c00 tx=0x7ff084009a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee52d916c90d4c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff0737fe640 1 -- 172.21.15.45:0/3822152729 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084045040 con 0x5649b6062250 2024-03-12T05:20:47.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff0737fe640 1 -- 172.21.15.45:0/3822152729 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff0840044f0 con 0x5649b6062250 2024-03-12T05:20:47.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.726+0000 7ff0737fe640 1 -- 172.21.15.45:0/3822152729 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084003c80 con 0x5649b6062250 2024-03-12T05:20:47.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.727+0000 7ff092822740 1 -- 172.21.15.45:0/3822152729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 msgr2=0x5649b606aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.727+0000 7ff092822740 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff084000c00 tx=0x7ff084009a70 comp rx=0 tx=0).stop 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 -- 172.21.15.45:0/3822152729 shutdown_connections 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b606b750 0x5649b6063fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b6062b70 0x5649b6063a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 --2- 172.21.15.45:0/3822152729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b606aec0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 -- 172.21.15.45:0/3822152729 >> 172.21.15.45:0/3822152729 conn(0x5649b605c980 msgr2=0x5649b6057a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 -- 172.21.15.45:0/3822152729 shutdown_connections 2024-03-12T05:20:47.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.728+0000 7ff092822740 1 -- 172.21.15.45:0/3822152729 wait complete. 2024-03-12T05:20:47.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.729+0000 7ff092822740 1 -- start start 2024-03-12T05:20:47.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.729+0000 7ff092822740 1 Processor -- start 2024-03-12T05:20:47.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.730+0000 7ff092822740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.730+0000 7ff092822740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062b70 0x5649b6070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff08d913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50380/0 (socket says 172.21.15.45:50380) 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff08d913640 1 -- 172.21.15.45:0/570467304 learned_addr learned my addr 172.21.15.45:0/570467304 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff08d112640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062b70 0x5649b6070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b606b750 0x5649b6071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b6071660 con 0x5649b6062b70 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b60717d0 con 0x5649b6062250 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649b6071990 con 0x5649b606b750 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff08e114640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b606b750 0x5649b6071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.731+0000 7ff092822740 10 client.0 _ll_register_callbacks cb 0x7ff0780d0f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 -- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b606b750 msgr2=0x5649b6071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b606b750 0x5649b6071120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 -- 172.21.15.45:0/570467304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062b70 msgr2=0x5649b6070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062b70 0x5649b6070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649b5fc86a0 con 0x5649b6062250 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff08d913640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff08402bc70 tx=0x7ff08402ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a675c5f9625c5ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff0717fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084045050 con 0x5649b6062250 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff084004310 con 0x5649b6062250 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.732+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649b60685c0 con 0x5649b6062250 2024-03-12T05:20:47.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.733+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0840048c0 con 0x5649b6062250 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.733+0000 7ff092822740 10 client.27494 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.733+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7ff088115290 con 0x5649b6062250 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.733+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5649b60685c0 con 0x5649b6062250 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff084052030 con 0x5649b6062250 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 10 client.27494 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 10 client.27494 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:20:47.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 20 client.27494 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b60d2c90 0x5649b60d5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 4 client.27494 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7ff0780cdce0 con 0x5649b60d2c90 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.734+0000 7ff08d112640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b60d2c90 0x5649b60d5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.735+0000 7ff0717fa640 10 client.27494 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.735+0000 7ff08d112640 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b60d2c90 0x5649b60d5080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46e60840b4b8b377 server_cookie=30ee4c7d96070a44 in_seq=0 out_seq=0 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.735+0000 7ff0717fa640 1 -- 172.21.15.45:0/570467304 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7ff0780cdce0 con 0x5649b60d2c90 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.735+0000 7ff0717fa640 10 client.27494 handle_command_reply: tid=0 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.735+0000 7ff092822740 1 client.27494 shutdown 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 msgr2=0x5649b60727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff08402bc70 tx=0x7ff08402ca80 comp rx=0 tx=0).stop 2024-03-12T05:20:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 shutdown_connections 2024-03-12T05:20:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5649b60d2c90 0x5649b60d5080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5649b6062b70 0x5649b6070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff0717fa640 0 client.27494 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5649b606b750 0x5649b6071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 --2- 172.21.15.45:0/570467304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b6062250 0x5649b60727b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 >> 172.21.15.45:0/570467304 conn(0x5649b605c980 msgr2=0x5649b6063a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 shutdown_connections 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 1 -- 172.21.15.45:0/570467304 wait complete. 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.736+0000 7ff092822740 20 client.27494 trim_cache size 0 max 16384 2024-03-12T05:20:47.737 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:47.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.737+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 msgr2=0x7ff0740a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.737+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 0x7ff0740a3ff0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff080009fd0 tx=0x7ff080004410 comp rx=0 tx=0).stop 2024-03-12T05:20:47.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.737+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 msgr2=0x7ff08819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.737+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 0x7ff08819e000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff08806a830 tx=0x7ff07800df70 comp rx=0 tx=0).stop 2024-03-12T05:20:47.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 shutdown_connections 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff0740a1b30 0x7ff0740a3ff0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08819f760 0x7ff08819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff08810cf00 0x7ff08819e000 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 --2- 172.21.15.45:0/3566063996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0881083f0 0x7ff08819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 >> 172.21.15.45:0/3566063996 conn(0x7ff08806c6c0 msgr2=0x7ff088071000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 shutdown_connections 2024-03-12T05:20:47.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.738+0000 7ff08fb77640 1 -- 172.21.15.45:0/3566063996 wait complete. 2024-03-12T05:20:47.752 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:47.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:20:47.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:47 smithi045 ceph-mon[32174]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:20:47.790 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:47 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2136060757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.931+0000 7faef09c7640 1 -- 172.21.15.45:0/3571093175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 msgr2=0x7faeec1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.931+0000 7faef09c7640 1 --2- 172.21.15.45:0/3571093175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1087d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faee4009cd0 tx=0x7faee402c0e0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 -- 172.21.15.45:0/3571093175 shutdown_connections 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 --2- 172.21.15.45:0/3571093175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec10cd10 0x7faeec10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 --2- 172.21.15.45:0/3571093175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec108d10 0x7faeec10c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 --2- 172.21.15.45:0/3571093175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1087d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 -- 172.21.15.45:0/3571093175 >> 172.21.15.45:0/3571093175 conn(0x7faeec06c6c0 msgr2=0x7faeec06cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 -- 172.21.15.45:0/3571093175 shutdown_connections 2024-03-12T05:20:47.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.932+0000 7faef09c7640 1 -- 172.21.15.45:0/3571093175 wait complete. 2024-03-12T05:20:47.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 Processor -- start 2024-03-12T05:20:47.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 -- start start 2024-03-12T05:20:47.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 0x7faeec1a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec1a70f0 con 0x7faeec108d10 2024-03-12T05:20:47.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec1a7260 con 0x7faeec10cd10 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faef09c7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec1a7420 con 0x7faeec1083f0 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faee9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faee9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43132/0 (socket says 172.21.15.45:43132) 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.933+0000 7faee9d9b640 1 -- 172.21.15.45:0/2184195553 learned_addr learned my addr 172.21.15.45:0/2184195553 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faeea59c640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faeead9d640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 0x7faeec1a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 -- 172.21.15.45:0/2184195553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 msgr2=0x7faeec1a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 -- 172.21.15.45:0/2184195553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 msgr2=0x7faeec1a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 0x7faeec1a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 -- 172.21.15.45:0/2184195553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee4009900 con 0x7faeec108d10 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faeea59c640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faeead9d640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 0x7faeec1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:47.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.934+0000 7faee9d9b640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7faee000bf60 tx=0x7faee000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.935+0000 7faedb7fe640 1 -- 172.21.15.45:0/2184195553 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0039d60 con 0x7faeec108d10 2024-03-12T05:20:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.935+0000 7faedb7fe640 1 -- 172.21.15.45:0/2184195553 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faee0009e40 con 0x7faeec108d10 2024-03-12T05:20:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.935+0000 7faedb7fe640 1 -- 172.21.15.45:0/2184195553 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0041960 con 0x7faeec108d10 2024-03-12T05:20:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.935+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec1a7700 con 0x7faeec108d10 2024-03-12T05:20:47.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.935+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec116380 con 0x7faeec108d10 2024-03-12T05:20:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.938+0000 7faef3597740 1 Processor -- start 2024-03-12T05:20:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 -- start start 2024-03-12T05:20:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac17210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac17750 0x559d3ac19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3abc9960 con 0x559d3ac0e250 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ab7d310 con 0x559d3ac0eb70 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faef3597740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ab7a840 con 0x559d3ac17750 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faedb7fe640 1 -- 172.21.15.45:0/2184195553 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7faee0041ac0 con 0x7faeec108d10 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43146/0 (socket says 172.21.15.45:43146) 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faeead9d640 1 -- 172.21.15.45:0/2680925897 learned_addr learned my addr 172.21.15.45:0/2680925897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.939+0000 7faeea59c640 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac17210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 -- 172.21.15.45:0/2680925897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac17750 msgr2=0x559d3ac19b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac17750 0x559d3ac19b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 -- 172.21.15.45:0/2680925897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 msgr2=0x559d3ac0e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 -- 172.21.15.45:0/2680925897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d3ab38060 con 0x559d3ac0eb70 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeead9d640 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faeea59c640 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac17210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faee4009bf0 tx=0x7faee4032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570ae8dabe50c030 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faed8ff9640 1 -- 172.21.15.45:0/2680925897 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4034440 con 0x559d3ac0eb70 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faed8ff9640 1 -- 172.21.15.45:0/2680925897 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faee40345a0 con 0x559d3ac0eb70 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.940+0000 7faed8ff9640 1 -- 172.21.15.45:0/2680925897 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4003970 con 0x559d3ac0eb70 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.941+0000 7faedb7fe640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 0x7faed40b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.941+0000 7faedb7fe640 1 -- 172.21.15.45:0/2184195553 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faee0033030 con 0x7faeec108d10 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.942+0000 7faeead9d640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 0x7faed40b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.942+0000 7faef3597740 1 -- 172.21.15.45:0/2680925897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 msgr2=0x559d3ac17210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.942+0000 7faef3597740 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac17210 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faee4009bf0 tx=0x7faee4032340 comp rx=0 tx=0).stop 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 -- 172.21.15.45:0/2680925897 shutdown_connections 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac17750 0x559d3ac19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac17210 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 --2- 172.21.15.45:0/2680925897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac0e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 -- 172.21.15.45:0/2680925897 >> 172.21.15.45:0/2680925897 conn(0x559d3ac08980 msgr2=0x559d3ac08d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 -- 172.21.15.45:0/2680925897 shutdown_connections 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faeead9d640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 0x7faed40b5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faedc005f60 tx=0x7faedc005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.943+0000 7faef3597740 1 -- 172.21.15.45:0/2680925897 wait complete. 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 -- start start 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 Processor -- start 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0e250 0x559d3ac14850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac14d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ab76020 con 0x559d3ac17750 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3a849430 con 0x559d3ac0e250 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.944+0000 7faef3597740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3a805e00 con 0x559d3ac0eb70 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faef3597740 10 client.0 _ll_register_callbacks cb 0x559d3ab1dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faee9d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac14d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43154/0 (socket says 172.21.15.45:43154) 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 -- 172.21.15.45:0/1872680739 learned_addr learned my addr 172.21.15.45:0/1872680739 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeea59c640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0e250 0x559d3ac14850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 -- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 msgr2=0x559d3ac14d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac14d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 -- 172.21.15.45:0/1872680739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0e250 msgr2=0x559d3ac14850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0e250 0x559d3ac14850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 -- 172.21.15.45:0/1872680739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec115f50 con 0x559d3ac17750 2024-03-12T05:20:47.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faeead9d640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7faedc0209e0 tx=0x7faedc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370279d4ab0ff24c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.945+0000 7faed1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faedc017070 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faedc011940 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d3a8757e0 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faedc011b00 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faef3597740 10 client.18015 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559d3ab746a0 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.946+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x559d3a8757e0 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faedc028ce0 con 0x559d3ac17750 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 10 client.18015 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 10 client.18015 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 20 client.18015 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.946 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac279c0 0x559d3ac29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 4 client.18015 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d3a8b2720 con 0x559d3ac279c0 2024-03-12T05:20:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.947+0000 7faeea59c640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac279c0 0x559d3ac29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.948+0000 7faed1ffb640 10 client.18015 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:47.947 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.948+0000 7faeea59c640 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac279c0 0x559d3ac29de0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13db2bdd64cd5c8c server_cookie=ac1b48ac3615c372 in_seq=0 out_seq=0 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.949+0000 7faed1ffb640 1 -- 172.21.15.45:0/1872680739 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d3a8b2720 con 0x559d3ac279c0 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.949+0000 7faed1ffb640 10 client.18015 handle_command_reply: tid=0 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.949+0000 7faef3597740 1 client.18015 shutdown 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.949+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 msgr2=0x559d3ac152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.949+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7faedc0209e0 tx=0x7faedc004730 comp rx=0 tx=0).stop 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 shutdown_connections 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac152d0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac279c0 0x559d3ac29de0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac14d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 --2- 172.21.15.45:0/1872680739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0e250 0x559d3ac14850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 >> 172.21.15.45:0/1872680739 conn(0x559d3ac08980 msgr2=0x559d3ac0a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faed1ffb640 0 client.18015 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:47.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 shutdown_connections 2024-03-12T05:20:47.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 1 -- 172.21.15.45:0/1872680739 wait complete. 2024-03-12T05:20:47.950 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.950+0000 7faef3597740 20 client.18015 trim_cache size 0 max 16384 2024-03-12T05:20:47.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:47 smithi130 ceph-mon[35398]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:20:47.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:47 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2136060757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:47.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.972+0000 7faef3597740 1 Processor -- start 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.972+0000 7faef3597740 1 -- start start 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac16f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac17750 0x559d3ac0fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3aafb130 con 0x559d3ac0e250 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ac09980 con 0x559d3ac17750 2024-03-12T05:20:47.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faef3597740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ac087e0 con 0x559d3ac0eb70 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33578/0 (socket says 172.21.15.45:33578) 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 -- 172.21.15.45:0/1168955094 learned_addr learned my addr 172.21.15.45:0/1168955094 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faeead9d640 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac17750 0x559d3ac0fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faeea59c640 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac16f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 -- 172.21.15.45:0/1168955094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac17750 msgr2=0x559d3ac0fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac17750 0x559d3ac0fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 -- 172.21.15.45:0/1168955094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 msgr2=0x559d3ac16f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac16f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.973+0000 7faee9d9b640 1 -- 172.21.15.45:0/1168955094 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d3ab10c20 con 0x559d3ac0eb70 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.974+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faee00b8270 tx=0x7faee0041410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8af426a11a59e79d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.974+0000 7faed3fff640 1 -- 172.21.15.45:0/1168955094 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00bc590 con 0x559d3ac0eb70 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.974+0000 7faed3fff640 1 -- 172.21.15.45:0/1168955094 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faee0032020 con 0x559d3ac0eb70 2024-03-12T05:20:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.974+0000 7faed3fff640 1 -- 172.21.15.45:0/1168955094 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00dc1d0 con 0x559d3ac0eb70 2024-03-12T05:20:47.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 -- 172.21.15.45:0/1168955094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 msgr2=0x559d3ac150f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faee00b8270 tx=0x7faee0041410 comp rx=0 tx=0).stop 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 -- 172.21.15.45:0/1168955094 shutdown_connections 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac17750 0x559d3ac0fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0eb70 0x559d3ac150f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 --2- 172.21.15.45:0/1168955094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac0e250 0x559d3ac16f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 -- 172.21.15.45:0/1168955094 >> 172.21.15.45:0/1168955094 conn(0x559d3ac08980 msgr2=0x559d3ac2e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 -- 172.21.15.45:0/1168955094 shutdown_connections 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.975+0000 7faef3597740 1 -- 172.21.15.45:0/1168955094 wait complete. 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faef3597740 1 -- start start 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faef3597740 1 Processor -- start 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faef3597740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 0x559d3ac1c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faef3597740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac1c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 0x559d3ac1c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 0x559d3ac1c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33582/0 (socket says 172.21.15.45:33582) 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.976+0000 7faeea59c640 1 -- 172.21.15.45:0/1884077880 learned_addr learned my addr 172.21.15.45:0/1884077880 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac1c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac1cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ac7cef0 con 0x559d3ac17750 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ac7d060 con 0x559d3ac0eb70 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d3ac7d220 con 0x559d3ac0e250 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faeead9d640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac1cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 -- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 msgr2=0x559d3ac1c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 0x559d3ac1c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 -- 172.21.15.45:0/1884077880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 msgr2=0x559d3ac1cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac1cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.977+0000 7faee9d9b640 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec116380 con 0x559d3ac0eb70 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faee9d9b640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac1c6d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faee0009fd0 tx=0x7faee00baaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66154d50e8ba7d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faef3597740 10 client.0 _ll_register_callbacks cb 0x7faee4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:47.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faee40345a0 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faed1ffb640 10 client.27512 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faed1ffb640 1 -- 172.21.15.45:0/1884077880 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee00d50d0 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faed1ffb640 1 -- 172.21.15.45:0/1884077880 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faee00bc590 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faed1ffb640 1 -- 172.21.15.45:0/1884077880 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faee00c06e0 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faef3597740 10 client.27512 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.978+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x559d3ab368c0 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faed1ffb640 1 -- 172.21.15.45:0/1884077880 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faee00b8c00 con 0x559d3ac0eb70 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 10 client.27512 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 10 client.27512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 20 client.27512 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac7e360 0x559d3ac80820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 4 client.27512 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d3a901890 con 0x559d3ac7e360 2024-03-12T05:20:47.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.979+0000 7faeea59c640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac7e360 0x559d3ac80820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.980+0000 7faeea59c640 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac7e360 0x559d3ac80820 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa2253375e854741 server_cookie=6fd24b1bef3ac50a in_seq=0 out_seq=0 2024-03-12T05:20:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.980+0000 7faed1ffb640 10 client.27512 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:47.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.980+0000 7faed1ffb640 1 -- 172.21.15.45:0/1884077880 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d3a901890 con 0x559d3ac7e360 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.980+0000 7faed1ffb640 10 client.27512 handle_command_reply: tid=0 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.980+0000 7faef3597740 1 client.27512 shutdown 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 msgr2=0x559d3ac1c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac1c6d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faee0009fd0 tx=0x7faee00baaa0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 shutdown_connections 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559d3ac17750 0x559d3ac1cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559d3ac7e360 0x559d3ac80820 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559d3ac0eb70 0x559d3ac1c6d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 --2- 172.21.15.45:0/1884077880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559d3ac0e250 0x559d3ac1c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 >> 172.21.15.45:0/1884077880 conn(0x559d3ac08980 msgr2=0x559d3ac0fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.982+0000 7faed1ffb640 0 client.27512 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.983+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 shutdown_connections 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.984+0000 7faef3597740 1 -- 172.21.15.45:0/1884077880 wait complete. 2024-03-12T05:20:47.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.984+0000 7faef3597740 20 client.27512 trim_cache size 0 max 16384 2024-03-12T05:20:47.982 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 msgr2=0x7faed40b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 0x7faed40b5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faedc005f60 tx=0x7faedc005f90 comp rx=0 tx=0).stop 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 msgr2=0x7faeec1a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7faee000bf60 tx=0x7faee000bf90 comp rx=0 tx=0).stop 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 shutdown_connections 2024-03-12T05:20:47.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.985+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faed40b2cb0 0x7faed40b5170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.986+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faeec108d10 0x7faeec1a2770 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.986+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeec10cd10 0x7faeec1a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.986+0000 7faef09c7640 1 --2- 172.21.15.45:0/2184195553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faeec1083f0 0x7faeec1a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.986+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 >> 172.21.15.45:0/2184195553 conn(0x7faeec06c6c0 msgr2=0x7faeec109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.987+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 shutdown_connections 2024-03-12T05:20:47.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:47.987+0000 7faef09c7640 1 -- 172.21.15.45:0/2184195553 wait complete. 2024-03-12T05:20:47.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:47 smithi116 ceph-mon[35486]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.4 MiB/s wr, 1.09k op/s 2024-03-12T05:20:47.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:47 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2136060757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:20:48.002 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:20:48.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:20:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 -- 172.21.15.45:0/198681749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108c80 msgr2=0x7f11d0116180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 --2- 172.21.15.45:0/198681749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108c80 0x7f11d0116180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009930 tx=0x7f11b802bc30 comp rx=0 tx=0).stop 2024-03-12T05:20:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 -- 172.21.15.45:0/198681749 shutdown_connections 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 --2- 172.21.15.45:0/198681749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d01166c0 0x7f11d0118ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 --2- 172.21.15.45:0/198681749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108c80 0x7f11d0116180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 --2- 172.21.15.45:0/198681749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108360 0x7f11d0108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 -- 172.21.15.45:0/198681749 >> 172.21.15.45:0/198681749 conn(0x7f11d006c260 msgr2=0x7f11d006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 -- 172.21.15.45:0/198681749 shutdown_connections 2024-03-12T05:20:48.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.224+0000 7f11d5548640 1 -- 172.21.15.45:0/198681749 wait complete. 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.225+0000 7f11d5548640 1 Processor -- start 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.225+0000 7f11d5548640 1 -- start start 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.225+0000 7f11d5548640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11d5548640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108c80 0x7f11d010c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11d5548640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d01166c0 0x7f11d010c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11d5548640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d01b2810 con 0x7f11d01166c0 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11d5548640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d01a1f20 con 0x7f11d0108360 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11d5548640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d01a20e0 con 0x7f11d0108c80 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50448/0 (socket says 172.21.15.45:50448) 2024-03-12T05:20:48.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 -- 172.21.15.45:0/2982732837 learned_addr learned my addr 172.21.15.45:0/2982732837 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:48.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11c7fff640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108c80 0x7f11d010c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 -- 172.21.15.45:0/2982732837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108c80 msgr2=0x7f11d010c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108c80 0x7f11d010c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 -- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d01166c0 msgr2=0x7f11d010c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d01166c0 0x7f11d010c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.226+0000 7f11ceffd640 1 -- 172.21.15.45:0/2982732837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b8009560 con 0x7f11d0108360 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11ceffd640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f11c000e9f0 tx=0x7f11c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11ccff9640 1 -- 172.21.15.45:0/2982732837 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0009720 con 0x7f11d0108360 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d01a23c0 con 0x7f11d0108360 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11ccff9640 1 -- 172.21.15.45:0/2982732837 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f11c000cd80 con 0x7f11d0108360 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11ccff9640 1 -- 172.21.15.45:0/2982732837 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0010430 con 0x7f11d0108360 2024-03-12T05:20:48.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.227+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d01a28a0 con 0x7f11d0108360 2024-03-12T05:20:48.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.231+0000 7f11d81f7740 1 Processor -- start 2024-03-12T05:20:48.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.232+0000 7f11ccff9640 1 -- 172.21.15.45:0/2982732837 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f11c0004fa0 con 0x7f11d0108360 2024-03-12T05:20:48.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.232+0000 7f11ccff9640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 0x7f11bc0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.233+0000 7f11ccff9640 1 -- 172.21.15.45:0/2982732837 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f11c0014070 con 0x7f11d0108360 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.233+0000 7f11d81f7740 1 -- start start 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951468210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895141a960 con 0x55895145fb70 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 0x7f11bc0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589513ce310 con 0x55895145f250 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11d81f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589513cb840 con 0x558951468750 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50454/0 (socket says 172.21.15.45:50454) 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.234+0000 7f11ceffd640 1 -- 172.21.15.45:0/357477943 learned_addr learned my addr 172.21.15.45:0/357477943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11c7fff640 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951468210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 -- 172.21.15.45:0/357477943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 msgr2=0x55895146ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 -- 172.21.15.45:0/357477943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 msgr2=0x558951468210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951468210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 -- 172.21.15.45:0/357477943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558951389060 con 0x55895145f250 2024-03-12T05:20:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11c7fff640 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951468210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:48.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11ceffd640 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f11c00971a0 tx=0x7f11c009a3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=156fa72ad4d1ab12 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11c57fa640 1 -- 172.21.15.45:0/357477943 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c009a7d0 con 0x55895145f250 2024-03-12T05:20:48.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11c57fa640 1 -- 172.21.15.45:0/357477943 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f11c009e920 con 0x55895145f250 2024-03-12T05:20:48.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.235+0000 7f11c57fa640 1 -- 172.21.15.45:0/357477943 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0097000 con 0x55895145f250 2024-03-12T05:20:48.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.236+0000 7f11d81f7740 1 -- 172.21.15.45:0/357477943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 msgr2=0x55895145f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.236+0000 7f11d81f7740 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f11c00971a0 tx=0x7f11c009a3d0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 0x7f11bc0a4160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f11c80092f0 tx=0x7f11c800e6c0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11d81f7740 1 -- 172.21.15.45:0/357477943 shutdown_connections 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11d81f7740 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951468210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11d81f7740 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11d81f7740 1 --2- 172.21.15.45:0/357477943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x55895145f630 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.238+0000 7f11d81f7740 1 -- 172.21.15.45:0/357477943 >> 172.21.15.45:0/357477943 conn(0x558951459980 msgr2=0x558951459d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 -- 172.21.15.45:0/357477943 shutdown_connections 2024-03-12T05:20:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 -- 172.21.15.45:0/357477943 wait complete. 2024-03-12T05:20:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 -- start start 2024-03-12T05:20:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 Processor -- start 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 0x558951465cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.239+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x5589514661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11d81f7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589513c7020 con 0x55895145f250 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11d81f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895109a430 con 0x55895145fb70 2024-03-12T05:20:48.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11d81f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558951056e00 con 0x558951468750 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11d81f7740 10 client.0 _ll_register_callbacks cb 0x55895136ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43222/0 (socket says 172.21.15.45:43222) 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 -- 172.21.15.45:0/4140154570 learned_addr learned my addr 172.21.15.45:0/4140154570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11c7fff640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 0x558951465cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x5589514661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 -- 172.21.15.45:0/4140154570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 msgr2=0x5589514661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x5589514661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 -- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 msgr2=0x558951465cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 0x558951465cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 -- 172.21.15.45:0/4140154570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d01a28a0 con 0x55895145f250 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11c7fff640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 0x558951465cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.240+0000 7f11ceffd640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f11c009a4d0 tx=0x7f11c0005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c376c8864d1f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0097000 con 0x55895145f250 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f11c00a2f70 con 0x55895145f250 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5589510c67e0 con 0x55895145f250 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c009a7d0 con 0x55895145f250 2024-03-12T05:20:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11d81f7740 10 client.18030 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.241+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5589513c56a0 con 0x55895145f250 2024-03-12T05:20:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5589510c67e0 con 0x55895145f250 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f11c001f060 con 0x55895145f250 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 10 client.18030 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 10 client.18030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 20 client.18030 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558951478840 0x55895147ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 4 client.18030 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558951103720 con 0x558951478840 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.242+0000 7f11c7fff640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558951478840 0x55895147ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.243+0000 7f11b67fc640 10 client.18030 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:48.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.243+0000 7f11c7fff640 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558951478840 0x55895147ac30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5c5cb5ca2576e server_cookie=e222eea7be3c6d03 in_seq=0 out_seq=0 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.244+0000 7f11b67fc640 1 -- 172.21.15.45:0/4140154570 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558951103720 con 0x558951478840 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.244+0000 7f11b67fc640 10 client.18030 handle_command_reply: tid=0 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.244+0000 7f11d81f7740 1 client.18030 shutdown 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 msgr2=0x558951465770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f11c009a4d0 tx=0x7f11c0005660 comp rx=0 tx=0).stop 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 shutdown_connections 2024-03-12T05:20:48.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558951478840 0x55895147ac30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x5589514661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145fb70 0x558951465cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 --2- 172.21.15.45:0/4140154570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145f250 0x558951465770 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11b67fc640 0 client.18030 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 >> 172.21.15.45:0/4140154570 conn(0x558951459980 msgr2=0x55895145b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 shutdown_connections 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.245+0000 7f11d81f7740 1 -- 172.21.15.45:0/4140154570 wait complete. 2024-03-12T05:20:48.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.246+0000 7f11d81f7740 20 client.18030 trim_cache size 0 max 16384 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.268+0000 7f11d81f7740 1 Processor -- start 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.268+0000 7f11d81f7740 1 -- start start 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 0x558951466010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558951468750 0x55895146f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895134c130 con 0x55895145fb70 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895144ba70 con 0x558951468750 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11d81f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558951478640 con 0x55895145f250 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43232/0 (socket says 172.21.15.45:43232) 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 -- 172.21.15.45:0/1626174845 learned_addr learned my addr 172.21.15.45:0/1626174845 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558951468750 0x55895146f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11ceffd640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 0x558951466010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 -- 172.21.15.45:0/1626174845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 msgr2=0x558951466010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 0x558951466010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 -- 172.21.15.45:0/1626174845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558951468750 msgr2=0x55895146f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558951468750 0x55895146f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11c7fff640 1 -- 172.21.15.45:0/1626174845 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558951361c20 con 0x55895145fb70 2024-03-12T05:20:48.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.269+0000 7f11ceffd640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 0x558951466010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:20:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.270+0000 7f11c7fff640 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f11b8005ab0 tx=0x7f11b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc71817e0e16bd80 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.270+0000 7f11c4ff9640 1 -- 172.21.15.45:0/1626174845 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8005220 con 0x55895145fb70 2024-03-12T05:20:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.270+0000 7f11c4ff9640 1 -- 172.21.15.45:0/1626174845 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558951103720 con 0x55895145fb70 2024-03-12T05:20:48.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.270+0000 7f11c4ff9640 1 -- 172.21.15.45:0/1626174845 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8004950 con 0x55895145fb70 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 -- 172.21.15.45:0/1626174845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 msgr2=0x558951458a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f11b8005ab0 tx=0x7f11b8009260 comp rx=0 tx=0).stop 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 -- 172.21.15.45:0/1626174845 shutdown_connections 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x558951458a00 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558951468750 0x55895146f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 --2- 172.21.15.45:0/1626174845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55895145f250 0x558951466010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.271+0000 7f11d81f7740 1 -- 172.21.15.45:0/1626174845 >> 172.21.15.45:0/1626174845 conn(0x558951459980 msgr2=0x5589514582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.272+0000 7f11d81f7740 1 -- 172.21.15.45:0/1626174845 shutdown_connections 2024-03-12T05:20:48.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.272+0000 7f11d81f7740 1 -- 172.21.15.45:0/1626174845 wait complete. 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.272+0000 7f11d81f7740 1 -- start start 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.272+0000 7f11d81f7740 1 Processor -- start 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x5589514c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x5589514c8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x5589514c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x5589514c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x5589514c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50476/0 (socket says 172.21.15.45:50476) 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x5589514c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43238/0 (socket says 172.21.15.45:43238) 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895146f080 con 0x55895145fb70 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11ceffd640 1 -- 172.21.15.45:0/360887391 learned_addr learned my addr 172.21.15.45:0/360887391 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895146f1f0 con 0x55895145f250 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895146f3b0 con 0x558951468750 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.273+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11d81f7740 10 client.0 _ll_register_callbacks cb 0x7f11c0083f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 -- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 msgr2=0x5589514c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x5589514c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 -- 172.21.15.45:0/360887391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 msgr2=0x5589514c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x5589514c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d01192c0 con 0x558951468750 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c009e920 con 0x558951468750 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11cf7fe640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146e980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55895145f030 tx=0x7f11c8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c68897df4950047 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11b67fc640 10 client.27511 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.274+0000 7f11b67fc640 1 -- 172.21.15.45:0/360887391 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011ea0 con 0x558951468750 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.275+0000 7f11b67fc640 1 -- 172.21.15.45:0/360887391 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f11c8012b40 con 0x558951468750 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.275+0000 7f11b67fc640 1 -- 172.21.15.45:0/360887391 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c8012d20 con 0x558951468750 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.275+0000 7f11d81f7740 10 client.27511 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:48.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.275+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5589513878c0 con 0x558951468750 2024-03-12T05:20:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11b67fc640 1 -- 172.21.15.45:0/360887391 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f11c801e040 con 0x558951468750 2024-03-12T05:20:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 10 client.27511 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 10 client.27511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 20 client.27511 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5589514d0e80 0x5589514d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 4 client.27511 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558951152890 con 0x5589514d0e80 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11ceffd640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5589514d0e80 0x5589514d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11b67fc640 10 client.27511 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.276+0000 7f11ceffd640 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5589514d0e80 0x5589514d3270 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5ab832327b1883e server_cookie=69f24336986e062c in_seq=0 out_seq=0 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11b67fc640 1 -- 172.21.15.45:0/360887391 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558951152890 con 0x5589514d0e80 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11b67fc640 10 client.27511 handle_command_reply: tid=0 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11d81f7740 1 client.27511 shutdown 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 msgr2=0x55895146e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146e980 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55895145f030 tx=0x7f11c8004a40 comp rx=0 tx=0).stop 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.277+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 shutdown_connections 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5589514d0e80 0x5589514d3270 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55895145fb70 0x5589514c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558951468750 0x55895146e980 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 --2- 172.21.15.45:0/360887391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55895145f250 0x5589514c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 >> 172.21.15.45:0/360887391 conn(0x558951459980 msgr2=0x558951458b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11b67fc640 0 client.27511 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 shutdown_connections 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 1 -- 172.21.15.45:0/360887391 wait complete. 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.278+0000 7f11d81f7740 20 client.27511 trim_cache size 0 max 16384 2024-03-12T05:20:48.277 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:20:48.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 msgr2=0x7f11bc0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 0x7f11bc0a4160 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f11c80092f0 tx=0x7f11c800e6c0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 msgr2=0x7f11d010be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f11c000e9f0 tx=0x7f11c000edf0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 shutdown_connections 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11bc0a1ca0 0x7f11bc0a4160 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d01166c0 0x7f11d010c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d0108c80 0x7f11d010c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 --2- 172.21.15.45:0/2982732837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d0108360 0x7f11d010be00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 >> 172.21.15.45:0/2982732837 conn(0x7f11d006c260 msgr2=0x7f11d0116de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 shutdown_connections 2024-03-12T05:20:48.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:48.279+0000 7f11d5548640 1 -- 172.21.15.45:0/2982732837 wait complete. 2024-03-12T05:20:49.294 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:20:49.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:20:49.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:20:49.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.472+0000 7f08b4255640 1 -- 172.21.15.45:0/1280385043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac101d40 msgr2=0x7f08ac105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.472+0000 7f08b4255640 1 --2- 172.21.15.45:0/1280385043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac101d40 0x7f08ac105d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009890 tx=0x7f08a802bb70 comp rx=0 tx=0).stop 2024-03-12T05:20:49.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 -- 172.21.15.45:0/1280385043 shutdown_connections 2024-03-12T05:20:49.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 --2- 172.21.15.45:0/1280385043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac106460 0x7f08ac10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 --2- 172.21.15.45:0/1280385043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac101d40 0x7f08ac105d80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 --2- 172.21.15.45:0/1280385043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 -- 172.21.15.45:0/1280385043 >> 172.21.15.45:0/1280385043 conn(0x7f08ac0fcc10 msgr2=0x7f08ac0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 -- 172.21.15.45:0/1280385043 shutdown_connections 2024-03-12T05:20:49.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.473+0000 7f08b4255640 1 -- 172.21.15.45:0/1280385043 wait complete. 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 Processor -- start 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 -- start start 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac106460 0x7f08ac10afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac1a38f0 0x7f08ac1a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac1a42e0 con 0x7f08ac1a38f0 2024-03-12T05:20:49.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac1a4450 con 0x7f08ac106460 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b4255640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac1a4610 con 0x7f08ac101390 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b1ff1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac1a38f0 0x7f08ac1a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.474+0000 7f08b17f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac106460 0x7f08ac10afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33638/0 (socket says 172.21.15.45:33638) 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 -- 172.21.15.45:0/3089167753 learned_addr learned my addr 172.21.15.45:0/3089167753 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 -- 172.21.15.45:0/3089167753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac106460 msgr2=0x7f08ac10afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac106460 0x7f08ac10afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 -- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac1a38f0 msgr2=0x7f08ac1a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac1a38f0 0x7f08ac1a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.475+0000 7f08b1ff1640 1 -- 172.21.15.45:0/3089167753 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8009490 con 0x7f08ac101390 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f08a000e9f0 tx=0x7f08a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f089effd640 1 -- 172.21.15.45:0/3089167753 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a0009700 con 0x7f08ac101390 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac1aa9a0 con 0x7f08ac101390 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f089effd640 1 -- 172.21.15.45:0/3089167753 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08a000cd80 con 0x7f08ac101390 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac1aae70 con 0x7f08ac101390 2024-03-12T05:20:49.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.476+0000 7f089effd640 1 -- 172.21.15.45:0/3089167753 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a0018840 con 0x7f08ac101390 2024-03-12T05:20:49.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 Processor -- start 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 -- start start 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a442e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a46d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d19a9f90 con 0x55d3d1a448b0 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d19a74c0 con 0x55d3d1a3b380 2024-03-12T05:20:49.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b6e22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d19a2ca0 con 0x55d3d1a3bd30 2024-03-12T05:20:49.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50508/0 (socket says 172.21.15.45:50508) 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.480+0000 7f08b27f2640 1 -- 172.21.15.45:0/411715830 learned_addr learned my addr 172.21.15.45:0/411715830 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b17f0640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a46d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f089effd640 1 -- 172.21.15.45:0/3089167753 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f08a00189a0 con 0x7f08ac101390 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b27f2640 1 -- 172.21.15.45:0/411715830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 msgr2=0x55d3d1a442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b27f2640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b27f2640 1 -- 172.21.15.45:0/411715830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 msgr2=0x55d3d1a46d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b27f2640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a46d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b27f2640 1 -- 172.21.15.45:0/411715830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3d19650b0 con 0x55d3d1a3b380 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.481+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:20:49.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f08b27f2640 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f08a400e480 tx=0x7f08a40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec16f4c9fde9e4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f0897fff640 1 -- 172.21.15.45:0/411715830 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a401b070 con 0x55d3d1a3b380 2024-03-12T05:20:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f0897fff640 1 -- 172.21.15.45:0/411715830 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08a4003e20 con 0x55d3d1a3b380 2024-03-12T05:20:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f0897fff640 1 -- 172.21.15.45:0/411715830 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4013040 con 0x55d3d1a3b380 2024-03-12T05:20:49.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f089effd640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 0x7f08980a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.482+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 0x7f08980a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.483+0000 7f089effd640 1 -- 172.21.15.45:0/3089167753 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f08a0014070 con 0x7f08ac101390 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.483+0000 7f08b6e22740 1 -- 172.21.15.45:0/411715830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 msgr2=0x55d3d1a3b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.483+0000 7f08b6e22740 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f08a400e480 tx=0x7f08a40095c0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.483+0000 7f08b1ff1640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 0x7f08980a4160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55d3d1a3ae50 tx=0x7f08a0096ed0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.484+0000 7f08b6e22740 1 -- 172.21.15.45:0/411715830 shutdown_connections 2024-03-12T05:20:49.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.484+0000 7f08b6e22740 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a46d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.484+0000 7f08b6e22740 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.484+0000 7f08b6e22740 1 --2- 172.21.15.45:0/411715830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a3b760 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.484+0000 7f08b6e22740 1 -- 172.21.15.45:0/411715830 >> 172.21.15.45:0/411715830 conn(0x55d3d1a35840 msgr2=0x55d3d1a35c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.485+0000 7f08b6e22740 1 -- 172.21.15.45:0/411715830 shutdown_connections 2024-03-12T05:20:49.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.485+0000 7f08b6e22740 1 -- 172.21.15.45:0/411715830 wait complete. 2024-03-12T05:20:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.486+0000 7f08b6e22740 1 -- start start 2024-03-12T05:20:49.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.486+0000 7f08b6e22740 1 Processor -- start 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.486+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a41870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.486+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b6e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1947f40 con 0x55d3d1a3b380 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b6e22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d16764b0 con 0x55d3d1a448b0 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b6e22740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1928670 con 0x55d3d1a3bd30 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b6e22740 10 client.0 _ll_register_callbacks cb 0x55d3d1995f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:49.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a41870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50524/0 (socket says 172.21.15.45:50524) 2024-03-12T05:20:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 -- 172.21.15.45:0/3053687853 learned_addr learned my addr 172.21.15.45:0/3053687853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 -- 172.21.15.45:0/3053687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 msgr2=0x55d3d1a41db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 -- 172.21.15.45:0/3053687853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 msgr2=0x55d3d1a41870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a41870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.487+0000 7f08b27f2640 1 -- 172.21.15.45:0/3053687853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a4009ce0 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08b27f2640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f08a400de90 tx=0x7f08a4010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b74aa2062c7ef295 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08957fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a401b070 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3d197cf40 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f08a4004e40 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08a4013070 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08b6e22740 10 client.27542 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.488+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d3d19a1320 con 0x55d3d1a448b0 2024-03-12T05:20:49.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08a4003b80 con 0x55d3d1a448b0 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f08a4005040 con 0x55d3d1a448b0 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 10 client.27542 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 10 client.27542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 20 client.27542 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1a65d80 0x55d3d1a681a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 4 client.27542 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3d197cf40 con 0x55d3d1a65d80 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.489+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1a65d80 0x55d3d1a681a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.490+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1a65d80 0x55d3d1a681a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=620c58eca91466ae server_cookie=d1520122f84ca451 in_seq=0 out_seq=0 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.490+0000 7f08957fa640 10 client.27542 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08957fa640 1 -- 172.21.15.45:0/3053687853 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d3d197cf40 con 0x55d3d1a65d80 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08957fa640 10 client.27542 handle_command_reply: tid=0 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 client.27542 shutdown 2024-03-12T05:20:49.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 msgr2=0x55d3d1a422f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f08a400de90 tx=0x7f08a4010040 comp rx=0 tx=0).stop 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 shutdown_connections 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1a65d80 0x55d3d1a681a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1a422f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3053687853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a41870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.491+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 >> 172.21.15.45:0/3053687853 conn(0x55d3d1a35840 msgr2=0x55d3d1a377a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.492+0000 7f08957fa640 0 client.27542 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.492+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 shutdown_connections 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.492+0000 7f08b6e22740 1 -- 172.21.15.45:0/3053687853 wait complete. 2024-03-12T05:20:49.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.492+0000 7f08b6e22740 20 client.27542 trim_cache size 0 max 16384 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.518+0000 7f08b6e22740 1 Processor -- start 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.518+0000 7f08b6e22740 1 -- start start 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a43fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a5c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1a61930 con 0x55d3d1a448b0 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a43fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b17f0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b17f0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33676/0 (socket says 172.21.15.45:33676) 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b17f0640 1 -- 172.21.15.45:0/3302644350 learned_addr learned my addr 172.21.15.45:0/3302644350 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1a3acd0 con 0x55d3d1a3b380 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.519+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1a31180 con 0x55d3d1a3bd30 2024-03-12T05:20:49.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b27f2640 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a5c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b17f0640 1 -- 172.21.15.45:0/3302644350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 msgr2=0x55d3d1a43fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a43fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b17f0640 1 -- 172.21.15.45:0/3302644350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 msgr2=0x55d3d1a5c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a5c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.520+0000 7f08b17f0640 1 -- 172.21.15.45:0/3302644350 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3d193dc00 con 0x55d3d1a3bd30 2024-03-12T05:20:49.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.521+0000 7f08b17f0640 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f08ac06a830 tx=0x7f08a802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c72f453ac43121f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.521+0000 7f08977fe640 1 -- 172.21.15.45:0/3302644350 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8044040 con 0x55d3d1a3bd30 2024-03-12T05:20:49.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.521+0000 7f08977fe640 1 -- 172.21.15.45:0/3302644350 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08a802ce90 con 0x55d3d1a3bd30 2024-03-12T05:20:49.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.522+0000 7f08977fe640 1 -- 172.21.15.45:0/3302644350 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a80462c0 con 0x55d3d1a3bd30 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.522+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 msgr2=0x55d3d1a41f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f08ac06a830 tx=0x7f08a802e040 comp rx=0 tx=0).stop 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 shutdown_connections 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a448b0 0x55d3d1a5c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a41f40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 --2- 172.21.15.45:0/3302644350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a3b380 0x55d3d1a43fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 >> 172.21.15.45:0/3302644350 conn(0x55d3d1a35840 msgr2=0x55d3d1a41830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 shutdown_connections 2024-03-12T05:20:49.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 -- 172.21.15.45:0/3302644350 wait complete. 2024-03-12T05:20:49.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.523+0000 7f08b6e22740 1 -- start start 2024-03-12T05:20:49.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.524+0000 7f08b6e22740 1 Processor -- start 2024-03-12T05:20:49.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.524+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a5c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.524+0000 7f08b6e22740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a5c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a5c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b27f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a5c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43296/0 (socket says 172.21.15.45:43296) 2024-03-12T05:20:49.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b27f2640 1 -- 172.21.15.45:0/2480203605 learned_addr learned my addr 172.21.15.45:0/2480203605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1ab57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1ab5cf0 con 0x55d3d1a3b380 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1ab5e60 con 0x55d3d1a448b0 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3d1ab6020 con 0x55d3d1a3bd30 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b17f0640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a5c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b6e22740 10 client.0 _ll_register_callbacks cb 0x7f08a4009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:49.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.525+0000 7f08b27f2640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1ab57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 -- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 msgr2=0x55d3d1ab57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1ab57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 -- 172.21.15.45:0/2480203605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 msgr2=0x55d3d1a5c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a5c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac10e840 con 0x55d3d1a3bd30 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08b17f0640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a5c8d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f08a8037040 tx=0x7f08a80348f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57a31360c80f9d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.526+0000 7f08957fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8004460 con 0x55d3d1a3bd30 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f08a8005040 con 0x55d3d1a3bd30 2024-03-12T05:20:49.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3d19c8900 con 0x55d3d1a3bd30 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08a8005200 con 0x55d3d1a3bd30 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08b6e22740 10 client.27526 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.527+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d3d1963910 con 0x55d3d1a3bd30 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.528+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08a802c9c0 con 0x55d3d1a3bd30 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.528+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f08a8034d20 con 0x55d3d1a3bd30 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 10 client.27526 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 10 client.27526 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 20 client.27526 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1abe720 0x55d3d1ac0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 4 client.27526 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:20:49.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d3d19c8900 con 0x55d3d1abe720 2024-03-12T05:20:49.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.529+0000 7f08b27f2640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1abe720 0x55d3d1ac0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.530+0000 7f08957fa640 10 client.27526 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.530+0000 7f08b27f2640 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1abe720 0x55d3d1ac0b40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d0fe122a6d5251 server_cookie=f2a50c63191008a5 in_seq=0 out_seq=0 2024-03-12T05:20:49.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.531+0000 7f08957fa640 1 -- 172.21.15.45:0/2480203605 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d3d19c8900 con 0x55d3d1abe720 2024-03-12T05:20:49.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.531+0000 7f08957fa640 10 client.27526 handle_command_reply: tid=0 2024-03-12T05:20:49.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.531+0000 7f08b6e22740 1 client.27526 shutdown 2024-03-12T05:20:49.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 msgr2=0x55d3d1a5c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a5c8d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f08a8037040 tx=0x7f08a80348f0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 shutdown_connections 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d3d1abe720 0x55d3d1ac0b40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d1a448b0 0x55d3d1ab57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3d1a3bd30 0x55d3d1a5c8d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 --2- 172.21.15.45:0/2480203605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d3d1a3b380 0x55d3d1a5c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.532+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 >> 172.21.15.45:0/2480203605 conn(0x55d3d1a35840 msgr2=0x55d3d1a348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.533+0000 7f08957fa640 0 client.27526 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.533+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 shutdown_connections 2024-03-12T05:20:49.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.533+0000 7f08b6e22740 1 -- 172.21.15.45:0/2480203605 wait complete. 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.534+0000 7f08b6e22740 20 client.27526 trim_cache size 0 max 16384 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "70201eeb-58fe-4b3c-93d8-fab6560754ae", 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:20:49.530 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:49.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.535+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 msgr2=0x7f08980a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 0x7f08980a4160 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55d3d1a3ae50 tx=0x7f08a0096ed0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 msgr2=0x7f08ac10aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f08a000e9f0 tx=0x7f08a000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 shutdown_connections 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08980a1ca0 0x7f08980a4160 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55d3d1a3ae50 tx=0x7f08a0096ed0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08ac1a38f0 0x7f08ac1a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08ac106460 0x7f08ac10afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 --2- 172.21.15.45:0/3089167753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08ac101390 0x7f08ac10aa60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.536+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 >> 172.21.15.45:0/3089167753 conn(0x7f08ac0fcc10 msgr2=0x7f08ac0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.537+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 shutdown_connections 2024-03-12T05:20:49.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.537+0000 7f08b4255640 1 -- 172.21.15.45:0/3089167753 wait complete. 2024-03-12T05:20:49.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:20:49.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:20:49.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.740+0000 7f4261f8a640 1 -- 172.21.15.45:0/919244203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c108100 msgr2=0x7f425c1145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.740+0000 7f4261f8a640 1 --2- 172.21.15.45:0/919244203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c108100 0x7f425c1145c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4254009ca0 tx=0x7f425402c190 comp rx=0 tx=0).stop 2024-03-12T05:20:49.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.740+0000 7f4261f8a640 1 -- 172.21.15.45:0/919244203 shutdown_connections 2024-03-12T05:20:49.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.740+0000 7f4261f8a640 1 --2- 172.21.15.45:0/919244203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c107760 0x7f425c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.741+0000 7f4261f8a640 1 --2- 172.21.15.45:0/919244203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c108100 0x7f425c1145c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.741+0000 7f4261f8a640 1 --2- 172.21.15.45:0/919244203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c10c640 0x7f425c10ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.741+0000 7f4261f8a640 1 -- 172.21.15.45:0/919244203 >> 172.21.15.45:0/919244203 conn(0x7f425c076fe0 msgr2=0x7f425c105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.741+0000 7f4261f8a640 1 -- 172.21.15.45:0/919244203 shutdown_connections 2024-03-12T05:20:49.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.741+0000 7f4261f8a640 1 -- 172.21.15.45:0/919244203 wait complete. 2024-03-12T05:20:49.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 Processor -- start 2024-03-12T05:20:49.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 -- start start 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c108100 0x7f425c1a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 0x7f425c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c19e090 con 0x7f425c10c640 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c19e200 con 0x7f425c107760 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.742+0000 7f4261f8a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c19e3c0 con 0x7f425c108100 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 0x7f425c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50552/0 (socket says 172.21.15.45:50552) 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 -- 172.21.15.45:0/4020246134 learned_addr learned my addr 172.21.15.45:0/4020246134 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 0x7f425c19da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43310/0 (socket says 172.21.15.45:43310) 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425affd640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c108100 0x7f425c1a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 -- 172.21.15.45:0/4020246134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c108100 msgr2=0x7f425c1a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c108100 0x7f425c1a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 -- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 msgr2=0x7f425c19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 0x7f425c19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 -- 172.21.15.45:0/4020246134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254009900 con 0x7f425c107760 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f425b7fe640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f424c00e9f0 tx=0x7f424c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.743+0000 7f4258ff9640 1 -- 172.21.15.45:0/4020246134 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c009700 con 0x7f425c107760 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.744+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c19e6a0 con 0x7f425c107760 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.744+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c1aa930 con 0x7f425c107760 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.744+0000 7f4258ff9640 1 -- 172.21.15.45:0/4020246134 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424c00cd80 con 0x7f425c107760 2024-03-12T05:20:49.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.744+0000 7f4258ff9640 1 -- 172.21.15.45:0/4020246134 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c018840 con 0x7f425c107760 2024-03-12T05:20:49.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 Processor -- start 2024-03-12T05:20:49.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 -- start start 2024-03-12T05:20:49.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4258ff9640 1 -- 172.21.15.45:0/4020246134 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f424c010b50 con 0x7f425c107760 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2471a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24725ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24689280 con 0x55be2471a2b0 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be246867b0 con 0x55be2471abd0 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.747+0000 7f4264c39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24681f90 con 0x55be247237b0 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50566/0 (socket says 172.21.15.45:50566) 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 -- 172.21.15.45:0/2621682608 learned_addr learned my addr 172.21.15.45:0/2621682608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425affd640 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24725ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425bfff640 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2471a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 -- 172.21.15.45:0/2621682608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 msgr2=0x55be24725ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24725ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 -- 172.21.15.45:0/2621682608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 msgr2=0x55be2471a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2471a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f4258ff9640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 0x7f42440a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 -- 172.21.15.45:0/2621682608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be24640570 con 0x55be2471abd0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.748+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f424c0cb070 tx=0x7f424c0d3e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b6d060eb79fd84f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.749+0000 7f4241ffb640 1 -- 172.21.15.45:0/2621682608 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c09a460 con 0x55be2471abd0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.749+0000 7f4258ff9640 1 -- 172.21.15.45:0/4020246134 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f424c014070 con 0x7f425c107760 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.749+0000 7f4241ffb640 1 -- 172.21.15.45:0/2621682608 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424c09a1d0 con 0x55be2471abd0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.749+0000 7f4241ffb640 1 -- 172.21.15.45:0/2621682608 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c09a690 con 0x55be2471abd0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.749+0000 7f425bfff640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 0x7f42440a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 -- 172.21.15.45:0/2621682608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 msgr2=0x55be24723270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f424c0cb070 tx=0x7f424c0d3e50 comp rx=0 tx=0).stop 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 -- 172.21.15.45:0/2621682608 shutdown_connections 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24725ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723270 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 --2- 172.21.15.45:0/2621682608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2471a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 -- 172.21.15.45:0/2621682608 >> 172.21.15.45:0/2621682608 conn(0x55be24714990 msgr2=0x55be24714d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f4264c39740 1 -- 172.21.15.45:0/2621682608 shutdown_connections 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.750+0000 7f425bfff640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 0x7f42440a40e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4254005e30 tx=0x7f425402e040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.751+0000 7f4264c39740 1 -- 172.21.15.45:0/2621682608 wait complete. 2024-03-12T05:20:49.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.751+0000 7f4264c39740 1 -- start start 2024-03-12T05:20:49.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.751+0000 7f4264c39740 1 Processor -- start 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f4264c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f4264c39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471abd0 0x55be24720510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f4264c39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24720b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:50578/0 (socket says 172.21.15.45:50578) 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f425b7fe640 1 -- 172.21.15.45:0/953781311 learned_addr learned my addr 172.21.15.45:0/953781311 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24355430 con 0x55be2471abd0 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f425affd640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471abd0 0x55be24720510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.752+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24311e00 con 0x55be2471a2b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24606fd0 con 0x55be247237b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f4264c39740 10 client.0 _ll_register_callbacks cb 0x55be2465cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425bfff640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24720b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 -- 172.21.15.45:0/953781311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 msgr2=0x55be24720b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24720b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 -- 172.21.15.45:0/953781311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471abd0 msgr2=0x55be24720510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471abd0 0x55be24720510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c1aa930 con 0x55be2471a2b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f425b7fe640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f424c0a2850 tx=0x7f424c0d3800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10cfab8ca0d0cf02 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.753+0000 7f423affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c099ff0 con 0x55be2471a2b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f424c0d0e40 con 0x55be2471a2b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be243817e0 con 0x55be2471a2b0 2024-03-12T05:20:49.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f424c0d3240 con 0x55be2471a2b0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f4264c39740 10 client.27560 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55be24680610 con 0x55be2471a2b0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424c0ae020 con 0x55be2471a2b0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.754+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f424c0d5050 con 0x55be2471a2b0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 10 client.27560 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 10 client.27560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 20 client.27560 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be24733620 0x55be24735ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 4 client.27560 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be243817e0 con 0x55be24733620 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f425affd640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be24733620 0x55be24735ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f425affd640 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be24733620 0x55be24735ac0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd04dda6e0d7fa79 server_cookie=584933827a860c61 in_seq=0 out_seq=0 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.755+0000 7f423affd640 10 client.27560 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.756+0000 7f423affd640 1 -- 172.21.15.45:0/953781311 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55be243817e0 con 0x55be24733620 2024-03-12T05:20:49.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.756+0000 7f423affd640 10 client.27560 handle_command_reply: tid=0 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.756+0000 7f4264c39740 1 client.27560 shutdown 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 msgr2=0x55be24723180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f424c0a2850 tx=0x7f424c0d3800 comp rx=0 tx=0).stop 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 shutdown_connections 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be24733620 0x55be24735ac0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471abd0 0x55be24720510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24720b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f423affd640 0 client.27560 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 --2- 172.21.15.45:0/953781311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471a2b0 0x55be24723180 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 >> 172.21.15.45:0/953781311 conn(0x55be24714990 msgr2=0x55be24716730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 shutdown_connections 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 1 -- 172.21.15.45:0/953781311 wait complete. 2024-03-12T05:20:49.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.757+0000 7f4264c39740 20 client.27560 trim_cache size 0 max 16384 2024-03-12T05:20:49.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.784+0000 7f4264c39740 1 Processor -- start 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.784+0000 7f4264c39740 1 -- start start 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.784+0000 7f4264c39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.784+0000 7f4264c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f4264c39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be2471b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f4264c39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be2472f600 con 0x55be2471a2b0 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f4264c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be2470ee20 con 0x55be2471abd0 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f4264c39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24726480 con 0x55be247237b0 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43336/0 (socket says 172.21.15.45:43336) 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 -- 172.21.15.45:0/2809531 learned_addr learned my addr 172.21.15.45:0/2809531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425affd640 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425bfff640 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be2471b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 -- 172.21.15.45:0/2809531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 msgr2=0x55be2471b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be2471b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 -- 172.21.15.45:0/2809531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 msgr2=0x55be24723180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.785+0000 7f425b7fe640 1 -- 172.21.15.45:0/2809531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be2461cac0 con 0x55be2471a2b0 2024-03-12T05:20:49.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.786+0000 7f425b7fe640 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f424c098940 tx=0x7f424c0d3e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f506c2354972730 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.788+0000 7f42417fa640 1 -- 172.21.15.45:0/2809531 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c085070 con 0x55be2471a2b0 2024-03-12T05:20:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.788+0000 7f42417fa640 1 -- 172.21.15.45:0/2809531 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424c0ae030 con 0x55be2471a2b0 2024-03-12T05:20:49.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.788+0000 7f42417fa640 1 -- 172.21.15.45:0/2809531 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c0ae400 con 0x55be2471a2b0 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 -- 172.21.15.45:0/2809531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 msgr2=0x55be24722c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f424c098940 tx=0x7f424c0d3e50 comp rx=0 tx=0).stop 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 -- 172.21.15.45:0/2809531 shutdown_connections 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be2471b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24723180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 --2- 172.21.15.45:0/2809531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be24722c40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.789+0000 7f4264c39740 1 -- 172.21.15.45:0/2809531 >> 172.21.15.45:0/2809531 conn(0x55be24714990 msgr2=0x55be2473a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.790+0000 7f4264c39740 1 -- 172.21.15.45:0/2809531 shutdown_connections 2024-03-12T05:20:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.790+0000 7f4264c39740 1 -- 172.21.15.45:0/2809531 wait complete. 2024-03-12T05:20:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.791+0000 7f4264c39740 1 -- start start 2024-03-12T05:20:49.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.791+0000 7f4264c39740 1 Processor -- start 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.791+0000 7f4264c39740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2472ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24726f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24727550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24727c20 con 0x55be2471a2b0 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24727d90 con 0x55be2471abd0 2024-03-12T05:20:49.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be24789ba0 con 0x55be247237b0 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f4264c39740 10 client.0 _ll_register_callbacks cb 0x7f424c0d3420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2472ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2472ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43340/0 (socket says 172.21.15.45:43340) 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425b7fe640 1 -- 172.21.15.45:0/3910281288 learned_addr learned my addr 172.21.15.45:0/3910281288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24726f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425bfff640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24727550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 -- 172.21.15.45:0/3910281288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 msgr2=0x55be24727550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24727550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 -- 172.21.15.45:0/3910281288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 msgr2=0x55be2472ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2472ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.792+0000 7f425affd640 1 -- 172.21.15.45:0/3910281288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55be24680610 con 0x55be2471abd0 2024-03-12T05:20:49.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f425affd640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24726f40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x55be247363d0 tx=0x7f4250009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=855fc5c88fe719cf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f423affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250017040 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4250003a00 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f424c09a1d0 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4250003be0 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.793+0000 7f4264c39740 10 client.27566 fetch_fsmap learned FSMap version 33 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55be246426a0 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424c09a1d0 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4250027420 con 0x55be2471abd0 2024-03-12T05:20:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f4264c39740 10 client.27566 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f4264c39740 10 client.27566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f4264c39740 20 client.27566 populate_metadata read hostname 'smithi045' 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.794+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be2478b480 0x55be2478d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.795+0000 7f4264c39740 4 client.27566 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.795+0000 7f425b7fe640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be2478b480 0x55be2478d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.795+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55be243be720 con 0x55be2478b480 2024-03-12T05:20:49.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.795+0000 7f423affd640 10 client.27566 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.795+0000 7f425b7fe640 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be2478b480 0x55be2478d8a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c30675eaae43f80 server_cookie=4b16c7d15cd74aae in_seq=0 out_seq=0 2024-03-12T05:20:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f423affd640 1 -- 172.21.15.45:0/3910281288 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55be243be720 con 0x55be2478b480 2024-03-12T05:20:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f423affd640 10 client.27566 handle_command_reply: tid=0 2024-03-12T05:20:49.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 client.27566 shutdown 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 msgr2=0x55be24726f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24726f40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x55be247363d0 tx=0x7f4250009f90 comp rx=0 tx=0).stop 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 shutdown_connections 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55be2478b480 0x55be2478d8a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be247237b0 0x55be24727550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be2471abd0 0x55be24726f40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f423affd640 0 client.27566 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 --2- 172.21.15.45:0/3910281288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55be2471a2b0 0x55be2472ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.796+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 >> 172.21.15.45:0/3910281288 conn(0x55be24714990 msgr2=0x55be2471b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.797+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 shutdown_connections 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.797+0000 7f4264c39740 1 -- 172.21.15.45:0/3910281288 wait complete. 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.797+0000 7f4264c39740 20 client.27566 trim_cache size 0 max 16384 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "70201eeb-58fe-4b3c-93d8-fab6560754ae": { 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "70201eeb-58fe-4b3c-93d8-fab6560754ae", 2024-03-12T05:20:49.796 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:20:49.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:20:49.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:20:49.797 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 msgr2=0x7f42440a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 0x7f42440a40e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4254005e30 tx=0x7f425402e040 comp rx=0 tx=0).stop 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 msgr2=0x7f425c1a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f424c00e9f0 tx=0x7f424c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 shutdown_connections 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42440a1c20 0x7f42440a40e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f425c10c640 0x7f425c19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f425c108100 0x7f425c1a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 --2- 172.21.15.45:0/4020246134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425c107760 0x7f425c1a3890 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:20:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.798+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 >> 172.21.15.45:0/4020246134 conn(0x7f425c076fe0 msgr2=0x7f425c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:20:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.799+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 shutdown_connections 2024-03-12T05:20:49.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:20:49.799+0000 7f4261f8a640 1 -- 172.21.15.45:0/4020246134 wait complete. 2024-03-12T05:20:49.815 INFO:tasks.cephfs.filesystem:scrub status for tag:70201eeb-58fe-4b3c-93d8-fab6560754ae - {'path': '/', 'tag': '70201eeb-58fe-4b3c-93d8-fab6560754ae', 'options': 'recursive,force'} 2024-03-12T05:20:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 525.0 s, 1223.6 tps, lat 81.582 ms stddev 35.098 2024-03-12T05:20:50.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:50 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:20:50.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:20:50.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:50 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:20:50.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:50.968 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:50 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:20:52.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:52 smithi116 ceph-mon[35486]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:20:52.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:52 smithi045 ceph-mon[32174]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:20:52.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:52 smithi130 ceph-mon[35398]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:20:54.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:54 smithi045 ceph-mon[32174]: pgmap v311: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:20:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 530.0 s, 1395.0 tps, lat 71.794 ms stddev 50.818 2024-03-12T05:20:54.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:54 smithi130 ceph-mon[35398]: pgmap v311: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:20:54.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:54 smithi116 ceph-mon[35486]: pgmap v311: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:20:56.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:56 smithi045 ceph-mon[32174]: pgmap v312: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 14 MiB/s wr, 1.59k op/s 2024-03-12T05:20:56.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:56 smithi130 ceph-mon[35398]: pgmap v312: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 14 MiB/s wr, 1.59k op/s 2024-03-12T05:20:56.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:56 smithi116 ceph-mon[35486]: pgmap v312: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 14 MiB/s wr, 1.59k op/s 2024-03-12T05:20:58.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:20:58 smithi045 ceph-mon[32174]: pgmap v313: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.2 MiB/s wr, 1.05k op/s 2024-03-12T05:20:58.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:20:58 smithi130 ceph-mon[35398]: pgmap v313: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.2 MiB/s wr, 1.05k op/s 2024-03-12T05:20:58.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:20:58 smithi116 ceph-mon[35486]: pgmap v313: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.2 MiB/s wr, 1.05k op/s 2024-03-12T05:20:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 535.0 s, 1555.2 tps, lat 64.379 ms stddev 30.627 2024-03-12T05:21:00.768 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:00 smithi130 ceph-mon[35398]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-03-12T05:21:00.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:00 smithi045 ceph-mon[32174]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-03-12T05:21:00.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:00 smithi116 ceph-mon[35486]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-03-12T05:21:02.793 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:02 smithi116 ceph-mon[35486]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:21:02.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:02 smithi045 ceph-mon[32174]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:21:02.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:02 smithi130 ceph-mon[35398]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:21:03.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:03.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:03.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:03.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:03 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:03.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:03.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:03.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:03.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:03 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:03.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:03.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:03.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:03.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:03 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:04.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:04 smithi045 ceph-mon[32174]: pgmap v316: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:21:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 540.0 s, 1364.0 tps, lat 73.338 ms stddev 29.636 2024-03-12T05:21:04.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:04 smithi130 ceph-mon[35398]: pgmap v316: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:21:04.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:04 smithi116 ceph-mon[35486]: pgmap v316: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:21:06.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:06 smithi045 ceph-mon[32174]: pgmap v317: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 1.75k op/s 2024-03-12T05:21:06.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:06 smithi130 ceph-mon[35398]: pgmap v317: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 1.75k op/s 2024-03-12T05:21:06.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:06 smithi116 ceph-mon[35486]: pgmap v317: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 15 MiB/s wr, 1.75k op/s 2024-03-12T05:21:08.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:08 smithi045 ceph-mon[32174]: pgmap v318: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:21:08.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:08 smithi130 ceph-mon[35398]: pgmap v318: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:21:08.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:08 smithi116 ceph-mon[35486]: pgmap v318: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:21:09.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:09 smithi045 ceph-mon[32174]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T05:21:09.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:09 smithi045 ceph-mon[32174]: pgmap v320: 97 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 70 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.9 MiB/s wr, 1.07k op/s 2024-03-12T05:21:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 545.0 s, 1254.6 tps, lat 79.644 ms stddev 44.918 2024-03-12T05:21:09.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:09 smithi130 ceph-mon[35398]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T05:21:09.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:09 smithi130 ceph-mon[35398]: pgmap v320: 97 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 70 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.9 MiB/s wr, 1.07k op/s 2024-03-12T05:21:09.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:09 smithi116 ceph-mon[35486]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T05:21:09.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:09 smithi116 ceph-mon[35486]: pgmap v320: 97 pgs: 10 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 70 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.9 MiB/s wr, 1.07k op/s 2024-03-12T05:21:11.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:11 smithi130 ceph-mon[35398]: pgmap v321: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:11.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:11 smithi116 ceph-mon[35486]: pgmap v321: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:12.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:11 smithi045 ceph-mon[32174]: pgmap v321: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:12.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:12 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:21:12.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:12 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:21:13.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:12 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:21:13.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:13 smithi130 ceph-mon[35398]: pgmap v322: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:13.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:13 smithi116 ceph-mon[35486]: pgmap v322: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:14.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:13 smithi045 ceph-mon[32174]: pgmap v322: 97 pgs: 22 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 39 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-03-12T05:21:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 550.0 s, 1396.8 tps, lat 71.693 ms stddev 22.358 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:14 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.244 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:14 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:15.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:14 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:16.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:16 smithi045 ceph-mon[32174]: pgmap v323: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:16.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:16 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:21:16.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:16 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:21:16.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:16 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:21:16.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:16 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:21:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:16 smithi130 ceph-mon[35398]: pgmap v323: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:16 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:21:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:16 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:21:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:16 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:21:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:16 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:21:16.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:16 smithi116 ceph-mon[35486]: pgmap v323: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:16.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:16 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-12T05:21:16.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:16 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi116 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-03-12T05:21:16.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:16 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-03-12T05:21:16.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:16 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi130 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-03-12T05:21:18.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:18 smithi045 ceph-mon[32174]: pgmap v324: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:18.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:18 smithi130 ceph-mon[35398]: pgmap v324: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:18.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:18 smithi116 ceph-mon[35486]: pgmap v324: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-03-12T05:21:19.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:21:19.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:21:19.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 555.0 s, 1333.8 tps, lat 74.085 ms stddev 30.063 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 -- 172.21.15.45:0/3459869728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009100 msgr2=0x7fa1d40094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/3459869728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009100 0x7fa1d40094e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009890 tx=0x7fa1d002bb70 comp rx=0 tx=0).stop 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 -- 172.21.15.45:0/3459869728 shutdown_connections 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/3459869728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d4009a20 0x7fa1d40a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/3459869728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d40a6ba0 0x7fa1d40a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/3459869728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009100 0x7fa1d40094e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 -- 172.21.15.45:0/3459869728 >> 172.21.15.45:0/3459869728 conn(0x7fa1d4093990 msgr2=0x7fa1d4095db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.232+0000 7fa1e0c53640 1 -- 172.21.15.45:0/3459869728 shutdown_connections 2024-03-12T05:21:20.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.233+0000 7fa1e0c53640 1 -- 172.21.15.45:0/3459869728 wait complete. 2024-03-12T05:21:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 Processor -- start 2024-03-12T05:21:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 -- start start 2024-03-12T05:21:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009a20 0x7fa1d4138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d40a6ba0 0x7fa1d41326e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d4132e70 con 0x7fa1d40a6ba0 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d4132fe0 con 0x7fa1d4009a20 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1e0c53640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d41331a0 con 0x7fa1d4009100 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.234+0000 7fa1db7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56966/0 (socket says 172.21.15.45:56966) 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 -- 172.21.15.45:0/2671903768 learned_addr learned my addr 172.21.15.45:0/2671903768 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 -- 172.21.15.45:0/2671903768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009a20 msgr2=0x7fa1d4138b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009a20 0x7fa1d4138b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 -- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d40a6ba0 msgr2=0x7fa1d41326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d40a6ba0 0x7fa1d41326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 -- 172.21.15.45:0/2671903768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0009490 con 0x7fa1d4009100 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.235+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0000c00 tx=0x7fa1d0004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.236+0000 7fa1d8ff9640 1 -- 172.21.15.45:0/2671903768 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0034850 con 0x7fa1d4009100 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.236+0000 7fa1d8ff9640 1 -- 172.21.15.45:0/2671903768 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1d002ce90 con 0x7fa1d4009100 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.236+0000 7fa1d8ff9640 1 -- 172.21.15.45:0/2671903768 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0003ce0 con 0x7fa1d4009100 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.236+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d4133420 con 0x7fa1d4009100 2024-03-12T05:21:20.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.237+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d413f5a0 con 0x7fa1d4009100 2024-03-12T05:21:20.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.240+0000 7fa1e4b64740 1 Processor -- start 2024-03-12T05:21:20.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.241+0000 7fa1d8ff9640 1 -- 172.21.15.45:0/2671903768 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa1d003b040 con 0x7fa1d4009100 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.241+0000 7fa1e4b64740 1 -- start start 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca83930280 con 0x55ca839c1490 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8392d7b0 con 0x55ca839c1db0 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca83928f90 con 0x55ca839ca990 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59648/0 (socket says 172.21.15.45:59648) 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 -- 172.21.15.45:0/1390715268 learned_addr learned my addr 172.21.15.45:0/1390715268 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1daffd640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 -- 172.21.15.45:0/1390715268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 msgr2=0x55ca839ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.242+0000 7fa1dbfff640 1 -- 172.21.15.45:0/1390715268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 msgr2=0x55ca839ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1dbfff640 1 -- 172.21.15.45:0/1390715268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca838e7570 con 0x55ca839c1490 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:20.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1d8ff9640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 0x7fa1cc0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc04fc90 tx=0x7fa1dc068df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29da92c6f24af0d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1d8ff9640 1 -- 172.21.15.45:0/2671903768 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa1d00ef0a0 con 0x7fa1d4009100 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1c5ffb640 1 -- 172.21.15.45:0/1390715268 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc06e040 con 0x55ca839c1490 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1c5ffb640 1 -- 172.21.15.45:0/1390715268 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1dc074530 con 0x55ca839c1490 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1c5ffb640 1 -- 172.21.15.45:0/1390715268 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc070520 con 0x55ca839c1490 2024-03-12T05:21:20.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.243+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 0x7fa1cc0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.244+0000 7fa1e4b64740 1 -- 172.21.15.45:0/1390715268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 msgr2=0x55ca839c1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.244+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc04fc90 tx=0x7fa1dc068df0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.244+0000 7fa1db7fe640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 0x7fa1cc0a41b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55ca839c0f60 tx=0x7fa1d0005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 -- 172.21.15.45:0/1390715268 shutdown_connections 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/1390715268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c1870 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 -- 172.21.15.45:0/1390715268 >> 172.21.15.45:0/1390715268 conn(0x55ca839bbbc0 msgr2=0x55ca839bbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 -- 172.21.15.45:0/1390715268 shutdown_connections 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 -- 172.21.15.45:0/1390715268 wait complete. 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.245+0000 7fa1e4b64740 1 -- start start 2024-03-12T05:21:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 Processor -- start 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca835fc430 con 0x55ca839c1db0 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca835b8e00 con 0x55ca839ca990 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca838adfd0 con 0x55ca839c1490 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59652/0 (socket says 172.21.15.45:59652) 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 -- 172.21.15.45:0/2964956312 learned_addr learned my addr 172.21.15.45:0/2964956312 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.246+0000 7fa1e4b64740 10 client.0 _ll_register_callbacks cb 0x55ca83903fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 -- 172.21.15.45:0/2964956312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 msgr2=0x55ca839ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 -- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 msgr2=0x55ca839c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 -- 172.21.15.45:0/2964956312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc068650 con 0x55ca839c1db0 2024-03-12T05:21:20.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1daffd640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011460 tx=0x7fa1c8011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61cb42f9483d058 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1baffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.247+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8012980 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca836287e0 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa1c8012ae0 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1c8012ca0 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1e4b64740 10 client.18081 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ca83927610 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1c8026020 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.248+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1c801a3f0 con 0x55ca839c1db0 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 10 client.18081 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 10 client.18081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 20 client.18081 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca839ebaf0 0x55ca839edf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 4 client.18081 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca83665720 con 0x55ca839ebaf0 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.249+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca839ebaf0 0x55ca839edf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.250+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca839ebaf0 0x55ca839edf10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf58694a2d795d7 server_cookie=957ed2199c2e2cc2 in_seq=0 out_seq=0 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.250+0000 7fa1baffd640 10 client.18081 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1baffd640 1 -- 172.21.15.45:0/2964956312 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ca83665720 con 0x55ca839ebaf0 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1baffd640 10 client.18081 handle_command_reply: tid=0 2024-03-12T05:21:20.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 client.18081 shutdown 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 msgr2=0x55ca839c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011460 tx=0x7fa1c8011920 comp rx=0 tx=0).stop 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 shutdown_connections 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca839ebaf0 0x55ca839edf10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839c7630 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/2964956312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 >> 172.21.15.45:0/2964956312 conn(0x55ca839bbbc0 msgr2=0x55ca839bd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.251+0000 7fa1baffd640 0 client.18081 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.252+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 shutdown_connections 2024-03-12T05:21:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.252+0000 7fa1e4b64740 1 -- 172.21.15.45:0/2964956312 wait complete. 2024-03-12T05:21:20.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.252+0000 7fa1e4b64740 20 client.18081 trim_cache size 0 max 16384 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.282+0000 7fa1e4b64740 1 Processor -- start 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.282+0000 7fa1e4b64740 1 -- start start 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.282+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca839c10f0 con 0x55ca839c1db0 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1daffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59654/0 (socket says 172.21.15.45:59654) 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1daffd640 1 -- 172.21.15.45:0/560374543 learned_addr learned my addr 172.21.15.45:0/560374543 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca839f3100 con 0x55ca839ca990 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca839c5b80 con 0x55ca839c1490 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 -- 172.21.15.45:0/560374543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 msgr2=0x55ca839b69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 -- 172.21.15.45:0/560374543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 msgr2=0x55ca839ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.283+0000 7fa1dbfff640 1 -- 172.21.15.45:0/560374543 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca839c2500 con 0x55ca839c1490 2024-03-12T05:21:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.284+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839c9d60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc064670 tx=0x7fa1dc068df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffc5918f889c9c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.284+0000 7fa1c57fa640 1 -- 172.21.15.45:0/560374543 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc06e040 con 0x55ca839c1490 2024-03-12T05:21:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.284+0000 7fa1c57fa640 1 -- 172.21.15.45:0/560374543 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1dc074b50 con 0x55ca839c1490 2024-03-12T05:21:20.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.284+0000 7fa1c57fa640 1 -- 172.21.15.45:0/560374543 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0678d0 con 0x55ca839c1490 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 msgr2=0x55ca839c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839c9d60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc064670 tx=0x7fa1dc068df0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 shutdown_connections 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1db0 0x55ca839ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839ca990 0x55ca839b69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/560374543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839c1490 0x55ca839c9d60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 >> 172.21.15.45:0/560374543 conn(0x55ca839bbbc0 msgr2=0x55ca839c2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 shutdown_connections 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.285+0000 7fa1e4b64740 1 -- 172.21.15.45:0/560374543 wait complete. 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.286+0000 7fa1e4b64740 1 -- start start 2024-03-12T05:21:20.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.286+0000 7fa1e4b64740 1 Processor -- start 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839c4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca839e3160 con 0x55ca839c1490 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca83a41ff0 con 0x55ca839c1db0 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca83a42130 con 0x55ca839ca990 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.287+0000 7fa1e4b64740 10 client.0 _ll_register_callbacks cb 0x7fa1c8009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59670/0 (socket says 172.21.15.45:59670) 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 -- 172.21.15.45:0/3344397159 learned_addr learned my addr 172.21.15.45:0/3344397159 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1daffd640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 -- 172.21.15.45:0/3344397159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 msgr2=0x55ca839c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 -- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 msgr2=0x55ca839c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.288+0000 7fa1dbfff640 1 -- 172.21.15.45:0/3344397159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca838e7570 con 0x55ca839c1490 2024-03-12T05:21:20.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.289+0000 7fa1dbfff640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc050150 tx=0x7fa1dc0671a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760a8b098433ce92 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.289+0000 7fa1baffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc099070 con 0x55ca839c1490 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa1dc068020 con 0x55ca839c1490 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1dc074530 con 0x55ca839c1490 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1dc06e040 con 0x55ca839c1490 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1e4b64740 10 client.18087 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:20.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.290+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ca838c3ac0 con 0x55ca839c1490 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa1dc0749b0 con 0x55ca839c1490 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1dc09c020 con 0x55ca839c1490 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 10 client.18087 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 20 client.18087 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca83a433d0 0x55ca83a457c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 4 client.18087 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fa1dc074530 con 0x55ca83a433d0 2024-03-12T05:21:20.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.291+0000 7fa1daffd640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca83a433d0 0x55ca83a457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1baffd640 10 client.18087 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:20.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1daffd640 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca83a433d0 0x55ca83a457c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dfe3cbac493aca9 server_cookie=8dcbde34cfa5b750 in_seq=0 out_seq=0 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1baffd640 1 -- 172.21.15.45:0/3344397159 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fa1dc074530 con 0x55ca83a433d0 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1baffd640 10 client.18087 handle_command_reply: tid=0 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 client.18087 shutdown 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 msgr2=0x55ca839c40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc050150 tx=0x7fa1dc0671a0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 shutdown_connections 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ca83a433d0 0x55ca83a457c0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.292+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ca839ca990 0x55ca839c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca839c1db0 0x55ca839c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 1 --2- 172.21.15.45:0/3344397159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ca839c1490 0x55ca839c40d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 >> 172.21.15.45:0/3344397159 conn(0x55ca839bbbc0 msgr2=0x55ca839b69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1baffd640 0 client.18087 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 shutdown_connections 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 1 -- 172.21.15.45:0/3344397159 wait complete. 2024-03-12T05:21:20.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e4b64740 20 client.18087 trim_cache size 0 max 16384 2024-03-12T05:21:20.290 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:20.290 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:21:20.290 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:21:20.290 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.293+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 msgr2=0x7fa1cc0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 0x7fa1cc0a41b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55ca839c0f60 tx=0x7fa1d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 msgr2=0x7fa1d4138620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0000c00 tx=0x7fa1d0004550 comp rx=0 tx=0).stop 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 shutdown_connections 2024-03-12T05:21:20.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa1cc0a1cf0 0x7fa1cc0a41b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1d40a6ba0 0x7fa1d41326e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1d4009a20 0x7fa1d4138b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 --2- 172.21.15.45:0/2671903768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1d4009100 0x7fa1d4138620 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 >> 172.21.15.45:0/2671903768 conn(0x7fa1d4093990 msgr2=0x7fa1d4095d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 shutdown_connections 2024-03-12T05:21:20.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.294+0000 7fa1e0c53640 1 -- 172.21.15.45:0/2671903768 wait complete. 2024-03-12T05:21:20.307 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:21:20.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:21:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.488+0000 7fefecba9640 1 -- 172.21.15.45:0/1717924965 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe402cc10 con 0x7fefe8069aa0 2024-03-12T05:21:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 -- 172.21.15.45:0/1717924965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe8069aa0 msgr2=0x7fefe8069ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 --2- 172.21.15.45:0/1717924965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe8069aa0 0x7fefe8069ee0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009900 tx=0x7fefe402bcd0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 -- 172.21.15.45:0/1717924965 shutdown_connections 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 --2- 172.21.15.45:0/1717924965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe8069aa0 0x7fefe8069ee0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 --2- 172.21.15.45:0/1717924965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8103ba0 0x7fefe8104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 --2- 172.21.15.45:0/1717924965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe81060e0 0x7fefe8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 -- 172.21.15.45:0/1717924965 >> 172.21.15.45:0/1717924965 conn(0x7fefe80771b0 msgr2=0x7fefe80775c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 -- 172.21.15.45:0/1717924965 shutdown_connections 2024-03-12T05:21:20.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.489+0000 7fefefe0f640 1 -- 172.21.15.45:0/1717924965 wait complete. 2024-03-12T05:21:20.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.490+0000 7fefefe0f640 1 Processor -- start 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.490+0000 7fefefe0f640 1 -- start start 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 0x7fefe81a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 0x7fefe819da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe819e0e0 con 0x7fefe81060e0 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe819e250 con 0x7fefe8103ba0 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefefe0f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe819e410 con 0x7fefe8069aa0 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57024/0 (socket says 172.21.15.45:57024) 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 -- 172.21.15.45:0/278449088 learned_addr learned my addr 172.21.15.45:0/278449088 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:20.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefed3aa640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 0x7fefe81a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefee3ac640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 0x7fefe819da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 -- 172.21.15.45:0/278449088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 msgr2=0x7fefe81a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 0x7fefe81a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 -- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 msgr2=0x7fefe819da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 0x7fefe819da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefedbab640 1 -- 172.21.15.45:0/278449088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4009560 con 0x7fefe8069aa0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefed3aa640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 0x7fefe81a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.491+0000 7fefee3ac640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 0x7fefe819da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefedbab640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fefe000eab0 tx=0x7fefe000ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000ce40 con 0x7fefe8069aa0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe819e6f0 con 0x7fefe8069aa0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe81aa980 con 0x7fefe8069aa0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fefe0011920 con 0x7fefe8069aa0 2024-03-12T05:21:20.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.492+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0010960 con 0x7fefe8069aa0 2024-03-12T05:21:20.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.493+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc005290 con 0x7fefe8069aa0 2024-03-12T05:21:20.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.495+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fefe0011470 con 0x7fefe8069aa0 2024-03-12T05:21:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.496+0000 7fefdaffd640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 0x7fefd40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.496+0000 7fefed3aa640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 0x7fefd40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.496+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fefe00cddc0 con 0x7fefe8069aa0 2024-03-12T05:21:20.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.497+0000 7fefed3aa640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 0x7fefd40a40e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefe4006010 tx=0x7fefe4005a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:20.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.502+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fefe0096c60 con 0x7fefe8069aa0 2024-03-12T05:21:20.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:20 smithi045 ceph-mon[32174]: pgmap v325: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:21:20.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:20 smithi045 ceph-mon[32174]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T05:21:20.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.863+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefcc005810 con 0x7fefe8069aa0 2024-03-12T05:21:20.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.865+0000 7fefdaffd640 1 -- 172.21.15.45:0/278449088 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fefe009a320 con 0x7fefe8069aa0 2024-03-12T05:21:20.860 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:21:20.860 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:21:20.861 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:21:20.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 msgr2=0x7fefd40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 0x7fefd40a40e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefe4006010 tx=0x7fefe4005a40 comp rx=0 tx=0).stop 2024-03-12T05:21:20.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 msgr2=0x7fefe81a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:20.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fefe000eab0 tx=0x7fefe000ef70 comp rx=0 tx=0).stop 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 shutdown_connections 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fefd40a1c20 0x7fefd40a40e0 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefe4006010 tx=0x7fefe4005a40 comp rx=0 tx=0).stop 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefe81060e0 0x7fefe819da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefe8103ba0 0x7fefe81a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 --2- 172.21.15.45:0/278449088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe8069aa0 0x7fefe81a38e0 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fefe000eab0 tx=0x7fefe000ef70 comp rx=0 tx=0).stop 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 >> 172.21.15.45:0/278449088 conn(0x7fefe80771b0 msgr2=0x7fefe8102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 shutdown_connections 2024-03-12T05:21:20.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:20.868+0000 7fefefe0f640 1 -- 172.21.15.45:0/278449088 wait complete. 2024-03-12T05:21:20.881 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:20.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:21:20.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:20 smithi130 ceph-mon[35398]: pgmap v325: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:21:20.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:20 smithi130 ceph-mon[35398]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T05:21:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:20 smithi116 ceph-mon[35486]: pgmap v325: 97 pgs: 97 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:21:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:20 smithi116 ceph-mon[35486]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T05:21:21.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 -- 172.21.15.45:0/3723283640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c100040 msgr2=0x7fbc1c1004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 --2- 172.21.15.45:0/3723283640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c100040 0x7fbc1c1004a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009ca0 tx=0x7fbc0c02c180 comp rx=0 tx=0).stop 2024-03-12T05:21:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 -- 172.21.15.45:0/3723283640 shutdown_connections 2024-03-12T05:21:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 --2- 172.21.15.45:0/3723283640 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c100040 0x7fbc1c1004a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 --2- 172.21.15.45:0/3723283640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 --2- 172.21.15.45:0/3723283640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 0x7fbc1c1064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 -- 172.21.15.45:0/3723283640 >> 172.21.15.45:0/3723283640 conn(0x7fbc1c077110 msgr2=0x7fbc1c077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 -- 172.21.15.45:0/3723283640 shutdown_connections 2024-03-12T05:21:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.058+0000 7fbc211bc640 1 -- 172.21.15.45:0/3723283640 wait complete. 2024-03-12T05:21:21.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 Processor -- start 2024-03-12T05:21:21.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 -- start start 2024-03-12T05:21:21.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 0x7fbc1c10e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a9f10 con 0x7fbc1c1009e0 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1aa050 con 0x7fbc1c106100 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.059+0000 7fbc211bc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1aa190 con 0x7fbc1c100040 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57054/0 (socket says 172.21.15.45:57054) 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/244893797 learned_addr learned my addr 172.21.15.45:0/244893797 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 0x7fbc1c10e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 -- 172.21.15.45:0/244893797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 msgr2=0x7fbc1c10dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 -- 172.21.15.45:0/244893797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 msgr2=0x7fbc1c10e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 0x7fbc1c10e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 -- 172.21.15.45:0/244893797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c009900 con 0x7fbc1c1009e0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.060+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10e000 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c02c6c0 tx=0x7fbc0c034740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.061+0000 7fbc03fff640 1 -- 172.21.15.45:0/244893797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c003d80 con 0x7fbc1c1009e0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.061+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c1aa3e0 con 0x7fbc1c1009e0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.061+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c1aa8e0 con 0x7fbc1c1009e0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.061+0000 7fbc03fff640 1 -- 172.21.15.45:0/244893797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc0c032b30 con 0x7fbc1c1009e0 2024-03-12T05:21:21.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.061+0000 7fbc03fff640 1 -- 172.21.15.45:0/244893797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c004560 con 0x7fbc1c1009e0 2024-03-12T05:21:21.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.064+0000 7fbc23d8c740 1 Processor -- start 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.064+0000 7fbc23d8c740 1 -- start start 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.064+0000 7fbc03fff640 1 -- 172.21.15.45:0/244893797 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbc0c03b040 con 0x7fbc1c1009e0 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a088960 con 0x555f9a0cd250 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a03c310 con 0x555f9a0cdb70 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a039840 con 0x555f9a0d6750 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1b59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59712/0 (socket says 172.21.15.45:59712) 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1b59e640 1 -- 172.21.15.45:0/1023906980 learned_addr learned my addr 172.21.15.45:0/1023906980 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc03fff640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 0x7fbc040a4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc03fff640 1 -- 172.21.15.45:0/244893797 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc0c0efea0 con 0x7fbc1c1009e0 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1023906980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 msgr2=0x555f9a0d8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1023906980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 msgr2=0x555f9a0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.065+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1023906980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f99ff7060 con 0x555f9a0cdb70 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.066+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 0x7fbc040a4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.066+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d6210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbc100114f0 tx=0x7fbc100119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217d2facbd95b303 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.066+0000 7fbc017fa640 1 -- 172.21.15.45:0/1023906980 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000f680 con 0x555f9a0cdb70 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.067+0000 7fbc017fa640 1 -- 172.21.15.45:0/1023906980 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc1000fc90 con 0x555f9a0cdb70 2024-03-12T05:21:21.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.067+0000 7fbc017fa640 1 -- 172.21.15.45:0/1023906980 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10019cf0 con 0x555f9a0cdb70 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.067+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 0x7fbc040a4150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400be10 tx=0x7fbc14004730 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1023906980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 msgr2=0x555f9a0d6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d6210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbc100114f0 tx=0x7fbc100119b0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1023906980 shutdown_connections 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d6210 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1023906980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1023906980 >> 172.21.15.45:0/1023906980 conn(0x555f9a0c7980 msgr2=0x555f9a0c7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1023906980 shutdown_connections 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1023906980 wait complete. 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.068+0000 7fbc23d8c740 1 -- start start 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 Processor -- start 2024-03-12T05:21:21.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a035020 con 0x555f9a0cd250 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc1a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc1a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35638/0 (socket says 172.21.15.45:35638) 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc1a59c640 1 -- 172.21.15.45:0/2644424656 learned_addr learned my addr 172.21.15.45:0/2644424656 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f99d08430 con 0x555f9a0cdb70 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.069+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f99cc4e00 con 0x555f9a0d6750 2024-03-12T05:21:21.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc23d8c740 10 client.0 _ll_register_callbacks cb 0x555f99fdccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 -- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 msgr2=0x555f9a0d4170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:21.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d4170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 -- 172.21.15.45:0/2644424656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 msgr2=0x555f9a0d36f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:21:21.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0d36f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc10009c90 con 0x555f9a0cdb70 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.070+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c02c790 tx=0x7fbc0c0bc9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e231a4924994025b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.073+0000 7fbbfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.073+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0bcb50 con 0x555f9a0cdb70 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.073+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f99d347e0 con 0x555f9a0cdb70 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.073+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc0c0c0ca0 con 0x555f9a0cdb70 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.073+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc0c0c52f0 con 0x555f9a0cdb70 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.074+0000 7fbc23d8c740 10 client.27593 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.074+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555f9a0336a0 con 0x555f9a0cdb70 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x555f99d347e0 con 0x555f9a0cdb70 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc0c0bcec0 con 0x555f9a0cdb70 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 10 client.27593 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 10 client.27593 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 20 client.27593 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a0e6a00 0x555f9a0e8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 4 client.27593 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f99d71720 con 0x555f9a0e6a00 2024-03-12T05:21:21.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.075+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a0e6a00 0x555f9a0e8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.076+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a0e6a00 0x555f9a0e8e20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd7a6e016bb1607b server_cookie=7bcd379d4cc97343 in_seq=0 out_seq=0 2024-03-12T05:21:21.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.076+0000 7fbbfeff5640 10 client.27593 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:21.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.077+0000 7fbbfeff5640 1 -- 172.21.15.45:0/2644424656 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555f99d71720 con 0x555f9a0e6a00 2024-03-12T05:21:21.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.077+0000 7fbbfeff5640 10 client.27593 handle_command_reply: tid=0 2024-03-12T05:21:21.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 client.27593 shutdown 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 msgr2=0x555f9a0d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c02c790 tx=0x7fbc0c0bc9e0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 shutdown_connections 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a0e6a00 0x555f9a0e8e20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0d4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3c30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/2644424656 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cd250 0x555f9a0d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 >> 172.21.15.45:0/2644424656 conn(0x555f9a0c7980 msgr2=0x555f9a0c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbbfeff5640 0 client.27593 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 shutdown_connections 2024-03-12T05:21:21.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.078+0000 7fbc23d8c740 1 -- 172.21.15.45:0/2644424656 wait complete. 2024-03-12T05:21:21.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.079+0000 7fbc23d8c740 20 client.27593 trim_cache size 0 max 16384 2024-03-12T05:21:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.101+0000 7fbc23d8c740 1 Processor -- start 2024-03-12T05:21:21.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.102+0000 7fbc23d8c740 1 -- start start 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 0x555f9a0cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f99fba130 con 0x555f9a0d6750 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc23d8c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a0c8bb0 con 0x555f9a0cd250 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35650/0 (socket says 172.21.15.45:35650) 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.103+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1468316127 learned_addr learned my addr 172.21.15.45:0/1468316127 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a0e2570 con 0x555f9a0cdb70 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 0x555f9a0cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1468316127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 msgr2=0x555f9a0d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1468316127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 msgr2=0x555f9a0cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 0x555f9a0cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/1468316127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f99fcfc20 con 0x555f9a0cd250 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 0x555f9a0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.104+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009810 tx=0x7fbc10019850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f490f02ce06e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.105+0000 7fbc00ff9640 1 -- 172.21.15.45:0/1468316127 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10019590 con 0x555f9a0cd250 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.105+0000 7fbc00ff9640 1 -- 172.21.15.45:0/1468316127 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc10019de0 con 0x555f9a0cd250 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.105+0000 7fbc00ff9640 1 -- 172.21.15.45:0/1468316127 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10005040 con 0x555f9a0cd250 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.106+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 msgr2=0x555f9a0d5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.106+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009810 tx=0x7fbc10019850 comp rx=0 tx=0).stop 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.107+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 shutdown_connections 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.107+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0d6750 0x555f9a0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.107+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0cdb70 0x555f9a0d3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.107+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/1468316127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0d5e20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.107+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 >> 172.21.15.45:0/1468316127 conn(0x555f9a0c7980 msgr2=0x555f9a0ed760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.108+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 shutdown_connections 2024-03-12T05:21:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.108+0000 7fbc23d8c740 1 -- 172.21.15.45:0/1468316127 wait complete. 2024-03-12T05:21:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.108+0000 7fbc23d8c740 1 -- start start 2024-03-12T05:21:21.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.108+0000 7fbc23d8c740 1 Processor -- start 2024-03-12T05:21:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc23d8c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cdb70 0x555f9a0dbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35652/0 (socket says 172.21.15.45:35652) 2024-03-12T05:21:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/648225328 learned_addr learned my addr 172.21.15.45:0/648225328 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0dc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.109+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a0dc950 con 0x555f9a0cdb70 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a0dcac0 con 0x555f9a0cd250 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f9a13cef0 con 0x555f9a0d6750 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cdb70 0x555f9a0dbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1b59e640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0dc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc23d8c740 10 client.0 _ll_register_callbacks cb 0x7fbc10009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 msgr2=0x555f9a0dc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0dc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/648225328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cdb70 msgr2=0x555f9a0dbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cdb70 0x555f9a0dbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc1ad9d640 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f9a0336a0 con 0x555f9a0cd250 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.110+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc1000fc90 con 0x555f9a0cd250 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbc1ad9d640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbc1001c040 tx=0x7fbc10005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4912cd93bea338b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbbfeff5640 10 client.27599 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbbfeff5640 1 -- 172.21.15.45:0/648225328 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000f920 con 0x555f9a0cd250 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbbfeff5640 1 -- 172.21.15.45:0/648225328 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc1000fa80 con 0x555f9a0cd250 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbbfeff5640 1 -- 172.21.15.45:0/648225328 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc100198c0 con 0x555f9a0cd250 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbc23d8c740 10 client.27599 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.111+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x555f99ff7060 con 0x555f9a0cd250 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbbfeff5640 1 -- 172.21.15.45:0/648225328 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc10019a20 con 0x555f9a0cd250 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 10 client.27599 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 10 client.27599 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 20 client.27599 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a13df20 0x555f9a140310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 4 client.27599 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:21.109 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555f99dc0890 con 0x555f9a13df20 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a13df20 0x555f9a140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbc1a59c640 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a13df20 0x555f9a140310 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c05924c737ad49a server_cookie=b33f36e6a960972e in_seq=0 out_seq=0 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.112+0000 7fbbfeff5640 10 client.27599 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbbfeff5640 1 -- 172.21.15.45:0/648225328 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555f99dc0890 con 0x555f9a13df20 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbbfeff5640 10 client.27599 handle_command_reply: tid=0 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 client.27599 shutdown 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 msgr2=0x555f9a0dd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbc1001c040 tx=0x7fbc10005dc0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 shutdown_connections 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555f9a13df20 0x555f9a140310 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555f9a0cdb70 0x555f9a0dbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.110 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbbfeff5640 0 client.27599 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f9a0d6750 0x555f9a0dc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 --2- 172.21.15.45:0/648225328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f9a0cd250 0x555f9a0dd9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 >> 172.21.15.45:0/648225328 conn(0x555f9a0c7980 msgr2=0x555f9a0d1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 shutdown_connections 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.113+0000 7fbc23d8c740 1 -- 172.21.15.45:0/648225328 wait complete. 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc23d8c740 20 client.27599 trim_cache size 0 max 16384 2024-03-12T05:21:21.111 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 msgr2=0x7fbc040a4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 0x7fbc040a4150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400be10 tx=0x7fbc14004730 comp rx=0 tx=0).stop 2024-03-12T05:21:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 msgr2=0x7fbc1c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10e000 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c02c6c0 tx=0x7fbc0c034740 comp rx=0 tx=0).stop 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 shutdown_connections 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc040a1c90 0x7fbc040a4150 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc1c1009e0 0x7fbc1c10e000 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc1c106100 0x7fbc1c10e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 --2- 172.21.15.45:0/244893797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc1c100040 0x7fbc1c10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.114+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 >> 172.21.15.45:0/244893797 conn(0x7fbc1c077110 msgr2=0x7fbc1c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.115+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 shutdown_connections 2024-03-12T05:21:21.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.115+0000 7fbc211bc640 1 -- 172.21.15.45:0/244893797 wait complete. 2024-03-12T05:21:21.124 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:21.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 -- 172.21.15.45:0/1170877606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 msgr2=0x7facd410de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 --2- 172.21.15.45:0/1170877606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 0x7facd410de50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7faccc009be0 tx=0x7faccc02bf10 comp rx=0 tx=0).stop 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 -- 172.21.15.45:0/1170877606 shutdown_connections 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 --2- 172.21.15.45:0/1170877606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 0x7facd410de50 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 --2- 172.21.15.45:0/1170877606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 0x7facd411cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 --2- 172.21.15.45:0/1170877606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.270+0000 7facdb5d0640 1 -- 172.21.15.45:0/1170877606 >> 172.21.15.45:0/1170877606 conn(0x7facd406c260 msgr2=0x7facd406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.272+0000 7facdb5d0640 1 -- 172.21.15.45:0/1170877606 shutdown_connections 2024-03-12T05:21:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.275+0000 7facdb5d0640 1 -- 172.21.15.45:0/1170877606 wait complete. 2024-03-12T05:21:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.276+0000 7facdb5d0640 1 Processor -- start 2024-03-12T05:21:21.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.276+0000 7facdb5d0640 1 -- start start 2024-03-12T05:21:21.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 0x7facd4113ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 0x7facd41185a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4118ae0 con 0x7facd410d9d0 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4118c50 con 0x7facd411a960 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facdb5d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4118e10 con 0x7facd410f9a0 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.277+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd9b6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 0x7facd41185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57110/0 (socket says 172.21.15.45:57110) 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 -- 172.21.15.45:0/2574719591 learned_addr learned my addr 172.21.15.45:0/2574719591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 -- 172.21.15.45:0/2574719591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 msgr2=0x7facd41185a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 0x7facd41185a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 -- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 msgr2=0x7facd4113ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 0x7facd4113ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 -- 172.21.15.45:0/2574719591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc009840 con 0x7facd410f9a0 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.278+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faccc036040 tx=0x7faccc033d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.279+0000 7facca7fc640 1 -- 172.21.15.45:0/2574719591 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc003b80 con 0x7facd410f9a0 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.279+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4119090 con 0x7facd410f9a0 2024-03-12T05:21:21.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.279+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd4119410 con 0x7facd410f9a0 2024-03-12T05:21:21.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.280+0000 7facca7fc640 1 -- 172.21.15.45:0/2574719591 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faccc003ce0 con 0x7facd410f9a0 2024-03-12T05:21:21.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.280+0000 7facca7fc640 1 -- 172.21.15.45:0/2574719591 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc034670 con 0x7facd410f9a0 2024-03-12T05:21:21.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.283+0000 7facde1a1740 1 Processor -- start 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 -- start start 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6b70 0x5581e53cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5381960 con 0x5581e53c6b70 2024-03-12T05:21:21.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5335310 con 0x5581e53cf750 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facde1a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5332840 con 0x5581e53c6250 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facca7fc640 1 -- 172.21.15.45:0/2574719591 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7faccc034890 con 0x7facd410f9a0 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facd9b6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facd9b6d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57124/0 (socket says 172.21.15.45:57124) 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.284+0000 7facd9b6d640 1 -- 172.21.15.45:0/3772584861 learned_addr learned my addr 172.21.15.45:0/3772584861 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd936c640 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6b70 0x5581e53cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facca7fc640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 0x7facc00a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 -- 172.21.15.45:0/3772584861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 msgr2=0x5581e53d1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facca7fc640 1 -- 172.21.15.45:0/2574719591 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faccc0f0400 con 0x7facd410f9a0 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 -- 172.21.15.45:0/3772584861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6b70 msgr2=0x5581e53cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6b70 0x5581e53cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 -- 172.21.15.45:0/3772584861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e52f0060 con 0x5581e53c6250 2024-03-12T05:21:21.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.285+0000 7facd9b6d640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 0x7facc00a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.287+0000 7facd9b6d640 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7facd0009b40 tx=0x7facd000e1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa6c874ccc9dfd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.288+0000 7facbf7fe640 1 -- 172.21.15.45:0/3772584861 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd001d400 con 0x5581e53c6250 2024-03-12T05:21:21.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.288+0000 7facd9b6d640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 0x7facc00a41a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7facd0020000 tx=0x7facd001d000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.288+0000 7facbf7fe640 1 -- 172.21.15.45:0/3772584861 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facd0018070 con 0x5581e53c6250 2024-03-12T05:21:21.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.288+0000 7facbf7fe640 1 -- 172.21.15.45:0/3772584861 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd00135d0 con 0x5581e53c6250 2024-03-12T05:21:21.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 -- 172.21.15.45:0/3772584861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 msgr2=0x5581e53c6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7facd0009b40 tx=0x7facd000e1d0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 -- 172.21.15.45:0/3772584861 shutdown_connections 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6b70 0x5581e53cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 --2- 172.21.15.45:0/3772584861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6250 0x5581e53c6630 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.289+0000 7facde1a1740 1 -- 172.21.15.45:0/3772584861 >> 172.21.15.45:0/3772584861 conn(0x5581e53c0980 msgr2=0x5581e53c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.290+0000 7facde1a1740 1 -- 172.21.15.45:0/3772584861 shutdown_connections 2024-03-12T05:21:21.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.290+0000 7facde1a1740 1 -- 172.21.15.45:0/3772584861 wait complete. 2024-03-12T05:21:21.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.290+0000 7facde1a1740 1 -- start start 2024-03-12T05:21:21.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.290+0000 7facde1a1740 1 Processor -- start 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.291+0000 7facde1a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.291+0000 7facde1a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.291+0000 7facde1a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53cf750 0x5581e53cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.291+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.291+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35678/0 (socket says 172.21.15.45:35678) 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 -- 172.21.15.45:0/723156050 learned_addr learned my addr 172.21.15.45:0/723156050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e532e020 con 0x5581e53c6250 2024-03-12T05:21:21.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5001430 con 0x5581e53c6b70 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e4fbde00 con 0x5581e53cf750 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd9b6d640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53cf750 0x5581e53cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd936c640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 -- 172.21.15.45:0/723156050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53cf750 msgr2=0x5581e53cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53cf750 0x5581e53cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 -- 172.21.15.45:0/723156050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 msgr2=0x5581e53cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd0009670 con 0x5581e53c6b70 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd936c640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facde1a1740 10 client.0 _ll_register_callbacks cb 0x5581e52d5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facd8b6b640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faccc0096f0 tx=0x7faccc004f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f602213e34cbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.292+0000 7facbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581e502d7e0 con 0x5581e53c6b70 2024-03-12T05:21:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facbcff9640 1 -- 172.21.15.45:0/723156050 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0c5850 con 0x5581e53c6b70 2024-03-12T05:21:21.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facbcff9640 1 -- 172.21.15.45:0/723156050 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faccc0edd40 con 0x5581e53c6b70 2024-03-12T05:21:21.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facbcff9640 1 -- 172.21.15.45:0/723156050 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faccc0edf00 con 0x5581e53c6b70 2024-03-12T05:21:21.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facde1a1740 10 client.27605 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.293+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5581e532c6a0 con 0x5581e53c6b70 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facbcff9640 1 -- 172.21.15.45:0/723156050 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faccc0f0070 con 0x5581e53c6b70 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 10 client.27605 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 10 client.27605 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 20 client.27605 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e53df600 0x5581e53e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 4 client.27605 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:21.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.294+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581e502d7e0 con 0x5581e53df600 2024-03-12T05:21:21.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.296+0000 7facd936c640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e53df600 0x5581e53e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.297+0000 7facd936c640 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e53df600 0x5581e53e1a60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8b9d33fc4d0d5598 server_cookie=2409e4a4fadd2ef6 in_seq=0 out_seq=0 2024-03-12T05:21:21.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.297+0000 7facbcff9640 10 client.27605 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:21.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facbcff9640 1 -- 172.21.15.45:0/723156050 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5581e502d7e0 con 0x5581e53df600 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facbcff9640 10 client.27605 handle_command_reply: tid=0 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 client.27605 shutdown 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 msgr2=0x5581e53ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faccc0096f0 tx=0x7faccc004f10 comp rx=0 tx=0).stop 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 shutdown_connections 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e53df600 0x5581e53e1a60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53cf750 0x5581e53cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53c6b70 0x5581e53ccbf0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 --2- 172.21.15.45:0/723156050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 >> 172.21.15.45:0/723156050 conn(0x5581e53c0980 msgr2=0x5581e53c26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facbcff9640 0 client.27605 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:21.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 shutdown_connections 2024-03-12T05:21:21.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.298+0000 7facde1a1740 1 -- 172.21.15.45:0/723156050 wait complete. 2024-03-12T05:21:21.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.299+0000 7facde1a1740 20 client.27605 trim_cache size 0 max 16384 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.319+0000 7facde1a1740 1 Processor -- start 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.319+0000 7facde1a1740 1 -- start start 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facd936c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e52b3130 con 0x5581e53c6250 2024-03-12T05:21:21.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facd936c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59772/0 (socket says 172.21.15.45:59772) 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facd936c640 1 -- 172.21.15.45:0/811952412 learned_addr learned my addr 172.21.15.45:0/811952412 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e53c13d0 con 0x5581e53cf750 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e53c04b0 con 0x5581e53c6b70 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facd936c640 1 -- 172.21.15.45:0/811952412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 msgr2=0x5581e53c79f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.320+0000 7facd936c640 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53c79f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.321+0000 7facd936c640 1 -- 172.21.15.45:0/811952412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 msgr2=0x5581e53c9910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:21.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.321+0000 7facd936c640 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53c9910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.321+0000 7facd936c640 1 -- 172.21.15.45:0/811952412 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e52c8c20 con 0x5581e53c6250 2024-03-12T05:21:21.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.321+0000 7facd936c640 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7facc4009ac0 tx=0x7facc4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a70ff6a778ce06 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.323+0000 7facbeffd640 1 -- 172.21.15.45:0/811952412 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4017040 con 0x5581e53c6250 2024-03-12T05:21:21.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.323+0000 7facbeffd640 1 -- 172.21.15.45:0/811952412 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facc401d070 con 0x5581e53c6250 2024-03-12T05:21:21.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.323+0000 7facbeffd640 1 -- 172.21.15.45:0/811952412 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4011440 con 0x5581e53c6250 2024-03-12T05:21:21.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.325+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 msgr2=0x5581e53cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.326+0000 7facde1a1740 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7facc4009ac0 tx=0x7facc4009600 comp rx=0 tx=0).stop 2024-03-12T05:21:21.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.328+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 shutdown_connections 2024-03-12T05:21:21.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.328+0000 7facde1a1740 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.328+0000 7facde1a1740 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.328+0000 7facde1a1740 1 --2- 172.21.15.45:0/811952412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53cede0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.328+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 >> 172.21.15.45:0/811952412 conn(0x5581e53c0980 msgr2=0x5581e53e63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.333+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 shutdown_connections 2024-03-12T05:21:21.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.333+0000 7facde1a1740 1 -- 172.21.15.45:0/811952412 wait complete. 2024-03-12T05:21:21.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.333+0000 7facde1a1740 1 -- start start 2024-03-12T05:21:21.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.333+0000 7facde1a1740 1 Processor -- start 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53d5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5435b50 con 0x5581e53c6250 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5435c90 con 0x5581e53cf750 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5435dd0 con 0x5581e53c6b70 2024-03-12T05:21:21.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facde1a1740 10 client.0 _ll_register_callbacks cb 0x7facd00093f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facd936c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53d5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.334+0000 7facd8b6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57136/0 (socket says 172.21.15.45:57136) 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 -- 172.21.15.45:0/2358512052 learned_addr learned my addr 172.21.15.45:0/2358512052 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 -- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 msgr2=0x5581e53d5bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d5bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 -- 172.21.15.45:0/2358512052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 msgr2=0x5581e53d5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53d5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 -- 172.21.15.45:0/2358512052 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc4009e70 con 0x5581e53c6b70 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facd8b6b640 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc0a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eed7480934328ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0d9070 con 0x5581e53c6b70 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faccc0d3070 con 0x5581e53c6b70 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.335+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facd0018070 con 0x5581e53c6b70 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faccc0b95f0 con 0x5581e53c6b70 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 10 client.27595 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5581e52ee8c0 con 0x5581e53c6b70 2024-03-12T05:21:21.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facd0018070 con 0x5581e53c6b70 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faccc03c070 con 0x5581e53c6b70 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 10 client.27595 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 10 client.27595 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 20 client.27595 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e54370c0 0x5581e54394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 4 client.27595 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.336+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5581e506a720 con 0x5581e54370c0 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.337+0000 7facd936c640 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e54370c0 0x5581e54394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.337+0000 7facd936c640 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e54370c0 0x5581e54394e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4aa8b6bbea485986 server_cookie=ffe77afa6cdb948a in_seq=0 out_seq=0 2024-03-12T05:21:21.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.337+0000 7facbcff9640 10 client.27595 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.337+0000 7facbcff9640 1 -- 172.21.15.45:0/2358512052 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5581e506a720 con 0x5581e54370c0 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facbcff9640 10 client.27595 handle_command_reply: tid=0 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 client.27595 shutdown 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 msgr2=0x5581e53d5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc0a8000 comp rx=0 tx=0).stop 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 shutdown_connections 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5581e54370c0 0x5581e54394e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581e53cf750 0x5581e53d5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581e53c6b70 0x5581e53d5670 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 --2- 172.21.15.45:0/2358512052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5581e53c6250 0x5581e53d5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 >> 172.21.15.45:0/2358512052 conn(0x5581e53c0980 msgr2=0x5581e53c79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facbcff9640 0 client.27595 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 shutdown_connections 2024-03-12T05:21:21.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 1 -- 172.21.15.45:0/2358512052 wait complete. 2024-03-12T05:21:21.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.338+0000 7facde1a1740 20 client.27595 trim_cache size 0 max 16384 2024-03-12T05:21:21.336 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:21.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 msgr2=0x7facc00a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 0x7facc00a41a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7facd0020000 tx=0x7facd001d000 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 msgr2=0x7facd4114220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faccc036040 tx=0x7faccc033d20 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 shutdown_connections 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facc00a1ce0 0x7facc00a41a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facd411a960 0x7facd41185a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7facd410f9a0 0x7facd4114220 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 --2- 172.21.15.45:0/2574719591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facd410d9d0 0x7facd4113ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 >> 172.21.15.45:0/2574719591 conn(0x7facd406c260 msgr2=0x7facd411c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 shutdown_connections 2024-03-12T05:21:21.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.339+0000 7facdb5d0640 1 -- 172.21.15.45:0/2574719591 wait complete. 2024-03-12T05:21:21.345 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:21.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 -- 172.21.15.45:0/3241978779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894108da0 msgr2=0x7f289410c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 --2- 172.21.15.45:0/3241978779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894108da0 0x7f289410c3b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2884009900 tx=0x7f288402bc40 comp rx=0 tx=0).stop 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 -- 172.21.15.45:0/3241978779 shutdown_connections 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 --2- 172.21.15.45:0/3241978779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894108da0 0x7f289410c3b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 --2- 172.21.15.45:0/3241978779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f289410cac0 0x7f2894079490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 --2- 172.21.15.45:0/3241978779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28941083f0 0x7f28941087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.519+0000 7f2898b48640 1 -- 172.21.15.45:0/3241978779 >> 172.21.15.45:0/3241978779 conn(0x7f289406c260 msgr2=0x7f289406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.520+0000 7f2898b48640 1 -- 172.21.15.45:0/3241978779 shutdown_connections 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.520+0000 7f2898b48640 1 -- 172.21.15.45:0/3241978779 wait complete. 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 Processor -- start 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 -- start start 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28941083f0 0x7f2894079190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894073830 0x7f2894073ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28941c3b20 con 0x7f2894073830 2024-03-12T05:21:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28941c3c60 con 0x7f289410cac0 2024-03-12T05:21:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.521+0000 7f2898b48640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28941c3dd0 con 0x7f28941083f0 2024-03-12T05:21:21.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f289259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28941083f0 0x7f2894079190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35716/0 (socket says 172.21.15.45:35716) 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 -- 172.21.15.45:0/3962534526 learned_addr learned my addr 172.21.15.45:0/3962534526 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2892d9d640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894073830 0x7f2894073ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 -- 172.21.15.45:0/3962534526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28941083f0 msgr2=0x7f2894079190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28941083f0 0x7f2894079190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 -- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894073830 msgr2=0x7f2894073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894073830 0x7f2894073ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.522+0000 7f2891d9b640 1 -- 172.21.15.45:0/3962534526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c007820 con 0x7f289410cac0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.523+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ab0 tx=0x7f2884005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.523+0000 7f28837fe640 1 -- 172.21.15.45:0/3962534526 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840043d0 con 0x7f289410cac0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.524+0000 7f28837fe640 1 -- 172.21.15.45:0/3962534526 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2884003aa0 con 0x7f289410cac0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.524+0000 7f28837fe640 1 -- 172.21.15.45:0/3962534526 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884005350 con 0x7f289410cac0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.524+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2884009560 con 0x7f289410cac0 2024-03-12T05:21:21.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.524+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28941c43a0 con 0x7f289410cac0 2024-03-12T05:21:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.527+0000 7f289b7fb740 1 Processor -- start 2024-03-12T05:21:21.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.527+0000 7f28837fe640 1 -- 172.21.15.45:0/3962534526 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f28840054b0 con 0x7f289410cac0 2024-03-12T05:21:21.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 -- start start 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 0x5616a10d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10d3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a1083960 con 0x5616a10c8b70 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a1037310 con 0x5616a10c8250 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289b7fb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a1034840 con 0x5616a10d1750 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f2892d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f2892d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35718/0 (socket says 172.21.15.45:35718) 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f2892d9d640 1 -- 172.21.15.45:0/2356023896 learned_addr learned my addr 172.21.15.45:0/2356023896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.528+0000 7f289259c640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 0x5616a10d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2891d9b640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10d3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 -- 172.21.15.45:0/2356023896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 msgr2=0x5616a10d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10d3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 -- 172.21.15.45:0/2356023896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 msgr2=0x5616a10d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 0x5616a10d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 -- 172.21.15.45:0/2356023896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616a0ff2060 con 0x5616a10c8250 2024-03-12T05:21:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f289259c640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 0x5616a10d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:21.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2892d9d640 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f288c0112f0 tx=0x7f288c0074e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af88f201d16fbbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2880ff9640 1 -- 172.21.15.45:0/2356023896 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c007e60 con 0x5616a10c8250 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2880ff9640 1 -- 172.21.15.45:0/2356023896 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f288c010040 con 0x5616a10c8250 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.529+0000 7f2880ff9640 1 -- 172.21.15.45:0/2356023896 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c004030 con 0x5616a10c8250 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f28837fe640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 0x7f287c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f28837fe640 1 -- 172.21.15.45:0/3962534526 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f28840efe50 con 0x7f289410cac0 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 -- 172.21.15.45:0/2356023896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 msgr2=0x5616a10c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289259c640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 0x7f287c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f288c0112f0 tx=0x7f288c0074e0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 -- 172.21.15.45:0/2356023896 shutdown_connections 2024-03-12T05:21:21.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8b70 0x5616a10d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10d3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2356023896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10c8630 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.530+0000 7f289b7fb740 1 -- 172.21.15.45:0/2356023896 >> 172.21.15.45:0/2356023896 conn(0x5616a10c2980 msgr2=0x5616a10c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.531+0000 7f289b7fb740 1 -- 172.21.15.45:0/2356023896 shutdown_connections 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.531+0000 7f289b7fb740 1 -- 172.21.15.45:0/2356023896 wait complete. 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.532+0000 7f289b7fb740 1 -- start start 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.532+0000 7f289259c640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 0x7f287c0a4160 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5616a10c7d20 tx=0x7f2888008040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.532+0000 7f289b7fb740 1 Processor -- start 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.533+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8250 0x5616a10ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.533+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.533+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.533+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.533+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35732/0 (socket says 172.21.15.45:35732) 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f2891d9b640 1 -- 172.21.15.45:0/4059247820 learned_addr learned my addr 172.21.15.45:0/4059247820 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f2892d9d640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a1030020 con 0x5616a10c8250 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a0d03430 con 0x5616a10c8b70 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a0cbfe00 con 0x5616a10d1750 2024-03-12T05:21:21.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f289b7fb740 10 client.0 _ll_register_callbacks cb 0x5616a0fd7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.534+0000 7f289259c640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8250 0x5616a10ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 -- 172.21.15.45:0/4059247820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 msgr2=0x5616a10cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 -- 172.21.15.45:0/4059247820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8250 msgr2=0x5616a10ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8250 0x5616a10ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c009de0 con 0x5616a10d1750 2024-03-12T05:21:21.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.535+0000 7f2892d9d640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10cf230 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f288c0112f0 tx=0x7f288c009970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ba98c1101b0f701 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f2879ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c004e50 con 0x5616a10d1750 2024-03-12T05:21:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f288c005040 con 0x5616a10d1750 2024-03-12T05:21:21.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616a0d2f7e0 con 0x5616a10d1750 2024-03-12T05:21:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f288c005200 con 0x5616a10d1750 2024-03-12T05:21:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.536+0000 7f289b7fb740 10 client.27607 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5616a102e6a0 con 0x5616a10d1750 2024-03-12T05:21:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f288c005810 con 0x5616a10d1750 2024-03-12T05:21:21.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f288c025020 con 0x5616a10d1750 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 10 client.27607 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 10 client.27607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 20 client.27607 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a10e1ad0 0x5616a10e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 4 client.27607 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.537+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616a0d2f7e0 con 0x5616a10e1ad0 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.538+0000 7f289259c640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a10e1ad0 0x5616a10e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.538+0000 7f2879ffb640 10 client.27607 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:21.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.538+0000 7f289259c640 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a10e1ad0 0x5616a10e3f10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2a8440a3f78d594 server_cookie=d8ca45572bf740a8 in_seq=0 out_seq=0 2024-03-12T05:21:21.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.539+0000 7f2879ffb640 1 -- 172.21.15.45:0/4059247820 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616a0d2f7e0 con 0x5616a10e1ad0 2024-03-12T05:21:21.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.539+0000 7f2879ffb640 10 client.27607 handle_command_reply: tid=0 2024-03-12T05:21:21.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.539+0000 7f289b7fb740 1 client.27607 shutdown 2024-03-12T05:21:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.540+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 msgr2=0x5616a10cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.540+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10cf230 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f288c0112f0 tx=0x7f288c009970 comp rx=0 tx=0).stop 2024-03-12T05:21:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 shutdown_connections 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a10e1ad0 0x5616a10e3f10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10d1750 0x5616a10cf230 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 --2- 172.21.15.45:0/4059247820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10c8250 0x5616a10ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 >> 172.21.15.45:0/4059247820 conn(0x5616a10c2980 msgr2=0x5616a10c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f2879ffb640 0 client.27607 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 shutdown_connections 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 1 -- 172.21.15.45:0/4059247820 wait complete. 2024-03-12T05:21:21.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.541+0000 7f289b7fb740 20 client.27607 trim_cache size 0 max 16384 2024-03-12T05:21:21.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.566+0000 7f289b7fb740 1 Processor -- start 2024-03-12T05:21:21.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.566+0000 7f289b7fb740 1 -- start start 2024-03-12T05:21:21.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8250 0x5616a10d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10e88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a0fb5130 con 0x5616a10d1750 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a10cfb00 con 0x5616a10c8b70 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f289b7fb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a10de690 con 0x5616a10c8250 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f2892d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f2892d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59810/0 (socket says 172.21.15.45:59810) 2024-03-12T05:21:21.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.567+0000 7f2892d9d640 1 -- 172.21.15.45:0/3412196254 learned_addr learned my addr 172.21.15.45:0/3412196254 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10e88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f289259c640 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8250 0x5616a10d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 -- 172.21.15.45:0/3412196254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8250 msgr2=0x5616a10d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8250 0x5616a10d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 -- 172.21.15.45:0/3412196254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 msgr2=0x5616a10c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.569+0000 7f2891d9b640 1 -- 172.21.15.45:0/3412196254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616a0fcac20 con 0x5616a10c8b70 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.570+0000 7f2891d9b640 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10e88b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2884000c00 tx=0x7f2884004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffbe38e9441fc876 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.570+0000 7f287bfff640 1 -- 172.21.15.45:0/3412196254 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840bc640 con 0x5616a10c8b70 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.570+0000 7f287bfff640 1 -- 172.21.15.45:0/3412196254 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f28840bc460 con 0x5616a10c8b70 2024-03-12T05:21:21.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.570+0000 7f287bfff640 1 -- 172.21.15.45:0/3412196254 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840c52a0 con 0x5616a10c8b70 2024-03-12T05:21:21.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.572+0000 7f289b7fb740 1 -- 172.21.15.45:0/3412196254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 msgr2=0x5616a10e88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.572+0000 7f289b7fb740 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10e88b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2884000c00 tx=0x7f2884004530 comp rx=0 tx=0).stop 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 -- 172.21.15.45:0/3412196254 shutdown_connections 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8b70 0x5616a10e88b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 --2- 172.21.15.45:0/3412196254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8250 0x5616a10d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 -- 172.21.15.45:0/3412196254 >> 172.21.15.45:0/3412196254 conn(0x5616a10c2980 msgr2=0x5616a10bda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 -- 172.21.15.45:0/3412196254 shutdown_connections 2024-03-12T05:21:21.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 -- 172.21.15.45:0/3412196254 wait complete. 2024-03-12T05:21:21.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.573+0000 7f289b7fb740 1 -- start start 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.574+0000 7f289b7fb740 1 Processor -- start 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.574+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.574+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.574+0000 7f289b7fb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35750/0 (socket says 172.21.15.45:35750) 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289b7fb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a10d8ee0 con 0x5616a10d1750 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f2891d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57168/0 (socket says 172.21.15.45:57168) 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289b7fb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a1138040 con 0x5616a10c8250 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616a11381d0 con 0x5616a10c8b70 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f2891d9b640 1 -- 172.21.15.45:0/2926426443 learned_addr learned my addr 172.21.15.45:0/2926426443 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289b7fb740 10 client.0 _ll_register_callbacks cb 0x7f288c009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 -- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 msgr2=0x5616a10d74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 -- 172.21.15.45:0/2926426443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 msgr2=0x5616a10d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.575+0000 7f289259c640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.576+0000 7f289259c640 1 -- 172.21.15.45:0/2926426443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28941c43a0 con 0x5616a10c8250 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.576+0000 7f2891d9b640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.576+0000 7f289259c640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x5616a10e4800 tx=0x7f28880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aec9a077b370bff server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.576+0000 7f2879ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888023840 con 0x5616a10c8250 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f28880239a0 con 0x5616a10c8250 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f288c010040 con 0x5616a10c8250 2024-03-12T05:21:21.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2888023b60 con 0x5616a10c8250 2024-03-12T05:21:21.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 10 client.27620 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5616a0ff08c0 con 0x5616a10c8250 2024-03-12T05:21:21.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f288803c020 con 0x5616a10c8250 2024-03-12T05:21:21.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2888016cc0 con 0x5616a10c8250 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 10 client.27620 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 10 client.27620 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 20 client.27620 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a1139a80 0x5616a113bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 4 client.27620 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.577+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616a0d6c720 con 0x5616a1139a80 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.578+0000 7f2891d9b640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a1139a80 0x5616a113bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.578+0000 7f2879ffb640 10 client.27620 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.578+0000 7f2891d9b640 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a1139a80 0x5616a113bea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ab14f3532ca0578 server_cookie=12fd3b5d12941503 in_seq=0 out_seq=0 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.578+0000 7f2879ffb640 1 -- 172.21.15.45:0/2926426443 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616a0d6c720 con 0x5616a1139a80 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.578+0000 7f2879ffb640 10 client.27620 handle_command_reply: tid=0 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 client.27620 shutdown 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 msgr2=0x5616a10d6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x5616a10e4800 tx=0x7f28880046c0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 shutdown_connections 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5616a1139a80 0x5616a113bea0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5616a10d1750 0x5616a10d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616a10c8b70 0x5616a10d74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 --2- 172.21.15.45:0/2926426443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616a10c8250 0x5616a10d6f90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f2879ffb640 0 client.27620 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 >> 172.21.15.45:0/2926426443 conn(0x5616a10c2980 msgr2=0x5616a10e88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 shutdown_connections 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 1 -- 172.21.15.45:0/2926426443 wait complete. 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.579+0000 7f289b7fb740 20 client.27620 trim_cache size 0 max 16384 2024-03-12T05:21:21.576 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.580+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 msgr2=0x7f287c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.580+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 0x7f287c0a4160 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5616a10c7d20 tx=0x7f2888008040 comp rx=0 tx=0).stop 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 msgr2=0x7f28940732f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ab0 tx=0x7f2884005e60 comp rx=0 tx=0).stop 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 shutdown_connections 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f287c0a1ca0 0x7f287c0a4160 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2894073830 0x7f2894073ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f289410cac0 0x7f28940732f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 --2- 172.21.15.45:0/3962534526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28941083f0 0x7f2894079190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.581+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 >> 172.21.15.45:0/3962534526 conn(0x7f289406c260 msgr2=0x7f2894071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.582+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 shutdown_connections 2024-03-12T05:21:21.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.582+0000 7f2898b48640 1 -- 172.21.15.45:0/3962534526 wait complete. 2024-03-12T05:21:21.596 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:21.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 -- 172.21.15.45:0/2175545259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102250 msgr2=0x7fa574102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/2175545259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102250 0x7fa574102630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa570009900 tx=0x7fa57002bcd0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 -- 172.21.15.45:0/2175545259 shutdown_connections 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/2175545259 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa574102b70 0x7fa57410b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/2175545259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa57410b670 0x7fa57410db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/2175545259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102250 0x7fa574102630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 -- 172.21.15.45:0/2175545259 >> 172.21.15.45:0/2175545259 conn(0x7fa5740fba90 msgr2=0x7fa5740fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 -- 172.21.15.45:0/2175545259 shutdown_connections 2024-03-12T05:21:21.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.771+0000 7fa57a6c3640 1 -- 172.21.15.45:0/2175545259 wait complete. 2024-03-12T05:21:21.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 Processor -- start 2024-03-12T05:21:21.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 -- start start 2024-03-12T05:21:21.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102b70 0x7fa574195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa57410b670 0x7fa574199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57419a2b0 con 0x7fa57410b670 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57419a420 con 0x7fa574102250 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa57a6c3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57419a5e0 con 0x7fa574102b70 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.773+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35778/0 (socket says 172.21.15.45:35778) 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 -- 172.21.15.45:0/1713901341 learned_addr learned my addr 172.21.15.45:0/1713901341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa579ec2640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa57410b670 0x7fa574199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa578ec0640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102b70 0x7fa574195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 -- 172.21.15.45:0/1713901341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102b70 msgr2=0x7fa574195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102b70 0x7fa574195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 -- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa57410b670 msgr2=0x7fa574199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa57410b670 0x7fa574199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 -- 172.21.15.45:0/1713901341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c009840 con 0x7fa574102250 2024-03-12T05:21:21.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.774+0000 7fa5796c1640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa570006010 tx=0x7fa570004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.775+0000 7fa56a7fc640 1 -- 172.21.15.45:0/1713901341 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570004440 con 0x7fa574102250 2024-03-12T05:21:21.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.775+0000 7fa56a7fc640 1 -- 172.21.15.45:0/1713901341 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa570003aa0 con 0x7fa574102250 2024-03-12T05:21:21.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.775+0000 7fa56a7fc640 1 -- 172.21.15.45:0/1713901341 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700053f0 con 0x7fa574102250 2024-03-12T05:21:21.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.775+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570009560 con 0x7fa574102250 2024-03-12T05:21:21.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.775+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57419aad0 con 0x7fa574102250 2024-03-12T05:21:21.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.778+0000 7fa56a7fc640 1 -- 172.21.15.45:0/1713901341 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa5700035f0 con 0x7fa574102250 2024-03-12T05:21:21.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.779+0000 7fa57e5d1740 1 Processor -- start 2024-03-12T05:21:21.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.779+0000 7fa57e5d1740 1 -- start start 2024-03-12T05:21:21.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.779+0000 7fa56a7fc640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 0x7fa5600a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa56a7fc640 1 -- 172.21.15.45:0/1713901341 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa5700f0020 con 0x7fa574102250 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d03c70 0x55c836d04050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa579ec2640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 0x7fa5600a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836cb6960 con 0x55c836d03c70 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836c6a310 con 0x55c836d0d170 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836c67840 con 0x55c836d04590 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa578ec0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa578ec0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57182/0 (socket says 172.21.15.45:57182) 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa578ec0640 1 -- 172.21.15.45:0/2258691140 learned_addr learned my addr 172.21.15.45:0/2258691140 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.780+0000 7fa5796c1640 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d03c70 0x55c836d04050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa579ec2640 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 -- 172.21.15.45:0/2258691140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 msgr2=0x55c836d0f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 -- 172.21.15.45:0/2258691140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d03c70 msgr2=0x55c836d04050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d03c70 0x55c836d04050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 -- 172.21.15.45:0/2258691140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c836c25060 con 0x55c836d04590 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa579ec2640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 0x7fa5600a4160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005fd0 tx=0x7fa56c013c90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa564005fd0 tx=0x7fa56400e2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49e4f55785bc0430 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.781+0000 7fa55f7fe640 1 -- 172.21.15.45:0/2258691140 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400ed20 con 0x55c836d04590 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.782+0000 7fa55f7fe640 1 -- 172.21.15.45:0/2258691140 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa56400ee80 con 0x55c836d04590 2024-03-12T05:21:21.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.782+0000 7fa55f7fe640 1 -- 172.21.15.45:0/2258691140 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564011470 con 0x55c836d04590 2024-03-12T05:21:21.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.782+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2258691140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 msgr2=0x55c836d0cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.782+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa564005fd0 tx=0x7fa56400e2f0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2258691140 shutdown_connections 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d0cc30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2258691140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d03c70 0x55c836d04050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2258691140 >> 172.21.15.45:0/2258691140 conn(0x55c836cfe3a0 msgr2=0x55c836cfe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2258691140 shutdown_connections 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2258691140 wait complete. 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 -- start start 2024-03-12T05:21:21.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.783+0000 7fa57e5d1740 1 Processor -- start 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d0a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836c63020 con 0x55c836d04590 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836936430 con 0x55c836d0d170 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8368f2e00 con 0x55c836d03c70 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa57e5d1740 10 client.0 _ll_register_callbacks cb 0x55c836c0acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57188/0 (socket says 172.21.15.45:57188) 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa5796c1640 1 -- 172.21.15.45:0/2934325577 learned_addr learned my addr 172.21.15.45:0/2934325577 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa579ec2640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.784+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d0a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 -- 172.21.15.45:0/2934325577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 msgr2=0x55c836d0abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 -- 172.21.15.45:0/2934325577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 msgr2=0x55c836d0a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d0a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 -- 172.21.15.45:0/2934325577 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5640097b0 con 0x55c836d03c70 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa579ec2640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa5796c1640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa570005940 tx=0x7fa5700bd500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24fb24cc346db033 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa55cff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700c5470 con 0x55c836d03c70 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8369627e0 con 0x55c836d03c70 2024-03-12T05:21:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa570005550 con 0x55c836d03c70 2024-03-12T05:21:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.785+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa570003c10 con 0x55c836d03c70 2024-03-12T05:21:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.786+0000 7fa57e5d1740 10 client.27622 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.786+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c836c616a0 con 0x55c836d03c70 2024-03-12T05:21:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.786+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa57009f020 con 0x55c836d03c70 2024-03-12T05:21:21.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.786+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa5700d0d20 con 0x55c836d03c70 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.786+0000 7fa57e5d1740 10 client.27622 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa57e5d1740 10 client.27622 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa57e5d1740 20 client.27622 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836d1d140 0x55c836d1f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa57e5d1740 4 client.27622 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c83699f720 con 0x55c836d1d140 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.787+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836d1d140 0x55c836d1f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa578ec0640 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836d1d140 0x55c836d1f530 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e14ef9dc8ac745d server_cookie=cb05d8d32da806bb in_seq=0 out_seq=0 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa55cff9640 10 client.27622 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa55cff9640 1 -- 172.21.15.45:0/2934325577 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c83699f720 con 0x55c836d1d140 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa55cff9640 10 client.27622 handle_command_reply: tid=0 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa57e5d1740 1 client.27622 shutdown 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 msgr2=0x55c836d0a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.788+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa570005940 tx=0x7fa5700bd500 comp rx=0 tx=0).stop 2024-03-12T05:21:21.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 shutdown_connections 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d0a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836d1d140 0x55c836d1f530 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d0abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa55cff9640 0 client.27622 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/2934325577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0a130 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 >> 172.21.15.45:0/2934325577 conn(0x55c836cfe3a0 msgr2=0x55c836d000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 shutdown_connections 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.789+0000 7fa57e5d1740 1 -- 172.21.15.45:0/2934325577 wait complete. 2024-03-12T05:21:21.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.790+0000 7fa57e5d1740 20 client.27622 trim_cache size 0 max 16384 2024-03-12T05:21:21.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.813+0000 7fa57e5d1740 1 Processor -- start 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.816+0000 7fa57e5d1740 1 -- start start 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d07390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836be8130 con 0x55c836d04590 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836cfdc70 con 0x55c836d0d170 2024-03-12T05:21:21.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836d0b2a0 con 0x55c836d03c70 2024-03-12T05:21:21.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35792/0 (socket says 172.21.15.45:35792) 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 -- 172.21.15.45:0/3747304261 learned_addr learned my addr 172.21.15.45:0/3747304261 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa578ec0640 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d07390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 -- 172.21.15.45:0/3747304261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 msgr2=0x55c836d0c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 -- 172.21.15.45:0/3747304261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 msgr2=0x55c836d07390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d07390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.817+0000 7fa579ec2640 1 -- 172.21.15.45:0/3747304261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c836bfdc20 con 0x55c836d0d170 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.818+0000 7fa579ec2640 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00fcb0 tx=0x7fa56c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2c3268c8926d543 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.818+0000 7fa55effd640 1 -- 172.21.15.45:0/3747304261 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c004550 con 0x55c836d0d170 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.818+0000 7fa55effd640 1 -- 172.21.15.45:0/3747304261 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa56c0046b0 con 0x55c836d0d170 2024-03-12T05:21:21.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.818+0000 7fa55effd640 1 -- 172.21.15.45:0/3747304261 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c02b410 con 0x55c836d0d170 2024-03-12T05:21:21.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 -- 172.21.15.45:0/3747304261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 msgr2=0x55c836d078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00fcb0 tx=0x7fa56c009500 comp rx=0 tx=0).stop 2024-03-12T05:21:21.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 -- 172.21.15.45:0/3747304261 shutdown_connections 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d04590 0x55c836d07390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d0d170 0x55c836d078d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/3747304261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d03c70 0x55c836d0c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.819+0000 7fa57e5d1740 1 -- 172.21.15.45:0/3747304261 >> 172.21.15.45:0/3747304261 conn(0x55c836cfe3a0 msgr2=0x55c836d05450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.820+0000 7fa57e5d1740 1 -- 172.21.15.45:0/3747304261 shutdown_connections 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.820+0000 7fa57e5d1740 1 -- 172.21.15.45:0/3747304261 wait complete. 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.820+0000 7fa57e5d1740 1 -- start start 2024-03-12T05:21:21.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 Processor -- start 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d6ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d0d170 0x55c836d695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836d6f410 con 0x55c836d0d170 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836d69af0 con 0x55c836d03c70 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836d69cb0 con 0x55c836d04590 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.821+0000 7fa57e5d1740 10 client.0 _ll_register_callbacks cb 0x7fa57406a270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa579ec2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d0d170 0x55c836d695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35808/0 (socket says 172.21.15.45:35808) 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 -- 172.21.15.45:0/491580317 learned_addr learned my addr 172.21.15.45:0/491580317 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa578ec0640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d6ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 -- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 msgr2=0x55c836d6ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d6ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 -- 172.21.15.45:0/491580317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d0d170 msgr2=0x55c836d695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d0d170 0x55c836d695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.822+0000 7fa5796c1640 1 -- 172.21.15.45:0/491580317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c00fa40 con 0x55c836d03c70 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa5796c1640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa570000c00 tx=0x7fa57002bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3243c8dd3b3a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa55cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700c5470 con 0x55c836d03c70 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa5700a0080 con 0x55c836d03c70 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa56400ee80 con 0x55c836d03c70 2024-03-12T05:21:21.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.823+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5700b8a70 con 0x55c836d03c70 2024-03-12T05:21:21.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 10 client.27638 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:21.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c836c238c0 con 0x55c836d03c70 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa56400ee80 con 0x55c836d03c70 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa5700d0d20 con 0x55c836d03c70 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 10 client.27638 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 10 client.27638 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 20 client.27638 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:21.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836cfadf0 0x55c836cfd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:21.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 4 client.27638 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:21.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.826+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c8369ee890 con 0x55c836cfadf0 2024-03-12T05:21:21.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.827+0000 7fa578ec0640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836cfadf0 0x55c836cfd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:21.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa55cff9640 10 client.27638 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:21.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa578ec0640 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836cfadf0 0x55c836cfd210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=834237d7d3b9bcf9 server_cookie=6175a63c60146d8e in_seq=0 out_seq=0 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa55cff9640 1 -- 172.21.15.45:0/491580317 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c8369ee890 con 0x55c836cfadf0 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa55cff9640 10 client.27638 handle_command_reply: tid=0 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 client.27638 shutdown 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 msgr2=0x55c836d6e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa570000c00 tx=0x7fa57002bcd0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 shutdown_connections 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c836d0d170 0x55c836d695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55c836cfadf0 0x55c836cfd210 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c836d04590 0x55c836d6ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 --2- 172.21.15.45:0/491580317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c836d03c70 0x55c836d6e7f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa55cff9640 0 client.27638 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:21.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.829+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 >> 172.21.15.45:0/491580317 conn(0x55c836cfe3a0 msgr2=0x55c836d0a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.830+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 shutdown_connections 2024-03-12T05:21:21.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.830+0000 7fa57e5d1740 1 -- 172.21.15.45:0/491580317 wait complete. 2024-03-12T05:21:21.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.830+0000 7fa57e5d1740 20 client.27638 trim_cache size 0 max 16384 2024-03-12T05:21:21.826 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:21.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.832+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 msgr2=0x7fa5600a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.832+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 0x7fa5600a4160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa56c005fd0 tx=0x7fa56c013c90 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.832+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 msgr2=0x7fa5741953f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.832+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa570006010 tx=0x7fa570004040 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 shutdown_connections 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa5600a1ca0 0x7fa5600a4160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa57410b670 0x7fa574199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa574102b70 0x7fa574195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 --2- 172.21.15.45:0/1713901341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa574102250 0x7fa5741953f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.833+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 >> 172.21.15.45:0/1713901341 conn(0x7fa5740fba90 msgr2=0x7fa5740fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.834+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 shutdown_connections 2024-03-12T05:21:21.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:21.835+0000 7fa57a6c3640 1 -- 172.21.15.45:0/1713901341 wait complete. 2024-03-12T05:21:21.847 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:21 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/278449088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:21.848 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:21.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:21:21.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:21 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/278449088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:21.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:21 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/278449088' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 -- 172.21.15.45:0/1163223731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c100c10 msgr2=0x7faf6c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 --2- 172.21.15.45:0/1163223731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c100c10 0x7faf6c104c50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7faf60009890 tx=0x7faf6002bb70 comp rx=0 tx=0).stop 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 -- 172.21.15.45:0/1163223731 shutdown_connections 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 --2- 172.21.15.45:0/1163223731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 --2- 172.21.15.45:0/1163223731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c100c10 0x7faf6c104c50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 --2- 172.21.15.45:0/1163223731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 0x7faf6c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 -- 172.21.15.45:0/1163223731 >> 172.21.15.45:0/1163223731 conn(0x7faf6c077180 msgr2=0x7faf6c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 -- 172.21.15.45:0/1163223731 shutdown_connections 2024-03-12T05:21:22.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.076+0000 7faf71f0d640 1 -- 172.21.15.45:0/1163223731 wait complete. 2024-03-12T05:21:22.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.077+0000 7faf71f0d640 1 Processor -- start 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 -- start start 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 0x7faf6c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 0x7faf6c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a3ca0 con 0x7faf6c100260 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf71f0d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a3e10 con 0x7faf6c199050 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf6affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf6affd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57240/0 (socket says 172.21.15.45:57240) 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.078+0000 7faf6affd640 1 -- 172.21.15.45:0/3298433987 learned_addr learned my addr 172.21.15.45:0/3298433987 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.079+0000 7faf6bfff640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 0x7faf6c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.079+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 0x7faf6c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.079+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a3fd0 con 0x7faf6c105330 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 -- 172.21.15.45:0/3298433987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 msgr2=0x7faf6c1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 0x7faf6c1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 -- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 msgr2=0x7faf6c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 0x7faf6c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 -- 172.21.15.45:0/3298433987 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60009490 con 0x7faf6c105330 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6bfff640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 0x7faf6c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf6affd640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faf60037040 tx=0x7faf60004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf68ff9640 1 -- 172.21.15.45:0/3298433987 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60034850 con 0x7faf6c105330 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf68ff9640 1 -- 172.21.15.45:0/3298433987 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf6002ce90 con 0x7faf6c105330 2024-03-12T05:21:22.076 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.080+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c1a4250 con 0x7faf6c105330 2024-03-12T05:21:22.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.081+0000 7faf68ff9640 1 -- 172.21.15.45:0/3298433987 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60032c60 con 0x7faf6c105330 2024-03-12T05:21:22.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.081+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c1a4750 con 0x7faf6c105330 2024-03-12T05:21:22.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.083+0000 7faf68ff9640 1 -- 172.21.15.45:0/3298433987 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7faf6003b040 con 0x7faf6c105330 2024-03-12T05:21:22.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 Processor -- start 2024-03-12T05:21:22.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 -- start start 2024-03-12T05:21:22.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36e5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3697960 con 0x55d6f36e5750 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f364b310 con 0x55d6f36dc250 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf74bba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3648840 con 0x55d6f36dcb70 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35824/0 (socket says 172.21.15.45:35824) 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.084+0000 7faf6bfff640 1 -- 172.21.15.45:0/3603403508 learned_addr learned my addr 172.21.15.45:0/3603403508 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6affd640 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 -- 172.21.15.45:0/3603403508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 msgr2=0x55d6f36e5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36e5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 -- 172.21.15.45:0/3603403508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 msgr2=0x55d6f36e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36e7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 -- 172.21.15.45:0/3603403508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6f3606060 con 0x55d6f36dc250 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6affd640 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf68ff9640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 0x7faf540a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf68ff9640 1 -- 172.21.15.45:0/3298433987 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faf600ef140 con 0x7faf6c105330 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6bfff640 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf640095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba586a231be1a519 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.085+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 0x7faf540a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.086+0000 7faf51ffb640 1 -- 172.21.15.45:0/3603403508 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64013040 con 0x55d6f36dc250 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.086+0000 7faf51ffb640 1 -- 172.21.15.45:0/3603403508 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf6400ddd0 con 0x55d6f36dc250 2024-03-12T05:21:22.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.086+0000 7faf51ffb640 1 -- 172.21.15.45:0/3603403508 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640165e0 con 0x55d6f36dc250 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 -- 172.21.15.45:0/3603403508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 msgr2=0x55d6f36dc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf640095c0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 0x7faf540a4160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55d6f36dbd20 tx=0x7faf5c005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 -- 172.21.15.45:0/3603403508 shutdown_connections 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 --2- 172.21.15.45:0/3603403508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dc250 0x55d6f36dc630 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 -- 172.21.15.45:0/3603403508 >> 172.21.15.45:0/3603403508 conn(0x55d6f36d6980 msgr2=0x55d6f36d6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 -- 172.21.15.45:0/3603403508 shutdown_connections 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.087+0000 7faf74bba740 1 -- 172.21.15.45:0/3603403508 wait complete. 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.088+0000 7faf74bba740 1 -- start start 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.088+0000 7faf74bba740 1 Processor -- start 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 0x55d6f36de620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3644020 con 0x55d6f36dc250 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 0x55d6f36de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 0x55d6f36de620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59916/0 (socket says 172.21.15.45:59916) 2024-03-12T05:21:22.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35840/0 (socket says 172.21.15.45:35840) 2024-03-12T05:21:22.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6b7fe640 1 -- 172.21.15.45:0/321783990 learned_addr learned my addr 172.21.15.45:0/321783990 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:22.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3317430 con 0x55d6f36e5750 2024-03-12T05:21:22.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf6affd640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f32d3e00 con 0x55d6f36dcb70 2024-03-12T05:21:22.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.089+0000 7faf74bba740 10 client.0 _ll_register_callbacks cb 0x55d6f35ebcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 -- 172.21.15.45:0/321783990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 msgr2=0x55d6f36deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36deb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 -- 172.21.15.45:0/321783990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 msgr2=0x55d6f36de620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 0x55d6f36de620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6400e520 con 0x55d6f36e5750 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf6bfff640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf64004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1732e3babf3ede81 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf4affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64004e40 con 0x55d6f36e5750 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faf64003970 con 0x55d6f36e5750 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.090+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6f33437e0 con 0x55d6f36e5750 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf64003b30 con 0x55d6f36e5750 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 10 client.27650 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:22.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d6f36426a0 con 0x55d6f36e5750 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf64026070 con 0x55d6f36e5750 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faf64017420 con 0x55d6f36e5750 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 10 client.27650 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 10 client.27650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 20 client.27650 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f36fe220 0x55d6f3700650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 4 client.27650 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.091+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6f33437e0 con 0x55d6f36fe220 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.092+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f36fe220 0x55d6f3700650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.092+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f36fe220 0x55d6f3700650 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e60b0736240dc0 server_cookie=20a53380f91c496d in_seq=0 out_seq=0 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.092+0000 7faf4affd640 10 client.27650 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.093+0000 7faf4affd640 1 -- 172.21.15.45:0/321783990 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d6f33437e0 con 0x55d6f36fe220 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.093+0000 7faf4affd640 10 client.27650 handle_command_reply: tid=0 2024-03-12T05:21:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.093+0000 7faf74bba740 1 client.27650 shutdown 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 msgr2=0x55d6f36df0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faf6400e870 tx=0x7faf64004cd0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 shutdown_connections 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f36fe220 0x55d6f3700650 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f36df0a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dcb70 0x55d6f36deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 --2- 172.21.15.45:0/321783990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dc250 0x55d6f36de620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 >> 172.21.15.45:0/321783990 conn(0x55d6f36d6980 msgr2=0x55d6f36d86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf4affd640 0 client.27650 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 shutdown_connections 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 1 -- 172.21.15.45:0/321783990 wait complete. 2024-03-12T05:21:22.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.094+0000 7faf74bba740 20 client.27650 trim_cache size 0 max 16384 2024-03-12T05:21:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.116+0000 7faf74bba740 1 Processor -- start 2024-03-12T05:21:22.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 -- start start 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dcb70 0x55d6f36eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36eb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f35c9130 con 0x55d6f36e5750 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f36d7980 con 0x55d6f36dcb70 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf74bba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f36e2c70 con 0x55d6f36dc250 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.117+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57268/0 (socket says 172.21.15.45:57268) 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 -- 172.21.15.45:0/1472203189 learned_addr learned my addr 172.21.15.45:0/1472203189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6affd640 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dcb70 0x55d6f36eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6bfff640 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 -- 172.21.15.45:0/1472203189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dcb70 msgr2=0x55d6f36eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dcb70 0x55d6f36eacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 -- 172.21.15.45:0/1472203189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 msgr2=0x55d6f36eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 -- 172.21.15.45:0/1472203189 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6f35dec20 con 0x55d6f36dc250 2024-03-12T05:21:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.118+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009fc0 tx=0x7faf5c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eccb1d5a62d51b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.119+0000 7faf517fa640 1 -- 172.21.15.45:0/1472203189 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c023030 con 0x55d6f36dc250 2024-03-12T05:21:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.119+0000 7faf517fa640 1 -- 172.21.15.45:0/1472203189 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf5c024040 con 0x55d6f36dc250 2024-03-12T05:21:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.119+0000 7faf517fa640 1 -- 172.21.15.45:0/1472203189 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c017070 con 0x55d6f36dc250 2024-03-12T05:21:22.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 -- 172.21.15.45:0/1472203189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 msgr2=0x55d6f36e4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009fc0 tx=0x7faf5c019000 comp rx=0 tx=0).stop 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 -- 172.21.15.45:0/1472203189 shutdown_connections 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36e5750 0x55d6f36eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36dcb70 0x55d6f36eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 --2- 172.21.15.45:0/1472203189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36e4950 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 -- 172.21.15.45:0/1472203189 >> 172.21.15.45:0/1472203189 conn(0x55d6f36d6980 msgr2=0x55d6f36dcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 -- 172.21.15.45:0/1472203189 shutdown_connections 2024-03-12T05:21:22.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.120+0000 7faf74bba740 1 -- 172.21.15.45:0/1472203189 wait complete. 2024-03-12T05:21:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.121+0000 7faf74bba740 1 -- start start 2024-03-12T05:21:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.121+0000 7faf74bba740 1 Processor -- start 2024-03-12T05:21:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.122+0000 7faf74bba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.122+0000 7faf74bba740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dcb70 0x55d6f36f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.122+0000 7faf74bba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f3754770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf74bba740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3754cb0 con 0x55d6f36dcb70 2024-03-12T05:21:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf74bba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3754df0 con 0x55d6f36e5750 2024-03-12T05:21:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf74bba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6f3754f30 con 0x55d6f36dc250 2024-03-12T05:21:22.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf74bba740 10 client.0 _ll_register_callbacks cb 0x7faf6400e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57278/0 (socket says 172.21.15.45:57278) 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 -- 172.21.15.45:0/869402702 learned_addr learned my addr 172.21.15.45:0/869402702 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 -- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 msgr2=0x55d6f3754770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f3754770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 -- 172.21.15.45:0/869402702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dcb70 msgr2=0x55d6f36f7c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dcb70 0x55d6f36f7c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.123+0000 7faf6b7fe640 1 -- 172.21.15.45:0/869402702 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c009da0 con 0x55d6f36dc250 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf6b7fe640 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009730 tx=0x7faf5c0053e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f2ebb44ad583ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf4affd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c023070 con 0x55d6f36dc250 2024-03-12T05:21:22.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf6400ddd0 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faf5c02a840 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf5c017050 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf74bba740 10 client.27646 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.124+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d6f36048c0 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faf6400ddd0 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faf5c031020 con 0x55d6f36dc250 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 10 client.27646 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 10 client.27646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 20 client.27646 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f37565f0 0x55d6f3758a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 4 client.27646 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:22.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d6f3380720 con 0x55d6f37565f0 2024-03-12T05:21:22.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.125+0000 7faf6affd640 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f37565f0 0x55d6f3758a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:22.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.126+0000 7faf6affd640 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f37565f0 0x55d6f3758a10 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b14fca8e95ba3bc2 server_cookie=64e0e9a8c09673ed in_seq=0 out_seq=0 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.126+0000 7faf4affd640 10 client.27646 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.126+0000 7faf4affd640 1 -- 172.21.15.45:0/869402702 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d6f3380720 con 0x55d6f37565f0 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf4affd640 10 client.27646 handle_command_reply: tid=0 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 client.27646 shutdown 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 msgr2=0x55d6f36f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009730 tx=0x7faf5c0053e0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 shutdown_connections 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d6f37565f0 0x55d6f3758a10 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d6f36dcb70 0x55d6f36f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6f36e5750 0x55d6f3754770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 --2- 172.21.15.45:0/869402702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6f36dc250 0x55d6f36f7700 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf4affd640 0 client.27646 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:22.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.127+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 >> 172.21.15.45:0/869402702 conn(0x55d6f36d6980 msgr2=0x55d6f36e1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.131+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 shutdown_connections 2024-03-12T05:21:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.131+0000 7faf74bba740 1 -- 172.21.15.45:0/869402702 wait complete. 2024-03-12T05:21:22.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.131+0000 7faf74bba740 20 client.27646 trim_cache size 0 max 16384 2024-03-12T05:21:22.126 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:22.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 msgr2=0x7faf540a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 0x7faf540a4160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55d6f36dbd20 tx=0x7faf5c005f90 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 msgr2=0x7faf6c198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faf60037040 tx=0x7faf60004360 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 shutdown_connections 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7faf540a1ca0 0x7faf540a4160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf6c199050 0x7faf6c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c105330 0x7faf6c198ac0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.132+0000 7faf71f0d640 1 --2- 172.21.15.45:0/3298433987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf6c100260 0x7faf6c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.133+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 >> 172.21.15.45:0/3298433987 conn(0x7faf6c077180 msgr2=0x7faf6c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.133+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 shutdown_connections 2024-03-12T05:21:22.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:22.133+0000 7faf71f0d640 1 -- 172.21.15.45:0/3298433987 wait complete. 2024-03-12T05:21:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:22 smithi045 ceph-mon[32174]: pgmap v327: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:22.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:22 smithi130 ceph-mon[35398]: pgmap v327: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:22.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:22 smithi116 ceph-mon[35486]: pgmap v327: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:23.146 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:21:23.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:21:23.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:21:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.349+0000 7f6173702640 1 -- 172.21.15.45:0/1503421991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c108da0 msgr2=0x7f616c10c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.349+0000 7f6173702640 1 --2- 172.21.15.45:0/1503421991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c108da0 0x7f616c10c3b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f615c009900 tx=0x7f615c02bc50 comp rx=0 tx=0).stop 2024-03-12T05:21:23.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.351+0000 7f6173702640 1 -- 172.21.15.45:0/1503421991 shutdown_connections 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.351+0000 7f6173702640 1 --2- 172.21.15.45:0/1503421991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c108da0 0x7f616c10c3b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.351+0000 7f6173702640 1 --2- 172.21.15.45:0/1503421991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f616c10cac0 0x7f616c079490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.351+0000 7f6173702640 1 --2- 172.21.15.45:0/1503421991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f616c1083f0 0x7f616c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.351+0000 7f6173702640 1 -- 172.21.15.45:0/1503421991 >> 172.21.15.45:0/1503421991 conn(0x7f616c06c260 msgr2=0x7f616c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.352+0000 7f6173702640 1 -- 172.21.15.45:0/1503421991 shutdown_connections 2024-03-12T05:21:23.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.352+0000 7f6173702640 1 -- 172.21.15.45:0/1503421991 wait complete. 2024-03-12T05:21:23.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 Processor -- start 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 -- start start 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f616c1083f0 0x7f616c083c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f616c10cac0 0x7f616c082000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1c3b20 con 0x7f616c082540 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1c3c60 con 0x7f616c10cac0 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.353+0000 7f6173702640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1c3da0 con 0x7f616c1083f0 2024-03-12T05:21:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59956/0 (socket says 172.21.15.45:59956) 2024-03-12T05:21:23.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 -- 172.21.15.45:0/2896096713 learned_addr learned my addr 172.21.15.45:0/2896096713 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 -- 172.21.15.45:0/2896096713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f616c1083f0 msgr2=0x7f616c083c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:21:23.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f616c1083f0 0x7f616c083c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 -- 172.21.15.45:0/2896096713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f616c10cac0 msgr2=0x7f616c082000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:21:23.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f616c10cac0 0x7f616c082000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.354+0000 7f6171c9f640 1 -- 172.21.15.45:0/2896096713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009840 con 0x7f616c082540 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f6171c9f640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f616800d950 tx=0x7f616800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f61627fc640 1 -- 172.21.15.45:0/2896096713 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168009a30 con 0x7f616c082540 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f61627fc640 1 -- 172.21.15.45:0/2896096713 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f61680044d0 con 0x7f616c082540 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f61627fc640 1 -- 172.21.15.45:0/2896096713 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168003c30 con 0x7f616c082540 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615c009560 con 0x7f616c082540 2024-03-12T05:21:23.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.356+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c1c4400 con 0x7f616c082540 2024-03-12T05:21:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.360+0000 7f61763b3740 1 Processor -- start 2024-03-12T05:21:23.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 -- start start 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ace2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ace8f0 0x561177aced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177a26f90 con 0x561177ab81e0 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177a244c0 con 0x561177ab8b90 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f61763b3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177a1fca0 con 0x561177ace8f0 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.361+0000 7f6170c9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59960/0 (socket says 172.21.15.45:59960) 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 -- 172.21.15.45:0/2447911846 learned_addr learned my addr 172.21.15.45:0/2447911846 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ace8f0 0x561177aced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f61627fc640 1 -- 172.21.15.45:0/2896096713 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6168003d90 con 0x7f616c082540 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 -- 172.21.15.45:0/2447911846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ace8f0 msgr2=0x561177aced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ace8f0 0x561177aced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.362+0000 7f6170c9d640 1 -- 172.21.15.45:0/2447911846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 msgr2=0x561177ace2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:23.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.363+0000 7f6170c9d640 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ace2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.363+0000 7f6170c9d640 1 -- 172.21.15.45:0/2447911846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611779e20b0 con 0x561177ab81e0 2024-03-12T05:21:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.363+0000 7f61627fc640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 0x7f61580a4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.363+0000 7f6170c9d640 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f615c009230 tx=0x7f615c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3588623943b08b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.363+0000 7f61627fc640 1 -- 172.21.15.45:0/2896096713 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f61680cd600 con 0x7f616c082540 2024-03-12T05:21:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.364+0000 7f61577fe640 1 -- 172.21.15.45:0/2447911846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c047070 con 0x561177ab81e0 2024-03-12T05:21:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.364+0000 7f61577fe640 1 -- 172.21.15.45:0/2447911846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f615c02cc40 con 0x561177ab81e0 2024-03-12T05:21:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.364+0000 7f61577fe640 1 -- 172.21.15.45:0/2447911846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0038e0 con 0x561177ab81e0 2024-03-12T05:21:23.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.364+0000 7f617149e640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 0x7f61580a4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 -- 172.21.15.45:0/2447911846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 msgr2=0x561177ab85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f615c009230 tx=0x7f615c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 -- 172.21.15.45:0/2447911846 shutdown_connections 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ace8f0 0x561177aced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ace2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 --2- 172.21.15.45:0/2447911846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab81e0 0x561177ab85c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.365+0000 7f61763b3740 1 -- 172.21.15.45:0/2447911846 >> 172.21.15.45:0/2447911846 conn(0x561177ab2840 msgr2=0x561177ab2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.366+0000 7f61763b3740 1 -- 172.21.15.45:0/2447911846 shutdown_connections 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.366+0000 7f617149e640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 0x7f61580a4190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561177ab8fd0 tx=0x7f6164009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.366+0000 7f61763b3740 1 -- 172.21.15.45:0/2447911846 wait complete. 2024-03-12T05:21:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.366+0000 7f61763b3740 1 -- start start 2024-03-12T05:21:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.366+0000 7f61763b3740 1 Processor -- start 2024-03-12T05:21:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 0x561177ad29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 0x561177ad2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611779c4f40 con 0x561177ab8b90 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611776f34b0 con 0x561177ace8f0 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611779a5670 con 0x561177ab81e0 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f61763b3740 10 client.0 _ll_register_callbacks cb 0x561177a12f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f617149e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f617149e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57310/0 (socket says 172.21.15.45:57310) 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f617149e640 1 -- 172.21.15.45:0/1696963092 learned_addr learned my addr 172.21.15.45:0/1696963092 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f6171c9f640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 0x561177ad2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.367+0000 7f6170c9d640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 0x561177ad29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 -- 172.21.15.45:0/1696963092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 msgr2=0x561177ad2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 -- 172.21.15.45:0/1696963092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 msgr2=0x561177ad29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 0x561177ad29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 -- 172.21.15.45:0/1696963092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c079d50 con 0x561177ace8f0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6170c9d640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 0x561177ad29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6171c9f640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 0x561177ad2ef0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f61680cacc0 tx=0x7f616809a9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba15777c465c0b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.368+0000 7f6154ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168096810 con 0x561177ace8f0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6168096970 con 0x561177ace8f0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611779f9f40 con 0x561177ace8f0 2024-03-12T05:21:23.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61680caf40 con 0x561177ace8f0 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f61763b3740 10 client.27659 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.369+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561177a1e320 con 0x561177ace8f0 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5611779f9f40 con 0x561177ace8f0 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6168004040 con 0x561177ace8f0 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 10 client.27659 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 10 client.27659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 20 client.27659 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177ae2f60 0x561177ae5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 4 client.27659 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:23.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561177a16310 con 0x561177ae2f60 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.370+0000 7f617149e640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177ae2f60 0x561177ae5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.371+0000 7f6154ff9640 10 client.27659 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.371+0000 7f617149e640 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177ae2f60 0x561177ae5350 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7daeb0f5390e90a1 server_cookie=b677ee939dc3397e in_seq=0 out_seq=0 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f6154ff9640 1 -- 172.21.15.45:0/1696963092 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561177a16310 con 0x561177ae2f60 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f6154ff9640 10 client.27659 handle_command_reply: tid=0 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 client.27659 shutdown 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 msgr2=0x561177ad2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 0x561177ad2ef0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f61680cacc0 tx=0x7f616809a9e0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 shutdown_connections 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177ae2f60 0x561177ae5350 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ab8b90 0x561177ad29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ace8f0 0x561177ad2ef0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 --2- 172.21.15.45:0/1696963092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 >> 172.21.15.45:0/1696963092 conn(0x561177ab2840 msgr2=0x561177ab47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.372+0000 7f6154ff9640 0 client.27659 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.373+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 shutdown_connections 2024-03-12T05:21:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.373+0000 7f61763b3740 1 -- 172.21.15.45:0/1696963092 wait complete. 2024-03-12T05:21:23.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.373+0000 7f61763b3740 20 client.27659 trim_cache size 0 max 16384 2024-03-12T05:21:23.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.406+0000 7f61763b3740 1 Processor -- start 2024-03-12T05:21:23.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.406+0000 7f61763b3740 1 -- start start 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ad2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177acc4d0 con 0x561177ace8f0 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177ab3b80 con 0x561177ab8b90 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f61763b3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177ab90e0 con 0x561177ab81e0 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f6171c9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f6171c9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59970/0 (socket says 172.21.15.45:59970) 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f6171c9f640 1 -- 172.21.15.45:0/3424168182 learned_addr learned my addr 172.21.15.45:0/3424168182 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.407+0000 7f6170c9d640 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ad2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 -- 172.21.15.45:0/3424168182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 msgr2=0x561177ad4b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 -- 172.21.15.45:0/3424168182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 msgr2=0x561177ad2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ad2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 -- 172.21.15.45:0/3424168182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611779bac00 con 0x561177ace8f0 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6170c9d640 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ad2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.408+0000 7f6171c9f640 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6168096b70 tx=0x7f616809e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c5fb37b2c8bae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.409+0000 7f6156ffd640 1 -- 172.21.15.45:0/3424168182 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168099690 con 0x561177ace8f0 2024-03-12T05:21:23.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.409+0000 7f6156ffd640 1 -- 172.21.15.45:0/3424168182 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6168099b00 con 0x561177ace8f0 2024-03-12T05:21:23.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.409+0000 7f6156ffd640 1 -- 172.21.15.45:0/3424168182 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168004040 con 0x561177ace8f0 2024-03-12T05:21:23.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 -- 172.21.15.45:0/3424168182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 msgr2=0x561177acfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6168096b70 tx=0x7f616809e2f0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 -- 172.21.15.45:0/3424168182 shutdown_connections 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177acfce0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab8b90 0x561177ad2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 --2- 172.21.15.45:0/3424168182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab81e0 0x561177ad4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 -- 172.21.15.45:0/3424168182 >> 172.21.15.45:0/3424168182 conn(0x561177ab2840 msgr2=0x561177ae9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 -- 172.21.15.45:0/3424168182 shutdown_connections 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.410+0000 7f61763b3740 1 -- 172.21.15.45:0/3424168182 wait complete. 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.411+0000 7f61763b3740 1 -- start start 2024-03-12T05:21:23.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.411+0000 7f61763b3740 1 Processor -- start 2024-03-12T05:21:23.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 0x561177ad8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 0x561177ad8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177ad8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177b39490 con 0x561177ace8f0 2024-03-12T05:21:23.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177b39600 con 0x561177ab81e0 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561177b397c0 con 0x561177ab8b90 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f61763b3740 10 client.0 _ll_register_callbacks cb 0x7f61640093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f617149e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 0x561177ad8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f617149e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 0x561177ad8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35912/0 (socket says 172.21.15.45:35912) 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f617149e640 1 -- 172.21.15.45:0/479599299 learned_addr learned my addr 172.21.15.45:0/479599299 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f6170c9d640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 0x561177ad8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.412+0000 7f6171c9f640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177ad8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 -- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 msgr2=0x561177ad8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 0x561177ad8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 -- 172.21.15.45:0/479599299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 msgr2=0x561177ad8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177ad8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 -- 172.21.15.45:0/479599299 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561177a1e320 con 0x561177ab8b90 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6171c9f640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177ad8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6154ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.413+0000 7f6170c9d640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 0x561177ad8980 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f615c02c730 tx=0x7f615c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=559f497ac2a68cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c047040 con 0x561177ab8b90 2024-03-12T05:21:23.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f615c0037f0 con 0x561177ab8b90 2024-03-12T05:21:23.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561177a45900 con 0x561177ab8b90 2024-03-12T05:21:23.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615c047040 con 0x561177ab8b90 2024-03-12T05:21:23.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f61763b3740 10 client.27658 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:23.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.414+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5611779e0910 con 0x561177ab8b90 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.416+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561177a45900 con 0x561177ab8b90 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f615c005280 con 0x561177ab8b90 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 10 client.27658 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 10 client.27658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 20 client.27658 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177b3abf0 0x561177b3d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 4 client.27658 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:21:23.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56117771f860 con 0x561177b3abf0 2024-03-12T05:21:23.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.417+0000 7f617149e640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177b3abf0 0x561177b3d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.418+0000 7f6154ff9640 10 client.27658 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.418+0000 7f617149e640 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177b3abf0 0x561177b3d010 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a88f2859fc890e server_cookie=eeab8baac612a762 in_seq=0 out_seq=0 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.418+0000 7f6154ff9640 1 -- 172.21.15.45:0/479599299 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56117771f860 con 0x561177b3abf0 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.418+0000 7f6154ff9640 10 client.27658 handle_command_reply: tid=0 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 client.27658 shutdown 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 msgr2=0x561177ad8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 0x561177ad8980 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f615c02c730 tx=0x7f615c004270 comp rx=0 tx=0).stop 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 shutdown_connections 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x561177b3abf0 0x561177b3d010 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561177ace8f0 0x561177ad8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561177ab8b90 0x561177ad8980 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 --2- 172.21.15.45:0/479599299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561177ab81e0 0x561177ad8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 >> 172.21.15.45:0/479599299 conn(0x561177ab2840 msgr2=0x561177ab3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f6154ff9640 0 client.27658 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 shutdown_connections 2024-03-12T05:21:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.419+0000 7f61763b3740 1 -- 172.21.15.45:0/479599299 wait complete. 2024-03-12T05:21:23.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.420+0000 7f61763b3740 20 client.27658 trim_cache size 0 max 16384 2024-03-12T05:21:23.416 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:23.416 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:21:23.416 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "e831b26d-e372-4564-94ba-aaa02aea8182", 2024-03-12T05:21:23.416 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:21:23.416 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.420+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 msgr2=0x7f61580a4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 0x7f61580a4190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561177ab8fd0 tx=0x7f6164009500 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 msgr2=0x7f616c0829d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f616800d950 tx=0x7f616800de10 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 shutdown_connections 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f61580a1cd0 0x7f61580a4190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616c082540 0x7f616c0829d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f616c10cac0 0x7f616c082000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 --2- 172.21.15.45:0/2896096713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f616c1083f0 0x7f616c083c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 >> 172.21.15.45:0/2896096713 conn(0x7f616c06c260 msgr2=0x7f616c070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 shutdown_connections 2024-03-12T05:21:23.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.421+0000 7f6173702640 1 -- 172.21.15.45:0/2896096713 wait complete. 2024-03-12T05:21:23.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:21:23.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 -- 172.21.15.45:0/1524610527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda010b500 msgr2=0x7fdda010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/1524610527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda010b500 0x7fdda010d9a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009ca0 tx=0x7fdd9802c180 comp rx=0 tx=0).stop 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 -- 172.21.15.45:0/1524610527 shutdown_connections 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/1524610527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda010b500 0x7fdda010d9a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/1524610527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda010aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/1524610527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda01020e0 0x7fdda01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 -- 172.21.15.45:0/1524610527 >> 172.21.15.45:0/1524610527 conn(0x7fdda00fb940 msgr2=0x7fdda00fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 -- 172.21.15.45:0/1524610527 shutdown_connections 2024-03-12T05:21:23.625 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.629+0000 7fdda4f2a640 1 -- 172.21.15.45:0/1524610527 wait complete. 2024-03-12T05:21:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 Processor -- start 2024-03-12T05:21:23.626 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 -- start start 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 0x7fdda01974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda019c3b0 con 0x7fdda010b500 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda019c520 con 0x7fdda0102a00 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdda4f2a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda019c6e0 con 0x7fdda01020e0 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdd9f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 0x7fdda01974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.630+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.628 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59986/0 (socket says 172.21.15.45:59986) 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9ffff640 1 -- 172.21.15.45:0/4122569859 learned_addr learned my addr 172.21.15.45:0/4122569859 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd97fff640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 -- 172.21.15.45:0/4122569859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 msgr2=0x7fdda0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 -- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 msgr2=0x7fdda019bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 -- 172.21.15.45:0/4122569859 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd98009900 con 0x7fdda01020e0 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd97fff640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 0x7fdda01974f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800b3f0 tx=0x7fdd8800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.631+0000 7fdd9d7fa640 1 -- 172.21.15.45:0/4122569859 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88011830 con 0x7fdda01020e0 2024-03-12T05:21:23.629 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.632+0000 7fdd9d7fa640 1 -- 172.21.15.45:0/4122569859 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdd88011e40 con 0x7fdda01020e0 2024-03-12T05:21:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.632+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda019c9c0 con 0x7fdda01020e0 2024-03-12T05:21:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.632+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda01a4240 con 0x7fdda01020e0 2024-03-12T05:21:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.632+0000 7fdd9d7fa640 1 -- 172.21.15.45:0/4122569859 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8800f2e0 con 0x7fdda01020e0 2024-03-12T05:21:23.630 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.635+0000 7fdd9d7fa640 1 -- 172.21.15.45:0/4122569859 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fdd88011990 con 0x7fdda01020e0 2024-03-12T05:21:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 Processor -- start 2024-03-12T05:21:23.631 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 -- start start 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a056f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a05702490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a05668280 con 0x562a057029d0 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a056657b0 con 0x562a056f94d0 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdda89fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a05660f90 con 0x562a056f9df0 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59994/0 (socket says 172.21.15.45:59994) 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdd97fff640 1 -- 172.21.15.45:0/606267309 learned_addr learned my addr 172.21.15.45:0/606267309 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.636+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a056f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a05702490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9ffff640 1 -- 172.21.15.45:0/606267309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 msgr2=0x562a05702490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a05702490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9ffff640 1 -- 172.21.15.45:0/606267309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 msgr2=0x562a05704dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9ffff640 1 -- 172.21.15.45:0/606267309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a0561f570 con 0x562a056f94d0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9d7fa640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 0x7fdd8c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd97fff640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.637+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 0x7fdd8c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.638+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a056f98b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdd98006010 tx=0x7fdd98005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcb48c082fedf8a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.638+0000 7fdd9f7fe640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 0x7fdd8c0a40e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x562a056f8fa0 tx=0x7fdd88095ac0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.638+0000 7fdd9d7fa640 1 -- 172.21.15.45:0/4122569859 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdd880cd1c0 con 0x7fdda01020e0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.638+0000 7fdd95ffb640 1 -- 172.21.15.45:0/606267309 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd98034640 con 0x562a056f94d0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.638+0000 7fdd95ffb640 1 -- 172.21.15.45:0/606267309 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdd98003e20 con 0x562a056f94d0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdd95ffb640 1 -- 172.21.15.45:0/606267309 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd98032d10 con 0x562a056f94d0 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 -- 172.21.15.45:0/606267309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 msgr2=0x562a056f98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a056f98b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdd98006010 tx=0x7fdd98005d80 comp rx=0 tx=0).stop 2024-03-12T05:21:23.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 -- 172.21.15.45:0/606267309 shutdown_connections 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a05704dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a05702490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 --2- 172.21.15.45:0/606267309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a056f98b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.639+0000 7fdda89fd740 1 -- 172.21.15.45:0/606267309 >> 172.21.15.45:0/606267309 conn(0x562a056f3c00 msgr2=0x562a056f3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.640+0000 7fdda89fd740 1 -- 172.21.15.45:0/606267309 shutdown_connections 2024-03-12T05:21:23.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.640+0000 7fdda89fd740 1 -- 172.21.15.45:0/606267309 wait complete. 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.640+0000 7fdda89fd740 1 -- start start 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 Processor -- start 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a056ff7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a056ffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a05334430 con 0x562a056f94d0 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a052f0e00 con 0x562a057029d0 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a055e5fd0 con 0x562a056f9df0 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdda89fd740 10 client.0 _ll_register_callbacks cb 0x562a0563bfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60000/0 (socket says 172.21.15.45:60000) 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdd9ffff640 1 -- 172.21.15.45:0/1711990170 learned_addr learned my addr 172.21.15.45:0/1711990170 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a056ff7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.641+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a056ffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 -- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 msgr2=0x562a056ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a056ffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 -- 172.21.15.45:0/1711990170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 msgr2=0x562a05702410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 -- 172.21.15.45:0/1711990170 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda010e620 con 0x562a056f9df0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a056ff7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdd90012cf0 tx=0x7fdd9000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed292a851eac1ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd7effd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x562a056f9df0 2024-03-12T05:21:23.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a053607e0 con 0x562a056f9df0 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.642+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdd900120d0 con 0x562a056f9df0 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd90012290 con 0x562a056f9df0 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 10 client.27670 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562a0565f610 con 0x562a056f9df0 2024-03-12T05:21:23.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562a053607e0 con 0x562a056f9df0 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdd90003c30 con 0x562a056f9df0 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 10 client.27670 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 10 client.27670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 20 client.27670 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a0571b430 0x562a0571d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 4 client.27670 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.643+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a0539d720 con 0x562a0571b430 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.644+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a0571b430 0x562a0571d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.646+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a0571b430 0x562a0571d850 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf33cff16892f5d server_cookie=b1e871295f1bb62d in_seq=0 out_seq=0 2024-03-12T05:21:23.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.646+0000 7fdd7effd640 10 client.27670 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdd7effd640 1 -- 172.21.15.45:0/1711990170 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562a0539d720 con 0x562a0571b430 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdd7effd640 10 client.27670 handle_command_reply: tid=0 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 client.27670 shutdown 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 msgr2=0x562a056ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a056ff7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdd90012cf0 tx=0x7fdd9000f790 comp rx=0 tx=0).stop 2024-03-12T05:21:23.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 shutdown_connections 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a0571b430 0x562a0571d850 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a056ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a056ff7a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1711990170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f94d0 0x562a05702410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 >> 172.21.15.45:0/1711990170 conn(0x562a056f3c00 msgr2=0x562a056f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.647+0000 7fdd7effd640 0 client.27670 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.648+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 shutdown_connections 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.648+0000 7fdda89fd740 1 -- 172.21.15.45:0/1711990170 wait complete. 2024-03-12T05:21:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.648+0000 7fdda89fd740 20 client.27670 trim_cache size 0 max 16384 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 Processor -- start 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 -- start start 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a05716c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a056faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a05716c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57382/0 (socket says 172.21.15.45:57382) 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a05716c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:35974/0 (socket says 172.21.15.45:35974) 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdd97fff640 1 -- 172.21.15.45:0/1044469619 learned_addr learned my addr 172.21.15.45:0/1044469619 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.675+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a056f4e30 con 0x562a057029d0 2024-03-12T05:21:23.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.676+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a056f8850 con 0x562a056f94d0 2024-03-12T05:21:23.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.676+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a056fdcf0 con 0x562a056f9df0 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.676+0000 7fdd97fff640 1 -- 172.21.15.45:0/1044469619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 msgr2=0x562a05716c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.676+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a05716c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.676+0000 7fdd97fff640 1 -- 172.21.15.45:0/1044469619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 msgr2=0x562a056faad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.677+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a056faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.677+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a056faad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.677+0000 7fdd97fff640 1 -- 172.21.15.45:0/1044469619 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a055fbac0 con 0x562a056f9df0 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.677+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a056faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.677+0000 7fdd97fff640 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdd90000c00 tx=0x7fdd9001c550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8fe1cbe0235d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.678+0000 7fdd957fa640 1 -- 172.21.15.45:0/1044469619 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017070 con 0x562a056f9df0 2024-03-12T05:21:23.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.678+0000 7fdd957fa640 1 -- 172.21.15.45:0/1044469619 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdd90018040 con 0x562a056f9df0 2024-03-12T05:21:23.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.678+0000 7fdd957fa640 1 -- 172.21.15.45:0/1044469619 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90005460 con 0x562a056f9df0 2024-03-12T05:21:23.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 msgr2=0x562a057221b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdd90000c00 tx=0x7fdd9001c550 comp rx=0 tx=0).stop 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 shutdown_connections 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a057029d0 0x562a056faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f9df0 0x562a057221b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 --2- 172.21.15.45:0/1044469619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a056f94d0 0x562a05716c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.679+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 >> 172.21.15.45:0/1044469619 conn(0x562a056f3c00 msgr2=0x562a056eebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.680+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 shutdown_connections 2024-03-12T05:21:23.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.680+0000 7fdda89fd740 1 -- 172.21.15.45:0/1044469619 wait complete. 2024-03-12T05:21:23.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.681+0000 7fdda89fd740 1 -- start start 2024-03-12T05:21:23.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.681+0000 7fdda89fd740 1 Processor -- start 2024-03-12T05:21:23.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.681+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f94d0 0x562a057123e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a0570ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0570f260 con 0x562a056f9df0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0570f3d0 con 0x562a057029d0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0570f590 con 0x562a056f94d0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdda89fd740 10 client.0 _ll_register_callbacks cb 0x7fdd90005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f94d0 0x562a057123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60020/0 (socket says 172.21.15.45:60020) 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 -- 172.21.15.45:0/2549000180 learned_addr learned my addr 172.21.15.45:0/2549000180 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a0570ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 -- 172.21.15.45:0/2549000180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f94d0 msgr2=0x562a057123e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f94d0 0x562a057123e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 -- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 msgr2=0x562a0570ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a0570ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd97fff640 1 -- 172.21.15.45:0/2549000180 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562a0565f610 con 0x562a056f9df0 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.682+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a0570ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdd97fff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdd90014000 tx=0x7fdd900051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec96d481125b351b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdd7effd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90017040 con 0x562a056f9df0 2024-03-12T05:21:23.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd98003e20 con 0x562a056f9df0 2024-03-12T05:21:23.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdd900055c0 con 0x562a056f9df0 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd900122c0 con 0x562a056f9df0 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdda89fd740 10 client.18195 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.683+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562a056216a0 con 0x562a056f9df0 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdd98003e20 con 0x562a056f9df0 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdd90010430 con 0x562a056f9df0 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 10 client.18195 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 10 client.18195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 20 client.18195 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a05772ad0 0x562a05774ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 4 client.18195 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:21:23.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.684+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562a053ec890 con 0x562a05772ad0 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.685+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a05772ad0 0x562a05774ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.685+0000 7fdd7effd640 10 client.18195 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.685+0000 7fdd9ffff640 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a05772ad0 0x562a05774ec0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7970f96dea4d289c server_cookie=f806a0fd164fd0ae in_seq=0 out_seq=0 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdd7effd640 1 -- 172.21.15.45:0/2549000180 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562a053ec890 con 0x562a05772ad0 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdd7effd640 10 client.18195 handle_command_reply: tid=0 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 client.18195 shutdown 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 msgr2=0x562a0570e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdd90014000 tx=0x7fdd900051d0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 shutdown_connections 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x562a05772ad0 0x562a05774ec0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a056f9df0 0x562a0570e790 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdd7effd640 0 client.18195 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:23.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a057029d0 0x562a0570ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 --2- 172.21.15.45:0/2549000180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a056f94d0 0x562a057123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 >> 172.21.15.45:0/2549000180 conn(0x562a056f3c00 msgr2=0x562a057221b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.686+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 shutdown_connections 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.687+0000 7fdda89fd740 1 -- 172.21.15.45:0/2549000180 wait complete. 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.687+0000 7fdda89fd740 20 client.18195 trim_cache size 0 max 16384 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "e831b26d-e372-4564-94ba-aaa02aea8182": { 2024-03-12T05:21:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:21:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "e831b26d-e372-4564-94ba-aaa02aea8182", 2024-03-12T05:21:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:21:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:21:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:21:23.685 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 msgr2=0x7fdd8c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 0x7fdd8c0a40e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x562a056f8fa0 tx=0x7fdd88095ac0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 msgr2=0x7fdda01974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 0x7fdda01974f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800b3f0 tx=0x7fdd8800b8b0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 shutdown_connections 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdd8c0a1c20 0x7fdd8c0a40e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdda010b500 0x7fdda019bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdda0102a00 0x7fdda0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 --2- 172.21.15.45:0/4122569859 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdda01020e0 0x7fdda01974f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 >> 172.21.15.45:0/4122569859 conn(0x7fdda00fb940 msgr2=0x7fdda00fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 shutdown_connections 2024-03-12T05:21:23.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:23.688+0000 7fdda4f2a640 1 -- 172.21.15.45:0/4122569859 wait complete. 2024-03-12T05:21:23.703 INFO:tasks.cephfs.filesystem:scrub status for tag:e831b26d-e372-4564-94ba-aaa02aea8182 - {'path': '/', 'tag': 'e831b26d-e372-4564-94ba-aaa02aea8182', 'options': 'recursive,force'} 2024-03-12T05:21:24.001 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: pgmap v328: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:24.001 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.001 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.001 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:24.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:24 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: pgmap v328: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:24.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:24 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: pgmap v328: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:24.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:24 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:24.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 560.0 s, 1186.4 tps, lat 84.948 ms stddev 65.344 2024-03-12T05:21:25.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:21:25.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:25 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:21:25.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:21:25.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:25 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:25.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:25.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: Adjusting osd_memory_target on smithi045 to 177.2M 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: Unable to set osd_memory_target on smithi045 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:21:25.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:25 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:21:26.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:26 smithi045 ceph-mon[32174]: pgmap v329: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:26.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:26 smithi130 ceph-mon[35398]: pgmap v329: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:26.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:26 smithi116 ceph-mon[35486]: pgmap v329: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:28.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:28 smithi045 ceph-mon[32174]: pgmap v330: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:28 smithi130 ceph-mon[35398]: pgmap v330: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:28 smithi116 ceph-mon[35486]: pgmap v330: 97 pgs: 97 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-03-12T05:21:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 565.0 s, 1487.6 tps, lat 67.343 ms stddev 39.304 2024-03-12T05:21:30.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:30 smithi045 ceph-mon[32174]: pgmap v331: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 21 MiB/s wr, 1.46k op/s 2024-03-12T05:21:30.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:30 smithi130 ceph-mon[35398]: pgmap v331: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 21 MiB/s wr, 1.46k op/s 2024-03-12T05:21:30.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:30 smithi116 ceph-mon[35486]: pgmap v331: 97 pgs: 97 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 21 MiB/s wr, 1.46k op/s 2024-03-12T05:21:32.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:32 smithi045 ceph-mon[32174]: pgmap v332: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-03-12T05:21:32.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:32 smithi130 ceph-mon[35398]: pgmap v332: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-03-12T05:21:32.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:32 smithi116 ceph-mon[35486]: pgmap v332: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-03-12T05:21:34.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:34 smithi045 ceph-mon[32174]: pgmap v333: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:21:34.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:34 smithi130 ceph-mon[35398]: pgmap v333: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:21:34.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:34 smithi116 ceph-mon[35486]: pgmap v333: 97 pgs: 97 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:21:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 570.0 s, 1047.3 tps, lat 68.163 ms stddev 35.410 2024-03-12T05:21:36.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:36 smithi045 ceph-mon[32174]: pgmap v334: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 28 MiB/s wr, 1.83k op/s 2024-03-12T05:21:36.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:36 smithi130 ceph-mon[35398]: pgmap v334: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 28 MiB/s wr, 1.83k op/s 2024-03-12T05:21:36.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:36 smithi116 ceph-mon[35486]: pgmap v334: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 28 MiB/s wr, 1.83k op/s 2024-03-12T05:21:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:38 smithi045 ceph-mon[32174]: pgmap v335: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-03-12T05:21:38.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:38.366 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:38.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:38 smithi130 ceph-mon[35398]: pgmap v335: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-03-12T05:21:38.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:38.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:38.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:38 smithi116 ceph-mon[35486]: pgmap v335: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 20 MiB/s wr, 1.30k op/s 2024-03-12T05:21:38.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:38.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:39.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:39 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:39.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:39 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:21:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:39 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:39 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:21:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:39 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:39 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:21:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 575.0 s, 524.9 tps, lat 245.127 ms stddev 947.109 2024-03-12T05:21:40.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:40 smithi130 ceph-mon[35398]: pgmap v336: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-03-12T05:21:40.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:40 smithi116 ceph-mon[35486]: pgmap v336: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-03-12T05:21:40.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:40 smithi045 ceph-mon[32174]: pgmap v336: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-03-12T05:21:42.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:42 smithi045 ceph-mon[32174]: pgmap v337: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 14 MiB/s wr, 939 op/s 2024-03-12T05:21:42.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:42 smithi130 ceph-mon[35398]: pgmap v337: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 14 MiB/s wr, 939 op/s 2024-03-12T05:21:42.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:42 smithi116 ceph-mon[35486]: pgmap v337: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 14 MiB/s wr, 939 op/s 2024-03-12T05:21:44.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:44 smithi130 ceph-mon[35398]: pgmap v338: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 10 MiB/s wr, 669 op/s 2024-03-12T05:21:44.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:44 smithi116 ceph-mon[35486]: pgmap v338: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 10 MiB/s wr, 669 op/s 2024-03-12T05:21:44.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:44 smithi045 ceph-mon[32174]: pgmap v338: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 10 MiB/s wr, 669 op/s 2024-03-12T05:21:44.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 580.0 s, 1325.9 tps, lat 75.243 ms stddev 60.009 2024-03-12T05:21:46.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:46 smithi130 ceph-mon[35398]: pgmap v339: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-03-12T05:21:46.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:46 smithi116 ceph-mon[35486]: pgmap v339: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-03-12T05:21:46.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:46 smithi045 ceph-mon[32174]: pgmap v339: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 20 MiB/s wr, 1.28k op/s 2024-03-12T05:21:48.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:48 smithi130 ceph-mon[35398]: pgmap v340: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 9.7 MiB/s wr, 623 op/s 2024-03-12T05:21:48.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:48 smithi116 ceph-mon[35486]: pgmap v340: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 9.7 MiB/s wr, 623 op/s 2024-03-12T05:21:48.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:48 smithi045 ceph-mon[32174]: pgmap v340: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 9.7 MiB/s wr, 623 op/s 2024-03-12T05:21:49.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 585.0 s, 1486.1 tps, lat 67.273 ms stddev 25.141 2024-03-12T05:21:50.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:50 smithi130 ceph-mon[35398]: pgmap v341: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 14 MiB/s wr, 899 op/s 2024-03-12T05:21:50.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:50 smithi116 ceph-mon[35486]: pgmap v341: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 14 MiB/s wr, 899 op/s 2024-03-12T05:21:50.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:50 smithi045 ceph-mon[32174]: pgmap v341: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 14 MiB/s wr, 899 op/s 2024-03-12T05:21:52.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:52 smithi130 ceph-mon[35398]: pgmap v342: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:52.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:52 smithi116 ceph-mon[35486]: pgmap v342: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:52.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:52 smithi045 ceph-mon[32174]: pgmap v342: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:53.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:21:53.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:21:53.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.885+0000 7f93d41a1640 1 -- 172.21.15.45:0/2484339673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc108100 msgr2=0x7f93cc1145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.885+0000 7f93d41a1640 1 --2- 172.21.15.45:0/2484339673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc108100 0x7f93cc1145c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009be0 tx=0x7f93c402bf80 comp rx=0 tx=0).stop 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 -- 172.21.15.45:0/2484339673 shutdown_connections 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 --2- 172.21.15.45:0/2484339673 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc107760 0x7f93cc107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 --2- 172.21.15.45:0/2484339673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc108100 0x7f93cc1145c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 --2- 172.21.15.45:0/2484339673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc10c640 0x7f93cc10ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 -- 172.21.15.45:0/2484339673 >> 172.21.15.45:0/2484339673 conn(0x7f93cc076fe0 msgr2=0x7f93cc105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 -- 172.21.15.45:0/2484339673 shutdown_connections 2024-03-12T05:21:53.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.886+0000 7f93d41a1640 1 -- 172.21.15.45:0/2484339673 wait complete. 2024-03-12T05:21:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.887+0000 7f93d41a1640 1 Processor -- start 2024-03-12T05:21:53.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.887+0000 7f93d41a1640 1 -- start start 2024-03-12T05:21:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc108100 0x7f93cc1a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 0x7f93cc19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc19e0e0 con 0x7f93cc108100 2024-03-12T05:21:53.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d1f3d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d1f3d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51360/0 (socket says 172.21.15.45:51360) 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d1f3d640 1 -- 172.21.15.45:0/3323999426 learned_addr learned my addr 172.21.15.45:0/3323999426 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d273e640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 0x7f93cc19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d173c640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc108100 0x7f93cc1a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc19e250 con 0x7f93cc10c640 2024-03-12T05:21:53.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.888+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc19e410 con 0x7f93cc107760 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 -- 172.21.15.45:0/3323999426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 msgr2=0x7f93cc19da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 0x7f93cc19da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 -- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc108100 msgr2=0x7f93cc1a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc108100 0x7f93cc1a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 -- 172.21.15.45:0/3323999426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4009840 con 0x7f93cc107760 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d273e640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 0x7f93cc19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:53.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e990 tx=0x7f93c800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.889+0000 7f93c2ffd640 1 -- 172.21.15.45:0/3323999426 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c800cbe0 con 0x7f93cc107760 2024-03-12T05:21:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.890+0000 7f93c2ffd640 1 -- 172.21.15.45:0/3323999426 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93c800cd40 con 0x7f93cc107760 2024-03-12T05:21:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.890+0000 7f93c2ffd640 1 -- 172.21.15.45:0/3323999426 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c8011470 con 0x7f93cc107760 2024-03-12T05:21:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.890+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc19e6f0 con 0x7f93cc107760 2024-03-12T05:21:53.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.890+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc1aa980 con 0x7f93cc107760 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.894+0000 7f93d6d7b740 1 Processor -- start 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.894+0000 7f93d6d7b740 1 -- start start 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93c2ffd640 1 -- 172.21.15.45:0/3323999426 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f93c8004eb0 con 0x7f93cc107760 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafee7b0 0x560caaff0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaf54280 con 0x560caafe5bd0 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaf517b0 con 0x560caafe52b0 2024-03-12T05:21:53.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaf4cf90 con 0x560caafee7b0 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d273e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d273e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60622/0 (socket says 172.21.15.45:60622) 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d273e640 1 -- 172.21.15.45:0/250790557 learned_addr learned my addr 172.21.15.45:0/250790557 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d173c640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafee7b0 0x560caaff0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.895+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 -- 172.21.15.45:0/250790557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafee7b0 msgr2=0x560caaff0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafee7b0 0x560caaff0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 -- 172.21.15.45:0/250790557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 msgr2=0x560caafee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 -- 172.21.15.45:0/250790557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560caaf0b570 con 0x560caafe52b0 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93d273e640 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009be0 tx=0x7f93c4003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282e6e34dfd1a6d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.896+0000 7f93c2ffd640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 0x7f93b80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.897+0000 7f93c2ffd640 1 -- 172.21.15.45:0/3323999426 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93c80cd700 con 0x7f93cc107760 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.897+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 0x7f93b80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.897+0000 7f93b7fff640 1 -- 172.21.15.45:0/250790557 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c402c920 con 0x560caafe52b0 2024-03-12T05:21:53.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.897+0000 7f93b7fff640 1 -- 172.21.15.45:0/250790557 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93c402ca80 con 0x560caafe52b0 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 -- 172.21.15.45:0/250790557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 msgr2=0x560caafe5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009be0 tx=0x7f93c4003e50 comp rx=0 tx=0).stop 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d1f3d640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 0x7f93b80a40e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f93c80caed0 tx=0x7f93c8000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 -- 172.21.15.45:0/250790557 shutdown_connections 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafee7b0 0x560caaff0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/250790557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafe5690 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.898+0000 7f93d6d7b740 1 -- 172.21.15.45:0/250790557 >> 172.21.15.45:0/250790557 conn(0x560caafdf990 msgr2=0x560caafdfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.900+0000 7f93d6d7b740 1 -- 172.21.15.45:0/250790557 shutdown_connections 2024-03-12T05:21:53.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.901+0000 7f93d6d7b740 1 -- 172.21.15.45:0/250790557 wait complete. 2024-03-12T05:21:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.901+0000 7f93d6d7b740 1 -- start start 2024-03-12T05:21:53.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.901+0000 7f93d6d7b740 1 Processor -- start 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafeb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d273e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d273e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60630/0 (socket says 172.21.15.45:60630) 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d273e640 1 -- 172.21.15.45:0/1867991936 learned_addr learned my addr 172.21.15.45:0/1867991936 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d173c640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafeb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caac20430 con 0x560caafe5bd0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caabdce00 con 0x560caafe52b0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaed1fd0 con 0x560caafebb80 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.902+0000 7f93d273e640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 -- 172.21.15.45:0/1867991936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 msgr2=0x560caafec000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafec000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 -- 172.21.15.45:0/1867991936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 msgr2=0x560caafeb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafeb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc114e50 con 0x560caafe52b0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d173c640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafeb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d6d7b740 10 client.0 _ll_register_callbacks cb 0x560caaf27fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560caac4c7e0 con 0x560caafe52b0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d273e640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f93c4034470 tx=0x7f93c40346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b430652159c02634 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93b57fa640 10 client.27692 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93b57fa640 1 -- 172.21.15.45:0/1867991936 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c404a450 con 0x560caafe52b0 2024-03-12T05:21:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93b57fa640 1 -- 172.21.15.45:0/1867991936 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93c404a5b0 con 0x560caafe52b0 2024-03-12T05:21:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93b57fa640 1 -- 172.21.15.45:0/1867991936 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93c404a770 con 0x560caafe52b0 2024-03-12T05:21:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d6d7b740 10 client.27692 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.903+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x560caaf4b610 con 0x560caafe52b0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93b57fa640 1 -- 172.21.15.45:0/1867991936 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93c403c050 con 0x560caafe52b0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 10 client.27692 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 10 client.27692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 20 client.27692 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560caaffe6d0 0x560cab000b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 4 client.27692 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560caac89720 con 0x560caaffe6d0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.904+0000 7f93d173c640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560caaffe6d0 0x560cab000b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.905+0000 7f93d173c640 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560caaffe6d0 0x560cab000b00 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca6ae1bd346c57c0 server_cookie=b5fe8a43dc578b87 in_seq=0 out_seq=0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.905+0000 7f93b57fa640 10 client.27692 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93b57fa640 1 -- 172.21.15.45:0/1867991936 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560caac89720 con 0x560caaffe6d0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93b57fa640 10 client.27692 handle_command_reply: tid=0 2024-03-12T05:21:53.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 client.27692 shutdown 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 msgr2=0x560caafee240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f93c4034470 tx=0x7f93c40346f0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 shutdown_connections 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560caaffe6d0 0x560cab000b00 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93b57fa640 0 client.27692 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe5bd0 0x560caafeb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.906+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/1867991936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe52b0 0x560caafee240 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.907+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 >> 172.21.15.45:0/1867991936 conn(0x560caafdf990 msgr2=0x560caafe1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.907+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 shutdown_connections 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.907+0000 7f93d6d7b740 1 -- 172.21.15.45:0/1867991936 wait complete. 2024-03-12T05:21:53.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.907+0000 7f93d6d7b740 20 client.27692 trim_cache size 0 max 16384 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.933+0000 7f93d6d7b740 1 Processor -- start 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.933+0000 7f93d6d7b740 1 -- start start 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.933+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caafe9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43794/0 (socket says 172.21.15.45:43794) 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 -- 172.21.15.45:0/172935530 learned_addr learned my addr 172.21.15.45:0/172935530 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d173c640 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caafe9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caafec670 con 0x560caafe52b0 2024-03-12T05:21:53.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caafe4de0 con 0x560caafe5bd0 2024-03-12T05:21:53.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caafdf7f0 con 0x560caafebb80 2024-03-12T05:21:53.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 -- 172.21.15.45:0/172935530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 msgr2=0x560caafea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 -- 172.21.15.45:0/172935530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 msgr2=0x560caafe9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caafe9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.934+0000 7f93d273e640 1 -- 172.21.15.45:0/172935530 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560caaee7ac0 con 0x560caafe52b0 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.935+0000 7f93d273e640 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f93c40344a0 tx=0x7f93c4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4630ac1732b605 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.935+0000 7f93b77fe640 1 -- 172.21.15.45:0/172935530 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4049e00 con 0x560caafe52b0 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.935+0000 7f93b77fe640 1 -- 172.21.15.45:0/172935530 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x560caac4c7e0 con 0x560caafe52b0 2024-03-12T05:21:53.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.935+0000 7f93b77fe640 1 -- 172.21.15.45:0/172935530 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c404a700 con 0x560caafe52b0 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 msgr2=0x560caafedd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f93c40344a0 tx=0x7f93c4005fd0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 shutdown_connections 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafebb80 0x560caafea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caafe9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/172935530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafe52b0 0x560caafedd70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 >> 172.21.15.45:0/172935530 conn(0x560caafdf990 msgr2=0x560cab0054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.936+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 shutdown_connections 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.937+0000 7f93d6d7b740 1 -- 172.21.15.45:0/172935530 wait complete. 2024-03-12T05:21:53.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.937+0000 7f93d6d7b740 1 -- start start 2024-03-12T05:21:53.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.937+0000 7f93d6d7b740 1 Processor -- start 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafe52b0 0x560caaff5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 0x560caaff2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaff29d0 con 0x560caafebb80 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x560caaff2b40 con 0x560caafe5bd0 2024-03-12T05:21:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cab054c20 con 0x560caafe52b0 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d6d7b740 10 client.0 _ll_register_callbacks cb 0x7f93c8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60656/0 (socket says 172.21.15.45:60656) 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 -- 172.21.15.45:0/517193401 learned_addr learned my addr 172.21.15.45:0/517193401 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d273e640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 0x560caaff2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 -- 172.21.15.45:0/517193401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafe52b0 msgr2=0x560caaff5a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafe52b0 0x560caaff5a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 -- 172.21.15.45:0/517193401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 msgr2=0x560caaff2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 0x560caaff2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d173c640 1 -- 172.21.15.45:0/517193401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560caaf4b610 con 0x560caafe5bd0 2024-03-12T05:21:53.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.939+0000 7f93d273e640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 0x560caaff2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93d173c640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005fd0 tx=0x7f93bc01c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bcde14723cc7c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc017040 con 0x560caafe5bd0 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93bc0037f0 con 0x560caafe5bd0 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93c402ca80 con 0x560caafe5bd0 2024-03-12T05:21:53.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.940+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93bc0039b0 con 0x560caafe5bd0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 10 client.27698 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560caaf0d6a0 con 0x560caafe5bd0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93bc031080 con 0x560caafe5bd0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93bc025b50 con 0x560caafe5bd0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 10 client.27698 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 10 client.27698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 20 client.27698 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560cab0563b0 0x560cab0587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 4 client.27698 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.941+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560caacd8890 con 0x560cab0563b0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93d273e640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560cab0563b0 0x560cab0587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93d273e640 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560cab0563b0 0x560cab0587c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b28837664fbc3b73 server_cookie=eca7a664329f09ca in_seq=0 out_seq=0 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93b57fa640 10 client.27698 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:53.938 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93b57fa640 1 -- 172.21.15.45:0/517193401 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560caacd8890 con 0x560cab0563b0 2024-03-12T05:21:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93b57fa640 10 client.27698 handle_command_reply: tid=0 2024-03-12T05:21:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.942+0000 7f93d6d7b740 1 client.27698 shutdown 2024-03-12T05:21:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 msgr2=0x560caaff1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.939 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005fd0 tx=0x7f93bc01c680 comp rx=0 tx=0).stop 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 shutdown_connections 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x560cab0563b0 0x560cab0587c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x560caafebb80 0x560caaff2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93b57fa640 0 client.27698 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560caafe5bd0 0x560caaff1dc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 --2- 172.21.15.45:0/517193401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560caafe52b0 0x560caaff5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.940 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.943+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 >> 172.21.15.45:0/517193401 conn(0x560caafdf990 msgr2=0x560caafe9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.944+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 shutdown_connections 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.944+0000 7f93d6d7b740 1 -- 172.21.15.45:0/517193401 wait complete. 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.944+0000 7f93d6d7b740 20 client.27698 trim_cache size 0 max 16384 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:21:53.941 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 msgr2=0x7f93b80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 0x7f93b80a40e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f93c80caed0 tx=0x7f93c8000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 msgr2=0x7f93cc1a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e990 tx=0x7f93c800ee50 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 shutdown_connections 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93b80a1c20 0x7f93b80a40e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc108100 0x7f93cc1a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93cc10c640 0x7f93cc19da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 --2- 172.21.15.45:0/3323999426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93cc107760 0x7f93cc1a38e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:53.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 >> 172.21.15.45:0/3323999426 conn(0x7f93cc076fe0 msgr2=0x7f93cc077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:53.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 shutdown_connections 2024-03-12T05:21:53.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:53.945+0000 7f93d41a1640 1 -- 172.21.15.45:0/3323999426 wait complete. 2024-03-12T05:21:53.961 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:21:53.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.141+0000 7f572632c640 1 -- 172.21.15.45:0/6147262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720101380 msgr2=0x7f5720101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.141+0000 7f572632c640 1 --2- 172.21.15.45:0/6147262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720101380 0x7f5720101760 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5710009930 tx=0x7f571002bc30 comp rx=0 tx=0).stop 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 -- 172.21.15.45:0/6147262 shutdown_connections 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 --2- 172.21.15.45:0/6147262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720106450 0x7f572010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 --2- 172.21.15.45:0/6147262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5720101d30 0x7f5720105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 --2- 172.21.15.45:0/6147262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720101380 0x7f5720101760 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 -- 172.21.15.45:0/6147262 >> 172.21.15.45:0/6147262 conn(0x7f5720078580 msgr2=0x7f5720078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 -- 172.21.15.45:0/6147262 shutdown_connections 2024-03-12T05:21:54.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.142+0000 7f572632c640 1 -- 172.21.15.45:0/6147262 wait complete. 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.143+0000 7f572632c640 1 Processor -- start 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.143+0000 7f572632c640 1 -- start start 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.143+0000 7f572632c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.143+0000 7f572632c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 0x7f5720197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f572632c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 0x7f572019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f572632c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572019c7b0 con 0x7f5720106450 2024-03-12T05:21:54.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f572632c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572019c920 con 0x7f5720101380 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f572632c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572019ca90 con 0x7f572019bdc0 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f571ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f571ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60684/0 (socket says 172.21.15.45:60684) 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f571ffff640 1 -- 172.21.15.45:0/2279034421 learned_addr learned my addr 172.21.15.45:0/2279034421 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f571f7fe640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 0x7f5720197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f57248c9640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 0x7f572019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f57248c9640 1 -- 172.21.15.45:0/2279034421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 msgr2=0x7f5720197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f57248c9640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.144+0000 7f57248c9640 1 -- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 msgr2=0x7f5720197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.145+0000 7f57248c9640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 0x7f5720197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.145+0000 7f57248c9640 1 -- 172.21.15.45:0/2279034421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5710009560 con 0x7f572019bdc0 2024-03-12T05:21:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.145+0000 7f571ffff640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:54.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.145+0000 7f571f7fe640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 0x7f5720197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.145+0000 7f57248c9640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 0x7f572019c270 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5718009fd0 tx=0x7f571800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.146+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571800ce20 con 0x7f572019bdc0 2024-03-12T05:21:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.146+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5718005020 con 0x7f572019bdc0 2024-03-12T05:21:54.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.148+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571800ce20 con 0x7f572019bdc0 2024-03-12T05:21:54.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.149+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57201a4070 con 0x7f572019bdc0 2024-03-12T05:21:54.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.149+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57201a44d0 con 0x7f572019bdc0 2024-03-12T05:21:54.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.151+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57200683c0 con 0x7f572019bdc0 2024-03-12T05:21:54.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.152+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5718005190 con 0x7f572019bdc0 2024-03-12T05:21:54.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.153+0000 7f571d7fa640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 0x7f57080a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.153+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5718014070 con 0x7f572019bdc0 2024-03-12T05:21:54.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.154+0000 7f571ffff640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 0x7f57080a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.157+0000 7f571ffff640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 0x7f57080a41a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5710005ab0 tx=0x7f5710000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.158+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5718095bc0 con 0x7f572019bdc0 2024-03-12T05:21:54.365 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:54 smithi045 ceph-mon[32174]: pgmap v343: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:54.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:54 smithi130 ceph-mon[35398]: pgmap v343: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:54.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:54 smithi116 ceph-mon[35486]: pgmap v343: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 18 MiB/s wr, 1.24k op/s 2024-03-12T05:21:54.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.561+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f572019cc20 con 0x7f572019bdc0 2024-03-12T05:21:54.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.564+0000 7f571d7fa640 1 -- 172.21.15.45:0/2279034421 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f5718099280 con 0x7f572019bdc0 2024-03-12T05:21:54.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:21:54.559 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:21:54.559 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 msgr2=0x7f57080a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 0x7f57080a41a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5710005ab0 tx=0x7f5710000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 msgr2=0x7f572019c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 0x7f572019c270 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5718009fd0 tx=0x7f571800ef10 comp rx=0 tx=0).stop 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 shutdown_connections 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f57080a1ce0 0x7f57080a41a0 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5710005ab0 tx=0x7f5710000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5720106450 0x7f5720197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f572019bdc0 0x7f572019c270 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 --2- 172.21.15.45:0/2279034421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5720101380 0x7f5720197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.567+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 >> 172.21.15.45:0/2279034421 conn(0x7f5720078580 msgr2=0x7f57200ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.568+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 shutdown_connections 2024-03-12T05:21:54.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.568+0000 7f572632c640 1 -- 172.21.15.45:0/2279034421 wait complete. 2024-03-12T05:21:54.583 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:54.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:21:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.760+0000 7f0c95222640 1 -- 172.21.15.45:0/3831392485 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8402cb70 con 0x7f0c901086c0 2024-03-12T05:21:54.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.760+0000 7f0c97226640 1 -- 172.21.15.45:0/3831392485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 msgr2=0x7f0c90108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.760+0000 7f0c97226640 1 --2- 172.21.15.45:0/3831392485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c90108b20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009900 tx=0x7f0c8402bc30 comp rx=0 tx=0).stop 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 -- 172.21.15.45:0/3831392485 shutdown_connections 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 --2- 172.21.15.45:0/3831392485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c90113e20 0x7f0c90116210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 --2- 172.21.15.45:0/3831392485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c90108b20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 --2- 172.21.15.45:0/3831392485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c9006d410 0x7f0c90108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 -- 172.21.15.45:0/3831392485 >> 172.21.15.45:0/3831392485 conn(0x7f0c9006c030 msgr2=0x7f0c9006c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 -- 172.21.15.45:0/3831392485 shutdown_connections 2024-03-12T05:21:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.761+0000 7f0c97226640 1 -- 172.21.15.45:0/3831392485 wait complete. 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 Processor -- start 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 -- start start 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c9006d410 0x7f0c901133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c90111f40 con 0x7f0c90113e20 2024-03-12T05:21:54.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c901120b0 con 0x7f0c9006d410 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c97226640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c90112270 con 0x7f0c901086c0 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c96a25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c96224640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c9006d410 0x7f0c901133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c96a25640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43848/0 (socket says 172.21.15.45:43848) 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51450/0 (socket says 172.21.15.45:51450) 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.762+0000 7f0c96a25640 1 -- 172.21.15.45:0/84205913 learned_addr learned my addr 172.21.15.45:0/84205913 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 -- 172.21.15.45:0/84205913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 msgr2=0x7f0c901114c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 -- 172.21.15.45:0/84205913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c9006d410 msgr2=0x7f0c901133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c9006d410 0x7f0c901133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 -- 172.21.15.45:0/84205913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84009560 con 0x7f0c90113e20 2024-03-12T05:21:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c95a23640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.763+0000 7f0c96a25640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800eb40 tx=0x7f0c8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.764+0000 7f0c837fe640 1 -- 172.21.15.45:0/84205913 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88004b20 con 0x7f0c90113e20 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.764+0000 7f0c837fe640 1 -- 172.21.15.45:0/84205913 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c88004c80 con 0x7f0c90113e20 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.764+0000 7f0c837fe640 1 -- 172.21.15.45:0/84205913 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88010690 con 0x7f0c90113e20 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.764+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c901b2ab0 con 0x7f0c90113e20 2024-03-12T05:21:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.764+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c901b2f40 con 0x7f0c90113e20 2024-03-12T05:21:54.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.769+0000 7f0c9b066740 1 Processor -- start 2024-03-12T05:21:54.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c837fe640 1 -- 172.21.15.45:0/84205913 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0c880107f0 con 0x7f0c90113e20 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c9b066740 1 -- start start 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e119840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e124d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.770+0000 7f0c9b066740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e0d4960 con 0x55f33e122960 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c9b066740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e088310 con 0x55f33e119d80 2024-03-12T05:21:54.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c9b066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e085840 con 0x55f33e119460 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60716/0 (socket says 172.21.15.45:60716) 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 -- 172.21.15.45:0/2580345175 learned_addr learned my addr 172.21.15.45:0/2580345175 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e124d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e119840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 -- 172.21.15.45:0/2580345175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 msgr2=0x55f33e119840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e119840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 -- 172.21.15.45:0/2580345175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 msgr2=0x55f33e124d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e124d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c96224640 1 -- 172.21.15.45:0/2580345175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f33e043060 con 0x55f33e119d80 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.771+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c96224640 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009cf0 tx=0x7f0c8c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef20782bbce4c67e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c80bfe640 1 -- 172.21.15.45:0/2580345175 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c009ea0 con 0x55f33e119d80 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c80bfe640 1 -- 172.21.15.45:0/2580345175 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c8c004230 con 0x55f33e119d80 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c80bfe640 1 -- 172.21.15.45:0/2580345175 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c019790 con 0x55f33e119d80 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c837fe640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 0x7f0c7c0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c837fe640 1 -- 172.21.15.45:0/84205913 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0c88014070 con 0x7f0c90113e20 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.772+0000 7f0c95a23640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 0x7f0c7c0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 -- 172.21.15.45:0/2580345175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 msgr2=0x55f33e122420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009cf0 tx=0x7f0c8c011810 comp rx=0 tx=0).stop 2024-03-12T05:21:54.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c95a23640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 0x7f0c7c0a41b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006010 tx=0x7f0c84045040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 -- 172.21.15.45:0/2580345175 shutdown_connections 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e124d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e122420 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2580345175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e119840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 -- 172.21.15.45:0/2580345175 >> 172.21.15.45:0/2580345175 conn(0x55f33e113b90 msgr2=0x55f33e113f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 -- 172.21.15.45:0/2580345175 shutdown_connections 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.773+0000 7f0c9b066740 1 -- 172.21.15.45:0/2580345175 wait complete. 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.774+0000 7f0c9b066740 1 -- start start 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.774+0000 7f0c9b066740 1 Processor -- start 2024-03-12T05:21:54.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.774+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e11fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c96224640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e122960 0x55f33e120330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c96224640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43858/0 (socket says 172.21.15.45:43858) 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c96224640 1 -- 172.21.15.45:0/2235610239 learned_addr learned my addr 172.21.15.45:0/2235610239 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e081020 con 0x55f33e119460 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33dd54430 con 0x55f33e119d80 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33dd10e00 con 0x55f33e122960 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e11fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c96a25640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e122960 0x55f33e120330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 10 client.0 _ll_register_callbacks cb 0x55f33e028cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 -- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e122960 msgr2=0x55f33e120330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e122960 0x55f33e120330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 -- 172.21.15.45:0/2235610239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 msgr2=0x55f33e11f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c95a23640 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c009950 con 0x55f33e119d80 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c96224640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.775+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f33dd807e0 con 0x55f33e119d80 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c95a23640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e11fdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000c00 tx=0x7f0c84009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71297f7c723d4bcf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c79ffb640 10 client.27716 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c79ffb640 1 -- 172.21.15.45:0/2235610239 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840044f0 con 0x55f33e119d80 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c79ffb640 1 -- 172.21.15.45:0/2235610239 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0c84005420 con 0x55f33e119d80 2024-03-12T05:21:54.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c79ffb640 1 -- 172.21.15.45:0/2235610239 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c840055e0 con 0x55f33e119d80 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c9b066740 10 client.27716 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.776+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55f33e07f6a0 con 0x55f33e119d80 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c79ffb640 1 -- 172.21.15.45:0/2235610239 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0c840035f0 con 0x55f33e119d80 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 10 client.27716 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 10 client.27716 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:21:54.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 20 client.27716 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e132b20 0x55f33e134f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 4 client.27716 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f33ddbd720 con 0x55f33e132b20 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c96224640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e132b20 0x55f33e134f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c96224640 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e132b20 0x55f33e134f80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d93988cabba4dafd server_cookie=a2bd38eaded52767 in_seq=0 out_seq=0 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.777+0000 7f0c79ffb640 10 client.27716 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c79ffb640 1 -- 172.21.15.45:0/2235610239 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f33ddbd720 con 0x55f33e132b20 2024-03-12T05:21:54.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c79ffb640 10 client.27716 handle_command_reply: tid=0 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 client.27716 shutdown 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 msgr2=0x55f33e11fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e11fdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000c00 tx=0x7f0c84009260 comp rx=0 tx=0).stop 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 shutdown_connections 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e132b20 0x55f33e134f80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e122960 0x55f33e120330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119d80 0x55f33e11fdf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c79ffb640 0 client.27716 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 --2- 172.21.15.45:0/2235610239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119460 0x55f33e11f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.778+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 >> 172.21.15.45:0/2235610239 conn(0x55f33e113b90 msgr2=0x55f33e1158e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.779+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 shutdown_connections 2024-03-12T05:21:54.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.779+0000 7f0c9b066740 1 -- 172.21.15.45:0/2235610239 wait complete. 2024-03-12T05:21:54.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.779+0000 7f0c9b066740 20 client.27716 trim_cache size 0 max 16384 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.805+0000 7f0c9b066740 1 Processor -- start 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.805+0000 7f0c9b066740 1 -- start start 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e121fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e006130 con 0x55f33e119d80 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e12e780 con 0x55f33e122960 2024-03-12T05:21:54.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c9b066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e1139c0 con 0x55f33e119460 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96224640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e121fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96224640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e121fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51494/0 (socket says 172.21.15.45:51494) 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96224640 1 -- 172.21.15.45:0/3072833481 learned_addr learned my addr 172.21.15.45:0/3072833481 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43872/0 (socket says 172.21.15.45:43872) 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60732/0 (socket says 172.21.15.45:60732) 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 -- 172.21.15.45:0/3072833481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 msgr2=0x55f33e121fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e121fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 -- 172.21.15.45:0/3072833481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 msgr2=0x55f33e11e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.806+0000 7f0c96a25640 1 -- 172.21.15.45:0/3072833481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f33e01bc20 con 0x55f33e122960 2024-03-12T05:21:54.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.807+0000 7f0c95a23640 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:54.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.807+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0c880cb400 tx=0x7f0c8809a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f6b5f6d46df6281 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.807+0000 7f0c7bfff640 1 -- 172.21.15.45:0/3072833481 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880b7070 con 0x55f33e122960 2024-03-12T05:21:54.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.807+0000 7f0c7bfff640 1 -- 172.21.15.45:0/3072833481 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c8809ea30 con 0x55f33e122960 2024-03-12T05:21:54.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.807+0000 7f0c7bfff640 1 -- 172.21.15.45:0/3072833481 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880b3180 con 0x55f33e122960 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 -- 172.21.15.45:0/3072833481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 msgr2=0x55f33e11cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0c880cb400 tx=0x7f0c8809a300 comp rx=0 tx=0).stop 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 -- 172.21.15.45:0/3072833481 shutdown_connections 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e119d80 0x55f33e11e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e122960 0x55f33e11cb10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3072833481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119460 0x55f33e121fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 -- 172.21.15.45:0/3072833481 >> 172.21.15.45:0/3072833481 conn(0x55f33e113b90 msgr2=0x55f33e11e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.808+0000 7f0c9b066740 1 -- 172.21.15.45:0/3072833481 shutdown_connections 2024-03-12T05:21:54.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.809+0000 7f0c9b066740 1 -- 172.21.15.45:0/3072833481 wait complete. 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.809+0000 7f0c9b066740 1 -- start start 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.809+0000 7f0c9b066740 1 Processor -- start 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e1831d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c96224640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c96224640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60748/0 (socket says 172.21.15.45:60748) 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c95a23640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51500/0 (socket says 172.21.15.45:51500) 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c96224640 1 -- 172.21.15.45:0/3092954620 learned_addr learned my addr 172.21.15.45:0/3092954620 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e183710 con 0x55f33e122960 2024-03-12T05:21:54.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e1831d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e183880 con 0x55f33e119460 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.810+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f33e183a40 con 0x55f33e119d80 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c9b066740 10 client.0 _ll_register_callbacks cb 0x7f0c8c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 -- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 msgr2=0x55f33e129190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 -- 172.21.15.45:0/3092954620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 msgr2=0x55f33e128c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f33e07f6a0 con 0x55f33e122960 2024-03-12T05:21:54.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c96a25640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e1831d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0c88005a40 tx=0x7f0c8809a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac9b14cf4080463 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:54.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.811+0000 7f0c79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.812+0000 7f0c96224640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.812+0000 7f0c95a23640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.812+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880a9070 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.812+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0c880b8070 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.812+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f33e11f6e0 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.813+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c880b7030 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.813+0000 7f0c9b066740 10 client.18237 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.813+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f0c901b2f40 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.813+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0c880cb680 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0c880ac030 con 0x55f33e122960 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 10 client.18237 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:54.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 10 client.18237 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 20 client.18237 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e18add0 0x55f33e18d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 4 client.18237 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f33e11f6e0 con 0x55f33e18add0 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.814+0000 7f0c96224640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e18add0 0x55f33e18d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c96224640 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e18add0 0x55f33e18d1c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=57e43a66b1dcf787 server_cookie=f21a7120838c1f5e in_seq=0 out_seq=0 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c79ffb640 10 client.18237 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c79ffb640 1 -- 172.21.15.45:0/3092954620 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f33e11f6e0 con 0x55f33e18add0 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c79ffb640 10 client.18237 handle_command_reply: tid=0 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 client.18237 shutdown 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 msgr2=0x55f33e1831d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e1831d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0c88005a40 tx=0x7f0c8809a370 comp rx=0 tx=0).stop 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 shutdown_connections 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55f33e18add0 0x55f33e18d1c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f33e122960 0x55f33e1831d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c79ffb640 0 client.18237 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f33e119d80 0x55f33e129190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 --2- 172.21.15.45:0/3092954620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f33e119460 0x55f33e128c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.815+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 >> 172.21.15.45:0/3092954620 conn(0x55f33e113b90 msgr2=0x55f33e114eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.816+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 shutdown_connections 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.816+0000 7f0c9b066740 1 -- 172.21.15.45:0/3092954620 wait complete. 2024-03-12T05:21:54.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.816+0000 7f0c9b066740 20 client.18237 trim_cache size 0 max 16384 2024-03-12T05:21:54.816 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:54.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 msgr2=0x7f0c7c0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 0x7f0c7c0a41b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006010 tx=0x7f0c84045040 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 msgr2=0x7f0c90111a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800eb40 tx=0x7f0c8800c2d0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 shutdown_connections 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0c7c0a1cf0 0x7f0c7c0a41b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c90113e20 0x7f0c90111a00 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c901086c0 0x7f0c901114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 --2- 172.21.15.45:0/84205913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c9006d410 0x7f0c901133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 >> 172.21.15.45:0/84205913 conn(0x7f0c9006c030 msgr2=0x7f0c90072170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 shutdown_connections 2024-03-12T05:21:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:54.817+0000 7f0c97226640 1 -- 172.21.15.45:0/84205913 wait complete. 2024-03-12T05:21:54.829 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:54.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:21:54.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 590.0 s, 1150.0 tps, lat 86.486 ms stddev 102.515 2024-03-12T05:21:55.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- 172.21.15.45:0/2067430894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 msgr2=0x7f7a880ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 --2- 172.21.15.45:0/2067430894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a880ffcd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009890 tx=0x7f7a7802bc00 comp rx=0 tx=0).stop 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- 172.21.15.45:0/2067430894 shutdown_connections 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 --2- 172.21.15.45:0/2067430894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88100210 0x7f7a88111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 --2- 172.21.15.45:0/2067430894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88111d80 0x7f7a88114170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 --2- 172.21.15.45:0/2067430894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a880ffcd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- 172.21.15.45:0/2067430894 >> 172.21.15.45:0/2067430894 conn(0x7f7a880751c0 msgr2=0x7f7a880755d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- 172.21.15.45:0/2067430894 shutdown_connections 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- 172.21.15.45:0/2067430894 wait complete. 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 Processor -- start 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.008+0000 7f7a8de02640 1 -- start start 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88100210 0x7f7a881a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 0x7f7a8819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8819e000 con 0x7f7a88111d80 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8819e170 con 0x7f7a880ff8f0 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8de02640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8819e330 con 0x7f7a88100210 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 0x7f7a8819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60762/0 (socket says 172.21.15.45:60762) 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a87fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88100210 0x7f7a881a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 -- 172.21.15.45:0/522275872 learned_addr learned my addr 172.21.15.45:0/522275872 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 0x7f7a8819d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43890/0 (socket says 172.21.15.45:43890) 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 -- 172.21.15.45:0/522275872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88100210 msgr2=0x7f7a881a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88100210 0x7f7a881a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 -- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 msgr2=0x7f7a8819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 0x7f7a8819d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 -- 172.21.15.45:0/522275872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78009490 con 0x7f7a880ff8f0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.009+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a78034720 tx=0x7f7a78034750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.010+0000 7f7a85ffb640 1 -- 172.21.15.45:0/522275872 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7802c7b0 con 0x7f7a880ff8f0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.010+0000 7f7a85ffb640 1 -- 172.21.15.45:0/522275872 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a7802c910 con 0x7f7a880ff8f0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.010+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8819e5b0 con 0x7f7a880ff8f0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.010+0000 7f7a85ffb640 1 -- 172.21.15.45:0/522275872 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7803d650 con 0x7f7a880ff8f0 2024-03-12T05:21:55.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.010+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881aa8a0 con 0x7f7a880ff8f0 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.013+0000 7f7a85ffb640 1 -- 172.21.15.45:0/522275872 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7a7803b040 con 0x7f7a880ff8f0 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 Processor -- start 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 -- start start 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a85ffb640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 0x7f7a740a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a466960 con 0x55662a4ab460 2024-03-12T05:21:55.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a41a310 con 0x55662a4b4960 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a91c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a417840 con 0x55662a4abd80 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.014+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a85ffb640 1 -- 172.21.15.45:0/522275872 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7a780ef400 con 0x7f7a880ff8f0 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43892/0 (socket says 172.21.15.45:43892) 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 -- 172.21.15.45:0/3069226285 learned_addr learned my addr 172.21.15.45:0/3069226285 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a87fff640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 0x7f7a740a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 -- 172.21.15.45:0/3069226285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 msgr2=0x55662a4b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 -- 172.21.15.45:0/3069226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 msgr2=0x55662a4b6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 -- 172.21.15.45:0/3069226285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55662a3d5060 con 0x55662a4ab460 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a87fff640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a800129c0 tx=0x7f7a80012e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6a5d29a46bcaf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a7effd640 1 -- 172.21.15.45:0/3069226285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004d20 con 0x55662a4ab460 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a8d601640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 0x7f7a740a4060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a80022000 tx=0x7f7a80020000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.015+0000 7f7a7effd640 1 -- 172.21.15.45:0/3069226285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a800123f0 con 0x55662a4ab460 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.016+0000 7f7a7effd640 1 -- 172.21.15.45:0/3069226285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80004030 con 0x55662a4ab460 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- 172.21.15.45:0/3069226285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 msgr2=0x55662a4ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7a800129c0 tx=0x7f7a80012e80 comp rx=0 tx=0).stop 2024-03-12T05:21:55.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- 172.21.15.45:0/3069226285 shutdown_connections 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 --2- 172.21.15.45:0/3069226285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ab840 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- 172.21.15.45:0/3069226285 >> 172.21.15.45:0/3069226285 conn(0x55662a4a5b90 msgr2=0x55662a4a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- 172.21.15.45:0/3069226285 shutdown_connections 2024-03-12T05:21:55.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- 172.21.15.45:0/3069226285 wait complete. 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 -- start start 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.017+0000 7f7a91c38740 1 Processor -- start 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a413020 con 0x55662a4ab460 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a0e6430 con 0x55662a4b4960 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a0a2e00 con 0x55662a4abd80 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a91c38740 10 client.0 _ll_register_callbacks cb 0x55662a3bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a8d601640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60778/0 (socket says 172.21.15.45:60778) 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a8d601640 1 -- 172.21.15.45:0/2981599310 learned_addr learned my addr 172.21.15.45:0/2981599310 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a87fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.018+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a87fff640 1 -- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 msgr2=0x55662a4b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a87fff640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a87fff640 1 -- 172.21.15.45:0/2981599310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 msgr2=0x55662a4b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a87fff640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a87fff640 1 -- 172.21.15.45:0/2981599310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a80012570 con 0x55662a4abd80 2024-03-12T05:21:55.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.019+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:55.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.021+0000 7f7a87fff640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b1dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7a88111410 tx=0x7f7a7000beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d72316d3a4d68273 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.021+0000 7f7a67fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70016070 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7a70003e50 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55662a14f720 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a70011440 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a91c38740 10 client.27721 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.024+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55662a4116a0 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a70011a50 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7a700257a0 con 0x55662a4abd80 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 10 client.27721 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 10 client.27721 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 20 client.27721 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a4c4bc0 0x55662a4c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 4 client.27721 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:55.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55662a14f720 con 0x55662a4c4bc0 2024-03-12T05:21:55.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.025+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a4c4bc0 0x55662a4c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.026+0000 7f7a67fff640 10 client.27721 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.026+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a4c4bc0 0x55662a4c7000 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=732d2776e6410bc9 server_cookie=603815c10359b7bb in_seq=0 out_seq=0 2024-03-12T05:21:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a67fff640 1 -- 172.21.15.45:0/2981599310 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55662a14f720 con 0x55662a4c4bc0 2024-03-12T05:21:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a67fff640 10 client.27721 handle_command_reply: tid=0 2024-03-12T05:21:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 client.27721 shutdown 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 msgr2=0x55662a4b1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b1dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7a88111410 tx=0x7f7a7000beb0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 shutdown_connections 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a4c4bc0 0x55662a4c7000 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b1dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2981599310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 >> 172.21.15.45:0/2981599310 conn(0x55662a4a5b90 msgr2=0x55662a4a78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a67fff640 0 client.27721 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 shutdown_connections 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 1 -- 172.21.15.45:0/2981599310 wait complete. 2024-03-12T05:21:55.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.027+0000 7f7a91c38740 20 client.27721 trim_cache size 0 max 16384 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.047+0000 7f7a91c38740 1 Processor -- start 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.047+0000 7f7a91c38740 1 -- start start 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.047+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.047+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.047+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a91c38740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a4a1340 con 0x55662a4ab460 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a91c38740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a398130 con 0x55662a4b4960 2024-03-12T05:21:55.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a91c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a4b6d50 con 0x55662a4abd80 2024-03-12T05:21:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43912/0 (socket says 172.21.15.45:43912) 2024-03-12T05:21:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 -- 172.21.15.45:0/2050758676 learned_addr learned my addr 172.21.15.45:0/2050758676 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8d601640 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a87fff640 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 -- 172.21.15.45:0/2050758676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 msgr2=0x55662a4b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 -- 172.21.15.45:0/2050758676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 msgr2=0x55662a4a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 -- 172.21.15.45:0/2050758676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55662a3adc20 con 0x55662a4ab460 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.048+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ecc90 tx=0x7f7a780b8580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef55985bfbd3d35 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.049+0000 7f7a7e7fc640 1 -- 172.21.15.45:0/2050758676 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780c48b0 con 0x55662a4ab460 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.049+0000 7f7a7e7fc640 1 -- 172.21.15.45:0/2050758676 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a780ba4c0 con 0x55662a4ab460 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.049+0000 7f7a7e7fc640 1 -- 172.21.15.45:0/2050758676 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780d6410 con 0x55662a4ab460 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- 172.21.15.45:0/2050758676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 msgr2=0x55662a4b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ecc90 tx=0x7f7a780b8580 comp rx=0 tx=0).stop 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- 172.21.15.45:0/2050758676 shutdown_connections 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 --2- 172.21.15.45:0/2050758676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4b0a00 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- 172.21.15.45:0/2050758676 >> 172.21.15.45:0/2050758676 conn(0x55662a4a5b90 msgr2=0x55662a4a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- 172.21.15.45:0/2050758676 shutdown_connections 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- 172.21.15.45:0/2050758676 wait complete. 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 -- start start 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.050+0000 7f7a91c38740 1 Processor -- start 2024-03-12T05:21:55.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4ba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4bad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a8ce00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43924/0 (socket says 172.21.15.45:43924) 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a8ce00640 1 -- 172.21.15.45:0/1266173161 learned_addr learned my addr 172.21.15.45:0/1266173161 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a51b0f0 con 0x55662a4ab460 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a51b230 con 0x55662a4b4960 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662a51b370 con 0x55662a4abd80 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a87fff640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.051+0000 7f7a8d601640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4bad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a91c38740 10 client.0 _ll_register_callbacks cb 0x7f7a80009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 -- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 msgr2=0x55662a4bad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4bad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 -- 172.21.15.45:0/1266173161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 msgr2=0x55662a4ba310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ba310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881aa8a0 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.052+0000 7f7a87fff640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4ba850 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a70025b00 tx=0x7f7a70005690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6780212f7c0698a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.053+0000 7f7a67fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.053+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70016070 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.053+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f7a70003e50 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.053+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55662a1127e0 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.054+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a70025880 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.054+0000 7f7a91c38740 10 client.27727 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.054+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55662a3d38c0 con 0x55662a4abd80 2024-03-12T05:21:55.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7a7003b080 con 0x55662a4abd80 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7a7002d030 con 0x55662a4abd80 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 10 client.27727 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 10 client.27727 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 20 client.27727 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a51ca90 0x55662a51eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 4 client.27727 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55662a19e890 con 0x55662a51ca90 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.055+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a51ca90 0x55662a51eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.056+0000 7f7a8ce00640 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a51ca90 0x55662a51eeb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eba7ba5af2ec5c server_cookie=3fb878bfd11dec29 in_seq=0 out_seq=0 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.056+0000 7f7a67fff640 10 client.27727 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.056+0000 7f7a67fff640 1 -- 172.21.15.45:0/1266173161 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55662a19e890 con 0x55662a51ca90 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.056+0000 7f7a67fff640 10 client.27727 handle_command_reply: tid=0 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.056+0000 7f7a91c38740 1 client.27727 shutdown 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 msgr2=0x55662a4ba850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4ba850 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7a70025b00 tx=0x7f7a70005690 comp rx=0 tx=0).stop 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 shutdown_connections 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55662a51ca90 0x55662a51eeb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55662a4b4960 0x55662a4bad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55662a4abd80 0x55662a4ba850 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 --2- 172.21.15.45:0/1266173161 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55662a4ab460 0x55662a4ba310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a67fff640 0 client.27727 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:21:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 >> 172.21.15.45:0/1266173161 conn(0x55662a4a5b90 msgr2=0x55662a4a4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 shutdown_connections 2024-03-12T05:21:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 1 -- 172.21.15.45:0/1266173161 wait complete. 2024-03-12T05:21:55.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.057+0000 7f7a91c38740 20 client.27727 trim_cache size 0 max 16384 2024-03-12T05:21:55.055 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 msgr2=0x7f7a740a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 0x7f7a740a4060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a80022000 tx=0x7f7a80020000 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 msgr2=0x7f7a881a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7a78034720 tx=0x7f7a78034750 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 shutdown_connections 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7a740a1ba0 0x7f7a740a4060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a88111d80 0x7f7a8819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a88100210 0x7f7a881a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 --2- 172.21.15.45:0/522275872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a880ff8f0 0x7f7a881a3800 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.058+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 >> 172.21.15.45:0/522275872 conn(0x7f7a880751c0 msgr2=0x7f7a8810f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.059+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 shutdown_connections 2024-03-12T05:21:55.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.059+0000 7f7a8de02640 1 -- 172.21.15.45:0/522275872 wait complete. 2024-03-12T05:21:55.072 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:55.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:21:55.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.249+0000 7f9813fff640 1 -- 172.21.15.45:0/1064280280 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818032340 con 0x7f981c100c10 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 -- 172.21.15.45:0/1064280280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c100c10 msgr2=0x7f981c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 --2- 172.21.15.45:0/1064280280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c100c10 0x7f981c104c50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9818009860 tx=0x7f981802bb90 comp rx=0 tx=0).stop 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 -- 172.21.15.45:0/1064280280 shutdown_connections 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 --2- 172.21.15.45:0/1064280280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 --2- 172.21.15.45:0/1064280280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c100c10 0x7f981c104c50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 --2- 172.21.15.45:0/1064280280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981c100260 0x7f981c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 -- 172.21.15.45:0/1064280280 >> 172.21.15.45:0/1064280280 conn(0x7f981c077180 msgr2=0x7f981c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 -- 172.21.15.45:0/1064280280 shutdown_connections 2024-03-12T05:21:55.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.250+0000 7f98237eb640 1 -- 172.21.15.45:0/1064280280 wait complete. 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 Processor -- start 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 -- start start 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981c100260 0x7f981c1a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c19de40 0x7f981c19e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19e8c0 con 0x7f981c105330 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19ea30 con 0x7f981c100260 2024-03-12T05:21:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.251+0000 7f98237eb640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19ebf0 con 0x7f981c19de40 2024-03-12T05:21:55.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.252+0000 7f9820d86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.252+0000 7f9820d86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43936/0 (socket says 172.21.15.45:43936) 2024-03-12T05:21:55.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.252+0000 7f9820d86640 1 -- 172.21.15.45:0/256870947 learned_addr learned my addr 172.21.15.45:0/256870947 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 -- 172.21.15.45:0/256870947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c19de40 msgr2=0x7f981c19e2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c19de40 0x7f981c19e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 -- 172.21.15.45:0/256870947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981c100260 msgr2=0x7f981c1a28c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981c100260 0x7f981c1a28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 -- 172.21.15.45:0/256870947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818009490 con 0x7f981c105330 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f9820d86640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9818037040 tx=0x7f98180045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.253+0000 7f98127fc640 1 -- 172.21.15.45:0/256870947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981802c530 con 0x7f981c105330 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.254+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c06aba0 con 0x7f981c105330 2024-03-12T05:21:55.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.254+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c06b100 con 0x7f981c105330 2024-03-12T05:21:55.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.254+0000 7f98127fc640 1 -- 172.21.15.45:0/256870947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f981802cb40 con 0x7f981c105330 2024-03-12T05:21:55.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.254+0000 7f98127fc640 1 -- 172.21.15.45:0/256870947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818032600 con 0x7f981c105330 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.257+0000 7f98263b7740 1 Processor -- start 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.257+0000 7f98127fc640 1 -- 172.21.15.45:0/256870947 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f981803b040 con 0x7f981c105330 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.257+0000 7f98263b7740 1 -- start start 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc29b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc2a4750 0x5652dc2a6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc256960 con 0x5652dc29bb70 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc20a310 con 0x5652dc2a4750 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98263b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc207840 con 0x5652dc29b250 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821d88640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc29b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9820d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc2a4750 0x5652dc2a6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43946/0 (socket says 172.21.15.45:43946) 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821d88640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc29b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51572/0 (socket says 172.21.15.45:51572) 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 -- 172.21.15.45:0/4138877605 learned_addr learned my addr 172.21.15.45:0/4138877605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f98127fc640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 0x7f98080a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 -- 172.21.15.45:0/4138877605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 msgr2=0x5652dc29b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc29b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 -- 172.21.15.45:0/4138877605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc2a4750 msgr2=0x5652dc2a6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc2a4750 0x5652dc2a6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.258+0000 7f9821587640 1 -- 172.21.15.45:0/4138877605 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652dc1c5060 con 0x5652dc29bb70 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f9821587640 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ba80 tx=0x7f980c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13caba5a76200eb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f98127fc640 1 -- 172.21.15.45:0/256870947 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f98180ef400 con 0x7f981c105330 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f9821d88640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 0x7f98080a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f98077fe640 1 -- 172.21.15.45:0/4138877605 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00dca0 con 0x5652dc29bb70 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f98077fe640 1 -- 172.21.15.45:0/4138877605 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f980c00de00 con 0x5652dc29bb70 2024-03-12T05:21:55.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.259+0000 7f98077fe640 1 -- 172.21.15.45:0/4138877605 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c010460 con 0x5652dc29bb70 2024-03-12T05:21:55.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 -- 172.21.15.45:0/4138877605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 msgr2=0x5652dc2a4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f9821d88640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 0x7f98080a4110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98140093b0 tx=0x7f9814004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ba80 tx=0x7f980c00bf40 comp rx=0 tx=0).stop 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 -- 172.21.15.45:0/4138877605 shutdown_connections 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2a4210 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc2a4750 0x5652dc2a6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 --2- 172.21.15.45:0/4138877605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc29b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.260+0000 7f98263b7740 1 -- 172.21.15.45:0/4138877605 >> 172.21.15.45:0/4138877605 conn(0x5652dc295980 msgr2=0x5652dc295d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.261+0000 7f98263b7740 1 -- 172.21.15.45:0/4138877605 shutdown_connections 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.261+0000 7f98263b7740 1 -- 172.21.15.45:0/4138877605 wait complete. 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.261+0000 7f98263b7740 1 -- start start 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.261+0000 7f98263b7740 1 Processor -- start 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc2a2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc203020 con 0x5652dc2a4750 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dbed6430 con 0x5652dc29bb70 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dbe92e00 con 0x5652dc29b250 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f98263b7740 10 client.0 _ll_register_callbacks cb 0x5652dc1aacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f9821587640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f9821587640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51582/0 (socket says 172.21.15.45:51582) 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f9820d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f9821587640 1 -- 172.21.15.45:0/1970858775 learned_addr learned my addr 172.21.15.45:0/1970858775 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.262+0000 7f9821d88640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc2a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 -- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 msgr2=0x5652dc2a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 -- 172.21.15.45:0/1970858775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 msgr2=0x5652dc2a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc2a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 -- 172.21.15.45:0/1970858775 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c00b6b0 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9820d86640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1c50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f98180ecc30 tx=0x7f98180f2b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4b9eb89a148dcf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9804ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180c4620 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652dbf027e0 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f98180bc810 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98180bba70 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f98263b7740 10 client.27746 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.263+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5652dc2016a0 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5652dbf027e0 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f98180035f0 con 0x5652dc29bb70 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 10 client.27746 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 10 client.27746 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:21:55.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 20 client.27746 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc2b4450 0x5652dc2b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 4 client.27746 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652dbf3f720 con 0x5652dc2b4450 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.264+0000 7f9821587640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc2b4450 0x5652dc2b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.265+0000 7f9804ff9640 10 client.27746 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.265+0000 7f9821587640 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc2b4450 0x5652dc2b6870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49500e22a8610f45 server_cookie=e70b63a79d4d440c in_seq=0 out_seq=0 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.265+0000 7f9804ff9640 1 -- 172.21.15.45:0/1970858775 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5652dbf3f720 con 0x5652dc2b4450 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.265+0000 7f9804ff9640 10 client.27746 handle_command_reply: tid=0 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.265+0000 7f98263b7740 1 client.27746 shutdown 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 msgr2=0x5652dc2a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1c50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f98180ecc30 tx=0x7f98180f2b00 comp rx=0 tx=0).stop 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 shutdown_connections 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc2b4450 0x5652dc2b6870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc2a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1c50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 --2- 172.21.15.45:0/1970858775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 >> 172.21.15.45:0/1970858775 conn(0x5652dc295980 msgr2=0x5652dc2976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f9804ff9640 0 client.27746 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 shutdown_connections 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 1 -- 172.21.15.45:0/1970858775 wait complete. 2024-03-12T05:21:55.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.266+0000 7f98263b7740 20 client.27746 trim_cache size 0 max 16384 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 Processor -- start 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 -- start start 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc29e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc188130 con 0x5652dc2a4750 2024-03-12T05:21:55.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc29ac40 con 0x5652dc29bb70 2024-03-12T05:21:55.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.286+0000 7f98263b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc2bba60 con 0x5652dc29b250 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51598/0 (socket says 172.21.15.45:51598) 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 -- 172.21.15.45:0/1163277764 learned_addr learned my addr 172.21.15.45:0/1163277764 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9820d86640 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821d88640 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc29e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 -- 172.21.15.45:0/1163277764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 msgr2=0x5652dc2a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 -- 172.21.15.45:0/1163277764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 msgr2=0x5652dc29e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc29e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 -- 172.21.15.45:0/1163277764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652dc19dc20 con 0x5652dc29b250 2024-03-12T05:21:55.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.287+0000 7f9821587640 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f980c00b3d0 tx=0x7f980c005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=157e91aafb20f421 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.288+0000 7f9806ffd640 1 -- 172.21.15.45:0/1163277764 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c00daa0 con 0x5652dc29b250 2024-03-12T05:21:55.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.288+0000 7f9806ffd640 1 -- 172.21.15.45:0/1163277764 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f980c02b070 con 0x5652dc29b250 2024-03-12T05:21:55.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.288+0000 7f9806ffd640 1 -- 172.21.15.45:0/1163277764 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c003850 con 0x5652dc29b250 2024-03-12T05:21:55.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.289+0000 7f98263b7740 1 -- 172.21.15.45:0/1163277764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 msgr2=0x5652dc2a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.289+0000 7f98263b7740 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f980c00b3d0 tx=0x7f980c005c50 comp rx=0 tx=0).stop 2024-03-12T05:21:55.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.291+0000 7f98263b7740 1 -- 172.21.15.45:0/1163277764 shutdown_connections 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.291+0000 7f98263b7740 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc2a4750 0x5652dc29e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.291+0000 7f98263b7740 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29bb70 0x5652dc2a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.291+0000 7f98263b7740 1 --2- 172.21.15.45:0/1163277764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc29b250 0x5652dc2a3e40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.292+0000 7f98263b7740 1 -- 172.21.15.45:0/1163277764 >> 172.21.15.45:0/1163277764 conn(0x5652dc295980 msgr2=0x5652dc29c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.292+0000 7f98263b7740 1 -- 172.21.15.45:0/1163277764 shutdown_connections 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.292+0000 7f98263b7740 1 -- 172.21.15.45:0/1163277764 wait complete. 2024-03-12T05:21:55.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.293+0000 7f98263b7740 1 -- start start 2024-03-12T05:21:55.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.293+0000 7f98263b7740 1 Processor -- start 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.293+0000 7f98263b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.293+0000 7f98263b7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2aa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.293+0000 7f98263b7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc2a4750 0x5652dc2aad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f98263b7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc30aa80 con 0x5652dc29bb70 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f98263b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc30abf0 con 0x5652dc29b250 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f98263b7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dc30adb0 con 0x5652dc2a4750 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f98263b7740 10 client.0 _ll_register_callbacks cb 0x7f980c00b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60816/0 (socket says 172.21.15.45:60816) 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 -- 172.21.15.45:0/2343040639 learned_addr learned my addr 172.21.15.45:0/2343040639 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821d88640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc2a4750 0x5652dc2aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9820d86640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2aa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 -- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc2a4750 msgr2=0x5652dc2aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc2a4750 0x5652dc2aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 -- 172.21.15.45:0/2343040639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 msgr2=0x5652dc2aa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2aa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 -- 172.21.15.45:0/2343040639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652dc1c5060 con 0x5652dc29b250 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9821587640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f980c00d770 tx=0x7f980c00b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a7af815d4a10e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.294+0000 7f9804ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c004450 con 0x5652dc29b250 2024-03-12T05:21:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f980c0045b0 con 0x5652dc29b250 2024-03-12T05:21:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f980c00de00 con 0x5652dc29b250 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f980c00dc50 con 0x5652dc29b250 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f98263b7740 10 client.27752 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5652dc1c38c0 con 0x5652dc29b250 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.295+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f980c040030 con 0x5652dc29b250 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f980c027150 con 0x5652dc29b250 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 10 client.27752 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 10 client.27752 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 20 client.27752 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc30c160 0x5652dc30e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 4 client.27752 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5652dbf8e890 con 0x5652dc30c160 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f9820d86640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc30c160 0x5652dc30e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f9804ff9640 10 client.27752 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.296+0000 7f9820d86640 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc30c160 0x5652dc30e580 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f769b8469df7c61 server_cookie=4edeceacd02b9c82 in_seq=0 out_seq=0 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f9804ff9640 1 -- 172.21.15.45:0/2343040639 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5652dbf8e890 con 0x5652dc30c160 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f9804ff9640 10 client.27752 handle_command_reply: tid=0 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 client.27752 shutdown 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 msgr2=0x5652dc2aa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f980c00d770 tx=0x7f980c00b9e0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 shutdown_connections 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5652dc30c160 0x5652dc30e580 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5652dc29bb70 0x5652dc2aa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f9804ff9640 0 client.27752 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5652dc2a4750 0x5652dc2aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 --2- 172.21.15.45:0/2343040639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5652dc29b250 0x5652dc2aa2c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.297+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 >> 172.21.15.45:0/2343040639 conn(0x5652dc295980 msgr2=0x5652dc2942f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 shutdown_connections 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98263b7740 1 -- 172.21.15.45:0/2343040639 wait complete. 2024-03-12T05:21:55.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98263b7740 20 client.27752 trim_cache size 0 max 16384 2024-03-12T05:21:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 msgr2=0x7f98080a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 0x7f98080a4110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98140093b0 tx=0x7f9814004ab0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 msgr2=0x7f981c1a2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.298+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9818037040 tx=0x7f98180045e0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 shutdown_connections 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f98080a1c50 0x7f98080a4110 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f981c105330 0x7f981c1a2e00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f981c19de40 0x7f981c19e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 --2- 172.21.15.45:0/256870947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f981c100260 0x7f981c1a28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 >> 172.21.15.45:0/256870947 conn(0x7f981c077180 msgr2=0x7f981c0684c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 shutdown_connections 2024-03-12T05:21:55.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.299+0000 7f98237eb640 1 -- 172.21.15.45:0/256870947 wait complete. 2024-03-12T05:21:55.313 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:55.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:21:55.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:55 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2279034421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:55.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:55 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2279034421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.504+0000 7fd109fef640 1 -- 172.21.15.45:0/1587855288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 msgr2=0x7fd10410db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.504+0000 7fd109fef640 1 --2- 172.21.15.45:0/1587855288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10410db00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009c10 tx=0x7fd0fc02bef0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 -- 172.21.15.45:0/1587855288 shutdown_connections 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 --2- 172.21.15.45:0/1587855288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd1041145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 --2- 172.21.15.45:0/1587855288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 0x7fd104107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 --2- 172.21.15.45:0/1587855288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10410db00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 -- 172.21.15.45:0/1587855288 >> 172.21.15.45:0/1587855288 conn(0x7fd104075620 msgr2=0x7fd104077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 -- 172.21.15.45:0/1587855288 shutdown_connections 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.505+0000 7fd109fef640 1 -- 172.21.15.45:0/1587855288 wait complete. 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 Processor -- start 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 -- start start 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 0x7fd104072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd104072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406c1e0 con 0x7fd10410d720 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406c350 con 0x7fd1041080d0 2024-03-12T05:21:55.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.506+0000 7fd109fef640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406c510 con 0x7fd104107710 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd104072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd1037fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 0x7fd104072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd104072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60828/0 (socket says 172.21.15.45:60828) 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd103fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd103fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43992/0 (socket says 172.21.15.45:43992) 2024-03-12T05:21:55.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.509+0000 7fd102ffd640 1 -- 172.21.15.45:0/379393224 learned_addr learned my addr 172.21.15.45:0/379393224 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 -- 172.21.15.45:0/379393224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 msgr2=0x7fd104072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 0x7fd104072890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 -- 172.21.15.45:0/379393224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 msgr2=0x7fd104072dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd104072dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 -- 172.21.15.45:0/379393224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f8009560 con 0x7fd10410d720 2024-03-12T05:21:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.510+0000 7fd103fff640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e970 tx=0x7fd0f400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.511+0000 7fd100ff9640 1 -- 172.21.15.45:0/379393224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4009620 con 0x7fd10410d720 2024-03-12T05:21:55.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.511+0000 7fd100ff9640 1 -- 172.21.15.45:0/379393224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0f40044d0 con 0x7fd10410d720 2024-03-12T05:21:55.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.511+0000 7fd100ff9640 1 -- 172.21.15.45:0/379393224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4004800 con 0x7fd10410d720 2024-03-12T05:21:55.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.511+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc009840 con 0x7fd10410d720 2024-03-12T05:21:55.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.511+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1040705f0 con 0x7fd10410d720 2024-03-12T05:21:55.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.514+0000 7fd10cbbd740 1 Processor -- start 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd100ff9640 1 -- 172.21.15.45:0/379393224 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd0f4004960 con 0x7fd10410d720 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 -- start start 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0dfc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0dfeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0dae960 con 0x5579f0dfc750 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0d62310 con 0x5579f0df3b70 2024-03-12T05:21:55.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.515+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0d5f840 con 0x5579f0df3250 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51640/0 (socket says 172.21.15.45:51640) 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 -- 172.21.15.45:0/3364494501 learned_addr learned my addr 172.21.15.45:0/3364494501 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd1037fe640 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0dfc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0dfeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd100ff9640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 0x7fd0ec0a4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 -- 172.21.15.45:0/3364494501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 msgr2=0x5579f0dfc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0dfc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 -- 172.21.15.45:0/3364494501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 msgr2=0x5579f0dfeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0dfeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd102ffd640 1 -- 172.21.15.45:0/3364494501 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579f0d1d060 con 0x5579f0df3250 2024-03-12T05:21:55.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.516+0000 7fd100ff9640 1 -- 172.21.15.45:0/379393224 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd0f4014070 con 0x7fd10410d720 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.517+0000 7fd103fff640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 0x7fd0ec0a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.517+0000 7fd102ffd640 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8012990 tx=0x7fd0f80098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfa7454c3ed3a384 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.517+0000 7fd0e9ffb640 1 -- 172.21.15.45:0/3364494501 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8004030 con 0x5579f0df3250 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.517+0000 7fd0e9ffb640 1 -- 172.21.15.45:0/3364494501 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0f80107d0 con 0x5579f0df3250 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.517+0000 7fd0e9ffb640 1 -- 172.21.15.45:0/3364494501 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8004030 con 0x5579f0df3250 2024-03-12T05:21:55.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 -- 172.21.15.45:0/3364494501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 msgr2=0x5579f0df3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8012990 tx=0x7fd0f80098f0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd103fff640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 0x7fd0ec0a4260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4096070 tx=0x7fd0f40052b0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 -- 172.21.15.45:0/3364494501 shutdown_connections 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0dfeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0dfc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/3364494501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0df3630 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.518+0000 7fd10cbbd740 1 -- 172.21.15.45:0/3364494501 >> 172.21.15.45:0/3364494501 conn(0x5579f0ded980 msgr2=0x5579f0dedd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.519+0000 7fd10cbbd740 1 -- 172.21.15.45:0/3364494501 shutdown_connections 2024-03-12T05:21:55.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.519+0000 7fd10cbbd740 1 -- 172.21.15.45:0/3364494501 wait complete. 2024-03-12T05:21:55.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.519+0000 7fd10cbbd740 1 -- start start 2024-03-12T05:21:55.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.520+0000 7fd10cbbd740 1 Processor -- start 2024-03-12T05:21:55.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.521+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44008/0 (socket says 172.21.15.45:44008) 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd102ffd640 1 -- 172.21.15.45:0/384264453 learned_addr learned my addr 172.21.15.45:0/384264453 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3b70 0x5579f0df9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0dfc750 0x5579f0dfa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0d5b020 con 0x5579f0df3250 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0a2e430 con 0x5579f0dfc750 2024-03-12T05:21:55.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.522+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f09eae00 con 0x5579f0df3b70 2024-03-12T05:21:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd10cbbd740 10 client.0 _ll_register_callbacks cb 0x5579f0d02cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd1037fe640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3b70 0x5579f0df9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0dfc750 0x5579f0dfa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 -- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3b70 msgr2=0x5579f0df9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3b70 0x5579f0df9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 -- 172.21.15.45:0/384264453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0dfc750 msgr2=0x5579f0dfa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0dfc750 0x5579f0dfa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.523+0000 7fd102ffd640 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f801a040 con 0x5579f0df3250 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.524+0000 7fd102ffd640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8010d90 tx=0x7fd0f80092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26971ff27d08fe7b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.525+0000 7fd0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.525+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8004030 con 0x5579f0df3250 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.525+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5579f0a5a7e0 con 0x5579f0df3250 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.525+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd0f8005030 con 0x5579f0df3250 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0f80051f0 con 0x5579f0df3250 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd10cbbd740 10 client.18282 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5579f0d596a0 con 0x5579f0df3250 2024-03-12T05:21:55.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0f8003cb0 con 0x5579f0df3250 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd0f8004820 con 0x5579f0df3250 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.526+0000 7fd10cbbd740 10 client.18282 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd10cbbd740 10 client.18282 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd10cbbd740 20 client.18282 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e0ca50 0x5579f0e0eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd10cbbd740 4 client.18282 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579f0a5a7e0 con 0x5579f0e0ca50 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.527+0000 7fd1037fe640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e0ca50 0x5579f0e0eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.528+0000 7fd1037fe640 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e0ca50 0x5579f0e0eed0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9de8378079893a29 server_cookie=78bd7f755f01bc8f in_seq=0 out_seq=0 2024-03-12T05:21:55.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.529+0000 7fd0e2ffd640 10 client.18282 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:55.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.531+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/384264453 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5579f0a5a7e0 con 0x5579f0e0ca50 2024-03-12T05:21:55.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.531+0000 7fd0e2ffd640 10 client.18282 handle_command_reply: tid=0 2024-03-12T05:21:55.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.532+0000 7fd10cbbd740 1 client.18282 shutdown 2024-03-12T05:21:55.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.532+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 msgr2=0x5579f0df9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.532+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8010d90 tx=0x7fd0f80092c0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 shutdown_connections 2024-03-12T05:21:55.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e0ca50 0x5579f0e0eed0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0dfc750 0x5579f0dfa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3b70 0x5579f0df9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/384264453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0df3250 0x5579f0df9710 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 >> 172.21.15.45:0/384264453 conn(0x5579f0ded980 msgr2=0x5579f0def6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.536+0000 7fd0e2ffd640 0 client.18282 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:55.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.537+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 shutdown_connections 2024-03-12T05:21:55.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.541+0000 7fd10cbbd740 1 -- 172.21.15.45:0/384264453 wait complete. 2024-03-12T05:21:55.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.543+0000 7fd10cbbd740 20 client.18282 trim_cache size 0 max 16384 2024-03-12T05:21:55.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.567+0000 7fd10cbbd740 1 Processor -- start 2024-03-12T05:21:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 -- start start 2024-03-12T05:21:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0df4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0df4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0ce0130 con 0x5579f0dfc750 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0dfa6e0 con 0x5579f0df3b70 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.568+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0e09600 con 0x5579f0df3250 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51660/0 (socket says 172.21.15.45:51660) 2024-03-12T05:21:55.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 -- 172.21.15.45:0/2231528840 learned_addr learned my addr 172.21.15.45:0/2231528840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 -- 172.21.15.45:0/2231528840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 msgr2=0x5579f0df4a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0df4a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 -- 172.21.15.45:0/2231528840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 msgr2=0x5579f0df4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0df4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.569+0000 7fd102ffd640 1 -- 172.21.15.45:0/2231528840 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579f0cf5c20 con 0x5579f0df3250 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.570+0000 7fd102ffd640 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80109e0 tx=0x7fd0f8009d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e71446de2ccc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.570+0000 7fd0e97fa640 1 -- 172.21.15.45:0/2231528840 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8012860 con 0x5579f0df3250 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.570+0000 7fd0e97fa640 1 -- 172.21.15.45:0/2231528840 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0f8004d20 con 0x5579f0df3250 2024-03-12T05:21:55.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.570+0000 7fd0e97fa640 1 -- 172.21.15.45:0/2231528840 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8004f40 con 0x5579f0df3250 2024-03-12T05:21:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.571+0000 7fd10cbbd740 1 -- 172.21.15.45:0/2231528840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 msgr2=0x5579f0dfbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.571+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80109e0 tx=0x7fd0f8009d70 comp rx=0 tx=0).stop 2024-03-12T05:21:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.573+0000 7fd10cbbd740 1 -- 172.21.15.45:0/2231528840 shutdown_connections 2024-03-12T05:21:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.573+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0df4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.573+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0df4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.573+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/2231528840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0dfbe40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.573+0000 7fd10cbbd740 1 -- 172.21.15.45:0/2231528840 >> 172.21.15.45:0/2231528840 conn(0x5579f0ded980 msgr2=0x5579f0e13830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.577+0000 7fd10cbbd740 1 -- 172.21.15.45:0/2231528840 shutdown_connections 2024-03-12T05:21:55.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.578+0000 7fd10cbbd740 1 -- 172.21.15.45:0/2231528840 wait complete. 2024-03-12T05:21:55.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.578+0000 7fd10cbbd740 1 -- start start 2024-03-12T05:21:55.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.579+0000 7fd10cbbd740 1 Processor -- start 2024-03-12T05:21:55.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.579+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0e01d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0e61fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0e624f0 con 0x5579f0dfc750 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0e62630 con 0x5579f0df3b70 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579f0e627a0 con 0x5579f0df3250 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd10cbbd740 10 client.0 _ll_register_callbacks cb 0x7fd0f8009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51674/0 (socket says 172.21.15.45:51674) 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 -- 172.21.15.45:0/4013416701 learned_addr learned my addr 172.21.15.45:0/4013416701 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0e61fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd1037fe640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0e01d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 -- 172.21.15.45:0/4013416701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 msgr2=0x5579f0e01d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0e01d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 -- 172.21.15.45:0/4013416701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 msgr2=0x5579f0e61fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0e61fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.580+0000 7fd102ffd640 1 -- 172.21.15.45:0/4013416701 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f8016040 con 0x5579f0df3250 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.581+0000 7fd102ffd640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8012940 tx=0x7fd0f8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb1a4b466a4ba0f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.581+0000 7fd0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.581+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80050a0 con 0x5579f0df3250 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.581+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd0f8005200 con 0x5579f0df3250 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.581+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd0f80107d0 con 0x5579f0df3250 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.582+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0f80053c0 con 0x5579f0df3250 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.582+0000 7fd10cbbd740 10 client.27757 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.582+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5579f0d1b8c0 con 0x5579f0df3250 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd0f8005520 con 0x5579f0df3250 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd0f8044070 con 0x5579f0df3250 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 10 client.27757 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 10 client.27757 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 20 client.27757 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e63a70 0x5579f0e65e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 4 client.27757 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:55.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fd0f80107d0 con 0x5579f0e63a70 2024-03-12T05:21:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.583+0000 7fd1037fe640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e63a70 0x5579f0e65e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.584+0000 7fd0e2ffd640 10 client.27757 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:55.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.585+0000 7fd1037fe640 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e63a70 0x5579f0e65e60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d07fc2dfa5bd56 server_cookie=cf9e3d57fd2528e4 in_seq=0 out_seq=0 2024-03-12T05:21:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.588+0000 7fd0e2ffd640 1 -- 172.21.15.45:0/4013416701 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fd0f80107d0 con 0x5579f0e63a70 2024-03-12T05:21:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.588+0000 7fd0e2ffd640 10 client.27757 handle_command_reply: tid=0 2024-03-12T05:21:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.588+0000 7fd10cbbd740 1 client.27757 shutdown 2024-03-12T05:21:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.589+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 msgr2=0x5579f0e01800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.589+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8012940 tx=0x7fd0f8000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.590+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 shutdown_connections 2024-03-12T05:21:55.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.590+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5579f0dfc750 0x5579f0e61fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.590+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5579f0e63a70 0x5579f0e65e60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.591+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5579f0df3b70 0x5579f0e01d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.591+0000 7fd10cbbd740 1 --2- 172.21.15.45:0/4013416701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5579f0df3250 0x5579f0e01800 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.591+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 >> 172.21.15.45:0/4013416701 conn(0x5579f0ded980 msgr2=0x5579f0df4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.593+0000 7fd0e2ffd640 0 client.27757 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:21:55.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.593+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 shutdown_connections 2024-03-12T05:21:55.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.593+0000 7fd10cbbd740 1 -- 172.21.15.45:0/4013416701 wait complete. 2024-03-12T05:21:55.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.594+0000 7fd10cbbd740 20 client.27757 trim_cache size 0 max 16384 2024-03-12T05:21:55.590 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 msgr2=0x7fd0ec0a4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 0x7fd0ec0a4260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4096070 tx=0x7fd0f40052b0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 msgr2=0x7fd10406ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e970 tx=0x7fd0f400ee30 comp rx=0 tx=0).stop 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 shutdown_connections 2024-03-12T05:21:55.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd0ec0a1da0 0x7fd0ec0a4260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd10410d720 0x7fd10406ba50 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1041080d0 0x7fd104072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 --2- 172.21.15.45:0/379393224 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd104107710 0x7fd104072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 >> 172.21.15.45:0/379393224 conn(0x7fd104075620 msgr2=0x7fd104075f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 shutdown_connections 2024-03-12T05:21:55.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.596+0000 7fd109fef640 1 -- 172.21.15.45:0/379393224 wait complete. 2024-03-12T05:21:55.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:55 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2279034421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:21:55.610 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:21:55.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 -- 172.21.15.45:0/3956798996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103f40 msgr2=0x7fc42c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 --2- 172.21.15.45:0/3956798996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103f40 0x7fc42c107f80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc42001a940 tx=0x7fc42003cc70 comp rx=0 tx=0).stop 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 -- 172.21.15.45:0/3956798996 shutdown_connections 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 --2- 172.21.15.45:0/3956798996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103f40 0x7fc42c107f80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 --2- 172.21.15.45:0/3956798996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c108660 0x7fc42c110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 --2- 172.21.15.45:0/3956798996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c103590 0x7fc42c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 -- 172.21.15.45:0/3956798996 >> 172.21.15.45:0/3956798996 conn(0x7fc42c0fedd0 msgr2=0x7fc42c1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.788+0000 7fc432296640 1 -- 172.21.15.45:0/3956798996 shutdown_connections 2024-03-12T05:21:55.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.789+0000 7fc432296640 1 -- 172.21.15.45:0/3956798996 wait complete. 2024-03-12T05:21:55.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.789+0000 7fc432296640 1 Processor -- start 2024-03-12T05:21:55.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.789+0000 7fc432296640 1 -- start start 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103590 0x7fc42c10f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c108660 0x7fc42c10f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c06e9e0 con 0x7fc42c103590 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c06eb50 con 0x7fc42c108660 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc432296640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c06ed10 con 0x7fc42c06e020 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc430833640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc430833640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51694/0 (socket says 172.21.15.45:51694) 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc430833640 1 -- 172.21.15.45:0/4127260527 learned_addr learned my addr 172.21.15.45:0/4127260527 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.790+0000 7fc42bfff640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103590 0x7fc42c10f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c108660 0x7fc42c10f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 -- 172.21.15.45:0/4127260527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c108660 msgr2=0x7fc42c10f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c108660 0x7fc42c10f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 -- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103590 msgr2=0x7fc42c10f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103590 0x7fc42c10f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 -- 172.21.15.45:0/4127260527 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc424009840 con 0x7fc42c06e020 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc430833640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc42400efd0 tx=0x7fc42400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.791+0000 7fc4297fa640 1 -- 172.21.15.45:0/4127260527 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424009b40 con 0x7fc42c06e020 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.792+0000 7fc4297fa640 1 -- 172.21.15.45:0/4127260527 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc424003b70 con 0x7fc42c06e020 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.792+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42001a570 con 0x7fc42c06e020 2024-03-12T05:21:55.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.792+0000 7fc4297fa640 1 -- 172.21.15.45:0/4127260527 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240107e0 con 0x7fc42c06e020 2024-03-12T05:21:55.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.792+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c1a2470 con 0x7fc42c06e020 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.794+0000 7fc4297fa640 1 -- 172.21.15.45:0/4127260527 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc424004020 con 0x7fc42c06e020 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc4297fa640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 0x7fc4140a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 Processor -- start 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc4297fa640 1 -- 172.21.15.45:0/4127260527 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc424014070 con 0x7fc42c06e020 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc42bfff640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 0x7fc4140a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 -- start start 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093379630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093382750 0x557093384b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557093334960 con 0x557093379b70 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570932e8310 con 0x557093382750 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc434e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570932e5840 con 0x557093379250 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.795+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44052/0 (socket says 172.21.15.45:44052) 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2462807458 learned_addr learned my addr 172.21.15.45:0/2462807458 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc430833640 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093382750 0x557093384b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42bfff640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 0x7fc4140a40e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc41c005d00 tx=0x7fc41c00b410 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42bfff640 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093379630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2462807458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 msgr2=0x557093379630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093379630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2462807458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093382750 msgr2=0x557093384b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093382750 0x557093384b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2462807458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570932a3060 con 0x557093379b70 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc42003d1b0 tx=0x7fc420005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6f5706d1bd0ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc4127fc640 1 -- 172.21.15.45:0/2462807458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200041c0 con 0x557093379b70 2024-03-12T05:21:55.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.796+0000 7fc4127fc640 1 -- 172.21.15.45:0/2462807458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc420004b70 con 0x557093379b70 2024-03-12T05:21:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc4127fc640 1 -- 172.21.15.45:0/2462807458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200054d0 con 0x557093379b70 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 -- 172.21.15.45:0/2462807458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 msgr2=0x557093382210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc42003d1b0 tx=0x7fc420005e80 comp rx=0 tx=0).stop 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 -- 172.21.15.45:0/2462807458 shutdown_connections 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x557093382210 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093382750 0x557093384b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 --2- 172.21.15.45:0/2462807458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093379630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 -- 172.21.15.45:0/2462807458 >> 172.21.15.45:0/2462807458 conn(0x557093373980 msgr2=0x557093373d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 -- 172.21.15.45:0/2462807458 shutdown_connections 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.797+0000 7fc434e61740 1 -- 172.21.15.45:0/2462807458 wait complete. 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.798+0000 7fc434e61740 1 -- start start 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.798+0000 7fc434e61740 1 Processor -- start 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.798+0000 7fc434e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x55709337fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093382750 0x557093380180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60898/0 (socket says 172.21.15.45:60898) 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 -- 172.21.15.45:0/1014344427 learned_addr learned my addr 172.21.15.45:0/1014344427 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570932e1020 con 0x557093379b70 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x55709337fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557092fb4430 con 0x557093379250 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557092f70e00 con 0x557093382750 2024-03-12T05:21:55.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc434e61740 10 client.0 _ll_register_callbacks cb 0x557093288cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc430833640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093382750 0x557093380180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 -- 172.21.15.45:0/1014344427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093382750 msgr2=0x557093380180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093382750 0x557093380180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 -- 172.21.15.45:0/1014344427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 msgr2=0x55709337fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x55709337fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.799+0000 7fc42bfff640 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c110a40 con 0x557093379250 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc42bfff640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0116f0 tx=0x7fc41c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efe2b1365839e574 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc40b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c01d900 con 0x557093379250 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc41c017070 con 0x557093379250 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557092fe07e0 con 0x557093379250 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc41c01da60 con 0x557093379250 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc434e61740 10 client.27776 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.800+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570932df6a0 con 0x557093379250 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc41c032020 con 0x557093379250 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc41c01ed10 con 0x557093379250 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 10 client.27776 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 10 client.27776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 20 client.27776 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x557093395010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 4 client.27776 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55709301d720 con 0x557093392bf0 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.801+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x557093395010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.802+0000 7fc40b7fe640 10 client.27776 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.802+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x557093395010 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87a7bacb8b147804 server_cookie=7626c57cfe45ce92 in_seq=0 out_seq=0 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.802+0000 7fc40b7fe640 1 -- 172.21.15.45:0/1014344427 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55709301d720 con 0x557093392bf0 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.802+0000 7fc40b7fe640 10 client.27776 handle_command_reply: tid=0 2024-03-12T05:21:55.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.802+0000 7fc434e61740 1 client.27776 shutdown 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 msgr2=0x55709337f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0116f0 tx=0x7fc41c004730 comp rx=0 tx=0).stop 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 shutdown_connections 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x557093395010 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093379b70 0x55709337fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc40b7fe640 0 client.27776 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093382750 0x557093380180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 --2- 172.21.15.45:0/1014344427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x55709337f700 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 >> 172.21.15.45:0/1014344427 conn(0x557093373980 msgr2=0x5570933756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 shutdown_connections 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 1 -- 172.21.15.45:0/1014344427 wait complete. 2024-03-12T05:21:55.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.803+0000 7fc434e61740 20 client.27776 trim_cache size 0 max 16384 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 Processor -- start 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 -- start start 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093381e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379b70 0x55709337ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557093266130 con 0x557093382750 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557093385420 con 0x557093379b70 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.821+0000 7fc434e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55709301d720 con 0x557093379250 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379b70 0x55709337ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44062/0 (socket says 172.21.15.45:44062) 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 -- 172.21.15.45:0/1722377988 learned_addr learned my addr 172.21.15.45:0/1722377988 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc42bfff640 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093381e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 -- 172.21.15.45:0/1722377988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 msgr2=0x557093381e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093381e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 -- 172.21.15.45:0/1722377988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379b70 msgr2=0x55709337ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379b70 0x55709337ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 -- 172.21.15.45:0/1722377988 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55709327bc20 con 0x557093382750 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc430833640 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc42409a8c0 tx=0x7fc42409a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cfaa565e33919b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc411ffb640 1 -- 172.21.15.45:0/1722377988 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240a30d0 con 0x557093382750 2024-03-12T05:21:55.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc411ffb640 1 -- 172.21.15.45:0/1722377988 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc4240b2070 con 0x557093382750 2024-03-12T05:21:55.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.822+0000 7fc411ffb640 1 -- 172.21.15.45:0/1722377988 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240042d0 con 0x557093382750 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 -- 172.21.15.45:0/1722377988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 msgr2=0x557093399990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc42409a8c0 tx=0x7fc42409a470 comp rx=0 tx=0).stop 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 -- 172.21.15.45:0/1722377988 shutdown_connections 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093399990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379b70 0x55709337ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.823+0000 7fc434e61740 1 --2- 172.21.15.45:0/1722377988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379250 0x557093381e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.824+0000 7fc434e61740 1 -- 172.21.15.45:0/1722377988 >> 172.21.15.45:0/1722377988 conn(0x557093373980 msgr2=0x55709336ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.824+0000 7fc434e61740 1 -- 172.21.15.45:0/1722377988 shutdown_connections 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.824+0000 7fc434e61740 1 -- 172.21.15.45:0/1722377988 wait complete. 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.824+0000 7fc434e61740 1 -- start start 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 Processor -- start 2024-03-12T05:21:55.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x557093387fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093388a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557093389f00 con 0x557093382750 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570933e9120 con 0x557093379250 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570933e9290 con 0x557093379b70 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc434e61740 10 client.0 _ll_register_callbacks cb 0x7fc4240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc42b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51738/0 (socket says 172.21.15.45:51738) 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2786482674 learned_addr learned my addr 172.21.15.45:0/2786482674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc42bfff640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x557093387fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.825+0000 7fc430833640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093388a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 msgr2=0x557093387fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x557093387fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2786482674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 msgr2=0x557093388a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:55.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093388a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 -- 172.21.15.45:0/2786482674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c1a2470 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42bfff640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x557093387fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc42b7fe640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x557093378d20 tx=0x7fc42003cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4351faf5e91f771d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc40b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42005c040 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc420004b70 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.826+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc4200050b0 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc42005eb70 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc434e61740 10 client.27775 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570932a18c0 con 0x557093379b70 2024-03-12T05:21:55.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc42006b020 con 0x557093379b70 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc42004d070 con 0x557093379b70 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc434e61740 10 client.27775 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc434e61740 10 client.27775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.827+0000 7fc434e61740 20 client.27775 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.828+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x5570933ec960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.828+0000 7fc434e61740 4 client.27775 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.828+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55709306c890 con 0x557093392bf0 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.828+0000 7fc42bfff640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x5570933ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.829+0000 7fc42bfff640 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x5570933ec960 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1afbfc693f015054 server_cookie=8afa322a3400583d in_seq=0 out_seq=0 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.829+0000 7fc40b7fe640 10 client.27775 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:55.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc40b7fe640 1 -- 172.21.15.45:0/2786482674 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55709306c890 con 0x557093392bf0 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc40b7fe640 10 client.27775 handle_command_reply: tid=0 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 client.27775 shutdown 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 msgr2=0x5570933884f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x557093378d20 tx=0x7fc42003cc70 comp rx=0 tx=0).stop 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 shutdown_connections 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557093392bf0 0x5570933ec960 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557093382750 0x557093388a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc40b7fe640 0 client.27775 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557093379b70 0x5570933884f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 --2- 172.21.15.45:0/2786482674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557093379250 0x557093387fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.830+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 >> 172.21.15.45:0/2786482674 conn(0x557093373980 msgr2=0x557093399990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.831+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 shutdown_connections 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.831+0000 7fc434e61740 1 -- 172.21.15.45:0/2786482674 wait complete. 2024-03-12T05:21:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.831+0000 7fc434e61740 20 client.27775 trim_cache size 0 max 16384 2024-03-12T05:21:55.830 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:21:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 msgr2=0x7fc4140a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 0x7fc4140a40e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc41c005d00 tx=0x7fc41c00b410 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 msgr2=0x7fc42c06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc42400efd0 tx=0x7fc42400c6a0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 shutdown_connections 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4140a1c20 0x7fc4140a40e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc42c06e020 0x7fc42c06e4a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc42c108660 0x7fc42c10f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 --2- 172.21.15.45:0/4127260527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc42c103590 0x7fc42c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 >> 172.21.15.45:0/4127260527 conn(0x7fc42c0fedd0 msgr2=0x7fc42c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 shutdown_connections 2024-03-12T05:21:55.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:55.832+0000 7fc432296640 1 -- 172.21.15.45:0/4127260527 wait complete. 2024-03-12T05:21:56.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:56 smithi130 ceph-mon[35398]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 28 MiB/s wr, 1.85k op/s 2024-03-12T05:21:56.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:56 smithi116 ceph-mon[35486]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 28 MiB/s wr, 1.85k op/s 2024-03-12T05:21:56.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:56 smithi045 ceph-mon[32174]: pgmap v344: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 28 MiB/s wr, 1.85k op/s 2024-03-12T05:21:56.843 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:21:56.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:21:56.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:21:57.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.007+0000 7f05e0763640 1 -- 172.21.15.45:0/2374291834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d8100730 msgr2=0x7f05d8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.007+0000 7f05e0763640 1 --2- 172.21.15.45:0/2374291834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d8100730 0x7f05d8100bb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009930 tx=0x7f05c802bc30 comp rx=0 tx=0).stop 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.008+0000 7f05e0763640 1 -- 172.21.15.45:0/2374291834 shutdown_connections 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.008+0000 7f05e0763640 1 --2- 172.21.15.45:0/2374291834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d81010f0 0x7f05d810cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.008+0000 7f05e0763640 1 --2- 172.21.15.45:0/2374291834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d8100730 0x7f05d8100bb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.008+0000 7f05e0763640 1 --2- 172.21.15.45:0/2374291834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 0x7f05d8105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.008+0000 7f05e0763640 1 -- 172.21.15.45:0/2374291834 >> 172.21.15.45:0/2374291834 conn(0x7f05d80765a0 msgr2=0x7f05d80fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.009+0000 7f05e0763640 1 -- 172.21.15.45:0/2374291834 shutdown_connections 2024-03-12T05:21:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.009+0000 7f05e0763640 1 -- 172.21.15.45:0/2374291834 wait complete. 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.009+0000 7f05e0763640 1 Processor -- start 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.009+0000 7f05e0763640 1 -- start start 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 0x7f05d819bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819c510 con 0x7f05d8100730 2024-03-12T05:21:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819c680 con 0x7f05d81010f0 2024-03-12T05:21:57.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05e0763640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d819c840 con 0x7f05d8105050 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05ddcfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05ddcfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60940/0 (socket says 172.21.15.45:60940) 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05ded00640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 0x7f05d819bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05ddcfe640 1 -- 172.21.15.45:0/1987215006 learned_addr learned my addr 172.21.15.45:0/1987215006 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44098/0 (socket says 172.21.15.45:44098) 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 -- 172.21.15.45:0/1987215006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 msgr2=0x7f05d819bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 0x7f05d819bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 -- 172.21.15.45:0/1987215006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 msgr2=0x7f05d8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 -- 172.21.15.45:0/1987215006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0009840 con 0x7f05d8100730 2024-03-12T05:21:57.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05ddcfe640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.010+0000 7f05de4ff640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f05d400e9f0 tx=0x7f05d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.011+0000 7f05cf7fe640 1 -- 172.21.15.45:0/1987215006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d400ccc0 con 0x7f05d8100730 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.011+0000 7f05cf7fe640 1 -- 172.21.15.45:0/1987215006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05d40044d0 con 0x7f05d8100730 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.011+0000 7f05cf7fe640 1 -- 172.21.15.45:0/1987215006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4003e60 con 0x7f05d8100730 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.011+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d4009490 con 0x7f05d8100730 2024-03-12T05:21:57.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.011+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c8009560 con 0x7f05d8100730 2024-03-12T05:21:57.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.014+0000 7f05e340c740 1 Processor -- start 2024-03-12T05:21:57.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.014+0000 7f05cf7fe640 1 -- 172.21.15.45:0/1987215006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f05d4003fe0 con 0x7f05d8100730 2024-03-12T05:21:57.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.015+0000 7f05e340c740 1 -- start start 2024-03-12T05:21:57.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.017+0000 7f05e340c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.017+0000 7f05e340c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.017+0000 7f05cf7fe640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 0x7f05c40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05e340c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc1f0c0 0x5636cfc214d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05e340c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfb7bf90 con 0x5636cfc1f0c0 2024-03-12T05:21:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05e340c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfb794c0 con 0x5636cfc0d320 2024-03-12T05:21:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05e340c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfb74ca0 con 0x5636cfc0dcd0 2024-03-12T05:21:57.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60950/0 (socket says 172.21.15.45:60950) 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 -- 172.21.15.45:0/19125378 learned_addr learned my addr 172.21.15.45:0/19125378 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc1f0c0 0x5636cfc214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ddcfe640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 0x7f05c40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05cf7fe640 1 -- 172.21.15.45:0/1987215006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f05d4014070 con 0x7f05d8100730 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05de4ff640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 -- 172.21.15.45:0/19125378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 msgr2=0x5636cfc1eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 -- 172.21.15.45:0/19125378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc1f0c0 msgr2=0x5636cfc214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc1f0c0 0x5636cfc214d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05ded00640 1 -- 172.21.15.45:0/19125378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636cfb370b0 con 0x5636cfc0d320 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.018+0000 7f05de4ff640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.019+0000 7f05ddcfe640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 0x7f05c40a40e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05d80fec50 tx=0x7f05c8000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.019+0000 7f05ded00640 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f05d0013ce0 tx=0x7f05d0009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9604c8f050378c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.019+0000 7f05ccff9640 1 -- 172.21.15.45:0/19125378 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0013dd0 con 0x5636cfc0d320 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.019+0000 7f05ccff9640 1 -- 172.21.15.45:0/19125378 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05d0003e20 con 0x5636cfc0d320 2024-03-12T05:21:57.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.019+0000 7f05ccff9640 1 -- 172.21.15.45:0/19125378 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0004630 con 0x5636cfc0d320 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- 172.21.15.45:0/19125378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 msgr2=0x5636cfc0d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f05d0013ce0 tx=0x7f05d0009cd0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- 172.21.15.45:0/19125378 shutdown_connections 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc1f0c0 0x5636cfc214d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 --2- 172.21.15.45:0/19125378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0d320 0x5636cfc0d700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- 172.21.15.45:0/19125378 >> 172.21.15.45:0/19125378 conn(0x5636cfc07840 msgr2=0x5636cfc07c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- 172.21.15.45:0/19125378 shutdown_connections 2024-03-12T05:21:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- 172.21.15.45:0/19125378 wait complete. 2024-03-12T05:21:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.022+0000 7f05e340c740 1 -- start start 2024-03-12T05:21:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 Processor -- start 2024-03-12T05:21:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc1c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 0x5636cfc1caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfb19f40 con 0x5636cfc0d320 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc1c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cf8484b0 con 0x5636cfc0dcd0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc1c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44110/0 (socket says 172.21.15.45:44110) 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfafa670 con 0x5636cfc1f0c0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05de4ff640 1 -- 172.21.15.45:0/3924134005 learned_addr learned my addr 172.21.15.45:0/3924134005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05e340c740 10 client.0 _ll_register_callbacks cb 0x5636cfb67f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05ded00640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 0x5636cfc1caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.023+0000 7f05ddcfe640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 -- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0dcd0 msgr2=0x5636cfc1c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 -- 172.21.15.45:0/3924134005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 msgr2=0x5636cfc1c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc1c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 -- 172.21.15.45:0/3924134005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d001b040 con 0x5636cfc1f0c0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05ded00640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 0x5636cfc1caa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05d0013ce0 tx=0x7f05d00045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd6ddb3318efd8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.024+0000 7f05c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0003970 con 0x5636cfc1f0c0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f05d0003ad0 con 0x5636cfc1f0c0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636cfb4ef40 con 0x5636cfc1f0c0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05d0003c90 con 0x5636cfc1f0c0 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05e340c740 10 client.27787 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.025+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5636cfb73320 con 0x5636cfc1f0c0 2024-03-12T05:21:57.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05d0035020 con 0x5636cfc1f0c0 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f05d0020b70 con 0x5636cfc1f0c0 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 10 client.27787 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 10 client.27787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 20 client.27787 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc2f410 0x5636cfc31830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 4 client.27787 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.026+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636cfb6b310 con 0x5636cfc2f410 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.027+0000 7f05de4ff640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc2f410 0x5636cfc31830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.027+0000 7f05c1ffb640 10 client.27787 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.028+0000 7f05de4ff640 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc2f410 0x5636cfc31830 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31acf492dd26a1bf server_cookie=bf54ad8e234ce824 in_seq=0 out_seq=0 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.028+0000 7f05c1ffb640 1 -- 172.21.15.45:0/3924134005 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5636cfb6b310 con 0x5636cfc2f410 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.028+0000 7f05c1ffb640 10 client.27787 handle_command_reply: tid=0 2024-03-12T05:21:57.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.028+0000 7f05e340c740 1 client.27787 shutdown 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 msgr2=0x5636cfc1caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 0x5636cfc1caa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f05d0013ce0 tx=0x7f05d00045e0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 shutdown_connections 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc2f410 0x5636cfc31830 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc1f0c0 0x5636cfc1caa0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc0dcd0 0x5636cfc1c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 --2- 172.21.15.45:0/3924134005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc1c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 >> 172.21.15.45:0/3924134005 conn(0x5636cfc07840 msgr2=0x5636cfc097a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05c1ffb640 0 client.27787 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 shutdown_connections 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 1 -- 172.21.15.45:0/3924134005 wait complete. 2024-03-12T05:21:57.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.029+0000 7f05e340c740 20 client.27787 trim_cache size 0 max 16384 2024-03-12T05:21:57.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 Processor -- start 2024-03-12T05:21:57.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 -- start start 2024-03-12T05:21:57.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0dcd0 0x5636cfc094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc26090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfb6b310 con 0x5636cfc0dcd0 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfc076d0 con 0x5636cfc1f0c0 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05e340c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfc2c0a0 con 0x5636cfc0d320 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51786/0 (socket says 172.21.15.45:51786) 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05de4ff640 1 -- 172.21.15.45:0/299935944 learned_addr learned my addr 172.21.15.45:0/299935944 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05ddcfe640 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0dcd0 0x5636cfc094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.051+0000 7f05ded00640 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc26090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 -- 172.21.15.45:0/299935944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 msgr2=0x5636cfc26090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc26090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 -- 172.21.15.45:0/299935944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0dcd0 msgr2=0x5636cfc094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0dcd0 0x5636cfc094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 -- 172.21.15.45:0/299935944 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636cfb0fc00 con 0x5636cfc0d320 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05de4ff640 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f05d4096bf0 tx=0x7f05d4003780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dbd9a217eaf0886 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05c3fff640 1 -- 172.21.15.45:0/299935944 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d4060ea0 con 0x5636cfc0d320 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05c3fff640 1 -- 172.21.15.45:0/299935944 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05d409e8e0 con 0x5636cfc0d320 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.052+0000 7f05c3fff640 1 -- 172.21.15.45:0/299935944 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d40cb530 con 0x5636cfc0d320 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.053+0000 7f05e340c740 1 -- 172.21.15.45:0/299935944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 msgr2=0x5636cfc1c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.053+0000 7f05e340c740 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f05d4096bf0 tx=0x7f05d4003780 comp rx=0 tx=0).stop 2024-03-12T05:21:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.054+0000 7f05e340c740 1 -- 172.21.15.45:0/299935944 shutdown_connections 2024-03-12T05:21:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.054+0000 7f05e340c740 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0dcd0 0x5636cfc094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.054+0000 7f05e340c740 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc26090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.054+0000 7f05e340c740 1 --2- 172.21.15.45:0/299935944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0d320 0x5636cfc1c8c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.054+0000 7f05e340c740 1 -- 172.21.15.45:0/299935944 >> 172.21.15.45:0/299935944 conn(0x5636cfc07840 msgr2=0x5636cfc08de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.055+0000 7f05e340c740 1 -- 172.21.15.45:0/299935944 shutdown_connections 2024-03-12T05:21:57.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.055+0000 7f05e340c740 1 -- 172.21.15.45:0/299935944 wait complete. 2024-03-12T05:21:57.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.056+0000 7f05e340c740 1 -- start start 2024-03-12T05:21:57.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.056+0000 7f05e340c740 1 Processor -- start 2024-03-12T05:21:57.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc25260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc257a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc85920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfc85e60 con 0x5636cfc0d320 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc25260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05ddcfe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05ded00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc85920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05de4ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc25260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44132/0 (socket says 172.21.15.45:44132) 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05de4ff640 1 -- 172.21.15.45:0/2202280853 learned_addr learned my addr 172.21.15.45:0/2202280853 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfc85fa0 con 0x5636cfc1f0c0 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636cfc860e0 con 0x5636cfc0dcd0 2024-03-12T05:21:57.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05e340c740 10 client.0 _ll_register_callbacks cb 0x7f05d0013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05ded00640 1 -- 172.21.15.45:0/2202280853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 msgr2=0x5636cfc257a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.057+0000 7f05ded00640 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc257a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05ded00640 1 -- 172.21.15.45:0/2202280853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 msgr2=0x5636cfc25260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05ded00640 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc25260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05ded00640 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c8009560 con 0x5636cfc1f0c0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05ded00640 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc85920 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f05d0005c10 tx=0x7f05d0013510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b3c155efe7553e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d00306f0 con 0x5636cfc1f0c0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f05d0036080 con 0x5636cfc1f0c0 2024-03-12T05:21:57.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.058+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636cfb9a900 con 0x5636cfc1f0c0 2024-03-12T05:21:57.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.059+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05d0030ec0 con 0x5636cfc1f0c0 2024-03-12T05:21:57.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.059+0000 7f05e340c740 10 client.27800 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.059+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5636cfb35910 con 0x5636cfc1f0c0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.059+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5636cfb9a900 con 0x5636cfc1f0c0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.059+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f05d0038070 con 0x5636cfc1f0c0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 10 client.27800 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 10 client.27800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 20 client.27800 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc875f0 0x5636cfc899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 4 client.27800 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5636cf874860 con 0x5636cfc875f0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.060+0000 7f05de4ff640 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc875f0 0x5636cfc899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.061+0000 7f05de4ff640 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc875f0 0x5636cfc899e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=813d4a3338b1dbda server_cookie=e7b11ec49bf53218 in_seq=0 out_seq=0 2024-03-12T05:21:57.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.061+0000 7f05c1ffb640 10 client.27800 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.062+0000 7f05c1ffb640 1 -- 172.21.15.45:0/2202280853 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5636cf874860 con 0x5636cfc875f0 2024-03-12T05:21:57.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.062+0000 7f05c1ffb640 10 client.27800 handle_command_reply: tid=0 2024-03-12T05:21:57.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.062+0000 7f05e340c740 1 client.27800 shutdown 2024-03-12T05:21:57.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.063+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 msgr2=0x5636cfc85920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.063+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc85920 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f05d0005c10 tx=0x7f05d0013510 comp rx=0 tx=0).stop 2024-03-12T05:21:57.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.063+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 shutdown_connections 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.063+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636cfc875f0 0x5636cfc899e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.063+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5636cfc1f0c0 0x5636cfc85920 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05c1ffb640 0 client.27800 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636cfc0dcd0 0x5636cfc257a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 1 --2- 172.21.15.45:0/2202280853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5636cfc0d320 0x5636cfc25260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 >> 172.21.15.45:0/2202280853 conn(0x5636cfc07840 msgr2=0x5636cfc1a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 shutdown_connections 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 1 -- 172.21.15.45:0/2202280853 wait complete. 2024-03-12T05:21:57.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.064+0000 7f05e340c740 20 client.27800 trim_cache size 0 max 16384 2024-03-12T05:21:57.062 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:21:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "11e72d80-fefe-417c-9fbc-061070db1c03", 2024-03-12T05:21:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:21:57.062 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.065+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 msgr2=0x7f05c40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.065+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 0x7f05c40a40e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05d80fec50 tx=0x7f05c8000bc0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.065+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 msgr2=0x7f05d8197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.065+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f05d400e9f0 tx=0x7f05d400edf0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 shutdown_connections 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05c40a1c20 0x7f05c40a40e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05d8105050 0x7f05d819bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d81010f0 0x7f05d8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 --2- 172.21.15.45:0/1987215006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05d8100730 0x7f05d8197650 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 >> 172.21.15.45:0/1987215006 conn(0x7f05d80765a0 msgr2=0x7f05d81032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 shutdown_connections 2024-03-12T05:21:57.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.066+0000 7f05e0763640 1 -- 172.21.15.45:0/1987215006 wait complete. 2024-03-12T05:21:57.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:21:57.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:21:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd391324640 1 -- 172.21.15.45:0/3393508570 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802c7b0 con 0x7fd38c0ffb70 2024-03-12T05:21:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 -- 172.21.15.45:0/3393508570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c0ffb70 msgr2=0x7fd38c111a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 --2- 172.21.15.45:0/3393508570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c0ffb70 0x7fd38c111a30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd388009860 tx=0x7fd38802bc10 comp rx=0 tx=0).stop 2024-03-12T05:21:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 -- 172.21.15.45:0/3393508570 shutdown_connections 2024-03-12T05:21:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 --2- 172.21.15.45:0/3393508570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c111f70 0x7fd38c114360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 --2- 172.21.15.45:0/3393508570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c0ffb70 0x7fd38c111a30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 --2- 172.21.15.45:0/3393508570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c0ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 -- 172.21.15.45:0/3393508570 >> 172.21.15.45:0/3393508570 conn(0x7fd38c076210 msgr2=0x7fd38c0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 -- 172.21.15.45:0/3393508570 shutdown_connections 2024-03-12T05:21:57.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.266+0000 7fd39458a640 1 -- 172.21.15.45:0/3393508570 wait complete. 2024-03-12T05:21:57.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 Processor -- start 2024-03-12T05:21:57.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 -- start start 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c1a3190 con 0x7fd38c0ff250 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c1a3300 con 0x7fd38c1a27a0 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd39458a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c1a3470 con 0x7fd38c111f70 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.267+0000 7fd392326640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd391b25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd391b25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51808/0 (socket says 172.21.15.45:51808) 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:32776/0 (socket says 172.21.15.45:32776) 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd391b25640 1 -- 172.21.15.45:0/2076417649 learned_addr learned my addr 172.21.15.45:0/2076417649 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392326640 1 -- 172.21.15.45:0/2076417649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 msgr2=0x7fd38c19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392326640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392326640 1 -- 172.21.15.45:0/2076417649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 msgr2=0x7fd38c1a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392326640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.268+0000 7fd392326640 1 -- 172.21.15.45:0/2076417649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd384009900 con 0x7fd38c0ff250 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd391b25640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:57.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd392b27640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:57.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd392326640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c19df30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd384009fd0 tx=0x7fd38400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd3837fe640 1 -- 172.21.15.45:0/2076417649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384003970 con 0x7fd38c0ff250 2024-03-12T05:21:57.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd3837fe640 1 -- 172.21.15.45:0/2076417649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd384004020 con 0x7fd38c0ff250 2024-03-12T05:21:57.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.269+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd388009490 con 0x7fd38c0ff250 2024-03-12T05:21:57.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.270+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c1aad70 con 0x7fd38c0ff250 2024-03-12T05:21:57.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.270+0000 7fd3837fe640 1 -- 172.21.15.45:0/2076417649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38400b960 con 0x7fd38c0ff250 2024-03-12T05:21:57.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 Processor -- start 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 -- start start 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bc1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2b30280 con 0x55bfb2bc12b0 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2b2d7b0 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.273+0000 7fd39723a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2b28f90 con 0x55bfb2bca7b0 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51810/0 (socket says 172.21.15.45:51810) 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 -- 172.21.15.45:0/889222883 learned_addr learned my addr 172.21.15.45:0/889222883 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd392b27640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bc1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd3837fe640 1 -- 172.21.15.45:0/2076417649 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd38400a040 con 0x7fd38c0ff250 2024-03-12T05:21:57.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 -- 172.21.15.45:0/889222883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 msgr2=0x55bfb2bca270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bca270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 -- 172.21.15.45:0/889222883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 msgr2=0x55bfb2bc1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bc1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 -- 172.21.15.45:0/889222883 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfb2ae7570 con 0x55bfb2bca7b0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.274+0000 7fd391b25640 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd388000c00 tx=0x7fd388004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d436eac7722b038 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd3837fe640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 0x7fd3780a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd3837fe640 1 -- 172.21.15.45:0/2076417649 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd3840cd8a0 con 0x7fd38c0ff250 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd380bfe640 1 -- 172.21.15.45:0/889222883 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd388034440 con 0x55bfb2bca7b0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd380bfe640 1 -- 172.21.15.45:0/889222883 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd388005040 con 0x55bfb2bca7b0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd380bfe640 1 -- 172.21.15.45:0/889222883 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802c810 con 0x55bfb2bca7b0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.275+0000 7fd392b27640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 0x7fd3780a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 -- 172.21.15.45:0/889222883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 msgr2=0x55bfb2bccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd388000c00 tx=0x7fd388004170 comp rx=0 tx=0).stop 2024-03-12T05:21:57.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd392b27640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 0x7fd3780a4060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd37c012560 tx=0x7fd37c005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 -- 172.21.15.45:0/889222883 shutdown_connections 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bccba0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 --2- 172.21.15.45:0/889222883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bc1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 -- 172.21.15.45:0/889222883 >> 172.21.15.45:0/889222883 conn(0x55bfb2bbb990 msgr2=0x55bfb2bbbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 -- 172.21.15.45:0/889222883 shutdown_connections 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.276+0000 7fd39723a740 1 -- 172.21.15.45:0/889222883 wait complete. 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 -- start start 2024-03-12T05:21:57.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 Processor -- start 2024-03-12T05:21:57.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bca260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bc7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb27fc430 con 0x55bfb2bc12b0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb27b8e00 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2aadfd0 con 0x55bfb2bca7b0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.277+0000 7fd39723a740 10 client.0 _ll_register_callbacks cb 0x55bfb2b03fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd392326640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:32792/0 (socket says 172.21.15.45:32792) 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 -- 172.21.15.45:0/2602671659 learned_addr learned my addr 172.21.15.45:0/2602671659 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 -- 172.21.15.45:0/2602671659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 msgr2=0x55bfb2bc7d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bc7d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 -- 172.21.15.45:0/2602671659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 msgr2=0x55bfb2bca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 -- 172.21.15.45:0/2602671659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c114df0 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd392326640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:57.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.278+0000 7fd391b25640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd38802bc50 tx=0x7fd3880054b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=437f85aa4443ecf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38804d070 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd388034440 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bfb28287e0 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd388005520 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd39723a740 10 client.27809 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55bfb2b27610 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.279+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd38802ca60 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd38805b070 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 10 client.27809 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 10 client.27809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 20 client.27809 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2bdca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 4 client.27809 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:21:57.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bfb28287e0 con 0x55bfb2bda610 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.280+0000 7fd392326640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2bdca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.281+0000 7fd375ffb640 10 client.27809 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.281+0000 7fd392326640 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2bdca20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=571e8e7fd1c210f6 server_cookie=d0e1ee7ffe5f61b in_seq=0 out_seq=0 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd375ffb640 1 -- 172.21.15.45:0/2602671659 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bfb28287e0 con 0x55bfb2bda610 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd375ffb640 10 client.27809 handle_command_reply: tid=0 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 client.27809 shutdown 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 msgr2=0x55bfb2bc75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd38802bc50 tx=0x7fd3880054b0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 shutdown_connections 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2bdca20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd375ffb640 0 client.27809 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bc7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bc75f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 --2- 172.21.15.45:0/2602671659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 >> 172.21.15.45:0/2602671659 conn(0x55bfb2bbb990 msgr2=0x55bfb2bbd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 shutdown_connections 2024-03-12T05:21:57.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.282+0000 7fd39723a740 1 -- 172.21.15.45:0/2602671659 wait complete. 2024-03-12T05:21:57.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.283+0000 7fd39723a740 20 client.27809 trim_cache size 0 max 16384 2024-03-12T05:21:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.313+0000 7fd39723a740 1 Processor -- start 2024-03-12T05:21:57.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.313+0000 7fd39723a740 1 -- start start 2024-03-12T05:21:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bd1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb28287e0 con 0x55bfb2bca7b0 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd392326640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd392326640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51830/0 (socket says 172.21.15.45:51830) 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd392326640 1 -- 172.21.15.45:0/2051632878 learned_addr learned my addr 172.21.15.45:0/2051632878 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd392b27640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd391b25640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bd1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2bbcc10 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.314+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2bc03a0 con 0x55bfb2bc12b0 2024-03-12T05:21:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 -- 172.21.15.45:0/2051632878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 msgr2=0x55bfb2bd60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 -- 172.21.15.45:0/2051632878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bca7b0 msgr2=0x55bfb2bd1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 -- 172.21.15.45:0/2051632878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfb2ac3ac0 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd392326640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd391b25640 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bd1600 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd38802c220 tx=0x7fd388005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce821de64706e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd377fff640 1 -- 172.21.15.45:0/2051632878 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38804d070 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd377fff640 1 -- 172.21.15.45:0/2051632878 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd388005520 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.315+0000 7fd377fff640 1 -- 172.21.15.45:0/2051632878 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38802c5b0 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 msgr2=0x55bfb2bd1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bd1600 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd38802c220 tx=0x7fd388005780 comp rx=0 tx=0).stop 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 shutdown_connections 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bd1600 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 --2- 172.21.15.45:0/2051632878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.316+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 >> 172.21.15.45:0/2051632878 conn(0x55bfb2bbb990 msgr2=0x55bfb2bc28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.317+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 shutdown_connections 2024-03-12T05:21:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.317+0000 7fd39723a740 1 -- 172.21.15.45:0/2051632878 wait complete. 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.317+0000 7fd39723a740 1 -- start start 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.317+0000 7fd39723a740 1 Processor -- start 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.317+0000 7fd39723a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bcfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2bd0ad0 con 0x55bfb2bc12b0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44196/0 (socket says 172.21.15.45:44196) 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2bd0c40 con 0x55bfb2bc1bd0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 -- 172.21.15.45:0/2782929766 learned_addr learned my addr 172.21.15.45:0/2782929766 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfb2c30b80 con 0x55bfb2bca7b0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd39723a740 10 client.0 _ll_register_callbacks cb 0x7fd384009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd391b25640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bcfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 -- 172.21.15.45:0/2782929766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 msgr2=0x55bfb2bd0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392b27640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 -- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 msgr2=0x55bfb2bcfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bcfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c1aad70 con 0x55bfb2bc12b0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd391b25640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bcfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392b27640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd392326640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd38409a1a0 tx=0x7fd38400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c59b9ec3c15399a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.318+0000 7fd375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.319+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38409e5f0 con 0x55bfb2bc12b0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.319+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd3840a2c40 con 0x55bfb2bc12b0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.319+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd388005040 con 0x55bfb2bc12b0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.319+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd384004190 con 0x55bfb2bc12b0 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.319+0000 7fd39723a740 10 client.18339 fetch_fsmap learned FSMap version 33 2024-03-12T05:21:57.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55bfb2b27610 con 0x55bfb2bc12b0 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd3840042f0 con 0x55bfb2bc12b0 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd3840961f0 con 0x55bfb2bc12b0 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd39723a740 10 client.18339 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd39723a740 10 client.18339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.320+0000 7fd39723a740 20 client.18339 populate_metadata read hostname 'smithi045' 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.321+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2c34400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.321+0000 7fd39723a740 4 client.18339 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.321+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fd388005040 con 0x55bfb2bda610 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.321+0000 7fd391b25640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2c34400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.322+0000 7fd391b25640 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2c34400 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc039a9d7150616 server_cookie=9fcd3c0f3711ba71 in_seq=0 out_seq=0 2024-03-12T05:21:57.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.322+0000 7fd375ffb640 10 client.18339 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.322+0000 7fd375ffb640 1 -- 172.21.15.45:0/2782929766 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fd388005040 con 0x55bfb2bda610 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.323+0000 7fd375ffb640 10 client.18339 handle_command_reply: tid=0 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.323+0000 7fd39723a740 1 client.18339 shutdown 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 msgr2=0x55bfb2bd1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd38409a1a0 tx=0x7fd38400cf30 comp rx=0 tx=0).stop 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 shutdown_connections 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55bfb2bda610 0x55bfb2c34400 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bfb2bca7b0 0x55bfb2bd0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bfb2bc1bd0 0x55bfb2bcfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 --2- 172.21.15.45:0/2782929766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55bfb2bc12b0 0x55bfb2bd1a30 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 >> 172.21.15.45:0/2782929766 conn(0x55bfb2bbb990 msgr2=0x55bfb2bc5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd375ffb640 0 client.18339 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.324+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 shutdown_connections 2024-03-12T05:21:57.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.325+0000 7fd39723a740 1 -- 172.21.15.45:0/2782929766 wait complete. 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.325+0000 7fd39723a740 20 client.18339 trim_cache size 0 max 16384 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "11e72d80-fefe-417c-9fbc-061070db1c03": { 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "11e72d80-fefe-417c-9fbc-061070db1c03", 2024-03-12T05:21:57.323 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:21:57.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:21:57.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:21:57.324 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:21:57.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 msgr2=0x7fd3780a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 0x7fd3780a4060 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd37c012560 tx=0x7fd37c005f90 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 msgr2=0x7fd38c19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c19df30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd384009fd0 tx=0x7fd38400c880 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 shutdown_connections 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3780a1ba0 0x7fd3780a4060 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c1a27a0 0x7fd38c1a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd38c111f70 0x7fd38c19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 --2- 172.21.15.45:0/2076417649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd38c0ff250 0x7fd38c19df30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 >> 172.21.15.45:0/2076417649 conn(0x7fd38c076210 msgr2=0x7fd38c110a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 shutdown_connections 2024-03-12T05:21:57.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:21:57.327+0000 7fd39458a640 1 -- 172.21.15.45:0/2076417649 wait complete. 2024-03-12T05:21:57.342 INFO:tasks.cephfs.filesystem:scrub status for tag:11e72d80-fefe-417c-9fbc-061070db1c03 - {'path': '/', 'tag': '11e72d80-fefe-417c-9fbc-061070db1c03', 'options': 'recursive,force'} 2024-03-12T05:21:57.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:57 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:57.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:57 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:57.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:57 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: pgmap v345: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 18 MiB/s wr, 1.23k op/s 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:21:58 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: pgmap v345: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 18 MiB/s wr, 1.23k op/s 2024-03-12T05:21:58.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:21:58.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:21:58 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: pgmap v345: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 18 MiB/s wr, 1.23k op/s 2024-03-12T05:21:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:21:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:58.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:21:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:21:58 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:21:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 595.0 s, 930.1 tps, lat 108.025 ms stddev 167.373 2024-03-12T05:22:00.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:00 smithi130 ceph-mon[35398]: pgmap v346: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 22 MiB/s wr, 1.48k op/s 2024-03-12T05:22:00.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:00 smithi116 ceph-mon[35486]: pgmap v346: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 22 MiB/s wr, 1.48k op/s 2024-03-12T05:22:00.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:00 smithi045 ceph-mon[32174]: pgmap v346: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 22 MiB/s wr, 1.48k op/s 2024-03-12T05:22:02.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:02 smithi130 ceph-mon[35398]: pgmap v347: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 23 MiB/s wr, 1.51k op/s 2024-03-12T05:22:02.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:02 smithi116 ceph-mon[35486]: pgmap v347: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 23 MiB/s wr, 1.51k op/s 2024-03-12T05:22:02.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:02 smithi045 ceph-mon[32174]: pgmap v347: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 23 MiB/s wr, 1.51k op/s 2024-03-12T05:22:04.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:04 smithi130 ceph-mon[35398]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:22:04.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:04 smithi116 ceph-mon[35486]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:22:04.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:04 smithi045 ceph-mon[32174]: pgmap v348: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 18 MiB/s wr, 1.17k op/s 2024-03-12T05:22:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 600.0 s, 1157.8 tps, lat 86.405 ms stddev 106.415 2024-03-12T05:22:06.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:06 smithi130 ceph-mon[35398]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-03-12T05:22:06.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:06 smithi116 ceph-mon[35486]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-03-12T05:22:06.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:06 smithi045 ceph-mon[32174]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 28 MiB/s wr, 1.74k op/s 2024-03-12T05:22:08.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:08 smithi130 ceph-mon[35398]: pgmap v350: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 19 MiB/s wr, 1.13k op/s 2024-03-12T05:22:08.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:08.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:08.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:08.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:08 smithi116 ceph-mon[35486]: pgmap v350: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 19 MiB/s wr, 1.13k op/s 2024-03-12T05:22:08.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:08.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:08.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:08.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:08 smithi045 ceph-mon[32174]: pgmap v350: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 19 MiB/s wr, 1.13k op/s 2024-03-12T05:22:08.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:08.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:08.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:09.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:09 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:09.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:09 smithi130 ceph-mon[35398]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T05:22:09.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:09 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:09.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:09 smithi116 ceph-mon[35486]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T05:22:09.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:09 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:09.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:09 smithi045 ceph-mon[32174]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T05:22:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 605.0 s, 997.0 tps, lat 97.056 ms stddev 106.965 2024-03-12T05:22:10.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:10 smithi130 ceph-mon[35398]: pgmap v352: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-03-12T05:22:10.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:10 smithi116 ceph-mon[35486]: pgmap v352: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-03-12T05:22:10.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:10 smithi045 ceph-mon[32174]: pgmap v352: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-03-12T05:22:12.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:12 smithi130 ceph-mon[35398]: pgmap v353: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:12.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:12 smithi116 ceph-mon[35486]: pgmap v353: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:12.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:12 smithi045 ceph-mon[32174]: pgmap v353: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:14.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:14 smithi130 ceph-mon[35398]: pgmap v354: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:14.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:14 smithi116 ceph-mon[35486]: pgmap v354: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:14.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:14 smithi045 ceph-mon[32174]: pgmap v354: 97 pgs: 19 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 66 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 22 MiB/s wr, 1.39k op/s 2024-03-12T05:22:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 610.0 s, 97.6 tps, lat 796.150 ms stddev 1481.086 2024-03-12T05:22:16.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:16 smithi130 ceph-mon[35398]: pgmap v355: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:16.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:16 smithi116 ceph-mon[35486]: pgmap v355: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:16.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:16 smithi045 ceph-mon[32174]: pgmap v355: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:18.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:18 smithi130 ceph-mon[35398]: pgmap v356: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:18.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:18 smithi116 ceph-mon[35486]: pgmap v356: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:18.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:18 smithi045 ceph-mon[32174]: pgmap v356: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 18 MiB/s wr, 1.38k op/s 2024-03-12T05:22:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 615.0 s, 675.9 tps, lat 185.852 ms stddev 709.664 2024-03-12T05:22:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:20 smithi045 ceph-mon[32174]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-03-12T05:22:20.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:20 smithi045 ceph-mon[32174]: osdmap e100: 12 total, 12 up, 12 in 2024-03-12T05:22:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:20 smithi130 ceph-mon[35398]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-03-12T05:22:20.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:20 smithi130 ceph-mon[35398]: osdmap e100: 12 total, 12 up, 12 in 2024-03-12T05:22:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:20 smithi116 ceph-mon[35486]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-03-12T05:22:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:20 smithi116 ceph-mon[35486]: osdmap e100: 12 total, 12 up, 12 in 2024-03-12T05:22:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:22 smithi045 ceph-mon[32174]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:22.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:22 smithi130 ceph-mon[35398]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:22.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:22 smithi116 ceph-mon[35486]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:24 smithi045 ceph-mon[32174]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:24 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:22:24.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 620.0 s, 1192.0 tps, lat 83.490 ms stddev 34.459 2024-03-12T05:22:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:24 smithi130 ceph-mon[35398]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:24 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:22:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:24 smithi116 ceph-mon[35486]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 17 MiB/s wr, 1.28k op/s 2024-03-12T05:22:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:24 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:22:26.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:26 smithi045 ceph-mon[32174]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:26.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:26 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:26.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:26 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:26.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:26 smithi130 ceph-mon[35398]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:26.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:26 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:26.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:26 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:26.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:26 smithi116 ceph-mon[35486]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:26.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:26 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:26.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:26 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:27.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:22:27.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:22:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/176211636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 msgr2=0x7f1a90096360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/176211636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a90096360 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009900 tx=0x7f1a9402bc40 comp rx=0 tx=0).stop 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/176211636 shutdown_connections 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/176211636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a9009b050 0x7f1a900a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/176211636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a90096930 0x7f1a9009a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/176211636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a90096360 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.522+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/176211636 >> 172.21.15.45:0/176211636 conn(0x7f1a9000ce50 msgr2=0x7f1a9000d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.523+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/176211636 shutdown_connections 2024-03-12T05:22:27.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.523+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/176211636 wait complete. 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 Processor -- start 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 -- start start 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a901326c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9009b050 0x7f1a90132c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90137920 con 0x7f1a90136f30 2024-03-12T05:22:27.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90137a90 con 0x7f1a9009b050 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e8b1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90137c00 con 0x7f1a90095f80 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e0b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a901326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e0b0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47254/0 (socket says 172.21.15.45:47254) 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a901326c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56828/0 (socket says 172.21.15.45:56828) 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2102024661 learned_addr learned my addr 172.21.15.45:0/2102024661 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.524+0000 7f1a9d0ae640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9009b050 0x7f1a90132c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2102024661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 msgr2=0x7f1a901326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a901326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2102024661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9009b050 msgr2=0x7f1a90132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9009b050 0x7f1a90132c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2102024661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a88009490 con 0x7f1a90136f30 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1a9804fc90 tx=0x7f1a98074ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a8effd640 1 -- 172.21.15.45:0/2102024661 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9806a900 con 0x7f1a90136f30 2024-03-12T05:22:27.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a8effd640 1 -- 172.21.15.45:0/2102024661 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a9807c040 con 0x7f1a90136f30 2024-03-12T05:22:27.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.525+0000 7f1a8effd640 1 -- 172.21.15.45:0/2102024661 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980822d0 con 0x7f1a90136f30 2024-03-12T05:22:27.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.526+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a94009560 con 0x7f1a90136f30 2024-03-12T05:22:27.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.526+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9013f420 con 0x7f1a90136f30 2024-03-12T05:22:27.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.530+0000 7f1a8effd640 1 -- 172.21.15.45:0/2102024661 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1a98067770 con 0x7f1a90136f30 2024-03-12T05:22:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1a8effd640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 0x7f1a840a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1aa27c7740 1 Processor -- start 2024-03-12T05:22:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1a8effd640 1 -- 172.21.15.45:0/2102024661 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a98137e60 con 0x7f1a90136f30 2024-03-12T05:22:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1aa27c7740 1 -- start start 2024-03-12T05:22:27.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 0x7f1a840a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.531+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721553450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721555d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7214b9280 con 0x55d721553990 2024-03-12T05:22:27.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7214b67b0 con 0x55d72154a490 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7214b1f90 con 0x55d72154adb0 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59740/0 (socket says 172.21.15.45:59740) 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 -- 172.21.15.45:0/469341503 learned_addr learned my addr 172.21.15.45:0/469341503 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d0ae640 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721553450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721555d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 -- 172.21.15.45:0/469341503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 msgr2=0x55d721553450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721553450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 -- 172.21.15.45:0/469341503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 msgr2=0x55d721555d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721555d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.532+0000 7f1a9d8af640 1 -- 172.21.15.45:0/469341503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d721470570 con 0x55d72154a490 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.533+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1a94035000 tx=0x7f1a94046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf69cfe1f9f98edc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.533+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 0x7f1a840a4160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a9402c250 tx=0x7f1a94004bd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.533+0000 7f1a83fff640 1 -- 172.21.15.45:0/469341503 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94004500 con 0x55d72154a490 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.533+0000 7f1a83fff640 1 -- 172.21.15.45:0/469341503 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a9402c9d0 con 0x55d72154a490 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.533+0000 7f1a83fff640 1 -- 172.21.15.45:0/469341503 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94005130 con 0x55d72154a490 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 -- 172.21.15.45:0/469341503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 msgr2=0x55d72154a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1a94035000 tx=0x7f1a94046000 comp rx=0 tx=0).stop 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 -- 172.21.15.45:0/469341503 shutdown_connections 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721555d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721553450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/469341503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154a490 0x55d72154a870 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 -- 172.21.15.45:0/469341503 >> 172.21.15.45:0/469341503 conn(0x55d721544bc0 msgr2=0x55d721544fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 -- 172.21.15.45:0/469341503 shutdown_connections 2024-03-12T05:22:27.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.534+0000 7f1aa27c7740 1 -- 172.21.15.45:0/469341503 wait complete. 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.535+0000 7f1aa27c7740 1 -- start start 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.535+0000 7f1aa27c7740 1 Processor -- start 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.535+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721550810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.535+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721550d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 0x55d72155f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721185430 con 0x55d721553990 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721141e00 con 0x55d721551290 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721436fd0 con 0x55d72154adb0 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721550d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1aa27c7740 10 client.0 _ll_register_callbacks cb 0x55d72148cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721550d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47270/0 (socket says 172.21.15.45:47270) 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9d0ae640 1 -- 172.21.15.45:0/2180429586 learned_addr learned my addr 172.21.15.45:0/2180429586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 0x55d72155f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2180429586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 msgr2=0x55d721550810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721550810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2180429586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 msgr2=0x55d721550d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721550d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.536+0000 7f1a9e0b0640 1 -- 172.21.15.45:0/2180429586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a94049050 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 0x55d72155f8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1a98135630 tx=0x7f1a98104470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1e08028ac7d7f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1a817fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a981359a0 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a98105e40 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d7211b17e0 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a98108c60 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1aa27c7740 10 client.27827 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.537+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d7214b0610 con 0x55d721551290 2024-03-12T05:22:27.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d7211b17e0 con 0x55d721551290 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1a98100960 con 0x55d721551290 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 10 client.27827 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 10 client.27827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 20 client.27827 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215640b0 0x55d7215664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 4 client.27827 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7211ee720 con 0x55d7215640b0 2024-03-12T05:22:27.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215640b0 0x55d7215664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.538+0000 7f1a817fa640 10 client.27827 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.539+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215640b0 0x55d7215664d0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c66bc99f035800fb server_cookie=e3a8ec59bfc9da07 in_seq=0 out_seq=0 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.539+0000 7f1a817fa640 1 -- 172.21.15.45:0/2180429586 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d7211ee720 con 0x55d7215640b0 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.539+0000 7f1a817fa640 10 client.27827 handle_command_reply: tid=0 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.539+0000 7f1aa27c7740 1 client.27827 shutdown 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 msgr2=0x55d72155f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 0x55d72155f8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1a98135630 tx=0x7f1a98104470 comp rx=0 tx=0).stop 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 shutdown_connections 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215640b0 0x55d7215664d0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d721550d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721551290 0x55d72155f8f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2180429586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d72154adb0 0x55d721550810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 >> 172.21.15.45:0/2180429586 conn(0x55d721544bc0 msgr2=0x55d721546910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1a817fa640 0 client.27827 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 shutdown_connections 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2180429586 wait complete. 2024-03-12T05:22:27.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.540+0000 7f1aa27c7740 20 client.27827 trim_cache size 0 max 16384 2024-03-12T05:22:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.568+0000 7f1aa27c7740 1 Processor -- start 2024-03-12T05:22:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.568+0000 7f1aa27c7740 1 -- start start 2024-03-12T05:22:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.568+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 0x55d72154f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d72154da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721556180 con 0x55d721553990 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d721540010 con 0x55d72154adb0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7211ee720 con 0x55d721551290 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 0x55d72154f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59750/0 (socket says 172.21.15.45:59750) 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 0x55d72154f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56850/0 (socket says 172.21.15.45:56850) 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 -- 172.21.15.45:0/95649429 learned_addr learned my addr 172.21.15.45:0/95649429 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 -- 172.21.15.45:0/95649429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 msgr2=0x55d72154f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 0x55d72154f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 -- 172.21.15.45:0/95649429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 msgr2=0x55d72154da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d72154da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 -- 172.21.15.45:0/95649429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d72144cac0 con 0x55d72154adb0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.569+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009900 tx=0x7f1a94005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3356469ffd3f93e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.570+0000 7f1a837fe640 1 -- 172.21.15.45:0/95649429 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94004920 con 0x55d72154adb0 2024-03-12T05:22:27.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.570+0000 7f1a837fe640 1 -- 172.21.15.45:0/95649429 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a94003e50 con 0x55d72154adb0 2024-03-12T05:22:27.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.570+0000 7f1a837fe640 1 -- 172.21.15.45:0/95649429 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9405b680 con 0x55d72154adb0 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 -- 172.21.15.45:0/95649429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 msgr2=0x55d72154f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009900 tx=0x7f1a94005fd0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 -- 172.21.15.45:0/95649429 shutdown_connections 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721553990 0x55d72154da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721551290 0x55d72154f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/95649429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d72154adb0 0x55d72154f0e0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009900 tx=0x7f1a94005fd0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 -- 172.21.15.45:0/95649429 >> 172.21.15.45:0/95649429 conn(0x55d721544bc0 msgr2=0x55d72155f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 -- 172.21.15.45:0/95649429 shutdown_connections 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.571+0000 7f1aa27c7740 1 -- 172.21.15.45:0/95649429 wait complete. 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.572+0000 7f1aa27c7740 1 -- start start 2024-03-12T05:22:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.572+0000 7f1aa27c7740 1 Processor -- start 2024-03-12T05:22:27.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721551290 0x55d7215590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 0x55d7215ba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72155af00 con 0x55d721551290 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d72155b070 con 0x55d721553990 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7215bab00 con 0x55d721559b70 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1aa27c7740 10 client.0 _ll_register_callbacks cb 0x7f1a88005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9e0b0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 0x55d7215ba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59752/0 (socket says 172.21.15.45:59752) 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 -- 172.21.15.45:0/2998256631 learned_addr learned my addr 172.21.15.45:0/2998256631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721551290 0x55d7215590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 -- 172.21.15.45:0/2998256631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 msgr2=0x55d7215ba5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 0x55d7215ba5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 -- 172.21.15.45:0/2998256631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721551290 msgr2=0x55d7215590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721551290 0x55d7215590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9d0ae640 1 -- 172.21.15.45:0/2998256631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d721470570 con 0x55d721553990 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.573+0000 7f1a9e0b0640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 0x55d7215ba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.574+0000 7f1a9d0ae640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1a88010960 tx=0x7f1a88010e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41995fe8726abaf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.574+0000 7f1a817fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:27.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.574+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a88004520 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.574+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a8800f9e0 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.574+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d721545540 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a8800fba0 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 10 client.27833 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d7214726a0 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a8802a020 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1a8801d4e0 con 0x55d721553990 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 10 client.27833 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 10 client.27833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 20 client.27833 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215bc3b0 0x55d7215be7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.575+0000 7f1aa27c7740 4 client.27833 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:22:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.576+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215bc3b0 0x55d7215be7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.576+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f1a9402c9d0 con 0x55d7215bc3b0 2024-03-12T05:22:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.576+0000 7f1a817fa640 10 client.27833 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:27.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.577+0000 7f1a9d8af640 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215bc3b0 0x55d7215be7d0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8c8d5b4af87a07 server_cookie=50391bdbe9c4fa7b in_seq=0 out_seq=0 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.577+0000 7f1a817fa640 1 -- 172.21.15.45:0/2998256631 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f1a9402c9d0 con 0x55d7215bc3b0 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.577+0000 7f1a817fa640 10 client.27833 handle_command_reply: tid=0 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.577+0000 7f1aa27c7740 1 client.27833 shutdown 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 msgr2=0x55d721559630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1a88010960 tx=0x7f1a88010e20 comp rx=0 tx=0).stop 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 shutdown_connections 2024-03-12T05:22:27.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d7215bc3b0 0x55d7215be7d0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d721559b70 0x55d7215ba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d721553990 0x55d721559630 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 --2- 172.21.15.45:0/2998256631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d721551290 0x55d7215590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 >> 172.21.15.45:0/2998256631 conn(0x55d721544bc0 msgr2=0x55d72154f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1a817fa640 0 client.27833 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 shutdown_connections 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 1 -- 172.21.15.45:0/2998256631 wait complete. 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.578+0000 7f1aa27c7740 20 client.27833 trim_cache size 0 max 16384 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:22:27.575 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 msgr2=0x7f1a840a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 0x7f1a840a4160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a9402c250 tx=0x7f1a94004bd0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 msgr2=0x7f1a901373e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1a9804fc90 tx=0x7f1a98074ef0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 shutdown_connections 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a840a1ca0 0x7f1a840a4160 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a90136f30 0x7f1a901373e0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a9009b050 0x7f1a90132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 --2- 172.21.15.45:0/2102024661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a90095f80 0x7f1a901326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 >> 172.21.15.45:0/2102024661 conn(0x7f1a9000ce50 msgr2=0x7f1a90093b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 shutdown_connections 2024-03-12T05:22:27.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.579+0000 7f1a9e8b1640 1 -- 172.21.15.45:0/2102024661 wait complete. 2024-03-12T05:22:27.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:22:27.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:22:27.591 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:27.593 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:22:27.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:22:27.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:22:27.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:22:27.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:27.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:22:27.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:22:27.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 -- 172.21.15.45:0/655732255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 msgr2=0x7fb37410b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 --2- 172.21.15.45:0/655732255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb37410b210 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009ca0 tx=0x7fb36c02c180 comp rx=0 tx=0).stop 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 -- 172.21.15.45:0/655732255 shutdown_connections 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 --2- 172.21.15.45:0/655732255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 0x7fb37410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 --2- 172.21.15.45:0/655732255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb37410b210 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 --2- 172.21.15.45:0/655732255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb3741027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 -- 172.21.15.45:0/655732255 >> 172.21.15.45:0/655732255 conn(0x7fb3740771e0 msgr2=0x7fb3740775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:27.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 -- 172.21.15.45:0/655732255 shutdown_connections 2024-03-12T05:22:27.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.770+0000 7fb37944d640 1 -- 172.21.15.45:0/655732255 wait complete. 2024-03-12T05:22:27.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.771+0000 7fb37944d640 1 Processor -- start 2024-03-12T05:22:27.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.771+0000 7fb37944d640 1 -- start start 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb374197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 0x7fb37419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37419c550 con 0x7fb374102410 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37419c6c0 con 0x7fb37410b750 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb37944d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37419c880 con 0x7fb374102d30 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb372ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb372ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47308/0 (socket says 172.21.15.45:47308) 2024-03-12T05:22:27.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb372ffd640 1 -- 172.21.15.45:0/3826383505 learned_addr learned my addr 172.21.15.45:0/3826383505 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb3727fc640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb374197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.772+0000 7fb3737fe640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 0x7fb37419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 -- 172.21.15.45:0/3826383505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 msgr2=0x7fb37419bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 0x7fb37419bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 -- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 msgr2=0x7fb374197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 -- 172.21.15.45:0/3826383505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c009900 con 0x7fb374102d30 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb372ffd640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb3727fc640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb374197bd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb36c02c6c0 tx=0x7fb36c034740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c003d80 con 0x7fb374102d30 2024-03-12T05:22:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb36c032b30 con 0x7fb374102d30 2024-03-12T05:22:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c004640 con 0x7fb374102d30 2024-03-12T05:22:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37419cb00 con 0x7fb374102d30 2024-03-12T05:22:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.773+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3741a4200 con 0x7fb374102d30 2024-03-12T05:22:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.775+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740683c0 con 0x7fb374102d30 2024-03-12T05:22:27.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.776+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb36c03b040 con 0x7fb374102d30 2024-03-12T05:22:27.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.777+0000 7fb35bfff640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 0x7fb35c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:27.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.777+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb36c0effe0 con 0x7fb374102d30 2024-03-12T05:22:27.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.777+0000 7fb372ffd640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 0x7fb35c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:27.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.778+0000 7fb372ffd640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 0x7fb35c0a4060 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb364011550 tx=0x7fb364005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:27.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:27.784+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb36c0b8f90 con 0x7fb374102d30 2024-03-12T05:22:28.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.196+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb374111110 con 0x7fb374102d30 2024-03-12T05:22:28.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.196+0000 7fb35bfff640 1 -- 172.21.15.45:0/3826383505 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fb36c0bc650 con 0x7fb374102d30 2024-03-12T05:22:28.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:22:28.192 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:22:28.192 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 msgr2=0x7fb35c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 0x7fb35c0a4060 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb364011550 tx=0x7fb364005f90 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 msgr2=0x7fb374197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb374197bd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb36c02c6c0 tx=0x7fb36c034740 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 shutdown_connections 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb35c0a1ba0 0x7fb35c0a4060 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb37410b750 0x7fb37419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb374102d30 0x7fb374197bd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 --2- 172.21.15.45:0/3826383505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb374102410 0x7fb374197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 >> 172.21.15.45:0/3826383505 conn(0x7fb3740771e0 msgr2=0x7fb3740fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 shutdown_connections 2024-03-12T05:22:28.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.200+0000 7fb37944d640 1 -- 172.21.15.45:0/3826383505 wait complete. 2024-03-12T05:22:28.213 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:22:28.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:22:28.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.400+0000 7f115bb77640 1 -- 172.21.15.45:0/2197844711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154111fc0 msgr2=0x7f11541143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.400+0000 7f115bb77640 1 --2- 172.21.15.45:0/2197844711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154111fc0 0x7f11541143b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f114c009cd0 tx=0x7f114c02c0e0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 -- 172.21.15.45:0/2197844711 shutdown_connections 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 --2- 172.21.15.45:0/2197844711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154111fc0 0x7f11541143b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 --2- 172.21.15.45:0/2197844711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1154075fd0 0x7f1154111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 --2- 172.21.15.45:0/2197844711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11540756b0 0x7f1154075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 -- 172.21.15.45:0/2197844711 >> 172.21.15.45:0/2197844711 conn(0x7f11540fedd0 msgr2=0x7f11541011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.401+0000 7f115bb77640 1 -- 172.21.15.45:0/2197844711 shutdown_connections 2024-03-12T05:22:28.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.402+0000 7f115bb77640 1 -- 172.21.15.45:0/2197844711 wait complete. 2024-03-12T05:22:28.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.403+0000 7f115bb77640 1 Processor -- start 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.403+0000 7f115bb77640 1 -- start start 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f115bb77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 0x7f11541a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f115bb77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 0x7f11541a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f115bb77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 0x7f115419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f1159913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 0x7f11541a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f1159913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 0x7f11541a3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56884/0 (socket says 172.21.15.45:56884) 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f1159112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 0x7f11541a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f1159112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 0x7f11541a3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59792/0 (socket says 172.21.15.45:59792) 2024-03-12T05:22:28.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.404+0000 7f115bb77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419df80 con 0x7f1154111fc0 2024-03-12T05:22:28.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115bb77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419e0f0 con 0x7f1154075fd0 2024-03-12T05:22:28.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 0x7f115419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f1159112640 1 -- 172.21.15.45:0/979623535 learned_addr learned my addr 172.21.15.45:0/979623535 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115419e2b0 con 0x7f11540756b0 2024-03-12T05:22:28.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 -- 172.21.15.45:0/979623535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 msgr2=0x7f11541a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 0x7f11541a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 -- 172.21.15.45:0/979623535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 msgr2=0x7f11541a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 0x7f11541a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.405+0000 7f115a114640 1 -- 172.21.15.45:0/979623535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150009490 con 0x7f1154111fc0 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f115a114640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 0x7f115419d930 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f114c02c6f0 tx=0x7f114c035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f114affd640 1 -- 172.21.15.45:0/979623535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0329e0 con 0x7f1154111fc0 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c009900 con 0x7f1154111fc0 2024-03-12T05:22:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115419e7b0 con 0x7f1154111fc0 2024-03-12T05:22:28.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f114affd640 1 -- 172.21.15.45:0/979623535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f114c035e90 con 0x7f1154111fc0 2024-03-12T05:22:28.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.406+0000 7f114affd640 1 -- 172.21.15.45:0/979623535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0046a0 con 0x7f1154111fc0 2024-03-12T05:22:28.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.410+0000 7f115e825740 1 Processor -- start 2024-03-12T05:22:28.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.410+0000 7f114affd640 1 -- 172.21.15.45:0/979623535 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f114c03b040 con 0x7f1154111fc0 2024-03-12T05:22:28.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.411+0000 7f115e825740 1 -- start start 2024-03-12T05:22:28.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.411+0000 7f114affd640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 0x7f11400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f114affd640 1 -- 172.21.15.45:0/979623535 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f114c0ef9a0 con 0x7f1154111fc0 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1b70 0x55b5d38fa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d38ac960 con 0x55b5d38f1250 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3860310 con 0x55b5d38f1b70 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f115e825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d385d840 con 0x55b5d38fa750 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f1159112640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 0x7f11400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.412+0000 7f1159913640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47332/0 (socket says 172.21.15.45:47332) 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 -- 172.21.15.45:0/3268227298 learned_addr learned my addr 172.21.15.45:0/3268227298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159112640 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1b70 0x55b5d38fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f115a114640 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 -- 172.21.15.45:0/3268227298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 msgr2=0x55b5d38fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 -- 172.21.15.45:0/3268227298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1b70 msgr2=0x55b5d38fa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1b70 0x55b5d38fa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159913640 1 -- 172.21.15.45:0/3268227298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5d381b060 con 0x55b5d38f1250 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.413+0000 7f1159112640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 0x7f11400a40e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1150009710 tx=0x7f1150013aa0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.414+0000 7f1159913640 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1144009c90 tx=0x7f1144011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f044d188ae77a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.414+0000 7f113ffff640 1 -- 172.21.15.45:0/3268227298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144005220 con 0x55b5d38f1250 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.414+0000 7f113ffff640 1 -- 172.21.15.45:0/3268227298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1144004c50 con 0x55b5d38f1250 2024-03-12T05:22:28.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.414+0000 7f113ffff640 1 -- 172.21.15.45:0/3268227298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144019690 con 0x55b5d38f1250 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.415+0000 7f115e825740 1 -- 172.21.15.45:0/3268227298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 msgr2=0x55b5d38f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.415+0000 7f115e825740 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1144009c90 tx=0x7f1144011810 comp rx=0 tx=0).stop 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 -- 172.21.15.45:0/3268227298 shutdown_connections 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1b70 0x55b5d38fa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 --2- 172.21.15.45:0/3268227298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1250 0x55b5d38f1630 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 -- 172.21.15.45:0/3268227298 >> 172.21.15.45:0/3268227298 conn(0x55b5d38eb980 msgr2=0x55b5d38ebd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 -- 172.21.15.45:0/3268227298 shutdown_connections 2024-03-12T05:22:28.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.417+0000 7f115e825740 1 -- 172.21.15.45:0/3268227298 wait complete. 2024-03-12T05:22:28.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.418+0000 7f115e825740 1 -- start start 2024-03-12T05:22:28.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 Processor -- start 2024-03-12T05:22:28.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3859020 con 0x55b5d38f1b70 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d352c430 con 0x55b5d38f1250 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d34e8e00 con 0x55b5d38fa750 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.419+0000 7f115e825740 10 client.0 _ll_register_callbacks cb 0x55b5d3800cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59798/0 (socket says 172.21.15.45:59798) 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 -- 172.21.15.45:0/381850597 learned_addr learned my addr 172.21.15.45:0/381850597 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f115a114640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 -- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 msgr2=0x55b5d38f81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38f81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 -- 172.21.15.45:0/381850597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 msgr2=0x55b5d38f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.420+0000 7f1159913640 1 -- 172.21.15.45:0/381850597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11440098f0 con 0x55b5d38f1250 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.421+0000 7f1159913640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f11440055b0 tx=0x7f11440048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=955ac2551db01050 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.421+0000 7f113d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:28.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.421+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144023840 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.421+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1144004dc0 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.422+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5d35587e0 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.422+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1144004a60 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.422+0000 7f115e825740 10 client.27851 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.422+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b5d38576a0 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b5d35587e0 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1144021040 con 0x55b5d38f1250 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 10 client.27851 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 10 client.27851 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 20 client.27851 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d390abb0 0x55b5d390cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 4 client.27851 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:28.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.423+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5d3595720 con 0x55b5d390abb0 2024-03-12T05:22:28.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.424+0000 7f1159112640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d390abb0 0x55b5d390cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.424+0000 7f1159112640 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d390abb0 0x55b5d390cfd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eb36a13cf250960a server_cookie=1991b49f738c307c in_seq=0 out_seq=0 2024-03-12T05:22:28.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.424+0000 7f113d7fa640 10 client.27851 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:22:28.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.425+0000 7f113d7fa640 1 -- 172.21.15.45:0/381850597 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b5d3595720 con 0x55b5d390abb0 2024-03-12T05:22:28.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.425+0000 7f113d7fa640 10 client.27851 handle_command_reply: tid=0 2024-03-12T05:22:28.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.425+0000 7f115e825740 1 client.27851 shutdown 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 msgr2=0x55b5d38f7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f11440055b0 tx=0x7f11440048f0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 shutdown_connections 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d390abb0 0x55b5d390cfd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38fa750 0x55b5d38f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f113d7fa640 0 client.27851 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:22:28.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 --2- 172.21.15.45:0/381850597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d38f7730 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 >> 172.21.15.45:0/381850597 conn(0x55b5d38eb980 msgr2=0x55b5d38ed6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 shutdown_connections 2024-03-12T05:22:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 1 -- 172.21.15.45:0/381850597 wait complete. 2024-03-12T05:22:28.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.426+0000 7f115e825740 20 client.27851 trim_cache size 0 max 16384 2024-03-12T05:22:28.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.448+0000 7f115e825740 1 Processor -- start 2024-03-12T05:22:28.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 -- start start 2024-03-12T05:22:28.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38fa750 0x55b5d38f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d37de130 con 0x55b5d38f1b70 2024-03-12T05:22:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d38f0d80 con 0x55b5d38fa750 2024-03-12T05:22:28.446 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.450+0000 7f115e825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3907670 con 0x55b5d38f1250 2024-03-12T05:22:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56906/0 (socket says 172.21.15.45:56906) 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 -- 172.21.15.45:0/290147193 learned_addr learned my addr 172.21.15.45:0/290147193 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f115a114640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38fa750 0x55b5d38f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159112640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 -- 172.21.15.45:0/290147193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38fa750 msgr2=0x55b5d38f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38fa750 0x55b5d38f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 -- 172.21.15.45:0/290147193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 msgr2=0x55b5d38f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159913640 1 -- 172.21.15.45:0/290147193 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5d37f3c20 con 0x55b5d38f1250 2024-03-12T05:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.451+0000 7f1159112640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.452+0000 7f1159913640 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1144011850 tx=0x7f1144009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8721989740cdb6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.452+0000 7f113f7fe640 1 -- 172.21.15.45:0/290147193 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400eea0 con 0x55b5d38f1250 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.452+0000 7f113f7fe640 1 -- 172.21.15.45:0/290147193 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1144005410 con 0x55b5d38f1250 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.452+0000 7f113f7fe640 1 -- 172.21.15.45:0/290147193 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144004290 con 0x55b5d38f1250 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 -- 172.21.15.45:0/290147193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 msgr2=0x55b5d38f9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1144011850 tx=0x7f1144009310 comp rx=0 tx=0).stop 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 -- 172.21.15.45:0/290147193 shutdown_connections 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38f1b70 0x55b5d38f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38fa750 0x55b5d38f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 --2- 172.21.15.45:0/290147193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1250 0x55b5d38f9e60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.453+0000 7f115e825740 1 -- 172.21.15.45:0/290147193 >> 172.21.15.45:0/290147193 conn(0x55b5d38eb980 msgr2=0x55b5d38f2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.454+0000 7f115e825740 1 -- 172.21.15.45:0/290147193 shutdown_connections 2024-03-12T05:22:28.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.454+0000 7f115e825740 1 -- 172.21.15.45:0/290147193 wait complete. 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.454+0000 7f115e825740 1 -- start start 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.454+0000 7f115e825740 1 Processor -- start 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d39001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 0x55b5d39006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 0x55b5d3900c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3961100 con 0x55b5d38fa750 2024-03-12T05:22:28.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3961240 con 0x55b5d38f1250 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5d3961400 con 0x55b5d38f1b70 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115e825740 10 client.0 _ll_register_callbacks cb 0x7f11440097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f1159112640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 0x55b5d39006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115a114640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 0x55b5d3900c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f1159112640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 0x55b5d39006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56918/0 (socket says 172.21.15.45:56918) 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f115a114640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 0x55b5d3900c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47346/0 (socket says 172.21.15.45:47346) 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f1159112640 1 -- 172.21.15.45:0/3735631227 learned_addr learned my addr 172.21.15.45:0/3735631227 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.455+0000 7f1159913640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d39001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 -- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 msgr2=0x55b5d39006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 0x55b5d39006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 -- 172.21.15.45:0/3735631227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 msgr2=0x55b5d3900c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 0x55b5d3900c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 -- 172.21.15.45:0/3735631227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5d38576a0 con 0x55b5d38f1250 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f1159913640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d39001a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f114402ebc0 tx=0x7f114402ea70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5685f9af91b4ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f113d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440055b0 con 0x55b5d38f1250 2024-03-12T05:22:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1144000e40 con 0x55b5d38f1250 2024-03-12T05:22:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.456+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1144004c50 con 0x55b5d38f1250 2024-03-12T05:22:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.457+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1144009b10 con 0x55b5d38f1250 2024-03-12T05:22:28.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.458+0000 7f115e825740 10 client.27857 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.458+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b5d38198c0 con 0x55b5d38f1250 2024-03-12T05:22:28.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.459+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f114404c020 con 0x55b5d38f1250 2024-03-12T05:22:28.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1144038550 con 0x55b5d38f1250 2024-03-12T05:22:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 10 client.27857 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 10 client.27857 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:22:28.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 20 client.27857 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d3962740 0x55b5d3964b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 4 client.27857 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:22:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.462+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b5d35e4890 con 0x55b5d3962740 2024-03-12T05:22:28.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.463+0000 7f1159112640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d3962740 0x55b5d3964b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.464+0000 7f1159112640 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d3962740 0x55b5d3964b30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=91028d2638fcf4c server_cookie=438f93ed00345568 in_seq=0 out_seq=0 2024-03-12T05:22:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.464+0000 7f113d7fa640 10 client.27857 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:22:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.464+0000 7f113d7fa640 1 -- 172.21.15.45:0/3735631227 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b5d35e4890 con 0x55b5d3962740 2024-03-12T05:22:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.464+0000 7f113d7fa640 10 client.27857 handle_command_reply: tid=0 2024-03-12T05:22:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.464+0000 7f115e825740 1 client.27857 shutdown 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 msgr2=0x55b5d39001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d39001a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f114402ebc0 tx=0x7f114402ea70 comp rx=0 tx=0).stop 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 shutdown_connections 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55b5d3962740 0x55b5d3964b30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b5d38fa750 0x55b5d3900c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5d38f1b70 0x55b5d39006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f113d7fa640 0 client.27857 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 --2- 172.21.15.45:0/3735631227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5d38f1250 0x55b5d39001a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 >> 172.21.15.45:0/3735631227 conn(0x55b5d38eb980 msgr2=0x55b5d38ea2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 shutdown_connections 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 1 -- 172.21.15.45:0/3735631227 wait complete. 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.465+0000 7f115e825740 20 client.27857 trim_cache size 0 max 16384 2024-03-12T05:22:28.461 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 msgr2=0x7f11400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 0x7f11400a40e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1150009710 tx=0x7f1150013aa0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 msgr2=0x7f115419d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 0x7f115419d930 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f114c02c6f0 tx=0x7f114c035ca0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 shutdown_connections 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f11400a1c20 0x7f11400a40e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1154111fc0 0x7f115419d930 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1154075fd0 0x7f11541a3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 --2- 172.21.15.45:0/979623535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11540756b0 0x7f11541a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 >> 172.21.15.45:0/979623535 conn(0x7f11540fedd0 msgr2=0x7f11541011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 shutdown_connections 2024-03-12T05:22:28.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.467+0000 7f115bb77640 1 -- 172.21.15.45:0/979623535 wait complete. 2024-03-12T05:22:28.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:28 smithi045 ceph-mon[32174]: pgmap v362: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:28.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:28 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3826383505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:22:28.481 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:22:28.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.662+0000 7fab6c925640 1 -- 172.21.15.45:0/485657580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6810d970 msgr2=0x7fab6810fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.662+0000 7fab6c925640 1 --2- 172.21.15.45:0/485657580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6810d970 0x7fab6810fd60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fab60009ca0 tx=0x7fab6002c120 comp rx=0 tx=0).stop 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 -- 172.21.15.45:0/485657580 shutdown_connections 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 --2- 172.21.15.45:0/485657580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab68100770 0x7fab6810d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 --2- 172.21.15.45:0/485657580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab6810d970 0x7fab6810fd60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 --2- 172.21.15.45:0/485657580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 -- 172.21.15.45:0/485657580 >> 172.21.15.45:0/485657580 conn(0x7fab680fb9d0 msgr2=0x7fab680fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 -- 172.21.15.45:0/485657580 shutdown_connections 2024-03-12T05:22:28.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.663+0000 7fab6c925640 1 -- 172.21.15.45:0/485657580 wait complete. 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.664+0000 7fab6c925640 1 Processor -- start 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.664+0000 7fab6c925640 1 -- start start 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.664+0000 7fab6c925640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab6c925640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 0x7fab68199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56932/0 (socket says 172.21.15.45:56932) 2024-03-12T05:22:28.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab677fe640 1 -- 172.21.15.45:0/3556892000 learned_addr learned my addr 172.21.15.45:0/3556892000 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 0x7fab6819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 0x7fab68199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6819e440 con 0x7fab6810d970 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6819e5b0 con 0x7fab68100770 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6819e770 con 0x7fab680ffdc0 2024-03-12T05:22:28.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab67fff640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 0x7fab6819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 -- 172.21.15.45:0/3556892000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 msgr2=0x7fab681995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 -- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 msgr2=0x7fab6819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 0x7fab6819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.665+0000 7fab66ffd640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60009900 con 0x7fab68100770 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab677fe640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab67fff640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 0x7fab6819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab66ffd640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 0x7fab68199b10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009f60 tx=0x7fab5c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab64ff9640 1 -- 172.21.15.45:0/3556892000 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c019070 con 0x7fab68100770 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab64ff9640 1 -- 172.21.15.45:0/3556892000 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab5c01b9d0 con 0x7fab68100770 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab64ff9640 1 -- 172.21.15.45:0/3556892000 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c01da70 con 0x7fab68100770 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6819ea50 con 0x7fab68100770 2024-03-12T05:22:28.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.666+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab68102120 con 0x7fab68100770 2024-03-12T05:22:28.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.669+0000 7fab64ff9640 1 -- 172.21.15.45:0/3556892000 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fab5c01bb40 con 0x7fab68100770 2024-03-12T05:22:28.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab64ff9640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 0x7fab500a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 Processor -- start 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab64ff9640 1 -- 172.21.15.45:0/3556892000 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab5c0d5520 con 0x7fab68100770 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 -- start start 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab677fe640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 0x7fab500a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe347b960 con 0x558fe34c0d80 2024-03-12T05:22:28.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe342f310 con 0x558fe34c9960 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab70831740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe342c840 con 0x558fe34c0460 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.670+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47370/0 (socket says 172.21.15.45:47370) 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 -- 172.21.15.45:0/1592165298 learned_addr learned my addr 172.21.15.45:0/1592165298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 -- 172.21.15.45:0/1592165298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 msgr2=0x558fe34c0840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c0840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 -- 172.21.15.45:0/1592165298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 msgr2=0x558fe34cbd50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34cbd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 -- 172.21.15.45:0/1592165298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fe33ea060 con 0x558fe34c0d80 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.671+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0aa330 tx=0x7fab5c0a2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56728bbe9aa03fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.672+0000 7fab677fe640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 0x7fab500a41a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fab58005fd0 tx=0x7fab5800b480 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.672+0000 7fab4dffb640 1 -- 172.21.15.45:0/1592165298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0a2170 con 0x558fe34c0d80 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.672+0000 7fab4dffb640 1 -- 172.21.15.45:0/1592165298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab5c0a6210 con 0x558fe34c0d80 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.672+0000 7fab4dffb640 1 -- 172.21.15.45:0/1592165298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c01dbd0 con 0x558fe34c0d80 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.673+0000 7fab70831740 1 -- 172.21.15.45:0/1592165298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 msgr2=0x558fe34c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.673+0000 7fab70831740 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0aa330 tx=0x7fab5c0a2040 comp rx=0 tx=0).stop 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.674+0000 7fab70831740 1 -- 172.21.15.45:0/1592165298 shutdown_connections 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.674+0000 7fab70831740 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0d80 0x558fe34c9420 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.674+0000 7fab70831740 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34cbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.674+0000 7fab70831740 1 --2- 172.21.15.45:0/1592165298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.674+0000 7fab70831740 1 -- 172.21.15.45:0/1592165298 >> 172.21.15.45:0/1592165298 conn(0x558fe34bab90 msgr2=0x558fe34baf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.675+0000 7fab70831740 1 -- 172.21.15.45:0/1592165298 shutdown_connections 2024-03-12T05:22:28.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.675+0000 7fab70831740 1 -- 172.21.15.45:0/1592165298 wait complete. 2024-03-12T05:22:28.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.675+0000 7fab70831740 1 -- start start 2024-03-12T05:22:28.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.675+0000 7fab70831740 1 Processor -- start 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 0x558fe34c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0d80 0x558fe34c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe3428020 con 0x558fe34c0460 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe30fb430 con 0x558fe34c9960 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe30b7e00 con 0x558fe34c0d80 2024-03-12T05:22:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 0x558fe34c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab70831740 10 client.0 _ll_register_callbacks cb 0x558fe33cfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 0x558fe34c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47380/0 (socket says 172.21.15.45:47380) 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab677fe640 1 -- 172.21.15.45:0/343966555 learned_addr learned my addr 172.21.15.45:0/343966555 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab66ffd640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0d80 0x558fe34c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.676+0000 7fab67fff640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 -- 172.21.15.45:0/343966555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0d80 msgr2=0x558fe34c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0d80 0x558fe34c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 -- 172.21.15.45:0/343966555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 msgr2=0x558fe34c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 0x558fe34c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 -- 172.21.15.45:0/343966555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab68102120 con 0x558fe34c9960 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab67fff640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34c75a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fab6002c730 tx=0x7fab60034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bdb4dd7a65649f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60032a00 con 0x558fe34c9960 2024-03-12T05:22:28.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab60003970 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fe31277e0 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.677+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab600348a0 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 10 client.27866 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fe34266a0 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab60004040 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab6003f480 con 0x558fe34c9960 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 10 client.27866 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 10 client.27866 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 20 client.27866 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe34eaf10 0x558fe34ed320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 4 client.27866 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:28.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.678+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fe31277e0 con 0x558fe34eaf10 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.679+0000 7fab677fe640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe34eaf10 0x558fe34ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.679+0000 7fab677fe640 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe34eaf10 0x558fe34ed320 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a4f796ac61ae481f server_cookie=569cd1efc8c3ded1 in_seq=0 out_seq=0 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.679+0000 7fab46ffd640 10 client.27866 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.680+0000 7fab46ffd640 1 -- 172.21.15.45:0/343966555 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558fe31277e0 con 0x558fe34eaf10 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.680+0000 7fab46ffd640 10 client.27866 handle_command_reply: tid=0 2024-03-12T05:22:28.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.680+0000 7fab70831740 1 client.27866 shutdown 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 msgr2=0x558fe34c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34c75a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fab6002c730 tx=0x7fab60034670 comp rx=0 tx=0).stop 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 shutdown_connections 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe34eaf10 0x558fe34ed320 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c9960 0x558fe34c75a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0d80 0x558fe34c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 --2- 172.21.15.45:0/343966555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c0460 0x558fe34c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 >> 172.21.15.45:0/343966555 conn(0x558fe34bab90 msgr2=0x558fe34bc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.683+0000 7fab46ffd640 0 client.27866 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:22:28.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.684+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 shutdown_connections 2024-03-12T05:22:28.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.684+0000 7fab70831740 1 -- 172.21.15.45:0/343966555 wait complete. 2024-03-12T05:22:28.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.684+0000 7fab70831740 20 client.27866 trim_cache size 0 max 16384 2024-03-12T05:22:28.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.708+0000 7fab70831740 1 Processor -- start 2024-03-12T05:22:28.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.708+0000 7fab70831740 1 -- start start 2024-03-12T05:22:28.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.708+0000 7fab70831740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab70831740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab70831740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe34c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab70831740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe33ad130 con 0x558fe34c9960 2024-03-12T05:22:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab70831740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe34c24e0 con 0x558fe34c0d80 2024-03-12T05:22:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab70831740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe34bf730 con 0x558fe34c0460 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59874/0 (socket says 172.21.15.45:59874) 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab66ffd640 1 -- 172.21.15.45:0/2799099070 learned_addr learned my addr 172.21.15.45:0/2799099070 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab67fff640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe34c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab677fe640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab677fe640 1 -- 172.21.15.45:0/2799099070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 msgr2=0x558fe34c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab677fe640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.709+0000 7fab677fe640 1 -- 172.21.15.45:0/2799099070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 msgr2=0x558fe34c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab677fe640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe34c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab677fe640 1 -- 172.21.15.45:0/2799099070 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fe33c2c20 con 0x558fe34c0460 2024-03-12T05:22:28.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab66ffd640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab67fff640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe34c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab677fe640 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c9250 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x558fe34edbd0 tx=0x7fab58013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3fe98022f64b294 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab4d7fa640 1 -- 172.21.15.45:0/2799099070 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58024070 con 0x558fe34c0460 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab4d7fa640 1 -- 172.21.15.45:0/2799099070 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab58015070 con 0x558fe34c0460 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.710+0000 7fab4d7fa640 1 -- 172.21.15.45:0/2799099070 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5801ea20 con 0x558fe34c0460 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.711+0000 7fab70831740 1 -- 172.21.15.45:0/2799099070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 msgr2=0x558fe34c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.711+0000 7fab70831740 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c9250 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x558fe34edbd0 tx=0x7fab58013000 comp rx=0 tx=0).stop 2024-03-12T05:22:28.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 -- 172.21.15.45:0/2799099070 shutdown_connections 2024-03-12T05:22:28.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe34c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 --2- 172.21.15.45:0/2799099070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34c9250 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 -- 172.21.15.45:0/2799099070 >> 172.21.15.45:0/2799099070 conn(0x558fe34bab90 msgr2=0x558fe34c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 -- 172.21.15.45:0/2799099070 shutdown_connections 2024-03-12T05:22:28.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.712+0000 7fab70831740 1 -- 172.21.15.45:0/2799099070 wait complete. 2024-03-12T05:22:28.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 -- start start 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 Processor -- start 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34e09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe3541410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe3541950 con 0x558fe34c9960 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.713+0000 7fab70831740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe3541a90 con 0x558fe34c0d80 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab70831740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fe3541bd0 con 0x558fe34c0460 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab70831740 10 client.0 _ll_register_callbacks cb 0x7fab5800b320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34e09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab67fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe3541410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59880/0 (socket says 172.21.15.45:59880) 2024-03-12T05:22:28.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 -- 172.21.15.45:0/1083908384 learned_addr learned my addr 172.21.15.45:0/1083908384 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34e09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:56974/0 (socket says 172.21.15.45:56974) 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 -- 172.21.15.45:0/1083908384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 msgr2=0x558fe34e09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34e09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 -- 172.21.15.45:0/1083908384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 msgr2=0x558fe3541410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe3541410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 -- 172.21.15.45:0/1083908384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5c0ba040 con 0x558fe34c0d80 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab67fff640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe3541410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.714+0000 7fab66ffd640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fab5c09e5d0 tx=0x7fab5c01bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=341dc4324c0146db server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0aa860 con 0x558fe34c0d80 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fab5c0d2e60 con 0x558fe34c0d80 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab5c0a6210 con 0x558fe34c0d80 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab5c0d3020 con 0x558fe34c0d80 2024-03-12T05:22:28.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab70831740 10 client.27872 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558fe34266a0 con 0x558fe34c0d80 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fab5c0a6210 con 0x558fe34c0d80 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.715+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab5c09dec0 con 0x558fe34c0d80 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 10 client.27872 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 10 client.27872 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 20 client.27872 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe3542fd0 0x558fe35453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 4 client.27872 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558fe3164720 con 0x558fe3542fd0 2024-03-12T05:22:28.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.716+0000 7fab677fe640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe3542fd0 0x558fe35453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.717+0000 7fab677fe640 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe3542fd0 0x558fe35453c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=86fa05f0f5542c4 server_cookie=da927cc8a4365dcb in_seq=0 out_seq=0 2024-03-12T05:22:28.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.717+0000 7fab46ffd640 10 client.27872 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.717+0000 7fab46ffd640 1 -- 172.21.15.45:0/1083908384 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558fe3164720 con 0x558fe3542fd0 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.717+0000 7fab46ffd640 10 client.27872 handle_command_reply: tid=0 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.717+0000 7fab70831740 1 client.27872 shutdown 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 msgr2=0x558fe34e0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fab5c09e5d0 tx=0x7fab5c01bf90 comp rx=0 tx=0).stop 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 shutdown_connections 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x558fe3542fd0 0x558fe35453c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558fe34c9960 0x558fe3541410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558fe34c0d80 0x558fe34e0ee0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab46ffd640 0 client.27872 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 --2- 172.21.15.45:0/1083908384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558fe34c0460 0x558fe34e09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.718+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 >> 172.21.15.45:0/1083908384 conn(0x558fe34bab90 msgr2=0x558fe34c5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.719+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 shutdown_connections 2024-03-12T05:22:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.719+0000 7fab70831740 1 -- 172.21.15.45:0/1083908384 wait complete. 2024-03-12T05:22:28.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.719+0000 7fab70831740 20 client.27872 trim_cache size 0 max 16384 2024-03-12T05:22:28.716 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 msgr2=0x7fab500a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 0x7fab500a41a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fab58005fd0 tx=0x7fab5800b480 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 msgr2=0x7fab68199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 0x7fab68199b10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009f60 tx=0x7fab5c009f90 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 shutdown_connections 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fab500a1ce0 0x7fab500a41a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab6810d970 0x7fab6819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab68100770 0x7fab68199b10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 --2- 172.21.15.45:0/3556892000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fab680ffdc0 0x7fab681995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.720+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 >> 172.21.15.45:0/3556892000 conn(0x7fab680fb9d0 msgr2=0x7fab6810fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.721+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 shutdown_connections 2024-03-12T05:22:28.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.721+0000 7fab6c925640 1 -- 172.21.15.45:0/3556892000 wait complete. 2024-03-12T05:22:28.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:28 smithi130 ceph-mon[35398]: pgmap v362: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:28.718 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:28 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3826383505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:22:28.734 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:22:28.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:22:28.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:28 smithi116 ceph-mon[35486]: pgmap v362: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-03-12T05:22:28.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:28 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3826383505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 -- 172.21.15.45:0/1572921553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc104480 msgr2=0x7f47bc107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/1572921553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc104480 0x7f47bc107fe0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009930 tx=0x7f47b002bc30 comp rx=0 tx=0).stop 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 -- 172.21.15.45:0/1572921553 shutdown_connections 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/1572921553 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc104480 0x7f47bc107fe0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/1572921553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 0x7f47bc110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/1572921553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 -- 172.21.15.45:0/1572921553 >> 172.21.15.45:0/1572921553 conn(0x7f47bc076ed0 msgr2=0x7f47bc0792f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 -- 172.21.15.45:0/1572921553 shutdown_connections 2024-03-12T05:22:28.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.920+0000 7f47c1a2e640 1 -- 172.21.15.45:0/1572921553 wait complete. 2024-03-12T05:22:28.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.921+0000 7f47c1a2e640 1 Processor -- start 2024-03-12T05:22:28.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.921+0000 7f47c1a2e640 1 -- start start 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 0x7f47bc19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 0x7f47bc199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19a130 con 0x7f47bc199660 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19a2a0 con 0x7f47bc103ad0 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47c1a2e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc19a460 con 0x7f47bc108710 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc19f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59902/0 (socket says 172.21.15.45:59902) 2024-03-12T05:22:28.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47baffd640 1 -- 172.21.15.45:0/2761715106 learned_addr learned my addr 172.21.15.45:0/2761715106 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 0x7f47bc199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.922+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 0x7f47bc19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 -- 172.21.15.45:0/2761715106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 msgr2=0x7f47bc19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 0x7f47bc19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 -- 172.21.15.45:0/2761715106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 msgr2=0x7f47bc19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 -- 172.21.15.45:0/2761715106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0009560 con 0x7f47bc199660 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 0x7f47bc199ae0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f47b400c9e0 tx=0x7f47b400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.923+0000 7f47c0a2c640 1 -- 172.21.15.45:0/2761715106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4004d20 con 0x7f47bc199660 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.924+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc1a6400 con 0x7f47bc199660 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.924+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc1a68e0 con 0x7f47bc199660 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.924+0000 7f47c0a2c640 1 -- 172.21.15.45:0/2761715106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f47b4004e80 con 0x7f47bc199660 2024-03-12T05:22:28.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.924+0000 7f47c0a2c640 1 -- 172.21.15.45:0/2761715106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b400f6b0 con 0x7f47bc199660 2024-03-12T05:22:28.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.928+0000 7f47c0a2c640 1 -- 172.21.15.45:0/2761715106 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f47b400f810 con 0x7f47bc199660 2024-03-12T05:22:28.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.928+0000 7f47c4600740 1 Processor -- start 2024-03-12T05:22:28.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.928+0000 7f47c4600740 1 -- start start 2024-03-12T05:22:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.928+0000 7f47c0a2c640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 0x7f47a40a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47c4600740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fad2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47c4600740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fadb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47c4600740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30faddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47c0a2c640 1 -- 172.21.15.45:0/2761715106 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f47b40cd8c0 con 0x7f47bc199660 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47c4600740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fa8d960 con 0x55f30fadb750 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47baffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fad2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47baffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fad2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57000/0 (socket says 172.21.15.45:57000) 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47baffd640 1 -- 172.21.15.45:0/137145935 learned_addr learned my addr 172.21.15.45:0/137145935 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 0x7f47a40a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.929+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fadb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fa41310 con 0x55f30fad2b70 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30faddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fa3e840 con 0x55f30fad2250 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 -- 172.21.15.45:0/137145935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 msgr2=0x55f30fad2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fad2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 -- 172.21.15.45:0/137145935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 msgr2=0x55f30fadb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fadb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 -- 172.21.15.45:0/137145935 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f30f9fc060 con 0x55f30fadb750 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.930+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30faddb40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f47b40caf80 tx=0x7f47b4005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af35699dfa7f2295 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.931+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 0x7f47a40a41a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009850 tx=0x7f47b0004890 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.931+0000 7f47a1ffb640 1 -- 172.21.15.45:0/137145935 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b40ba0d0 con 0x55f30fadb750 2024-03-12T05:22:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.931+0000 7f47a1ffb640 1 -- 172.21.15.45:0/137145935 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f47b40a2c00 con 0x55f30fadb750 2024-03-12T05:22:28.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.931+0000 7f47a1ffb640 1 -- 172.21.15.45:0/137145935 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b409e170 con 0x55f30fadb750 2024-03-12T05:22:28.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 msgr2=0x55f30faddb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30faddb40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f47b40caf80 tx=0x7f47b4005ae0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 shutdown_connections 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30faddb40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fadb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 --2- 172.21.15.45:0/137145935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fad2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 >> 172.21.15.45:0/137145935 conn(0x55f30facc980 msgr2=0x55f30faccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.932+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 shutdown_connections 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.933+0000 7f47c4600740 1 -- 172.21.15.45:0/137145935 wait complete. 2024-03-12T05:22:28.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.933+0000 7f47c4600740 1 -- start start 2024-03-12T05:22:28.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.933+0000 7f47c4600740 1 Processor -- start 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.934+0000 7f47c4600740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.934+0000 7f47c4600740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30fad8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.934+0000 7f47c4600740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fad91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.934+0000 7f47c4600740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fa3a020 con 0x55f30fad2b70 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.934+0000 7f47c4600740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30f70d430 con 0x55f30fad2250 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47c4600740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30f6c9e00 con 0x55f30fadb750 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47baffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59916/0 (socket says 172.21.15.45:59916) 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47baffd640 1 -- 172.21.15.45:0/1867842956 learned_addr learned my addr 172.21.15.45:0/1867842956 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30fad8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fad91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47c4600740 10 client.0 _ll_register_callbacks cb 0x55f30f9e1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 -- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 msgr2=0x55f30fad91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fad91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 -- 172.21.15.45:0/1867842956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 msgr2=0x55f30fad8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47ba7fc640 1 -- 172.21.15.45:0/1867842956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc1a68e0 con 0x55f30fad2b70 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.935+0000 7f47baffd640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.936+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30fad8c90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009230 tx=0x7f47b002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8161d2dbf52f9634 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.936+0000 7f479affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.936+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0052070 con 0x55f30fad2b70 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.936+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f47b002de40 con 0x55f30fad2b70 2024-03-12T05:22:28.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.936+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f30f7397e0 con 0x55f30fad2b70 2024-03-12T05:22:28.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47b0048070 con 0x55f30fad2b70 2024-03-12T05:22:28.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 10 client.18405 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f30fa386a0 con 0x55f30fad2b70 2024-03-12T05:22:28.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f30f7397e0 con 0x55f30fad2b70 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f47b0055030 con 0x55f30fad2b70 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 10 client.18405 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 10 client.18405 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 20 client.18405 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30faf4110 0x55f30faf6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 4 client.18405 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.937+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f30f776720 con 0x55f30faf4110 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.938+0000 7f47baffd640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30faf4110 0x55f30faf6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.938+0000 7f47baffd640 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30faf4110 0x55f30faf6530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ff291591c809bdc server_cookie=48680e06ad42b23f in_seq=0 out_seq=0 2024-03-12T05:22:28.934 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.938+0000 7f479affd640 10 client.18405 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.939+0000 7f479affd640 1 -- 172.21.15.45:0/1867842956 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f30f776720 con 0x55f30faf4110 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.939+0000 7f479affd640 10 client.18405 handle_command_reply: tid=0 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.939+0000 7f47c4600740 1 client.18405 shutdown 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 msgr2=0x55f30fad8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30fad8c90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009230 tx=0x7f47b002bc30 comp rx=0 tx=0).stop 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 shutdown_connections 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30faf4110 0x55f30faf6530 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30fad8c90 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fad91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.935 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 --2- 172.21.15.45:0/1867842956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fad8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 >> 172.21.15.45:0/1867842956 conn(0x55f30facc980 msgr2=0x55f30face6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f479affd640 0 client.18405 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:22:28.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.940+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 shutdown_connections 2024-03-12T05:22:28.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.941+0000 7f47c4600740 1 -- 172.21.15.45:0/1867842956 wait complete. 2024-03-12T05:22:28.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.941+0000 7f47c4600740 20 client.18405 trim_cache size 0 max 16384 2024-03-12T05:22:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.963+0000 7f47c4600740 1 Processor -- start 2024-03-12T05:22:28.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.963+0000 7f47c4600740 1 -- start start 2024-03-12T05:22:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47c4600740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47c4600740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47c4600740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30fad5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47baffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47baffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57028/0 (socket says 172.21.15.45:57028) 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59928/0 (socket says 172.21.15.45:59928) 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47bb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30fad5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47bb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30fad5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47462/0 (socket says 172.21.15.45:47462) 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.964+0000 7f47baffd640 1 -- 172.21.15.45:0/47352134 learned_addr learned my addr 172.21.15.45:0/47352134 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30f9bf130 con 0x55f30fadb750 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fafb780 con 0x55f30fad2b70 2024-03-12T05:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fac8080 con 0x55f30fad2250 2024-03-12T05:22:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47ba7fc640 1 -- 172.21.15.45:0/47352134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 msgr2=0x55f30fadae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47ba7fc640 1 -- 172.21.15.45:0/47352134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 msgr2=0x55f30fad5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30fad5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47ba7fc640 1 -- 172.21.15.45:0/47352134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f30f9d4c20 con 0x55f30fad2b70 2024-03-12T05:22:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.965+0000 7f47baffd640 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.966+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f47b002c240 tx=0x7f47b0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c94d0c309e20dc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.966+0000 7f47a17fa640 1 -- 172.21.15.45:0/47352134 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0052020 con 0x55f30fad2b70 2024-03-12T05:22:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.966+0000 7f47a17fa640 1 -- 172.21.15.45:0/47352134 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f47b00040a0 con 0x55f30fad2b70 2024-03-12T05:22:28.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.966+0000 7f47a17fa640 1 -- 172.21.15.45:0/47352134 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0048040 con 0x55f30fad2b70 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.967+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 msgr2=0x55f30fad8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.967+0000 7f47c4600740 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f47b002c240 tx=0x7f47b0009260 comp rx=0 tx=0).stop 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 shutdown_connections 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fadb750 0x55f30fad5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2b70 0x55f30fad8ff0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.963 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 --2- 172.21.15.45:0/47352134 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fad2250 0x55f30fadae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 >> 172.21.15.45:0/47352134 conn(0x55f30facc980 msgr2=0x55f30fad3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 shutdown_connections 2024-03-12T05:22:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 -- 172.21.15.45:0/47352134 wait complete. 2024-03-12T05:22:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.968+0000 7f47c4600740 1 -- start start 2024-03-12T05:22:28.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 Processor -- start 2024-03-12T05:22:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fae9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fb4a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30faea8c0 con 0x55f30fad2b70 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30faeaa30 con 0x55f30fad2250 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30fb4ac20 con 0x55f30fadb750 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47c4600740 10 client.0 _ll_register_callbacks cb 0x7f47b4096040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47ba7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47466/0 (socket says 172.21.15.45:47466) 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47ba7fc640 1 -- 172.21.15.45:0/3532230930 learned_addr learned my addr 172.21.15.45:0/3532230930 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47baffd640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fae9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fb4a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47baffd640 1 -- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 msgr2=0x55f30fb4a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47baffd640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fb4a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.969+0000 7f47baffd640 1 -- 172.21.15.45:0/3532230930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 msgr2=0x55f30faea0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47baffd640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47baffd640 1 -- 172.21.15.45:0/3532230930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac012040 con 0x55f30fad2250 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47bb7fe640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fb4a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47baffd640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fae9b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f47ac005fd0 tx=0x7f47ac01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e67c0816181515e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f479affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac017040 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47b40a2c00 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f47ac025680 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.970+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47ac025840 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 10 client.27887 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f30fa386a0 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f47b40a2c00 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f47ac011aa0 con 0x55f30fad2250 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 10 client.27887 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 10 client.27887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 20 client.27887 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30fb4c4d0 0x55f30fb4e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:28.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 4 client.27887 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:22:28.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.971+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f30f7c5890 con 0x55f30fb4c4d0 2024-03-12T05:22:28.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.972+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30fb4c4d0 0x55f30fb4e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:28.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.972+0000 7f47ba7fc640 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30fb4c4d0 0x55f30fb4e8f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=648b07c2b52a7519 server_cookie=386b0c4496395678 in_seq=0 out_seq=0 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.972+0000 7f479affd640 10 client.27887 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.973+0000 7f479affd640 1 -- 172.21.15.45:0/3532230930 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f30f7c5890 con 0x55f30fb4c4d0 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.973+0000 7f479affd640 10 client.27887 handle_command_reply: tid=0 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.973+0000 7f47c4600740 1 client.27887 shutdown 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.973+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 msgr2=0x55f30fae9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fae9b70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f47ac005fd0 tx=0x7f47ac01c910 comp rx=0 tx=0).stop 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 shutdown_connections 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55f30fb4c4d0 0x55f30fb4e8f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f479affd640 0 client.27887 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f30fad2b70 0x55f30faea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30fadb750 0x55f30fb4a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 --2- 172.21.15.45:0/3532230930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f30fad2250 0x55f30fae9b70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 >> 172.21.15.45:0/3532230930 conn(0x55f30facc980 msgr2=0x55f30facddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.974+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 shutdown_connections 2024-03-12T05:22:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.975+0000 7f47c4600740 1 -- 172.21.15.45:0/3532230930 wait complete. 2024-03-12T05:22:28.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.975+0000 7f47c4600740 20 client.27887 trim_cache size 0 max 16384 2024-03-12T05:22:28.972 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 msgr2=0x7f47a40a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 0x7f47a40a41a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009850 tx=0x7f47b0004890 comp rx=0 tx=0).stop 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 msgr2=0x7f47bc199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 0x7f47bc199ae0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f47b400c9e0 tx=0x7f47b400cea0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 shutdown_connections 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f47a40a1ce0 0x7f47a40a41a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc199660 0x7f47bc199ae0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47bc108710 0x7f47bc19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 --2- 172.21.15.45:0/2761715106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc103ad0 0x7f47bc19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:28.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.976+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 >> 172.21.15.45:0/2761715106 conn(0x7f47bc076ed0 msgr2=0x7f47bc0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:28.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.977+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 shutdown_connections 2024-03-12T05:22:28.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:28.977+0000 7f47c1a2e640 1 -- 172.21.15.45:0/2761715106 wait complete. 2024-03-12T05:22:28.995 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:22:28.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 -- 172.21.15.45:0/1457204019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec1002a0 msgr2=0x7f54ec100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/1457204019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec1002a0 0x7f54ec100680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009900 tx=0x7f54dc02bc40 comp rx=0 tx=0).stop 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 -- 172.21.15.45:0/1457204019 shutdown_connections 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/1457204019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54ec105370 0x7f54ec10cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/1457204019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec100c50 0x7f54ec104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/1457204019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec1002a0 0x7f54ec100680 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 -- 172.21.15.45:0/1457204019 >> 172.21.15.45:0/1457204019 conn(0x7f54ec076400 msgr2=0x7f54ec076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 -- 172.21.15.45:0/1457204019 shutdown_connections 2024-03-12T05:22:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.175+0000 7f54f3d4f640 1 -- 172.21.15.45:0/1457204019 wait complete. 2024-03-12T05:22:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.176+0000 7f54f3d4f640 1 Processor -- start 2024-03-12T05:22:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.176+0000 7f54f3d4f640 1 -- start start 2024-03-12T05:22:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.176+0000 7f54f3d4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54ec1002a0 0x7f54ec0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f3d4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec105370 0x7f54ec072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f3d4f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f3d4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec06c660 con 0x7f54ec1002a0 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f3d4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec06c7d0 con 0x7f54ec105370 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f3d4f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec0703d0 con 0x7f54ec06ba20 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57042/0 (socket says 172.21.15.45:57042) 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 -- 172.21.15.45:0/755778369 learned_addr learned my addr 172.21.15.45:0/755778369 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 -- 172.21.15.45:0/755778369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec105370 msgr2=0x7f54ec072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec105370 0x7f54ec072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 -- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54ec1002a0 msgr2=0x7f54ec0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54ec1002a0 0x7f54ec0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.177+0000 7f54f22ec640 1 -- 172.21.15.45:0/755778369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc009560 con 0x7f54ec06ba20 2024-03-12T05:22:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.178+0000 7f54f22ec640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f54e400eaa0 tx=0x7f54e400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.178+0000 7f54e2ffd640 1 -- 172.21.15.45:0/755778369 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e400cdb0 con 0x7f54ec06ba20 2024-03-12T05:22:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.178+0000 7f54e2ffd640 1 -- 172.21.15.45:0/755778369 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54e4005020 con 0x7f54ec06ba20 2024-03-12T05:22:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.178+0000 7f54e2ffd640 1 -- 172.21.15.45:0/755778369 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e40116d0 con 0x7f54ec06ba20 2024-03-12T05:22:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.178+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ec0706b0 con 0x7f54ec06ba20 2024-03-12T05:22:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.179+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ec070b70 con 0x7f54ec06ba20 2024-03-12T05:22:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.181+0000 7f54e2ffd640 1 -- 172.21.15.45:0/755778369 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f54e4005190 con 0x7f54ec06ba20 2024-03-12T05:22:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.182+0000 7f54f69fe740 1 Processor -- start 2024-03-12T05:22:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 -- start start 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 0x5622f54f1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f54fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f54ac960 con 0x5622f54fa750 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f5460310 con 0x5622f54f1250 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f69fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f545d840 con 0x5622f54f1b70 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54e2ffd640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 0x7f54d80a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f1aeb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f1aeb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57056/0 (socket says 172.21.15.45:57056) 2024-03-12T05:22:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f1aeb640 1 -- 172.21.15.45:0/3937433647 learned_addr learned my addr 172.21.15.45:0/3937433647 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.183+0000 7f54f12ea640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 0x5622f54f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f12ea640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f54fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 -- 172.21.15.45:0/3937433647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 msgr2=0x5622f54f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 0x5622f54f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 -- 172.21.15.45:0/3937433647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 msgr2=0x5622f54fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f54fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54e2ffd640 1 -- 172.21.15.45:0/755778369 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54e40cd9f0 con 0x7f54ec06ba20 2024-03-12T05:22:29.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 -- 172.21.15.45:0/3937433647 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622f541b060 con 0x5622f54f1b70 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f22ec640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 0x7f54d80a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f12ea640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 0x5622f54f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009430 tx=0x7f54dc02c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fbb418c7eac3a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54d7fff640 1 -- 172.21.15.45:0/3937433647 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc004740 con 0x5622f54f1b70 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54d7fff640 1 -- 172.21.15.45:0/3937433647 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54dc0048a0 con 0x5622f54f1b70 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.184+0000 7f54d7fff640 1 -- 172.21.15.45:0/3937433647 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc003850 con 0x5622f54f1b70 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.185+0000 7f54f69fe740 1 -- 172.21.15.45:0/3937433647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 msgr2=0x5622f54fa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.185+0000 7f54f69fe740 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009430 tx=0x7f54dc02c710 comp rx=0 tx=0).stop 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f22ec640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 0x7f54d80a4230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54e400efd0 tx=0x7f54e4005440 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 -- 172.21.15.45:0/3937433647 shutdown_connections 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f54fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1b70 0x5622f54fa210 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 --2- 172.21.15.45:0/3937433647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1250 0x5622f54f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 -- 172.21.15.45:0/3937433647 >> 172.21.15.45:0/3937433647 conn(0x5622f54eb980 msgr2=0x5622f54ebd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 -- 172.21.15.45:0/3937433647 shutdown_connections 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.186+0000 7f54f69fe740 1 -- 172.21.15.45:0/3937433647 wait complete. 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.187+0000 7f54f69fe740 1 -- start start 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.187+0000 7f54f69fe740 1 Processor -- start 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.187+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.187+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f69fe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f5459020 con 0x5622f54f1250 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f69fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f512c430 con 0x5622f54f1b70 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f69fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f50e8e00 con 0x5622f54fa750 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:59980/0 (socket says 172.21.15.45:59980) 2024-03-12T05:22:29.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47496/0 (socket says 172.21.15.45:47496) 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 -- 172.21.15.45:0/2472486368 learned_addr learned my addr 172.21.15.45:0/2472486368 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f69fe740 10 client.0 _ll_register_callbacks cb 0x5622f5400cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 -- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 msgr2=0x5622f54f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 -- 172.21.15.45:0/2472486368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 msgr2=0x5622f54f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f1aeb640 1 -- 172.21.15.45:0/2472486368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ec10d750 con 0x5622f54f1b70 2024-03-12T05:22:29.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.188+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f54dc02ca70 tx=0x7f54dc02caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c5df18105bd9382 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc004520 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622f51587e0 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54dc02de40 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54dc003a00 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54f69fe740 10 client.27896 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5622f54576a0 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.189+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54dc003b60 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f54dc03c070 con 0x5622f54f1b70 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 10 client.27896 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 10 client.27896 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 20 client.27896 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:29.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f550a570 0x5622f550c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 4 client.27896 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.190+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622f51587e0 con 0x5622f550a570 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.191+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f550a570 0x5622f550c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.192+0000 7f54d57fa640 10 client.27896 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.192+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f550a570 0x5622f550c9e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edc792085b6e55ff server_cookie=28b81a72fac12b4f in_seq=0 out_seq=0 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.193+0000 7f54d57fa640 1 -- 172.21.15.45:0/2472486368 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5622f51587e0 con 0x5622f550a570 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.193+0000 7f54d57fa640 10 client.27896 handle_command_reply: tid=0 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.193+0000 7f54f69fe740 1 client.27896 shutdown 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.195+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 msgr2=0x5622f54f7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.195+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f54dc02ca70 tx=0x7f54dc02caa0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 shutdown_connections 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f550a570 0x5622f550c9e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f7cf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2472486368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 >> 172.21.15.45:0/2472486368 conn(0x5622f54eb980 msgr2=0x5622f54ed6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54d57fa640 0 client.27896 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:22:29.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 shutdown_connections 2024-03-12T05:22:29.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 1 -- 172.21.15.45:0/2472486368 wait complete. 2024-03-12T05:22:29.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.196+0000 7f54f69fe740 20 client.27896 trim_cache size 0 max 16384 2024-03-12T05:22:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.221+0000 7f54f69fe740 1 Processor -- start 2024-03-12T05:22:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.221+0000 7f54f69fe740 1 -- start start 2024-03-12T05:22:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.221+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f69fe740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f53de130 con 0x5622f54f1250 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f69fe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f54eb4b0 con 0x5622f54f1b70 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f69fe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f54f25f0 con 0x5622f54fa750 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47506/0 (socket says 172.21.15.45:47506) 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 -- 172.21.15.45:0/2489823195 learned_addr learned my addr 172.21.15.45:0/2489823195 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 -- 172.21.15.45:0/2489823195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 msgr2=0x5622f54f4f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:22:29.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f4f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 -- 172.21.15.45:0/2489823195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 msgr2=0x5622f54f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.222+0000 7f54f12ea640 1 -- 172.21.15.45:0/2489823195 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622f53f3c20 con 0x5622f54f1250 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.223+0000 7f54f12ea640 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f54ec06d0f0 tx=0x7f54e8009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eae6aa69effb0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.223+0000 7f54d77fe640 1 -- 172.21.15.45:0/2489823195 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8012040 con 0x5622f54f1250 2024-03-12T05:22:29.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.223+0000 7f54d77fe640 1 -- 172.21.15.45:0/2489823195 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54e8013040 con 0x5622f54f1250 2024-03-12T05:22:29.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.223+0000 7f54d77fe640 1 -- 172.21.15.45:0/2489823195 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8004a00 con 0x5622f54f1250 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 -- 172.21.15.45:0/2489823195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 msgr2=0x5622f54f9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f54ec06d0f0 tx=0x7f54e8009890 comp rx=0 tx=0).stop 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 -- 172.21.15.45:0/2489823195 shutdown_connections 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54fa750 0x5622f54f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f54f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.224+0000 7f54f69fe740 1 --2- 172.21.15.45:0/2489823195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54f1250 0x5622f54f9ee0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.225+0000 7f54f69fe740 1 -- 172.21.15.45:0/2489823195 >> 172.21.15.45:0/2489823195 conn(0x5622f54eb980 msgr2=0x5622f54f2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.225+0000 7f54f69fe740 1 -- 172.21.15.45:0/2489823195 shutdown_connections 2024-03-12T05:22:29.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.225+0000 7f54f69fe740 1 -- 172.21.15.45:0/2489823195 wait complete. 2024-03-12T05:22:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.225+0000 7f54f69fe740 1 -- start start 2024-03-12T05:22:29.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.226+0000 7f54f69fe740 1 Processor -- start 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.226+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.226+0000 7f54f69fe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f5500760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f12ea640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57092/0 (socket says 172.21.15.45:57092) 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f12ea640 1 -- 172.21.15.45:0/332824531 learned_addr learned my addr 172.21.15.45:0/332824531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f5500760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f5500ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f5560c80 con 0x5622f54fa750 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f5560df0 con 0x5622f54f1b70 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622f5560fb0 con 0x5622f54f1250 2024-03-12T05:22:29.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.227+0000 7f54f69fe740 10 client.0 _ll_register_callbacks cb 0x7f54e8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f5500ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 -- 172.21.15.45:0/332824531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 msgr2=0x5622f5500760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f5500760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 -- 172.21.15.45:0/332824531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 msgr2=0x5622f5500ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f5500ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e800dce0 con 0x5622f54f1250 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54f12ea640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f54e8006010 tx=0x7f54e8009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8775ea5082ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.228+0000 7f54d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e8020070 con 0x5622f54f1250 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54e8004a30 con 0x5622f54f1250 2024-03-12T05:22:29.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54dc0048a0 con 0x5622f54f1250 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54e8012040 con 0x5622f54f1250 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54f69fe740 10 client.27883 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.229+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5622f54198c0 con 0x5622f54f1250 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54dc0048a0 con 0x5622f54f1250 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f54e8018040 con 0x5622f54f1250 2024-03-12T05:22:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 10 client.27883 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 10 client.27883 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 20 client.27883 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f55626d0 0x5622f5564af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 4 client.27883 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.230+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5622f5195720 con 0x5622f55626d0 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.231+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f55626d0 0x5622f5564af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.226 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.231+0000 7f54d57fa640 10 client.27883 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:22:29.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.232+0000 7f54f1aeb640 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f55626d0 0x5622f5564af0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f54e1e97e253261 server_cookie=fd84de55c6321d0b in_seq=0 out_seq=0 2024-03-12T05:22:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.232+0000 7f54d57fa640 1 -- 172.21.15.45:0/332824531 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5622f5195720 con 0x5622f55626d0 2024-03-12T05:22:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.232+0000 7f54d57fa640 10 client.27883 handle_command_reply: tid=0 2024-03-12T05:22:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 client.27883 shutdown 2024-03-12T05:22:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 msgr2=0x5622f5500220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f54e8006010 tx=0x7f54e8009f10 comp rx=0 tx=0).stop 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 shutdown_connections 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5622f54fa750 0x5622f5500ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x5622f55626d0 0x5622f5564af0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622f54f1b70 0x5622f5500760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54d57fa640 0 client.27883 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 --2- 172.21.15.45:0/332824531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f54f1250 0x5622f5500220 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 >> 172.21.15.45:0/332824531 conn(0x5622f54eb980 msgr2=0x5622f54ea2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 shutdown_connections 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 1 -- 172.21.15.45:0/332824531 wait complete. 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.233+0000 7f54f69fe740 20 client.27883 trim_cache size 0 max 16384 2024-03-12T05:22:29.229 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.234+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 msgr2=0x7f54d80a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.234+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 0x7f54d80a4230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54e400efd0 tx=0x7f54e4005440 comp rx=0 tx=0).stop 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 msgr2=0x7f54ec06bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f54e400eaa0 tx=0x7f54e400ef60 comp rx=0 tx=0).stop 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 shutdown_connections 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54d80a1d70 0x7f54d80a4230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54ec06ba20 0x7f54ec06bed0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54ec105370 0x7f54ec072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 --2- 172.21.15.45:0/755778369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f54ec1002a0 0x7f54ec0728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 >> 172.21.15.45:0/755778369 conn(0x7f54ec076400 msgr2=0x7f54ec0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 shutdown_connections 2024-03-12T05:22:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.235+0000 7f54f3d4f640 1 -- 172.21.15.45:0/755778369 wait complete. 2024-03-12T05:22:29.248 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:22:29.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.430+0000 7f490d3b6640 1 -- 172.21.15.45:0/1912621130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 msgr2=0x7f4908101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.430+0000 7f490d3b6640 1 --2- 172.21.15.45:0/1912621130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 0x7f4908101670 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009cd0 tx=0x7f48f802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.431+0000 7f490d3b6640 1 -- 172.21.15.45:0/1912621130 shutdown_connections 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.431+0000 7f490d3b6640 1 --2- 172.21.15.45:0/1912621130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908101c40 0x7f4908105c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.431+0000 7f490d3b6640 1 --2- 172.21.15.45:0/1912621130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4908106360 0x7f490810df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.431+0000 7f490d3b6640 1 --2- 172.21.15.45:0/1912621130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 0x7f4908101670 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.431+0000 7f490d3b6640 1 -- 172.21.15.45:0/1912621130 >> 172.21.15.45:0/1912621130 conn(0x7f49080fcb10 msgr2=0x7f49080fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.433+0000 7f490d3b6640 1 -- 172.21.15.45:0/1912621130 shutdown_connections 2024-03-12T05:22:29.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.433+0000 7f490d3b6640 1 -- 172.21.15.45:0/1912621130 wait complete. 2024-03-12T05:22:29.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.434+0000 7f490d3b6640 1 Processor -- start 2024-03-12T05:22:29.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.435+0000 7f490d3b6640 1 -- start start 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 0x7f49081976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 0x7f490819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490819c930 con 0x7f4908106360 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490819caa0 con 0x7f4908101290 2024-03-12T05:22:29.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f490d3b6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490819cc10 con 0x7f490819bf40 2024-03-12T05:22:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.436+0000 7f48fe7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.437+0000 7f49077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 0x7f490819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.437+0000 7f48fe7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47538/0 (socket says 172.21.15.45:47538) 2024-03-12T05:22:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.437+0000 7f48fe7fc640 1 -- 172.21.15.45:0/3563680157 learned_addr learned my addr 172.21.15.45:0/3563680157 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.437+0000 7f49077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 0x7f490819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57124/0 (socket says 172.21.15.45:57124) 2024-03-12T05:22:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 -- 172.21.15.45:0/3563680157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 msgr2=0x7f490819c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 0x7f490819c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 -- 172.21.15.45:0/3563680157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 msgr2=0x7f49081976d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:22:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 0x7f49081976d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 -- 172.21.15.45:0/3563680157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8009900 con 0x7f4908106360 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.438+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f48f000e9c0 tx=0x7f48f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.439+0000 7f4904ff9640 1 -- 172.21.15.45:0/3563680157 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0009700 con 0x7f4908106360 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.439+0000 7f4904ff9640 1 -- 172.21.15.45:0/3563680157 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48f000cd80 con 0x7f4908106360 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.439+0000 7f4904ff9640 1 -- 172.21.15.45:0/3563680157 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0010430 con 0x7f4908106360 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.439+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49080733b0 con 0x7f4908106360 2024-03-12T05:22:29.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.439+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4908073810 con 0x7f4908106360 2024-03-12T05:22:29.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.442+0000 7f4904ff9640 1 -- 172.21.15.45:0/3563680157 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f48f0004f40 con 0x7f4908106360 2024-03-12T05:22:29.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.443+0000 7f490ff87740 1 Processor -- start 2024-03-12T05:22:29.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.443+0000 7f490ff87740 1 -- start start 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8856420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8858d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc8808960 con 0x564cc8856960 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc87bc310 con 0x564cc884dd80 2024-03-12T05:22:29.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f490ff87740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc87b9840 con 0x564cc884d460 2024-03-12T05:22:29.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f49077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f49077fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57126/0 (socket says 172.21.15.45:57126) 2024-03-12T05:22:29.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.444+0000 7f49077fe640 1 -- 172.21.15.45:0/3663008816 learned_addr learned my addr 172.21.15.45:0/3663008816 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.440 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f4904ff9640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 0x7f48f40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f4906ffd640 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8856420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f49077fe640 1 -- 172.21.15.45:0/3663008816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 msgr2=0x564cc8856420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f49077fe640 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8856420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f49077fe640 1 -- 172.21.15.45:0/3663008816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 msgr2=0x564cc8858d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f49077fe640 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8858d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.445+0000 7f49077fe640 1 -- 172.21.15.45:0/3663008816 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc8777060 con 0x564cc884d460 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.446+0000 7f4904ff9640 1 -- 172.21.15.45:0/3563680157 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48f0014070 con 0x7f4908106360 2024-03-12T05:22:29.441 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.446+0000 7f49077fe640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 0x7f48f40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.446+0000 7f4906ffd640 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8856420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:29.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.446+0000 7f49077fe640 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f49000099e0 tx=0x7f4900009ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fee2b333af3f409e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.447+0000 7f48fd7fa640 1 -- 172.21.15.45:0/3663008816 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900013470 con 0x564cc884d460 2024-03-12T05:22:29.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.447+0000 7f48fd7fa640 1 -- 172.21.15.45:0/3663008816 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4900013b00 con 0x564cc884d460 2024-03-12T05:22:29.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.448+0000 7f490ff87740 1 -- 172.21.15.45:0/3663008816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 msgr2=0x564cc884d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f49000099e0 tx=0x7f4900009ea0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.444 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 -- 172.21.15.45:0/3663008816 shutdown_connections 2024-03-12T05:22:29.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8858d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8856420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 --2- 172.21.15.45:0/3663008816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc884d840 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f490ff87740 1 -- 172.21.15.45:0/3663008816 >> 172.21.15.45:0/3663008816 conn(0x564cc8847b90 msgr2=0x564cc8847f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.445 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.449+0000 7f49077fe640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 0x7f48f40a4160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f490001e000 tx=0x7f49000127a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.450+0000 7f490ff87740 1 -- 172.21.15.45:0/3663008816 shutdown_connections 2024-03-12T05:22:29.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.450+0000 7f490ff87740 1 -- 172.21.15.45:0/3663008816 wait complete. 2024-03-12T05:22:29.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.450+0000 7f490ff87740 1 -- start start 2024-03-12T05:22:29.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.450+0000 7f490ff87740 1 Processor -- start 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8854350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc87b5020 con 0x564cc8856960 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc8488430 con 0x564cc884dd80 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc8444e00 con 0x564cc884d460 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f490ff87740 10 client.0 _ll_register_callbacks cb 0x564cc875ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f48fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f48fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60014/0 (socket says 172.21.15.45:60014) 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f48fe7fc640 1 -- 172.21.15.45:0/2851192409 learned_addr learned my addr 172.21.15.45:0/2851192409 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.451+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 -- 172.21.15.45:0/2851192409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 msgr2=0x564cc88538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 -- 172.21.15.45:0/2851192409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 msgr2=0x564cc8854350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8854350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 -- 172.21.15.45:0/2851192409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4900009640 con 0x564cc884dd80 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f48f0000c00 tx=0x7f48f009a220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91fa2a4035f157c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f009e950 con 0x564cc884dd80 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564cc84b47e0 con 0x564cc884dd80 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48f00a2fa0 con 0x564cc884dd80 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.452+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f00cb5a0 con 0x564cc884dd80 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 10 client.27911 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:29.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564cc87b36a0 con 0x564cc884dd80 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48f007d020 con 0x564cc884dd80 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48f001f060 con 0x564cc884dd80 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 10 client.27911 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 10 client.27911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 20 client.27911 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc8869200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 4 client.27911 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.453+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564cc84f1720 con 0x564cc8866de0 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.454+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc8869200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.454+0000 7f48ee7fc640 10 client.27911 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.455+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc8869200 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a741fadb1c8a617 server_cookie=c771ba095915beab in_seq=0 out_seq=0 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.456+0000 7f48ee7fc640 1 -- 172.21.15.45:0/2851192409 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564cc84f1720 con 0x564cc8866de0 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.456+0000 7f48ee7fc640 10 client.27911 handle_command_reply: tid=0 2024-03-12T05:22:29.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.456+0000 7f490ff87740 1 client.27911 shutdown 2024-03-12T05:22:29.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.457+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 msgr2=0x564cc8853e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.457+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f48f0000c00 tx=0x7f48f009a220 comp rx=0 tx=0).stop 2024-03-12T05:22:29.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.457+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 shutdown_connections 2024-03-12T05:22:29.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc8869200 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc8856960 0x564cc8854350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f48ee7fc640 0 client.27911 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8853e10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 --2- 172.21.15.45:0/2851192409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 >> 172.21.15.45:0/2851192409 conn(0x564cc8847b90 msgr2=0x564cc88498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 shutdown_connections 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 1 -- 172.21.15.45:0/2851192409 wait complete. 2024-03-12T05:22:29.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.458+0000 7f490ff87740 20 client.27911 trim_cache size 0 max 16384 2024-03-12T05:22:29.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.480+0000 7f490ff87740 1 Processor -- start 2024-03-12T05:22:29.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f490ff87740 1 -- start start 2024-03-12T05:22:29.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f490ff87740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f490ff87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8850b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47546/0 (socket says 172.21.15.45:47546) 2024-03-12T05:22:29.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.481+0000 7f4906ffd640 1 -- 172.21.15.45:0/2858874781 learned_addr learned my addr 172.21.15.45:0/2858874781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f490ff87740 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc8856960 0x564cc8851070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc873a130 con 0x564cc884d460 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc88548a0 con 0x564cc884dd80 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc8848980 con 0x564cc8856960 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8850b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f49077fe640 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc8856960 0x564cc8851070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.482+0000 7f4906ffd640 1 -- 172.21.15.45:0/2858874781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc8856960 msgr2=0x564cc8851070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.483+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc8856960 0x564cc8851070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.483+0000 7f4906ffd640 1 -- 172.21.15.45:0/2858874781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 msgr2=0x564cc8850b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.483+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8850b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.483+0000 7f4906ffd640 1 -- 172.21.15.45:0/2858874781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc874fc20 con 0x564cc884d460 2024-03-12T05:22:29.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.483+0000 7f4906ffd640 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009ca0 tx=0x7f48f802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a323bbcf7171eb5d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.485+0000 7f48fcff9640 1 -- 172.21.15.45:0/2858874781 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8034640 con 0x564cc884d460 2024-03-12T05:22:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.485+0000 7f48fcff9640 1 -- 172.21.15.45:0/2858874781 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48f8048070 con 0x564cc884d460 2024-03-12T05:22:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.485+0000 7f48fcff9640 1 -- 172.21.15.45:0/2858874781 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8032a90 con 0x564cc884d460 2024-03-12T05:22:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 msgr2=0x564cc8856000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009ca0 tx=0x7f48f802c890 comp rx=0 tx=0).stop 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 shutdown_connections 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc8856960 0x564cc8851070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc884dd80 0x564cc8850b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 --2- 172.21.15.45:0/2858874781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884d460 0x564cc8856000 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 >> 172.21.15.45:0/2858874781 conn(0x564cc8847b90 msgr2=0x564cc8849870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.486+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 shutdown_connections 2024-03-12T05:22:29.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.487+0000 7f490ff87740 1 -- 172.21.15.45:0/2858874781 wait complete. 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.487+0000 7f490ff87740 1 -- start start 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.487+0000 7f490ff87740 1 Processor -- start 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.487+0000 7f490ff87740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.487+0000 7f490ff87740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884dd80 0x564cc88b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f48fe7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884dd80 0x564cc88b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f490ff87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 0x564cc886ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57156/0 (socket says 172.21.15.45:57156) 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 -- 172.21.15.45:0/3224552823 learned_addr learned my addr 172.21.15.45:0/3224552823 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:29.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc886d300 con 0x564cc884dd80 2024-03-12T05:22:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc886d470 con 0x564cc8856960 2024-03-12T05:22:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc886d630 con 0x564cc884d460 2024-03-12T05:22:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f49077fe640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 0x564cc886ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f490ff87740 10 client.0 _ll_register_callbacks cb 0x7f4900009410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 -- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 msgr2=0x564cc886ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 0x564cc886ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 -- 172.21.15.45:0/3224552823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884dd80 msgr2=0x564cc88b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884dd80 0x564cc88b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.488+0000 7f4906ffd640 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490810e740 con 0x564cc884d460 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f49077fe640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 0x564cc886ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f4906ffd640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f48f802c630 tx=0x7f48f8032f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38db33169aae62d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f48ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:22:29.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8034a60 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f48f8034bc0 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.489+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4900013b00 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f8034dc0 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f490ff87740 10 client.27895 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564cc87758c0 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f48f8032a80 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48f8004270 con 0x564cc884d460 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.490+0000 7f490ff87740 10 client.27895 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f490ff87740 10 client.27895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f490ff87740 20 client.27895 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc88c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f490ff87740 4 client.27895 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:22:29.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4900013b00 con 0x564cc8866de0 2024-03-12T05:22:29.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.491+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc88c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.492+0000 7f48fe7fc640 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc88c11c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=802d17cfa03a5f80 server_cookie=f1d9b54050b3370 in_seq=0 out_seq=0 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f48ee7fc640 10 client.27895 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f48ee7fc640 1 -- 172.21.15.45:0/3224552823 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4900013b00 con 0x564cc8866de0 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f48ee7fc640 10 client.27895 handle_command_reply: tid=0 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 client.27895 shutdown 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 msgr2=0x564cc88b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f48f802c630 tx=0x7f48f8032f40 comp rx=0 tx=0).stop 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 shutdown_connections 2024-03-12T05:22:29.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cc8866de0 0x564cc88c11c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cc884dd80 0x564cc88b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cc8856960 0x564cc886ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 --2- 172.21.15.45:0/3224552823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cc884d460 0x564cc88b75f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 >> 172.21.15.45:0/3224552823 conn(0x564cc8847b90 msgr2=0x564cc88618c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.493+0000 7f48ee7fc640 0 client.27895 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.494+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 shutdown_connections 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.494+0000 7f490ff87740 1 -- 172.21.15.45:0/3224552823 wait complete. 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.494+0000 7f490ff87740 20 client.27895 trim_cache size 0 max 16384 2024-03-12T05:22:29.490 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 msgr2=0x7f48f40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 0x7f48f40a4160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f490001e000 tx=0x7f49000127a0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 msgr2=0x7f4908197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f48f000e9c0 tx=0x7f48f000ee80 comp rx=0 tx=0).stop 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 shutdown_connections 2024-03-12T05:22:29.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f48f40a1ca0 0x7f48f40a4160 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4908106360 0x7f4908197c10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f490819bf40 0x7f490819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 --2- 172.21.15.45:0/3563680157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4908101290 0x7f49081976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 >> 172.21.15.45:0/3563680157 conn(0x7f49080fcb10 msgr2=0x7f49080fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.495+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 shutdown_connections 2024-03-12T05:22:29.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:29.496+0000 7f490d3b6640 1 -- 172.21.15.45:0/3563680157 wait complete. 2024-03-12T05:22:29.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 625.0 s, 1139.8 tps, lat 87.958 ms stddev 28.959 2024-03-12T05:22:30.512 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:22:30.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:22:30.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:22:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.701+0000 7f646c8c9640 1 -- 172.21.15.45:0/420995445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 msgr2=0x7f6468110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.701+0000 7f646c8c9640 1 --2- 172.21.15.45:0/420995445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 0x7f6468110230 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6460009cd0 tx=0x7f646002c0e0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 -- 172.21.15.45:0/420995445 shutdown_connections 2024-03-12T05:22:30.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 --2- 172.21.15.45:0/420995445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 0x7f6468110230 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 --2- 172.21.15.45:0/420995445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103f40 0x7f6468107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 --2- 172.21.15.45:0/420995445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468103590 0x7f6468103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 -- 172.21.15.45:0/420995445 >> 172.21.15.45:0/420995445 conn(0x7f64680fedd0 msgr2=0x7f64681011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.702+0000 7f646c8c9640 1 -- 172.21.15.45:0/420995445 shutdown_connections 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.703+0000 7f646c8c9640 1 -- 172.21.15.45:0/420995445 wait complete. 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.704+0000 7f646c8c9640 1 Processor -- start 2024-03-12T05:22:30.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.704+0000 7f646c8c9640 1 -- start start 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.704+0000 7f646c8c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.704+0000 7f646c8c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 0x7f646819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646c8c9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 0x7f6468199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60042/0 (socket says 172.21.15.45:60042) 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 -- 172.21.15.45:0/131895209 learned_addr learned my addr 172.21.15.45:0/131895209 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646819a230 con 0x7f6468108660 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646819a3a0 con 0x7f6468103590 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646819a510 con 0x7f6468199780 2024-03-12T05:22:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f6466d9d640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 0x7f6468199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 -- 172.21.15.45:0/131895209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 msgr2=0x7f6468199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 0x7f6468199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 -- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 msgr2=0x7f646819fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 0x7f646819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6458009490 con 0x7f6468103590 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f6466d9d640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 0x7f6468199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:22:30.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.705+0000 7f646659c640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c890 tx=0x7f645c00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.706+0000 7f64577fe640 1 -- 172.21.15.45:0/131895209 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c004220 con 0x7f6468103590 2024-03-12T05:22:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.706+0000 7f64577fe640 1 -- 172.21.15.45:0/131895209 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f645c007e60 con 0x7f6468103590 2024-03-12T05:22:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.706+0000 7f64577fe640 1 -- 172.21.15.45:0/131895209 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c004a00 con 0x7f6468103590 2024-03-12T05:22:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.706+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6460009900 con 0x7f6468103590 2024-03-12T05:22:30.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.706+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64681a67c0 con 0x7f6468103590 2024-03-12T05:22:30.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.709+0000 7f64577fe640 1 -- 172.21.15.45:0/131895209 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f645c0079b0 con 0x7f6468103590 2024-03-12T05:22:30.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 Processor -- start 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 -- start start 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a6101700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a6112af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a606ff90 con 0x55e7a6101320 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a606d4c0 con 0x55e7a6101cd0 2024-03-12T05:22:30.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.710+0000 7f646f570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a6068ca0 con 0x55e7a61130c0 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6466d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a6101700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f64577fe640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 0x7f64500a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6466d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a6101700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47580/0 (socket says 172.21.15.45:47580) 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6466d9d640 1 -- 172.21.15.45:0/1255403839 learned_addr learned my addr 172.21.15.45:0/1255403839 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f646659c640 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a6112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6466d9d640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 0x7f64500a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f64577fe640 1 -- 172.21.15.45:0/131895209 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f645c0cd540 con 0x7f6468103590 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 -- 172.21.15.45:0/1255403839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 msgr2=0x55e7a6112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a6112af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 -- 172.21.15.45:0/1255403839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 msgr2=0x55e7a6101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a6101700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 -- 172.21.15.45:0/1255403839 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a602b0b0 con 0x55e7a61130c0 2024-03-12T05:22:30.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.711+0000 7f6465d9b640 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61154d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f64680ff360 tx=0x7f645800e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6192f63268ab74e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.712+0000 7f6466d9d640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 0x7f64500a4270 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6460033000 tx=0x7f646002c0e0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.712+0000 7f6454ff9640 1 -- 172.21.15.45:0/1255403839 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645800ed50 con 0x55e7a61130c0 2024-03-12T05:22:30.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.712+0000 7f6454ff9640 1 -- 172.21.15.45:0/1255403839 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f64580044d0 con 0x55e7a61130c0 2024-03-12T05:22:30.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.712+0000 7f6454ff9640 1 -- 172.21.15.45:0/1255403839 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458010690 con 0x55e7a61130c0 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 -- 172.21.15.45:0/1255403839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 msgr2=0x55e7a61154d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61154d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f64680ff360 tx=0x7f645800e350 comp rx=0 tx=0).stop 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 -- 172.21.15.45:0/1255403839 shutdown_connections 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61154d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a6112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 --2- 172.21.15.45:0/1255403839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a6101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 -- 172.21.15.45:0/1255403839 >> 172.21.15.45:0/1255403839 conn(0x55e7a60fb840 msgr2=0x55e7a60fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 -- 172.21.15.45:0/1255403839 shutdown_connections 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.713+0000 7f646f570740 1 -- 172.21.15.45:0/1255403839 wait complete. 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646f570740 1 -- start start 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646f570740 1 Processor -- start 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646f570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646f570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a610cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.714+0000 7f646659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47590/0 (socket says 172.21.15.45:47590) 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646f570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61126a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646f570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a600df40 con 0x55e7a6101320 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646f570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a5d3c4b0 con 0x55e7a6101cd0 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646f570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a5fee670 con 0x55e7a61130c0 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646f570740 10 client.0 _ll_register_callbacks cb 0x55e7a605bf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:30.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 -- 172.21.15.45:0/3214506649 learned_addr learned my addr 172.21.15.45:0/3214506649 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f6465d9b640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a610cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f6466d9d640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 -- 172.21.15.45:0/3214506649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 msgr2=0x55e7a61126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 -- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 msgr2=0x55e7a610cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a610cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 -- 172.21.15.45:0/3214506649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6458009850 con 0x55e7a6101320 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f646659c640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f645c096ab0 tx=0x7f645c069540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dc47663636cfa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.715+0000 7f644dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:22:30.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.716+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c09a7e0 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.716+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f645c09a0e0 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.716+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7a6042f40 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.716+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f645c09e230 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.716+0000 7f646f570740 10 client.18447 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.717+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e7a6067320 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.717+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e7a6042f40 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f645c016030 con 0x55e7a6101320 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 10 client.18447 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 10 client.18447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 20 client.18447 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a612be30 0x55e7a612e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 4 client.18447 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:30.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7a605f310 con 0x55e7a612be30 2024-03-12T05:22:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.718+0000 7f6465d9b640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a612be30 0x55e7a612e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.719+0000 7f6465d9b640 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a612be30 0x55e7a612e220 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b1590247d3bcf server_cookie=7948d5d9d47e23ab in_seq=0 out_seq=0 2024-03-12T05:22:30.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.719+0000 7f644dffb640 10 client.18447 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.719+0000 7f644dffb640 1 -- 172.21.15.45:0/3214506649 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e7a605f310 con 0x55e7a612be30 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.719+0000 7f644dffb640 10 client.18447 handle_command_reply: tid=0 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 client.18447 shutdown 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 msgr2=0x55e7a610c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f645c096ab0 tx=0x7f645c069540 comp rx=0 tx=0).stop 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 shutdown_connections 2024-03-12T05:22:30.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a612be30 0x55e7a612e220 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.720+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a61130c0 0x55e7a61126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101cd0 0x55e7a610cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 1 --2- 172.21.15.45:0/3214506649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101320 0x55e7a610c980 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 >> 172.21.15.45:0/3214506649 conn(0x55e7a60fb840 msgr2=0x55e7a60fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f644dffb640 0 client.18447 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 shutdown_connections 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 1 -- 172.21.15.45:0/3214506649 wait complete. 2024-03-12T05:22:30.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.721+0000 7f646f570740 20 client.18447 trim_cache size 0 max 16384 2024-03-12T05:22:30.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:30 smithi130 ceph-mon[35398]: pgmap v363: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 15 MiB/s wr, 1.23k op/s 2024-03-12T05:22:30.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:30 smithi116 ceph-mon[35486]: pgmap v363: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 15 MiB/s wr, 1.23k op/s 2024-03-12T05:22:30.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.749+0000 7f646f570740 1 Processor -- start 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 -- start start 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101320 0x55e7a61125c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 0x55e7a6103fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a60fca50 con 0x55e7a61130c0 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a60fc410 con 0x55e7a6101320 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646f570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a6102560 con 0x55e7a6101cd0 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f6465d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f646659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101320 0x55e7a61125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f6465d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57216/0 (socket says 172.21.15.45:57216) 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f6465d9b640 1 -- 172.21.15.45:0/4040794896 learned_addr learned my addr 172.21.15.45:0/4040794896 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.750+0000 7f6466d9d640 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 0x55e7a6103fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 -- 172.21.15.45:0/4040794896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101320 msgr2=0x55e7a61125c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101320 0x55e7a61125c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 -- 172.21.15.45:0/4040794896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 msgr2=0x55e7a6103fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 0x55e7a6103fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 -- 172.21.15.45:0/4040794896 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a6003c00 con 0x55e7a6101cd0 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6466d9d640 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 0x55e7a6103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:30.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f6465d9b640 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f64680ff360 tx=0x7f64580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148314d7818ea37c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f644ffff640 1 -- 172.21.15.45:0/4040794896 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458003a10 con 0x55e7a6101cd0 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f644ffff640 1 -- 172.21.15.45:0/4040794896 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f64580096c0 con 0x55e7a6101cd0 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.751+0000 7f644ffff640 1 -- 172.21.15.45:0/4040794896 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6458018430 con 0x55e7a6101cd0 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 -- 172.21.15.45:0/4040794896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 msgr2=0x55e7a60fa760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f64680ff360 tx=0x7f64580046c0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 -- 172.21.15.45:0/4040794896 shutdown_connections 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a61130c0 0x55e7a6103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101cd0 0x55e7a60fa760 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 --2- 172.21.15.45:0/4040794896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a6101320 0x55e7a61125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.752+0000 7f646f570740 1 -- 172.21.15.45:0/4040794896 >> 172.21.15.45:0/4040794896 conn(0x55e7a60fb840 msgr2=0x55e7a60fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.753+0000 7f646f570740 1 -- 172.21.15.45:0/4040794896 shutdown_connections 2024-03-12T05:22:30.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.753+0000 7f646f570740 1 -- 172.21.15.45:0/4040794896 wait complete. 2024-03-12T05:22:30.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 -- start start 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 Processor -- start 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 0x55e7a617c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 0x55e7a617ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a61130c0 0x55e7a6176920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a6176ee0 con 0x55e7a6101cd0 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a6177050 con 0x55e7a61130c0 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a6177210 con 0x55e7a6101320 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.754+0000 7f646f570740 10 client.0 _ll_register_callbacks cb 0x7f6458009390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f6465d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 0x55e7a617ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f6465d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 0x55e7a617ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47594/0 (socket says 172.21.15.45:47594) 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f6465d9b640 1 -- 172.21.15.45:0/57742683 learned_addr learned my addr 172.21.15.45:0/57742683 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 0x55e7a617c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f6466d9d640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a61130c0 0x55e7a6176920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 -- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a61130c0 msgr2=0x55e7a6176920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a61130c0 0x55e7a6176920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 -- 172.21.15.45:0/57742683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 msgr2=0x55e7a617ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 0x55e7a617ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.755+0000 7f646659c640 1 -- 172.21.15.45:0/57742683 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6468110a40 con 0x55e7a6101320 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f646659c640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 0x55e7a617c4e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f645c099d90 tx=0x7f645c0961b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b7854768402722d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f644dffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c0af070 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f645c09a7e0 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7a608e900 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f645c099a40 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f646f570740 10 client.27907 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f645801c040 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.756+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e7a608e900 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f645c07c050 con 0x55e7a6101320 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 10 client.27907 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 10 client.27907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 20 client.27907 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a6184030 0x55e7a6186450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 4 client.27907 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e7a5d68860 con 0x55e7a6184030 2024-03-12T05:22:30.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.757+0000 7f6465d9b640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a6184030 0x55e7a6186450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.758+0000 7f6465d9b640 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a6184030 0x55e7a6186450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb0589cdc521c11c server_cookie=2ab19836b83a9258 in_seq=0 out_seq=0 2024-03-12T05:22:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.758+0000 7f644dffb640 10 client.27907 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.760+0000 7f644dffb640 1 -- 172.21.15.45:0/57742683 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e7a5d68860 con 0x55e7a6184030 2024-03-12T05:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.760+0000 7f644dffb640 10 client.27907 handle_command_reply: tid=0 2024-03-12T05:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.760+0000 7f646f570740 1 client.27907 shutdown 2024-03-12T05:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.760+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 msgr2=0x55e7a617c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.760+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 0x55e7a617c4e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f645c099d90 tx=0x7f645c0961b0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 shutdown_connections 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e7a6184030 0x55e7a6186450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e7a6101cd0 0x55e7a617ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7a61130c0 0x55e7a6176920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 --2- 172.21.15.45:0/57742683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e7a6101320 0x55e7a617c4e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 >> 172.21.15.45:0/57742683 conn(0x55e7a60fb840 msgr2=0x55e7a6106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.761+0000 7f644dffb640 0 client.27907 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.762+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 shutdown_connections 2024-03-12T05:22:30.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.762+0000 7f646f570740 1 -- 172.21.15.45:0/57742683 wait complete. 2024-03-12T05:22:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.763+0000 7f646f570740 20 client.27907 trim_cache size 0 max 16384 2024-03-12T05:22:30.760 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:22:30.760 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:22:30.760 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "087de293-e73a-4b71-97d2-976a9ab62888", 2024-03-12T05:22:30.760 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:22:30.760 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:22:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 msgr2=0x7f64500a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 0x7f64500a4270 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6460033000 tx=0x7f646002c0e0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 msgr2=0x7f646819f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c890 tx=0x7f645c00cd50 comp rx=0 tx=0).stop 2024-03-12T05:22:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 shutdown_connections 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64500a1db0 0x7f64500a4270 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6468108660 0x7f646819fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6468199780 0x7f6468199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 --2- 172.21.15.45:0/131895209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6468103590 0x7f646819f610 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 >> 172.21.15.45:0/131895209 conn(0x7f64680fedd0 msgr2=0x7f64681001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 shutdown_connections 2024-03-12T05:22:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.768+0000 7f646c8c9640 1 -- 172.21.15.45:0/131895209 wait complete. 2024-03-12T05:22:30.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:30 smithi045 ceph-mon[32174]: pgmap v363: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 15 MiB/s wr, 1.23k op/s 2024-03-12T05:22:30.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:22:30.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:22:30.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.972+0000 7f942a7fc640 1 -- 172.21.15.45:0/2802805351 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941802c710 con 0x7f94240a4d50 2024-03-12T05:22:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 -- 172.21.15.45:0/2802805351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240a4d50 msgr2=0x7f94240a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 --2- 172.21.15.45:0/2802805351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240a4d50 0x7f94240a51d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9418009890 tx=0x7f941802bb70 comp rx=0 tx=0).stop 2024-03-12T05:22:30.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 -- 172.21.15.45:0/2802805351 shutdown_connections 2024-03-12T05:22:30.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 --2- 172.21.15.45:0/2802805351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 0x7f94240b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 --2- 172.21.15.45:0/2802805351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240a4d50 0x7f94240a51d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 --2- 172.21.15.45:0/2802805351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240af0e0 0x7f94240af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 -- 172.21.15.45:0/2802805351 >> 172.21.15.45:0/2802805351 conn(0x7f942401a640 msgr2=0x7f942401aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 -- 172.21.15.45:0/2802805351 shutdown_connections 2024-03-12T05:22:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.974+0000 7f9430ae6640 1 -- 172.21.15.45:0/2802805351 wait complete. 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.975+0000 7f9430ae6640 1 Processor -- start 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.975+0000 7f9430ae6640 1 -- start start 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240a4d50 0x7f94240b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 0x7f94240b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94240b3a10 con 0x7f94240a4d50 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94240b3b80 con 0x7f94240a5710 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f9430ae6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9424140550 con 0x7f94240af0e0 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f942affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 0x7f94240b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f942bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f942bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57244/0 (socket says 172.21.15.45:57244) 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f942bfff640 1 -- 172.21.15.45:0/3776749994 learned_addr learned my addr 172.21.15.45:0/3776749994 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.976+0000 7f942b7fe640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240a4d50 0x7f94240b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 -- 172.21.15.45:0/3776749994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 msgr2=0x7f94240b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 0x7f94240b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 -- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240a4d50 msgr2=0x7f94240b5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240a4d50 0x7f94240b5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 -- 172.21.15.45:0/3776749994 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c04f750 con 0x7f94240af0e0 2024-03-12T05:22:30.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.977+0000 7f942bfff640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f942c0506c0 tx=0x7f942c072d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.978+0000 7f9428ff9640 1 -- 172.21.15.45:0/3776749994 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c070c90 con 0x7f94240af0e0 2024-03-12T05:22:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.978+0000 7f9428ff9640 1 -- 172.21.15.45:0/3776749994 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f942c070df0 con 0x7f94240af0e0 2024-03-12T05:22:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.978+0000 7f9428ff9640 1 -- 172.21.15.45:0/3776749994 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c061e30 con 0x7f94240af0e0 2024-03-12T05:22:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.978+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418009490 con 0x7f94240af0e0 2024-03-12T05:22:30.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.978+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9424140b30 con 0x7f94240af0e0 2024-03-12T05:22:30.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.982+0000 7f94349fa740 1 Processor -- start 2024-03-12T05:22:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.984+0000 7f9428ff9640 1 -- 172.21.15.45:0/3776749994 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f942c0615c0 con 0x7f94240af0e0 2024-03-12T05:22:30.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f9428ff9640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 0x7f941c0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f94349fa740 1 -- start start 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f9428ff9640 1 -- 172.21.15.45:0/3776749994 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f942c1306b0 con 0x7f94240af0e0 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f942affd640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 0x7f941c0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f94349fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 0x558ddabd7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.985+0000 7f94349fa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7db0 0x558ddabe0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f94349fa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabe2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f94349fa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddab46280 con 0x558ddabe0990 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f94349fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddab437b0 con 0x558ddabd7490 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f94349fa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddab3ef90 con 0x558ddabd7db0 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabe2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 0x558ddabd7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabe2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47632/0 (socket says 172.21.15.45:47632) 2024-03-12T05:22:30.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942bfff640 1 -- 172.21.15.45:0/1029235490 learned_addr learned my addr 172.21.15.45:0/1029235490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 -- 172.21.15.45:0/1029235490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7db0 msgr2=0x558ddabe0450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:22:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7db0 0x558ddabe0450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 -- 172.21.15.45:0/1029235490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 msgr2=0x558ddabe2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabe2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.986+0000 7f942b7fe640 1 -- 172.21.15.45:0/1029235490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ddaafd570 con 0x558ddabd7490 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.987+0000 7f942b7fe640 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 0x558ddabd7870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9424018920 tx=0x7f942000bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f9ee16ef14b665c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.987+0000 7f9415ffb640 1 -- 172.21.15.45:0/1029235490 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420003dd0 con 0x558ddabd7490 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.987+0000 7f9415ffb640 1 -- 172.21.15.45:0/1029235490 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f942000be30 con 0x558ddabd7490 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.988+0000 7f9415ffb640 1 -- 172.21.15.45:0/1029235490 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200117c0 con 0x558ddabd7490 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.988+0000 7f94349fa740 1 -- 172.21.15.45:0/1029235490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 msgr2=0x558ddabd7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.988+0000 7f94349fa740 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 0x558ddabd7870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9424018920 tx=0x7f942000bb20 comp rx=0 tx=0).stop 2024-03-12T05:22:30.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.988+0000 7f942affd640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 0x7f941c0a4230 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9418005940 tx=0x7f9418005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 -- 172.21.15.45:0/1029235490 shutdown_connections 2024-03-12T05:22:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabe2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7db0 0x558ddabe0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 --2- 172.21.15.45:0/1029235490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7490 0x558ddabd7870 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 -- 172.21.15.45:0/1029235490 >> 172.21.15.45:0/1029235490 conn(0x558ddabd1bc0 msgr2=0x558ddabd1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 -- 172.21.15.45:0/1029235490 shutdown_connections 2024-03-12T05:22:30.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.994+0000 7f94349fa740 1 -- 172.21.15.45:0/1029235490 wait complete. 2024-03-12T05:22:30.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 -- start start 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 Processor -- start 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabe0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabdd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dda812430 con 0x558ddabd7db0 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dda7cee00 con 0x558ddabe0990 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddaac3fd0 con 0x558ddabd7490 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.995+0000 7f94349fa740 10 client.0 _ll_register_callbacks cb 0x558ddab19fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:30.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60124/0 (socket says 172.21.15.45:60124) 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 -- 172.21.15.45:0/3178189216 learned_addr learned my addr 172.21.15.45:0/3178189216 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942b7fe640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabe0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942affd640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabdd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 -- 172.21.15.45:0/3178189216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 msgr2=0x558ddabe0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabe0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 -- 172.21.15.45:0/3178189216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 msgr2=0x558ddabdd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabdd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 -- 172.21.15.45:0/3178189216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94200041d0 con 0x558ddabe0990 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f942bfff640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f942c071010 tx=0x7f942c0fd4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=715079996f7e6f59 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.996+0000 7f940affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c0fc7e0 con 0x558ddabe0990 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558dda83e7e0 con 0x558ddabe0990 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f942c0fc940 con 0x558ddabe0990 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f942c0fcb00 con 0x558ddabe0990 2024-03-12T05:22:30.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f94349fa740 10 client.27947 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558ddab3d610 con 0x558ddabe0990 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.997+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f942c109080 con 0x558ddabe0990 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f942c0fc250 con 0x558ddabe0990 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 10 client.27947 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 10 client.27947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 20 client.27947 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddabf0bf0 0x558ddabf3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 4 client.27947 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558dda87b720 con 0x558ddabf0bf0 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.998+0000 7f942b7fe640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddabf0bf0 0x558ddabf3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.999+0000 7f940affd640 10 client.27947 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.999+0000 7f942b7fe640 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddabf0bf0 0x558ddabf3010 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c459944398c6bfe8 server_cookie=c2be46bda936c386 in_seq=0 out_seq=0 2024-03-12T05:22:30.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.999+0000 7f940affd640 1 -- 172.21.15.45:0/3178189216 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558dda87b720 con 0x558ddabf0bf0 2024-03-12T05:22:30.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.999+0000 7f940affd640 10 client.27947 handle_command_reply: tid=0 2024-03-12T05:22:30.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:30.999+0000 7f94349fa740 1 client.27947 shutdown 2024-03-12T05:22:30.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 msgr2=0x558ddabddb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f942c071010 tx=0x7f942c0fd4d0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 shutdown_connections 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddabf0bf0 0x558ddabf3010 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabdd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabddb00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 --2- 172.21.15.45:0/3178189216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabe0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 >> 172.21.15.45:0/3178189216 conn(0x558ddabd1bc0 msgr2=0x558ddabd3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f940affd640 0 client.27947 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 shutdown_connections 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.000+0000 7f94349fa740 1 -- 172.21.15.45:0/3178189216 wait complete. 2024-03-12T05:22:30.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.001+0000 7f94349fa740 20 client.27947 trim_cache size 0 max 16384 2024-03-12T05:22:31.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 Processor -- start 2024-03-12T05:22:31.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 -- start start 2024-03-12T05:22:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabe0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabe7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddabdbc00 con 0x558ddabd7db0 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddabcd230 con 0x558ddabe0990 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f94349fa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddabe7e90 con 0x558ddabd7490 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.029+0000 7f942b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57276/0 (socket says 172.21.15.45:57276) 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 -- 172.21.15.45:0/517117306 learned_addr learned my addr 172.21.15.45:0/517117306 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942affd640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabe0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942bfff640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabe7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 -- 172.21.15.45:0/517117306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 msgr2=0x558ddabe7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabe7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 -- 172.21.15.45:0/517117306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 msgr2=0x558ddabe0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabe0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 -- 172.21.15.45:0/517117306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94240051f0 con 0x558ddabd7490 2024-03-12T05:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942bfff640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabe7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.030+0000 7f942b7fe640 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f942001af90 tx=0x7f942000b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da99fdd5fd6b5294 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.031+0000 7f94157fa640 1 -- 172.21.15.45:0/517117306 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942001e040 con 0x558ddabd7490 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.031+0000 7f94157fa640 1 -- 172.21.15.45:0/517117306 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f942001f040 con 0x558ddabd7490 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.031+0000 7f94157fa640 1 -- 172.21.15.45:0/517117306 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942001a960 con 0x558ddabd7490 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.033+0000 7f94349fa740 1 -- 172.21.15.45:0/517117306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 msgr2=0x558ddabdfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:31.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.033+0000 7f94349fa740 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f942001af90 tx=0x7f942000b650 comp rx=0 tx=0).stop 2024-03-12T05:22:31.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.034+0000 7f94349fa740 1 -- 172.21.15.45:0/517117306 shutdown_connections 2024-03-12T05:22:31.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.034+0000 7f94349fa740 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabd7db0 0x558ddabe0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.034+0000 7f94349fa740 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabe0990 0x558ddabe7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.034+0000 7f94349fa740 1 --2- 172.21.15.45:0/517117306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabdfcf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.034+0000 7f94349fa740 1 -- 172.21.15.45:0/517117306 >> 172.21.15.45:0/517117306 conn(0x558ddabd1bc0 msgr2=0x558ddabd1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:31.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.036+0000 7f94349fa740 1 -- 172.21.15.45:0/517117306 shutdown_connections 2024-03-12T05:22:31.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.036+0000 7f94349fa740 1 -- 172.21.15.45:0/517117306 wait complete. 2024-03-12T05:22:31.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.036+0000 7f94349fa740 1 -- start start 2024-03-12T05:22:31.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.037+0000 7f94349fa740 1 Processor -- start 2024-03-12T05:22:31.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.037+0000 7f94349fa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabd9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.037+0000 7f94349fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.037+0000 7f94349fa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabda8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f94349fa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddac47120 con 0x558ddabe0990 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f94349fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddac47290 con 0x558ddabd7db0 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f94349fa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ddac47450 con 0x558ddabd7490 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f94349fa740 10 client.0 _ll_register_callbacks cb 0x7f9420004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:60136/0 (socket says 172.21.15.45:60136) 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 -- 172.21.15.45:0/694754631 learned_addr learned my addr 172.21.15.45:0/694754631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942b7fe640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabd9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942bfff640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabda8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 -- 172.21.15.45:0/694754631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 msgr2=0x558ddabd9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabd9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 -- 172.21.15.45:0/694754631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 msgr2=0x558ddabda8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:22:31.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabda8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942affd640 1 -- 172.21.15.45:0/694754631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ddab3d610 con 0x558ddabd7db0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.038+0000 7f942b7fe640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabd9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f942affd640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9418009890 tx=0x7f941802bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=857c293d1f00358 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f940affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94180037f0 con 0x558ddabd7db0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9418003950 con 0x558ddabd7db0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f942000be30 con 0x558ddabd7db0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9418003b10 con 0x558ddabd7db0 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.039+0000 7f94349fa740 10 client.27953 fetch_fsmap learned FSMap version 33 2024-03-12T05:22:31.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558ddaafd570 con 0x558ddabd7db0 2024-03-12T05:22:31.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f941802ce90 con 0x558ddabd7db0 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f941803fbc0 con 0x558ddabd7db0 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 10 client.27953 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 10 client.27953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 20 client.27953 populate_metadata read hostname 'smithi045' 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddac48c50 0x558ddac4b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 4 client.27953 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.040+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f942000be30 con 0x558ddac48c50 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.041+0000 7f942b7fe640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddac48c50 0x558ddac4b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.041+0000 7f942b7fe640 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddac48c50 0x558ddac4b070 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417a3579f6b1eb61 server_cookie=430450f214ac874b in_seq=0 out_seq=0 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.042+0000 7f940affd640 10 client.27953 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.042+0000 7f940affd640 1 -- 172.21.15.45:0/694754631 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f942000be30 con 0x558ddac48c50 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.042+0000 7f940affd640 10 client.27953 handle_command_reply: tid=0 2024-03-12T05:22:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.042+0000 7f94349fa740 1 client.27953 shutdown 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 msgr2=0x558ddabda390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9418009890 tx=0x7f941802bb70 comp rx=0 tx=0).stop 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 shutdown_connections 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558ddac48c50 0x558ddac4b070 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f940affd640 0 client.27953 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558ddabe0990 0x558ddabda8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558ddabd7db0 0x558ddabda390 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 --2- 172.21.15.45:0/694754631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ddabd7490 0x558ddabd9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 >> 172.21.15.45:0/694754631 conn(0x558ddabd1bc0 msgr2=0x558ddabd88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 shutdown_connections 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 1 -- 172.21.15.45:0/694754631 wait complete. 2024-03-12T05:22:31.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.043+0000 7f94349fa740 20 client.27953 trim_cache size 0 max 16384 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "087de293-e73a-4b71-97d2-976a9ab62888": { 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "087de293-e73a-4b71-97d2-976a9ab62888", 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:22:31.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:22:31.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:22:31.041 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:22:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 msgr2=0x7f941c0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 0x7f941c0a4230 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9418005940 tx=0x7f9418005850 comp rx=0 tx=0).stop 2024-03-12T05:22:31.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 msgr2=0x7f94240b3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f942c0506c0 tx=0x7f942c072d80 comp rx=0 tx=0).stop 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 shutdown_connections 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f941c0a1d70 0x7f941c0a4230 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9418005940 tx=0x7f9418005850 comp rx=0 tx=0).stop 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94240af0e0 0x7f94240b3280 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94240a5710 0x7f94240b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 --2- 172.21.15.45:0/3776749994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94240a4d50 0x7f94240b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.044+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 >> 172.21.15.45:0/3776749994 conn(0x7f942401a640 msgr2=0x7f94240a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.045+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 shutdown_connections 2024-03-12T05:22:31.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:22:31.045+0000 7f9430ae6640 1 -- 172.21.15.45:0/3776749994 wait complete. 2024-03-12T05:22:31.059 INFO:tasks.cephfs.filesystem:scrub status for tag:087de293-e73a-4b71-97d2-976a9ab62888 - {'path': '/', 'tag': '087de293-e73a-4b71-97d2-976a9ab62888', 'options': 'recursive,force'} 2024-03-12T05:22:31.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:31 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:22:31.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:31 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:31.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:31 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:22:31.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:31 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:31.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:31 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:22:31.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:31 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:22:32.642 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:22:32.643 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.643 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.643 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:32 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-03-12T05:22:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:22:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:22:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:22:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:22:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:32 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 13 MiB/s wr, 1.13k op/s 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:32.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:32 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:22:34.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:34 smithi130 ceph-mon[35398]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-03-12T05:22:34.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:34 smithi116 ceph-mon[35486]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-03-12T05:22:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:34 smithi045 ceph-mon[32174]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-03-12T05:22:34.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 630.0 s, 1262.0 tps, lat 79.294 ms stddev 29.387 2024-03-12T05:22:36.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:36 smithi130 ceph-mon[35398]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-03-12T05:22:36.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:36 smithi116 ceph-mon[35486]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-03-12T05:22:36.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:36 smithi045 ceph-mon[32174]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 18 MiB/s wr, 1.57k op/s 2024-03-12T05:22:38.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:38 smithi130 ceph-mon[35398]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:22:38.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:38 smithi116 ceph-mon[35486]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:22:38.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:38 smithi045 ceph-mon[32174]: pgmap v367: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:22:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 635.0 s, 1375.0 tps, lat 72.793 ms stddev 26.132 2024-03-12T05:22:40.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:40 smithi045 ceph-mon[32174]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-03-12T05:22:40.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:40 smithi130 ceph-mon[35398]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-03-12T05:22:40.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:40 smithi116 ceph-mon[35486]: pgmap v368: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 14 MiB/s wr, 1.24k op/s 2024-03-12T05:22:42.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:42 smithi130 ceph-mon[35398]: pgmap v369: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-03-12T05:22:42.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:42 smithi116 ceph-mon[35486]: pgmap v369: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-03-12T05:22:42.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:42 smithi045 ceph-mon[32174]: pgmap v369: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 17 MiB/s wr, 1.49k op/s 2024-03-12T05:22:43.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:43.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:43.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:43 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:43.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:43.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:43.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:43 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:43.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:43.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:43.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:43 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:44.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:44 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:22:44.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:44 smithi130 ceph-mon[35398]: pgmap v370: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-12T05:22:44.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:44 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:22:44.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:44 smithi116 ceph-mon[35486]: pgmap v370: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-12T05:22:44.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:44 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:22:44.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:44 smithi045 ceph-mon[32174]: pgmap v370: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-12T05:22:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 640.0 s, 1422.0 tps, lat 70.249 ms stddev 24.638 2024-03-12T05:22:46.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:46 smithi130 ceph-mon[35398]: pgmap v371: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 18 MiB/s wr, 1.76k op/s 2024-03-12T05:22:46.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:46 smithi116 ceph-mon[35486]: pgmap v371: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 18 MiB/s wr, 1.76k op/s 2024-03-12T05:22:46.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:46 smithi045 ceph-mon[32174]: pgmap v371: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 18 MiB/s wr, 1.76k op/s 2024-03-12T05:22:48.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:48 smithi130 ceph-mon[35398]: pgmap v372: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-03-12T05:22:48.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:48 smithi116 ceph-mon[35486]: pgmap v372: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-03-12T05:22:48.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:48 smithi045 ceph-mon[32174]: pgmap v372: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 13 MiB/s wr, 1.24k op/s 2024-03-12T05:22:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 645.0 s, 1331.0 tps, lat 75.152 ms stddev 42.318 2024-03-12T05:22:50.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:50 smithi045 ceph-mon[32174]: pgmap v373: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 15 MiB/s wr, 1.46k op/s 2024-03-12T05:22:50.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:50 smithi130 ceph-mon[35398]: pgmap v373: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 15 MiB/s wr, 1.46k op/s 2024-03-12T05:22:50.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:50 smithi116 ceph-mon[35486]: pgmap v373: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 15 MiB/s wr, 1.46k op/s 2024-03-12T05:22:52.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:52 smithi045 ceph-mon[32174]: pgmap v374: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:22:52.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:52 smithi130 ceph-mon[35398]: pgmap v374: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:22:52.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:52 smithi116 ceph-mon[35486]: pgmap v374: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:22:54.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:54 smithi045 ceph-mon[32174]: pgmap v375: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:22:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 650.0 s, 1328.2 tps, lat 75.332 ms stddev 30.081 2024-03-12T05:22:54.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:54 smithi130 ceph-mon[35398]: pgmap v375: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:22:54.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:54 smithi116 ceph-mon[35486]: pgmap v375: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 12 MiB/s wr, 1.22k op/s 2024-03-12T05:22:56.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:56 smithi045 ceph-mon[32174]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 17 MiB/s wr, 1.74k op/s 2024-03-12T05:22:56.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:56 smithi130 ceph-mon[35398]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 17 MiB/s wr, 1.74k op/s 2024-03-12T05:22:56.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:56 smithi116 ceph-mon[35486]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 17 MiB/s wr, 1.74k op/s 2024-03-12T05:22:58.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:58 smithi045 ceph-mon[32174]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-03-12T05:22:58.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:58 smithi130 ceph-mon[35398]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-03-12T05:22:58.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:58 smithi116 ceph-mon[35486]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-03-12T05:22:59.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:22:59 smithi045 ceph-mon[32174]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-03-12T05:22:59.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 655.0 s, 1339.2 tps, lat 74.661 ms stddev 26.608 2024-03-12T05:22:59.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:22:59 smithi130 ceph-mon[35398]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-03-12T05:22:59.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:22:59 smithi116 ceph-mon[35486]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-03-12T05:23:01.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:23:01.061 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.235+0000 7f543852e640 1 -- 172.21.15.45:0/26570867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430101d40 msgr2=0x7f5430105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.235+0000 7f543852e640 1 --2- 172.21.15.45:0/26570867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430101d40 0x7f5430105d80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5420009860 tx=0x7f542002bb70 comp rx=0 tx=0).stop 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 -- 172.21.15.45:0/26570867 shutdown_connections 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 --2- 172.21.15.45:0/26570867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430106460 0x7f543010e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 --2- 172.21.15.45:0/26570867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430101d40 0x7f5430105d80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 --2- 172.21.15.45:0/26570867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5430101390 0x7f5430101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 -- 172.21.15.45:0/26570867 >> 172.21.15.45:0/26570867 conn(0x7f54300fcc50 msgr2=0x7f54300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 -- 172.21.15.45:0/26570867 shutdown_connections 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 -- 172.21.15.45:0/26570867 wait complete. 2024-03-12T05:23:01.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 Processor -- start 2024-03-12T05:23:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.236+0000 7f543852e640 1 -- start start 2024-03-12T05:23:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430101390 0x7f54301976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430106460 0x7f5430197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.233 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543019c980 con 0x7f543019bf90 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543019caf0 con 0x7f5430101390 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f543852e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543019cc60 con 0x7f5430106460 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50796/0 (socket says 172.21.15.45:50796) 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 -- 172.21.15.45:0/1877907748 learned_addr learned my addr 172.21.15.45:0/1877907748 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5435ac9640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430106460 0x7f5430197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f54362ca640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430101390 0x7f54301976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 -- 172.21.15.45:0/1877907748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430106460 msgr2=0x7f5430197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430106460 0x7f5430197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 -- 172.21.15.45:0/1877907748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430101390 msgr2=0x7f54301976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430101390 0x7f54301976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.237+0000 7f5436acb640 1 -- 172.21.15.45:0/1877907748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c009560 con 0x7f543019bf90 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f5436acb640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f542800eb10 tx=0x7f542800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f54277fe640 1 -- 172.21.15.45:0/1877907748 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428004b20 con 0x7f543019bf90 2024-03-12T05:23:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f54277fe640 1 -- 172.21.15.45:0/1877907748 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5428004c80 con 0x7f543019bf90 2024-03-12T05:23:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5420009490 con 0x7f543019bf90 2024-03-12T05:23:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f54277fe640 1 -- 172.21.15.45:0/1877907748 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5428010690 con 0x7f543019bf90 2024-03-12T05:23:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.238+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54301a4520 con 0x7f543019bf90 2024-03-12T05:23:01.235 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.241+0000 7f54277fe640 1 -- 172.21.15.45:0/1877907748 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5428005020 con 0x7f543019bf90 2024-03-12T05:23:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.242+0000 7f54277fe640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 0x7f541c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.236 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.242+0000 7f54277fe640 1 -- 172.21.15.45:0/1877907748 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5428014070 con 0x7f543019bf90 2024-03-12T05:23:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.242+0000 7f54362ca640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 0x7f541c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 Processor -- start 2024-03-12T05:23:01.237 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 -- start start 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 0x5592abec6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec6bd0 0x5592abecf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abe35280 con 0x5592abec62b0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abe327b0 con 0x5592abecf7b0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f543b1d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abe2df90 con 0x5592abec6bd0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f54362ca640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 0x7f541c0a40e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f542c009530 tx=0x7f542c009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f5436acb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f5436acb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51094/0 (socket says 172.21.15.45:51094) 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f5436acb640 1 -- 172.21.15.45:0/2112604616 learned_addr learned my addr 172.21.15.45:0/2112604616 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f5435ac9640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec6bd0 0x5592abecf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.243+0000 7f54362ca640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 0x5592abec6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 -- 172.21.15.45:0/2112604616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec6bd0 msgr2=0x5592abecf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec6bd0 0x5592abecf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 -- 172.21.15.45:0/2112604616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 msgr2=0x5592abec6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 0x5592abec6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 -- 172.21.15.45:0/2112604616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592abdec570 con 0x5592abecf7b0 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f54362ca640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 0x5592abec6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5436acb640 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5428097290 tx=0x7f542800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23cb67df6f2b2f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5424bfe640 1 -- 172.21.15.45:0/2112604616 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54280a3060 con 0x5592abecf7b0 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5424bfe640 1 -- 172.21.15.45:0/2112604616 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5428098bc0 con 0x5592abecf7b0 2024-03-12T05:23:01.240 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.244+0000 7f5424bfe640 1 -- 172.21.15.45:0/2112604616 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54280a2e30 con 0x5592abecf7b0 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 -- 172.21.15.45:0/2112604616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 msgr2=0x5592abed1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5428097290 tx=0x7f542800cf70 comp rx=0 tx=0).stop 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 -- 172.21.15.45:0/2112604616 shutdown_connections 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abecf7b0 0x5592abed1ba0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec6bd0 0x5592abecf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 --2- 172.21.15.45:0/2112604616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abec62b0 0x5592abec6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 -- 172.21.15.45:0/2112604616 >> 172.21.15.45:0/2112604616 conn(0x5592abec0990 msgr2=0x5592abec0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 -- 172.21.15.45:0/2112604616 shutdown_connections 2024-03-12T05:23:01.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.245+0000 7f543b1d7740 1 -- 172.21.15.45:0/2112604616 wait complete. 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.246+0000 7f543b1d7740 1 -- start start 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.246+0000 7f543b1d7740 1 Processor -- start 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abecf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abec8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abb01430 con 0x5592abecf7b0 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f54362ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abecf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f54362ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abecf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50794/0 (socket says 172.21.15.45:50794) 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f54362ca640 1 -- 172.21.15.45:0/593947777 learned_addr learned my addr 172.21.15.45:0/593947777 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51096/0 (socket says 172.21.15.45:51096) 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592ababde00 con 0x5592abec6bd0 2024-03-12T05:23:01.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abdb2fd0 con 0x5592abec62b0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5436acb640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abec8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f543b1d7740 10 client.0 _ll_register_callbacks cb 0x5592abe08fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 -- 172.21.15.45:0/593947777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 msgr2=0x5592abecf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abecf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 -- 172.21.15.45:0/593947777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 msgr2=0x5592abec8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abec8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.247+0000 7f5435ac9640 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54301a4520 con 0x5592abec6bd0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5435ac9640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f542002c0b0 tx=0x7f5420004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1a23d2bfe414f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420034850 con 0x5592abec6bd0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f542002c510 con 0x5592abec6bd0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592abb2d7e0 con 0x5592abec6bd0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5420004500 con 0x5592abec6bd0 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f543b1d7740 10 client.27965 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:01.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592abe2c610 con 0x5592abec6bd0 2024-03-12T05:23:01.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.248+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5592abb2d7e0 con 0x5592abec6bd0 2024-03-12T05:23:01.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5420045020 con 0x5592abec6bd0 2024-03-12T05:23:01.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 10 client.27965 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:01.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 10 client.27965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:01.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 20 client.27965 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:01.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abee8140 0x5592abeea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 4 client.27965 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592abb6a720 con 0x5592abee8140 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.249+0000 7f54362ca640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abee8140 0x5592abeea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f5419ffb640 10 client.27965 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f54362ca640 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abee8140 0x5592abeea560 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c09d851048b4f02 server_cookie=61bf63613131587 in_seq=0 out_seq=0 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f5419ffb640 1 -- 172.21.15.45:0/593947777 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5592abb6a720 con 0x5592abee8140 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f5419ffb640 10 client.27965 handle_command_reply: tid=0 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 client.27965 shutdown 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 msgr2=0x5592abec85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f542002c0b0 tx=0x7f5420004250 comp rx=0 tx=0).stop 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 shutdown_connections 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abee8140 0x5592abeea560 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abec8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abec85c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 --2- 172.21.15.45:0/593947777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abecf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 >> 172.21.15.45:0/593947777 conn(0x5592abec0990 msgr2=0x5592abec2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.250+0000 7f5419ffb640 0 client.27965 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:01.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.251+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 shutdown_connections 2024-03-12T05:23:01.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.251+0000 7f543b1d7740 1 -- 172.21.15.45:0/593947777 wait complete. 2024-03-12T05:23:01.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.251+0000 7f543b1d7740 20 client.27965 trim_cache size 0 max 16384 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.275+0000 7f543b1d7740 1 Processor -- start 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.275+0000 7f543b1d7740 1 -- start start 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.275+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.275+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abecf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.275+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abed4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f543b1d7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abec5cf0 con 0x5592abecf7b0 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f543b1d7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abec1560 con 0x5592abec6bd0 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f543b1d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abebae20 con 0x5592abec62b0 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50800/0 (socket says 172.21.15.45:50800) 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 -- 172.21.15.45:0/204977761 learned_addr learned my addr 172.21.15.45:0/204977761 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f5435ac9640 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abecf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f5436acb640 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abed4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 -- 172.21.15.45:0/204977761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 msgr2=0x5592abecf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abecf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 -- 172.21.15.45:0/204977761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 msgr2=0x5592abed4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abed4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.276+0000 7f54362ca640 1 -- 172.21.15.45:0/204977761 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592abdc8ac0 con 0x5592abec62b0 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.277+0000 7f54362ca640 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f542c01c450 tx=0x7f542c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fe89798ff0475 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.277+0000 7f541bfff640 1 -- 172.21.15.45:0/204977761 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c01dea0 con 0x5592abec62b0 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.277+0000 7f541bfff640 1 -- 172.21.15.45:0/204977761 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f542c0278d0 con 0x5592abec62b0 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.277+0000 7f541bfff640 1 -- 172.21.15.45:0/204977761 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c022040 con 0x5592abec62b0 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 -- 172.21.15.45:0/204977761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 msgr2=0x5592abeced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f542c01c450 tx=0x7f542c017000 comp rx=0 tx=0).stop 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 -- 172.21.15.45:0/204977761 shutdown_connections 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abed4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abecf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 --2- 172.21.15.45:0/204977761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abeced00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 -- 172.21.15.45:0/204977761 >> 172.21.15.45:0/204977761 conn(0x5592abec0990 msgr2=0x5592abed2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 -- 172.21.15.45:0/204977761 shutdown_connections 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.278+0000 7f543b1d7740 1 -- 172.21.15.45:0/204977761 wait complete. 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 -- start start 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 Processor -- start 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abee45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abee1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abee23d0 con 0x5592abecf7b0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abee2540 con 0x5592abec6bd0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592abee2700 con 0x5592abec62b0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f543b1d7740 10 client.0 _ll_register_callbacks cb 0x7f5428096b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f5435ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f54362ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abee45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f5435ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51126/0 (socket says 172.21.15.45:51126) 2024-03-12T05:23:01.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f5435ac9640 1 -- 172.21.15.45:0/344795 learned_addr learned my addr 172.21.15.45:0/344795 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.279+0000 7f5436acb640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abee1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 -- 172.21.15.45:0/344795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 msgr2=0x5592abee45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abee45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 -- 172.21.15.45:0/344795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 msgr2=0x5592abee1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abee1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 -- 172.21.15.45:0/344795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c017070 con 0x5592abec6bd0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5435ac9640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5592abec9b90 tx=0x7f5420003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be352eabb2445204 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5419ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420005240 con 0x5592abec6bd0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54200053a0 con 0x5592abec6bd0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5428098bc0 con 0x5592abec6bd0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.280+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5420004500 con 0x5592abec6bd0 2024-03-12T05:23:01.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 10 client.27971 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592abdee6a0 con 0x5592abec6bd0 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5420064020 con 0x5592abec6bd0 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5420056030 con 0x5592abec6bd0 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 10 client.27971 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 10 client.27971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 20 client.27971 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abf407c0 0x5592abf42be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 4 client.27971 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5592abbb9890 con 0x5592abf407c0 2024-03-12T05:23:01.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.281+0000 7f54362ca640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abf407c0 0x5592abf42be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f5419ffb640 10 client.27971 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f54362ca640 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abf407c0 0x5592abf42be0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63ab2877e5308785 server_cookie=6cb38f11ce1f48f3 in_seq=0 out_seq=0 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f5419ffb640 1 -- 172.21.15.45:0/344795 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5592abbb9890 con 0x5592abf407c0 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f5419ffb640 10 client.27971 handle_command_reply: tid=0 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f543b1d7740 1 client.27971 shutdown 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 msgr2=0x5592abee1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.282+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5592abec9b90 tx=0x7f5420003820 comp rx=0 tx=0).stop 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 shutdown_connections 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592abf407c0 0x5592abf42be0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592abecf7b0 0x5592abee1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f5419ffb640 0 client.27971 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592abec6bd0 0x5592abee1950 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 --2- 172.21.15.45:0/344795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592abec62b0 0x5592abee45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 >> 172.21.15.45:0/344795 conn(0x5592abec0990 msgr2=0x5592abec1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 shutdown_connections 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 1 -- 172.21.15.45:0/344795 wait complete. 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.283+0000 7f543b1d7740 20 client.27971 trim_cache size 0 max 16384 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:23:01.282 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:23:01.283 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 msgr2=0x7f541c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 0x7f541c0a40e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f542c009530 tx=0x7f542c009260 comp rx=0 tx=0).stop 2024-03-12T05:23:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 msgr2=0x7f543019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f542800eb10 tx=0x7f542800c2d0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 shutdown_connections 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f541c0a1c20 0x7f541c0a40e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f543019bf90 0x7f543019c440 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5430106460 0x7f5430197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 --2- 172.21.15.45:0/1877907748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5430101390 0x7f54301976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 >> 172.21.15.45:0/1877907748 conn(0x7f54300fcc50 msgr2=0x7f54300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 shutdown_connections 2024-03-12T05:23:01.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.284+0000 7f543852e640 1 -- 172.21.15.45:0/1877907748 wait complete. 2024-03-12T05:23:01.294 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:23:01.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:23:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.477+0000 7f3c9dac0640 1 -- 172.21.15.45:0/2681279770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810d9e0 msgr2=0x7f3c9810de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.477+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/2681279770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810d9e0 0x7f3c9810de60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3c8801a910 tx=0x7f3c8803cc80 comp rx=0 tx=0).stop 2024-03-12T05:23:01.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 -- 172.21.15.45:0/2681279770 shutdown_connections 2024-03-12T05:23:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/2681279770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9811a960 0x7f3c9811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/2681279770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810d9e0 0x7f3c9810de60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/2681279770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9810f9b0 0x7f3c9810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 -- 172.21.15.45:0/2681279770 >> 172.21.15.45:0/2681279770 conn(0x7f3c9806c270 msgr2=0x7f3c9806c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.478+0000 7f3c9dac0640 1 -- 172.21.15.45:0/2681279770 shutdown_connections 2024-03-12T05:23:01.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.480+0000 7f3c9dac0640 1 -- 172.21.15.45:0/2681279770 wait complete. 2024-03-12T05:23:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.481+0000 7f3c9dac0640 1 Processor -- start 2024-03-12T05:23:01.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 -- start start 2024-03-12T05:23:01.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 0x7f3c981160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810f9b0 0x7f3c981165e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98119f50 con 0x7f3c9810d9e0 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9811a0c0 con 0x7f3c9811a960 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.482+0000 7f3c9dac0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9811a280 con 0x7f3c9810f9b0 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51132/0 (socket says 172.21.15.45:51132) 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 -- 172.21.15.45:0/3089393000 learned_addr learned my addr 172.21.15.45:0/3089393000 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c977fe640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 0x7f3c981160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 -- 172.21.15.45:0/3089393000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810f9b0 msgr2=0x7f3c981165e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810f9b0 0x7f3c981165e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 -- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 msgr2=0x7f3c981160a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 0x7f3c981160a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c97fff640 1 -- 172.21.15.45:0/3089393000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8801a570 con 0x7f3c9811a960 2024-03-12T05:23:01.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.483+0000 7f3c977fe640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 0x7f3c981160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:01.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.484+0000 7f3c97fff640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00b3f0 tx=0x7f3c8c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.484+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c011830 con 0x7f3c9811a960 2024-03-12T05:23:01.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.484+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9811a560 con 0x7f3c9811a960 2024-03-12T05:23:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.484+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c8c011e40 con 0x7f3c9811a960 2024-03-12T05:23:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.484+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00f2e0 con 0x7f3c9811a960 2024-03-12T05:23:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.485+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981aaa30 con 0x7f3c9811a960 2024-03-12T05:23:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.487+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c98067f70 con 0x7f3c9811a960 2024-03-12T05:23:01.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.491+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3c8c011990 con 0x7f3c9811a960 2024-03-12T05:23:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.491+0000 7f3c94ff9640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 0x7f3c800a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.492+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3c8c0cd2d0 con 0x7f3c9811a960 2024-03-12T05:23:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.492+0000 7f3c977fe640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 0x7f3c800a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:01.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.493+0000 7f3c977fe640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 0x7f3c800a4270 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3c90005fd0 tx=0x7f3c90009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:01.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.499+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c8c096220 con 0x7f3c9811a960 2024-03-12T05:23:01.948 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.952+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9810aa90 con 0x7f3c9811a960 2024-03-12T05:23:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:23:01.950 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:23:01.950 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:23:01.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.953+0000 7f3c94ff9640 1 -- 172.21.15.45:0/3089393000 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f3c8c0998e0 con 0x7f3c9811a960 2024-03-12T05:23:01.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.957+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 msgr2=0x7f3c800a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.957+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 0x7f3c800a4270 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3c90005fd0 tx=0x7f3c90009500 comp rx=0 tx=0).stop 2024-03-12T05:23:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.957+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 msgr2=0x7f3c98119950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:01.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.957+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00b3f0 tx=0x7f3c8c00b8b0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 shutdown_connections 2024-03-12T05:23:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c800a1db0 0x7f3c800a4270 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9811a960 0x7f3c98119950 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c9810f9b0 0x7f3c981165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 --2- 172.21.15.45:0/3089393000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c9810d9e0 0x7f3c981160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:01.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.959+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 >> 172.21.15.45:0/3089393000 conn(0x7f3c9806c270 msgr2=0x7f3c9811c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:01.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.962+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 shutdown_connections 2024-03-12T05:23:01.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:01.962+0000 7f3c9dac0640 1 -- 172.21.15.45:0/3089393000 wait complete. 2024-03-12T05:23:01.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:01 smithi130 ceph-mon[35398]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 14 MiB/s wr, 1.45k op/s 2024-03-12T05:23:01.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:01 smithi045 ceph-mon[32174]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 14 MiB/s wr, 1.45k op/s 2024-03-12T05:23:01.981 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:01.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:23:01.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:01 smithi116 ceph-mon[35486]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 14 MiB/s wr, 1.45k op/s 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 -- 172.21.15.45:0/3129868122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96280957e0 msgr2=0x7f9628004730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 --2- 172.21.15.45:0/3129868122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96280957e0 0x7f9628004730 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9630066e50 tx=0x7f96300a8070 comp rx=0 tx=0).stop 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 -- 172.21.15.45:0/3129868122 shutdown_connections 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 --2- 172.21.15.45:0/3129868122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96280957e0 0x7f9628004730 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 --2- 172.21.15.45:0/3129868122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f96280952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 --2- 172.21.15.45:0/3129868122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9628099bb0 0x7f9628099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 -- 172.21.15.45:0/3129868122 >> 172.21.15.45:0/3129868122 conn(0x7f962800a730 msgr2=0x7f962800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 -- 172.21.15.45:0/3129868122 shutdown_connections 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.159+0000 7f9636b54640 1 -- 172.21.15.45:0/3129868122 wait complete. 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.160+0000 7f9636b54640 1 Processor -- start 2024-03-12T05:23:02.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.160+0000 7f9636b54640 1 -- start start 2024-03-12T05:23:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9628099bb0 0x7f962812e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9635351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9635b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9635351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50838/0 (socket says 172.21.15.45:50838) 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9635b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51146/0 (socket says 172.21.15.45:51146) 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9635351640 1 -- 172.21.15.45:0/945139819 learned_addr learned my addr 172.21.15.45:0/945139819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636353640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9628099bb0 0x7f962812e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962812ed80 con 0x7f96280957e0 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962812eef0 con 0x7f9628094e40 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.161+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962812f0b0 con 0x7f9628099bb0 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 -- 172.21.15.45:0/945139819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9628099bb0 msgr2=0x7f962812e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9628099bb0 0x7f962812e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 -- 172.21.15.45:0/945139819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 msgr2=0x7f9628129ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963004f750 con 0x7f96280957e0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635b52640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9635351640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f962c00d950 tx=0x7f962c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9626ffd640 1 -- 172.21.15.45:0/945139819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c009c30 con 0x7f96280957e0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962812f390 con 0x7f96280957e0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9626ffd640 1 -- 172.21.15.45:0/945139819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f962c009d90 con 0x7f96280957e0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.162+0000 7f9626ffd640 1 -- 172.21.15.45:0/945139819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00f6b0 con 0x7f96280957e0 2024-03-12T05:23:02.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.163+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9628136c10 con 0x7f96280957e0 2024-03-12T05:23:02.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.166+0000 7f9626ffd640 1 -- 172.21.15.45:0/945139819 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f962c010aa0 con 0x7f96280957e0 2024-03-12T05:23:02.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.167+0000 7f9626ffd640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 0x7f961c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.167+0000 7f9626ffd640 1 -- 172.21.15.45:0/945139819 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f962c0cd440 con 0x7f96280957e0 2024-03-12T05:23:02.162 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.168+0000 7f963a985740 1 Processor -- start 2024-03-12T05:23:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.168+0000 7f9635b52640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 0x7f961c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 -- start start 2024-03-12T05:23:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.163 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27ef420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27ef960 0x563cb27f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb27a1960 con 0x563cb27ef960 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb2755310 con 0x563cb27e6460 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f963a985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb2752840 con 0x563cb27e6d80 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f9635b52640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 0x7f961c0a4060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9620009fc0 tx=0x7f9620005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f9635b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f9635b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51152/0 (socket says 172.21.15.45:51152) 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f9635b52640 1 -- 172.21.15.45:0/585002761 learned_addr learned my addr 172.21.15.45:0/585002761 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.169+0000 7f9636353640 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27ef960 0x563cb27f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 -- 172.21.15.45:0/585002761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 msgr2=0x563cb27ef420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27ef420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 -- 172.21.15.45:0/585002761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27ef960 msgr2=0x563cb27f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27ef960 0x563cb27f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 -- 172.21.15.45:0/585002761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cb2710060 con 0x563cb27e6460 2024-03-12T05:23:02.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f9635b52640 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f96200042c0 tx=0x7f9620014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b5b3bd6980494d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.170+0000 7f961bfff640 1 -- 172.21.15.45:0/585002761 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000f480 con 0x563cb27e6460 2024-03-12T05:23:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.171+0000 7f961bfff640 1 -- 172.21.15.45:0/585002761 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9620021070 con 0x563cb27e6460 2024-03-12T05:23:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.171+0000 7f961bfff640 1 -- 172.21.15.45:0/585002761 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962001d980 con 0x563cb27e6460 2024-03-12T05:23:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.171+0000 7f963a985740 1 -- 172.21.15.45:0/585002761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 msgr2=0x563cb27e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f96200042c0 tx=0x7f9620014000 comp rx=0 tx=0).stop 2024-03-12T05:23:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 -- 172.21.15.45:0/585002761 shutdown_connections 2024-03-12T05:23:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27ef960 0x563cb27f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 --2- 172.21.15.45:0/585002761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27e6460 0x563cb27e6840 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.172+0000 7f963a985740 1 -- 172.21.15.45:0/585002761 >> 172.21.15.45:0/585002761 conn(0x563cb27e0b90 msgr2=0x563cb27e0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.173+0000 7f963a985740 1 -- 172.21.15.45:0/585002761 shutdown_connections 2024-03-12T05:23:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.173+0000 7f963a985740 1 -- 172.21.15.45:0/585002761 wait complete. 2024-03-12T05:23:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.173+0000 7f963a985740 1 -- start start 2024-03-12T05:23:02.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.173+0000 7f963a985740 1 Processor -- start 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27ed440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb274e020 con 0x563cb27e6d80 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb2421430 con 0x563cb27ef960 2024-03-12T05:23:02.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb23dde00 con 0x563cb27e6460 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f963a985740 10 client.0 _ll_register_callbacks cb 0x563cb26f5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f9635b52640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f9635b52640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50864/0 (socket says 172.21.15.45:50864) 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f9635b52640 1 -- 172.21.15.45:0/2527066600 learned_addr learned my addr 172.21.15.45:0/2527066600 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f9636353640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27ed440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.174+0000 7f9635351640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 -- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 msgr2=0x563cb27ec9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 -- 172.21.15.45:0/2527066600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 msgr2=0x563cb27ecf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27ecf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 -- 172.21.15.45:0/2527066600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9620017050 con 0x563cb27ef960 2024-03-12T05:23:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9635b52640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:02.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9635351640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27ecf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:02.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f9636353640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27ed440 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9630050150 tx=0x7f96300668e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc94c0b6219fe6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f96197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:02.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300b6070 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.175+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cb244d7e0 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f96300b0b40 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96300b0d00 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 10 client.27989 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563cb274c6a0 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563cb244d7e0 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f96300b1700 con 0x563cb27ef960 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 10 client.27989 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 10 client.27989 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 20 client.27989 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb27fff30 0x563cb2802350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 4 client.27989 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.176+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cb248a720 con 0x563cb27fff30 2024-03-12T05:23:02.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.177+0000 7f9635b52640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb27fff30 0x563cb2802350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.177+0000 7f96197fa640 10 client.27989 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:02.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.178+0000 7f9635b52640 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb27fff30 0x563cb2802350 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=30082f5a12997295 server_cookie=efe9b0b1eefc336 in_seq=0 out_seq=0 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.178+0000 7f96197fa640 1 -- 172.21.15.45:0/2527066600 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563cb248a720 con 0x563cb27fff30 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.178+0000 7f96197fa640 10 client.27989 handle_command_reply: tid=0 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.178+0000 7f963a985740 1 client.27989 shutdown 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 msgr2=0x563cb27ed440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27ed440 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9630050150 tx=0x7f96300668e0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 shutdown_connections 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb27fff30 0x563cb2802350 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27ecf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27ed440 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f96197fa640 0 client.27989 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 --2- 172.21.15.45:0/2527066600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ec9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 >> 172.21.15.45:0/2527066600 conn(0x563cb27e0b90 msgr2=0x563cb27e28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.179+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 shutdown_connections 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.180+0000 7f963a985740 1 -- 172.21.15.45:0/2527066600 wait complete. 2024-03-12T05:23:02.176 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.180+0000 7f963a985740 20 client.27989 trim_cache size 0 max 16384 2024-03-12T05:23:02.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 Processor -- start 2024-03-12T05:23:02.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 -- start start 2024-03-12T05:23:02.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27eb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.202+0000 7f963a985740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb26d3130 con 0x563cb27e6d80 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f963a985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb27e85d0 con 0x563cb27ef960 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f963a985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb27e5550 con 0x563cb27e6460 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9636353640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50880/0 (socket says 172.21.15.45:50880) 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 -- 172.21.15.45:0/503565994 learned_addr learned my addr 172.21.15.45:0/503565994 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635351640 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 -- 172.21.15.45:0/503565994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 msgr2=0x563cb27eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 -- 172.21.15.45:0/503565994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 msgr2=0x563cb27eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.203+0000 7f9635b52640 1 -- 172.21.15.45:0/503565994 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cb26e8c20 con 0x563cb27e6460 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.204+0000 7f9635b52640 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9620003e50 tx=0x7f96200096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=595ed9ec5ef06622 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.204+0000 7f961b7fe640 1 -- 172.21.15.45:0/503565994 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962001d3f0 con 0x563cb27e6460 2024-03-12T05:23:02.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.204+0000 7f961b7fe640 1 -- 172.21.15.45:0/503565994 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96200043b0 con 0x563cb27e6460 2024-03-12T05:23:02.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.204+0000 7f961b7fe640 1 -- 172.21.15.45:0/503565994 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96200334f0 con 0x563cb27e6460 2024-03-12T05:23:02.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- 172.21.15.45:0/503565994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 msgr2=0x563cb27ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9620003e50 tx=0x7f96200096c0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- 172.21.15.45:0/503565994 shutdown_connections 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27e6d80 0x563cb27eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 --2- 172.21.15.45:0/503565994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6460 0x563cb27ef0f0 secure :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9620003e50 tx=0x7f96200096c0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- 172.21.15.45:0/503565994 >> 172.21.15.45:0/503565994 conn(0x563cb27e0b90 msgr2=0x563cb2806cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- 172.21.15.45:0/503565994 shutdown_connections 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- 172.21.15.45:0/503565994 wait complete. 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.205+0000 7f963a985740 1 -- start start 2024-03-12T05:23:02.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 Processor -- start 2024-03-12T05:23:02.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27fff30 0x563cb2856440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb27f6190 con 0x563cb27fff30 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb27f6300 con 0x563cb27ef960 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cb2856980 con 0x563cb27e6d80 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f963a985740 10 client.0 _ll_register_callbacks cb 0x7f9620009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f9636353640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27fff30 0x563cb2856440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f9635351640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f9635351640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51180/0 (socket says 172.21.15.45:51180) 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f9635351640 1 -- 172.21.15.45:0/4028714128 learned_addr learned my addr 172.21.15.45:0/4028714128 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.206+0000 7f9635b52640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 -- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 msgr2=0x563cb27f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 -- 172.21.15.45:0/4028714128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27fff30 msgr2=0x563cb2856440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27fff30 0x563cb2856440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 -- 172.21.15.45:0/4028714128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962002a040 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f9635351640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f962c09a080 tx=0x7f962c09a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea7507e9a91c700 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f96197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0b0020 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.207+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cb27ec7f0 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f962c099740 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f962c0caf60 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f963a985740 10 client.27995 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563cb270e8c0 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563cb27ec7f0 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.208+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f962c0cc040 con 0x563cb27ef960 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 10 client.27995 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 10 client.27995 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:23:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 20 client.27995 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb2858240 0x563cb285a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 4 client.27995 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f9620021070 con 0x563cb2858240 2024-03-12T05:23:02.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.209+0000 7f9635b52640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb2858240 0x563cb285a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f96197fa640 10 client.27995 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f9635b52640 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb2858240 0x563cb285a660 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dcd7e42cd02e6f07 server_cookie=b68a1f43f1ef9e16 in_seq=0 out_seq=0 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f96197fa640 1 -- 172.21.15.45:0/4028714128 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f9620021070 con 0x563cb2858240 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f96197fa640 10 client.27995 handle_command_reply: tid=0 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 client.27995 shutdown 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 msgr2=0x563cb27f5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f962c09a080 tx=0x7f962c09a230 comp rx=0 tx=0).stop 2024-03-12T05:23:02.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 shutdown_connections 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x563cb2858240 0x563cb285a660 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563cb27fff30 0x563cb2856440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563cb27ef960 0x563cb27f5a30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f96197fa640 0 client.27995 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 --2- 172.21.15.45:0/4028714128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563cb27e6d80 0x563cb27f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 >> 172.21.15.45:0/4028714128 conn(0x563cb27e0b90 msgr2=0x563cb27eb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 shutdown_connections 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.210+0000 7f963a985740 1 -- 172.21.15.45:0/4028714128 wait complete. 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.211+0000 7f963a985740 20 client.27995 trim_cache size 0 max 16384 2024-03-12T05:23:02.208 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.211+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 msgr2=0x7f961c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.211+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 0x7f961c0a4060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9620009fc0 tx=0x7f9620005f90 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.211+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 msgr2=0x7f962812a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.211+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f962c00d950 tx=0x7f962c00de10 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 shutdown_connections 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f961c0a1ba0 0x7f961c0a4060 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96280957e0 0x7f962812a400 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9628099bb0 0x7f962812e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 --2- 172.21.15.45:0/945139819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9628094e40 0x7f9628129ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 >> 172.21.15.45:0/945139819 conn(0x7f962800a730 msgr2=0x7f9628093ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 shutdown_connections 2024-03-12T05:23:02.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.212+0000 7f9636b54640 1 -- 172.21.15.45:0/945139819 wait complete. 2024-03-12T05:23:02.225 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:02.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:23:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.411+0000 7f6f06b6c640 1 -- 172.21.15.45:0/2822515204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 msgr2=0x7f6f00110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.406 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.411+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/2822515204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f00110230 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009be0 tx=0x7f6ef802bf10 comp rx=0 tx=0).stop 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.412+0000 7f6f06b6c640 1 -- 172.21.15.45:0/2822515204 shutdown_connections 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.412+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/2822515204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f00110230 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.412+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/2822515204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00103f40 0x7f6f00107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.412+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/2822515204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 0x7f6f00103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.412+0000 7f6f06b6c640 1 -- 172.21.15.45:0/2822515204 >> 172.21.15.45:0/2822515204 conn(0x7f6f000fedf0 msgr2=0x7f6f00101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.413+0000 7f6f06b6c640 1 -- 172.21.15.45:0/2822515204 shutdown_connections 2024-03-12T05:23:02.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.413+0000 7f6f06b6c640 1 -- 172.21.15.45:0/2822515204 wait complete. 2024-03-12T05:23:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 Processor -- start 2024-03-12T05:23:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 -- start start 2024-03-12T05:23:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 0x7f6f0019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00199610 0x7f6f00199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0019a000 con 0x7f6f00103590 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0019a170 con 0x7f6f00199610 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f06b6c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0019a330 con 0x7f6f00108660 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6efffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6efffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50900/0 (socket says 172.21.15.45:50900) 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6f04908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 0x7f6f0019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.414+0000 7f6efffff640 1 -- 172.21.15.45:0/3928955943 learned_addr learned my addr 172.21.15.45:0/3928955943 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6f05109640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00199610 0x7f6f00199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 -- 172.21.15.45:0/3928955943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00199610 msgr2=0x7f6f00199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00199610 0x7f6f00199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 -- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 msgr2=0x7f6f0019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 0x7f6f0019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 -- 172.21.15.45:0/3928955943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8009840 con 0x7f6f00108660 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.415+0000 7f6efffff640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c970 tx=0x7f6ef400ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.416+0000 7f6efdffb640 1 -- 172.21.15.45:0/3928955943 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4004440 con 0x7f6f00108660 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.416+0000 7f6efdffb640 1 -- 172.21.15.45:0/3928955943 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ef4007c90 con 0x7f6f00108660 2024-03-12T05:23:02.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.416+0000 7f6efdffb640 1 -- 172.21.15.45:0/3928955943 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef400f450 con 0x7f6f00108660 2024-03-12T05:23:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.416+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f001a6320 con 0x7f6f00108660 2024-03-12T05:23:02.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.416+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f001a6830 con 0x7f6f00108660 2024-03-12T05:23:02.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.420+0000 7f6f0981e740 1 Processor -- start 2024-03-12T05:23:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.420+0000 7f6f0981e740 1 -- start start 2024-03-12T05:23:02.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59527e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 0x55de59530a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59533350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de594da960 con 0x55de59527a60 2024-03-12T05:23:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de5948e310 con 0x55de59530f60 2024-03-12T05:23:02.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f0981e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de5948b840 con 0x55de59528380 2024-03-12T05:23:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f05109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59527e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f05109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59527e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50904/0 (socket says 172.21.15.45:50904) 2024-03-12T05:23:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f05109640 1 -- 172.21.15.45:0/4200499502 learned_addr learned my addr 172.21.15.45:0/4200499502 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.421+0000 7f6f04908640 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 0x55de59530a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 -- 172.21.15.45:0/4200499502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 msgr2=0x55de59533350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59533350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 -- 172.21.15.45:0/4200499502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 msgr2=0x55de59527e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59527e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 -- 172.21.15.45:0/4200499502 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de59449060 con 0x55de59528380 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6f04908640 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 0x55de59530a20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0004600 tx=0x7f6ef000bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819883a20174434e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6ee6ffd640 1 -- 172.21.15.45:0/4200499502 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0003da0 con 0x55de59528380 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6ee6ffd640 1 -- 172.21.15.45:0/4200499502 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ef000be30 con 0x55de59528380 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6ee6ffd640 1 -- 172.21.15.45:0/4200499502 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011660 con 0x55de59528380 2024-03-12T05:23:02.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.422+0000 7f6efdffb640 1 -- 172.21.15.45:0/3928955943 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6ef40035f0 con 0x7f6f00108660 2024-03-12T05:23:02.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.423+0000 7f6efdffb640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 0x7f6ee80a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.423+0000 7f6f05109640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 0x7f6ee80a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6efdffb640 1 -- 172.21.15.45:0/3928955943 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6ef40cd100 con 0x7f6f00108660 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- 172.21.15.45:0/4200499502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 msgr2=0x55de59530a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 0x55de59530a20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0004600 tx=0x7f6ef000bb20 comp rx=0 tx=0).stop 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- 172.21.15.45:0/4200499502 shutdown_connections 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59533350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59528380 0x55de59530a20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 --2- 172.21.15.45:0/4200499502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59527e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- 172.21.15.45:0/4200499502 >> 172.21.15.45:0/4200499502 conn(0x55de59522190 msgr2=0x55de59522570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- 172.21.15.45:0/4200499502 shutdown_connections 2024-03-12T05:23:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- 172.21.15.45:0/4200499502 wait complete. 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.424+0000 7f6f0981e740 1 -- start start 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f0981e740 1 Processor -- start 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f05109640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 0x7f6ee80a41f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8036040 tx=0x7f6ef802cb00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de5952ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.425+0000 7f6f04908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51208/0 (socket says 172.21.15.45:51208) 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f04908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50910/0 (socket says 172.21.15.45:50910) 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 -- 172.21.15.45:0/1071812236 learned_addr learned my addr 172.21.15.45:0/1071812236 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f0981e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de59487020 con 0x55de59527a60 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de5915a430 con 0x55de59528380 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de59116e00 con 0x55de59530f60 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f0981e740 10 client.0 _ll_register_callbacks cb 0x55de5942ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f05109640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de5952ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 -- 172.21.15.45:0/1071812236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 msgr2=0x55de5952ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de5952ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 -- 172.21.15.45:0/1071812236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 msgr2=0x55de5952dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6efffff640 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef0004230 con 0x55de59528380 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.426+0000 7f6f04908640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6efffff640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a2090 tx=0x7f6ef40056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4281c2d72037a1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6edffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef409df70 con 0x55de59528380 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6ef40a25c0 con 0x55de59528380 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de591867e0 con 0x55de59528380 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ef40b2b50 con 0x55de59528380 2024-03-12T05:23:02.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.427+0000 7f6f0981e740 10 client.28001 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55de594856a0 con 0x55de59528380 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55de591867e0 con 0x55de59528380 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6ef4095b70 con 0x55de59528380 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 10 client.28001 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 10 client.28001 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 20 client.28001 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59541180 0x55de595435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 4 client.28001 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.428+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de591c3720 con 0x55de59541180 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.429+0000 7f6f04908640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59541180 0x55de595435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.430+0000 7f6f04908640 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59541180 0x55de595435c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2b8315bfa805f7c9 server_cookie=e6fb4ced5d0814c in_seq=0 out_seq=0 2024-03-12T05:23:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.430+0000 7f6edffff640 10 client.28001 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:02.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6edffff640 1 -- 172.21.15.45:0/1071812236 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55de591c3720 con 0x55de59541180 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6edffff640 10 client.28001 handle_command_reply: tid=0 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 client.28001 shutdown 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 msgr2=0x55de5952e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a2090 tx=0x7f6ef40056d0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 shutdown_connections 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59541180 0x55de595435c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6edffff640 0 client.28001 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de5952ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59528380 0x55de5952e4e0 secure :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a2090 tx=0x7f6ef40056d0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1071812236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de5952dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.431+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 >> 172.21.15.45:0/1071812236 conn(0x55de59522190 msgr2=0x55de59523650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.432+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 shutdown_connections 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.432+0000 7f6f0981e740 1 -- 172.21.15.45:0/1071812236 wait complete. 2024-03-12T05:23:02.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.432+0000 7f6f0981e740 20 client.28001 trim_cache size 0 max 16384 2024-03-12T05:23:02.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.464+0000 7f6f0981e740 1 Processor -- start 2024-03-12T05:23:02.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.464+0000 7f6f0981e740 1 -- start start 2024-03-12T05:23:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59527a60 0x55de5952e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59537e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de5940c130 con 0x55de59541180 2024-03-12T05:23:02.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f05109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f05109640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50918/0 (socket says 172.21.15.45:50918) 2024-03-12T05:23:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f05109640 1 -- 172.21.15.45:0/3492069843 learned_addr learned my addr 172.21.15.45:0/3492069843 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6efffff640 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59537e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de5953de90 con 0x55de59530f60 2024-03-12T05:23:02.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.465+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de59540f80 con 0x55de59527a60 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.466+0000 7f6f05109640 1 -- 172.21.15.45:0/3492069843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59527a60 msgr2=0x55de5952e840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.466+0000 7f6f05109640 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59527a60 0x55de5952e840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.466+0000 7f6f05109640 1 -- 172.21.15.45:0/3492069843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 msgr2=0x55de59537e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.466+0000 7f6f05109640 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59537e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.466+0000 7f6f05109640 1 -- 172.21.15.45:0/3492069843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de59421c20 con 0x55de59541180 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.467+0000 7f6f05109640 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8033930 tx=0x7f6ef8042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57248b96e3fffea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.467+0000 7f6ee67fc640 1 -- 172.21.15.45:0/3492069843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8051070 con 0x55de59541180 2024-03-12T05:23:02.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.467+0000 7f6ee67fc640 1 -- 172.21.15.45:0/3492069843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ef8044070 con 0x55de59541180 2024-03-12T05:23:02.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.467+0000 7f6ee67fc640 1 -- 172.21.15.45:0/3492069843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8034430 con 0x55de59541180 2024-03-12T05:23:02.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.468+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 msgr2=0x55de59538350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.468+0000 7f6f0981e740 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8033930 tx=0x7f6ef8042000 comp rx=0 tx=0).stop 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 shutdown_connections 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59541180 0x55de59538350 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59530f60 0x55de59537e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 --2- 172.21.15.45:0/3492069843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59527a60 0x55de5952e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 >> 172.21.15.45:0/3492069843 conn(0x55de59522190 msgr2=0x55de5952c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 shutdown_connections 2024-03-12T05:23:02.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 -- 172.21.15.45:0/3492069843 wait complete. 2024-03-12T05:23:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.469+0000 7f6f0981e740 1 -- start start 2024-03-12T05:23:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.470+0000 7f6f0981e740 1 Processor -- start 2024-03-12T05:23:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.470+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59538220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.470+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de595365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f0981e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6efffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de595365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6efffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de595365c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50938/0 (socket says 172.21.15.45:50938) 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f05109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f05109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51222/0 (socket says 172.21.15.45:51222) 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f04908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59538220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f04908640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59538220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50928/0 (socket says 172.21.15.45:50928) 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6efffff640 1 -- 172.21.15.45:0/1199019252 learned_addr learned my addr 172.21.15.45:0/1199019252 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de59537240 con 0x55de59527a60 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de595373b0 con 0x55de59541180 2024-03-12T05:23:02.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de595976f0 con 0x55de59530f60 2024-03-12T05:23:02.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.471+0000 7f6f0981e740 10 client.0 _ll_register_callbacks cb 0x7f6ef0004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 -- 172.21.15.45:0/1199019252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 msgr2=0x55de595365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de595365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 -- 172.21.15.45:0/1199019252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 msgr2=0x55de59538220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59538220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f00110a40 con 0x55de59541180 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6f05109640 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8057040 tx=0x7f6ef8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c81802b4205641c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.472+0000 7f6edffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8051050 con 0x55de59541180 2024-03-12T05:23:02.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6ef8004460 con 0x55de59541180 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ef000be30 con 0x55de59541180 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ef8034640 con 0x55de59541180 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6f0981e740 10 client.28007 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.473+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55de594478c0 con 0x55de59541180 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ef8052040 con 0x55de59541180 2024-03-12T05:23:02.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6ef8065050 con 0x55de59541180 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 10 client.28007 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 10 client.28007 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 20 client.28007 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59598ad0 0x55de5959aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 4 client.28007 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:02.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.474+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f6ef000be30 con 0x55de59598ad0 2024-03-12T05:23:02.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.479+0000 7f6f04908640 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59598ad0 0x55de5959aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.490+0000 7f6f04908640 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59598ad0 0x55de5959aef0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=923706a36c4ff749 server_cookie=2568acb160a127d5 in_seq=0 out_seq=0 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.490+0000 7f6edffff640 10 client.28007 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.490+0000 7f6edffff640 1 -- 172.21.15.45:0/1199019252 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f6ef000be30 con 0x55de59598ad0 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.490+0000 7f6edffff640 10 client.28007 handle_command_reply: tid=0 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.490+0000 7f6f0981e740 1 client.28007 shutdown 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 msgr2=0x55de59536b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8057040 tx=0x7f6ef8004270 comp rx=0 tx=0).stop 2024-03-12T05:23:02.486 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 shutdown_connections 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55de59598ad0 0x55de5959aef0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55de59541180 0x55de59536b40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55de59530f60 0x55de595365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 --2- 172.21.15.45:0/1199019252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55de59527a60 0x55de59538220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 >> 172.21.15.45:0/1199019252 conn(0x55de59522190 msgr2=0x55de595182f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6edffff640 0 client.28007 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 shutdown_connections 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 1 -- 172.21.15.45:0/1199019252 wait complete. 2024-03-12T05:23:02.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.491+0000 7f6f0981e740 20 client.28007 trim_cache size 0 max 16384 2024-03-12T05:23:02.488 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:02.488 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 msgr2=0x7f6ee80a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 0x7f6ee80a41f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8036040 tx=0x7f6ef802cb00 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 msgr2=0x7f6f0019f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400c970 tx=0x7f6ef400ce30 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 shutdown_connections 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6ee80a1d30 0x7f6ee80a41f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f00199610 0x7f6f00199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f00108660 0x7f6f0019f9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 --2- 172.21.15.45:0/3928955943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f00103590 0x7f6f0019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 >> 172.21.15.45:0/3928955943 conn(0x7f6f000fedf0 msgr2=0x7f6f001010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 shutdown_connections 2024-03-12T05:23:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.492+0000 7f6f06b6c640 1 -- 172.21.15.45:0/3928955943 wait complete. 2024-03-12T05:23:02.505 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:02.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:23:02.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.692+0000 7f0bfc38a640 1 -- 172.21.15.45:0/47552356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf41166c0 msgr2=0x7f0bf4118ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.692+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/47552356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf41166c0 0x7f0bf4118ab0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009900 tx=0x7f0bf002bc40 comp rx=0 tx=0).stop 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.692+0000 7f0bfc38a640 1 -- 172.21.15.45:0/47552356 shutdown_connections 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.693+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/47552356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf4108c80 0x7f0bf4116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.693+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/47552356 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf41166c0 0x7f0bf4118ab0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.693+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/47552356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.693+0000 7f0bfc38a640 1 -- 172.21.15.45:0/47552356 >> 172.21.15.45:0/47552356 conn(0x7f0bf406c260 msgr2=0x7f0bf406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.695+0000 7f0bfc38a640 1 -- 172.21.15.45:0/47552356 shutdown_connections 2024-03-12T05:23:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.695+0000 7f0bfc38a640 1 -- 172.21.15.45:0/47552356 wait complete. 2024-03-12T05:23:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 Processor -- start 2024-03-12T05:23:02.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 -- start start 2024-03-12T05:23:02.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf410be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 0x7f0bf410c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 0x7f0bf410c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41b2690 con 0x7f0bf41166c0 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41b27d0 con 0x7f0bf4108360 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfc38a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41b2940 con 0x7f0bf4108c80 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bfa126640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf410be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.696+0000 7f0bf9925640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 0x7f0bf410c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bf9925640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 0x7f0bf410c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50958/0 (socket says 172.21.15.45:50958) 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bf9925640 1 -- 172.21.15.45:0/524040524 learned_addr learned my addr 172.21.15.45:0/524040524 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa927640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 0x7f0bf410c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 -- 172.21.15.45:0/524040524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 msgr2=0x7f0bf410c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 0x7f0bf410c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 -- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 msgr2=0x7f0bf410c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 0x7f0bf410c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 -- 172.21.15.45:0/524040524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0009560 con 0x7f0bf4108360 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa927640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 0x7f0bf410c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.697+0000 7f0bfa126640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf410be00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0be400ea20 tx=0x7f0be400eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.698+0000 7f0beb7fe640 1 -- 172.21.15.45:0/524040524 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be400ccc0 con 0x7f0bf4108360 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.698+0000 7f0beb7fe640 1 -- 172.21.15.45:0/524040524 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0be400ce20 con 0x7f0bf4108360 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.698+0000 7f0beb7fe640 1 -- 172.21.15.45:0/524040524 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4018810 con 0x7f0bf4108360 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.698+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf41b2c20 con 0x7f0bf4108360 2024-03-12T05:23:02.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.698+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf41b30e0 con 0x7f0bf4108360 2024-03-12T05:23:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.701+0000 7f0beb7fe640 1 -- 172.21.15.45:0/524040524 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0be4018970 con 0x7f0bf4108360 2024-03-12T05:23:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.702+0000 7f0bff03a740 1 Processor -- start 2024-03-12T05:23:02.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.702+0000 7f0bff03a740 1 -- start start 2024-03-12T05:23:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea2d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0beb7fe640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 0x7f0be00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea36210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea36750 0x55b92ea38b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0beb7fe640 1 -- 172.21.15.45:0/524040524 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0be4014070 con 0x7f0bf4108360 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea2d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bf9925640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea36210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea2d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50960/0 (socket says 172.21.15.45:50960) 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bfa927640 1 -- 172.21.15.45:0/2763328469 learned_addr learned my addr 172.21.15.45:0/2763328469 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e9e8960 con 0x55b92ea36750 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e99c310 con 0x55b92ea2db70 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e999840 con 0x55b92ea2d250 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bfa927640 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea36750 0x55b92ea38b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.703+0000 7f0bfa126640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 0x7f0be00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 -- 172.21.15.45:0/2763328469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 msgr2=0x55b92ea2d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea2d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 -- 172.21.15.45:0/2763328469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea36750 msgr2=0x55b92ea38b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea36750 0x55b92ea38b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 -- 172.21.15.45:0/2763328469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b92e957060 con 0x55b92ea2db70 2024-03-12T05:23:02.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea36210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0bf410d600 tx=0x7f0bec0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=957d22e2bf08a380 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0be8bfe640 1 -- 172.21.15.45:0/2763328469 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec003b70 con 0x55b92ea2db70 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0bfa126640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 0x7f0be00a40e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0be409a670 tx=0x7f0be4010d20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0be8bfe640 1 -- 172.21.15.45:0/2763328469 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0bec003cd0 con 0x55b92ea2db70 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.704+0000 7f0be8bfe640 1 -- 172.21.15.45:0/2763328469 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec012990 con 0x55b92ea2db70 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 msgr2=0x55b92ea36210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea36210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0bf410d600 tx=0x7f0bec0117b0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 shutdown_connections 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea36750 0x55b92ea38b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea36210 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2763328469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea2d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 >> 172.21.15.45:0/2763328469 conn(0x55b92ea27980 msgr2=0x55b92ea27d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 shutdown_connections 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.705+0000 7f0bff03a740 1 -- 172.21.15.45:0/2763328469 wait complete. 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.706+0000 7f0bff03a740 1 -- start start 2024-03-12T05:23:02.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 Processor -- start 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50974/0 (socket says 172.21.15.45:50974) 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bfa927640 1 -- 172.21.15.45:0/1751000314 learned_addr learned my addr 172.21.15.45:0/1751000314 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea33bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea340f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e995020 con 0x55b92ea2d250 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e668430 con 0x55b92ea2db70 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.707+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e624e00 con 0x55b92ea36750 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bf9925640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea33bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bff03a740 10 client.0 _ll_register_callbacks cb 0x55b92e93ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 -- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 msgr2=0x55b92ea340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 -- 172.21.15.45:0/1751000314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 msgr2=0x55b92ea33bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea33bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec009ba0 con 0x55b92ea2d250 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bfa927640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0036040 tx=0x7f0bf0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdd4f6aed9e6cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.708+0000 7f0bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.709+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0004480 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.709+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0bf002de40 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.709+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b92e6947e0 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.710+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bf00035f0 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.710+0000 7f0bff03a740 10 client.18537 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.710+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b92e9936a0 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0bf0003c00 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0bf00052e0 con 0x55b92ea2d250 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 10 client.18537 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 10 client.18537 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:23:02.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 20 client.18537 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea48d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 4 client.18537 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:02.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b92e6947e0 con 0x55b92ea468a0 2024-03-12T05:23:02.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.711+0000 7f0bf9925640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea48d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.712+0000 7f0bf9925640 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea48d60 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c50ecc2c08c4e357 server_cookie=197e322c7227a22a in_seq=0 out_seq=0 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.712+0000 7f0bddffb640 10 client.18537 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bddffb640 1 -- 172.21.15.45:0/1751000314 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b92e6947e0 con 0x55b92ea468a0 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bddffb640 10 client.18537 handle_command_reply: tid=0 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 client.18537 shutdown 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 msgr2=0x55b92ea33670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0036040 tx=0x7f0bf0009260 comp rx=0 tx=0).stop 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 shutdown_connections 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea48d60 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2db70 0x55b92ea33bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bddffb640 0 client.18537 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 --2- 172.21.15.45:0/1751000314 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2d250 0x55b92ea33670 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.713+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 >> 172.21.15.45:0/1751000314 conn(0x55b92ea27980 msgr2=0x55b92ea296d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.714+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 shutdown_connections 2024-03-12T05:23:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.714+0000 7f0bff03a740 1 -- 172.21.15.45:0/1751000314 wait complete. 2024-03-12T05:23:02.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.714+0000 7f0bff03a740 20 client.18537 trim_cache size 0 max 16384 2024-03-12T05:23:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.737+0000 7f0bff03a740 1 Processor -- start 2024-03-12T05:23:02.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.737+0000 7f0bff03a740 1 -- start start 2024-03-12T05:23:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea32f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea36750 0x55b92ea308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:50978/0 (socket says 172.21.15.45:50978) 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bfa927640 1 -- 172.21.15.45:0/3962321743 learned_addr learned my addr 172.21.15.45:0/3962321743 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e91a130 con 0x55b92ea2db70 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92e6947e0 con 0x55b92ea36750 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92ea2cba0 con 0x55b92ea2d250 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.738+0000 7f0bf9925640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea32f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea36750 0x55b92ea308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 -- 172.21.15.45:0/3962321743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea36750 msgr2=0x55b92ea308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea36750 0x55b92ea308d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 -- 172.21.15.45:0/3962321743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 msgr2=0x55b92ea32f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea32f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 -- 172.21.15.45:0/3962321743 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b92e92fc20 con 0x55b92ea2d250 2024-03-12T05:23:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bf9925640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea32f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.739+0000 7f0bfa927640 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00048d0 tx=0x7f0bf002c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544690f2f8939c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.740+0000 7f0bdffff640 1 -- 172.21.15.45:0/3962321743 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0005810 con 0x55b92ea2d250 2024-03-12T05:23:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.740+0000 7f0bdffff640 1 -- 172.21.15.45:0/3962321743 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0bf00434f0 con 0x55b92ea2d250 2024-03-12T05:23:02.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.740+0000 7f0bdffff640 1 -- 172.21.15.45:0/3962321743 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0043c10 con 0x55b92ea2d250 2024-03-12T05:23:02.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.741+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 msgr2=0x55b92ea35da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.741+0000 7f0bff03a740 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00048d0 tx=0x7f0bf002c5e0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 shutdown_connections 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea32f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea36750 0x55b92ea308d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 --2- 172.21.15.45:0/3962321743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea2d250 0x55b92ea35da0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 >> 172.21.15.45:0/3962321743 conn(0x55b92ea27980 msgr2=0x55b92ea26b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 shutdown_connections 2024-03-12T05:23:02.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.742+0000 7f0bff03a740 1 -- 172.21.15.45:0/3962321743 wait complete. 2024-03-12T05:23:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.743+0000 7f0bff03a740 1 -- start start 2024-03-12T05:23:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.743+0000 7f0bff03a740 1 Processor -- start 2024-03-12T05:23:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea3bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bfa927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51290/0 (socket says 172.21.15.45:51290) 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92ea3c6d0 con 0x55b92ea2db70 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92ea3c840 con 0x55b92ea2d250 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bff03a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b92ea3ca00 con 0x55b92ea36750 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bfa927640 1 -- 172.21.15.45:0/2036334415 learned_addr learned my addr 172.21.15.45:0/2036334415 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.744+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea3bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bff03a740 10 client.0 _ll_register_callbacks cb 0x7f0bf0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 -- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 msgr2=0x55b92ea3c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea3c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 -- 172.21.15.45:0/2036334415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 msgr2=0x55b92ea3bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea3bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 -- 172.21.15.45:0/2036334415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b92e957060 con 0x55b92ea2d250 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bfa927640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0004a60 tx=0x7f0bf0043750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a576d8d7a55c222 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bddffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.745+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0004480 con 0x55b92ea2d250 2024-03-12T05:23:02.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.746+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bec003cd0 con 0x55b92ea2d250 2024-03-12T05:23:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.746+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0bf0000e40 con 0x55b92ea2d250 2024-03-12T05:23:02.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.746+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bf0004fd0 con 0x55b92ea2d250 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.747+0000 7f0bff03a740 10 client.28025 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.747+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b92e9936a0 con 0x55b92ea2d250 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.747+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0bf00035f0 con 0x55b92ea2d250 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0bf0034440 con 0x55b92ea2d250 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 10 client.28025 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 10 client.28025 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 20 client.28025 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea9ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 4 client.28025 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:02.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.748+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f0bec003cd0 con 0x55b92ea468a0 2024-03-12T05:23:02.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.749+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea9ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.750+0000 7f0bddffb640 10 client.28025 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:02.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.749+0000 7f0bf9925640 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea9ff70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=42cec8b7a07d36ab server_cookie=b71cfe99c4ad2f89 in_seq=0 out_seq=0 2024-03-12T05:23:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.750+0000 7f0bddffb640 1 -- 172.21.15.45:0/2036334415 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f0bec003cd0 con 0x55b92ea468a0 2024-03-12T05:23:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.750+0000 7f0bddffb640 10 client.28025 handle_command_reply: tid=0 2024-03-12T05:23:02.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 client.28025 shutdown 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 msgr2=0x55b92ea3d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0004a60 tx=0x7f0bf0043750 comp rx=0 tx=0).stop 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 shutdown_connections 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55b92ea468a0 0x55b92ea9ff70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b92ea2db70 0x55b92ea3bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b92ea36750 0x55b92ea3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 --2- 172.21.15.45:0/2036334415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b92ea2d250 0x55b92ea3d810 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 >> 172.21.15.45:0/2036334415 conn(0x55b92ea27980 msgr2=0x55b92ea33610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bddffb640 0 client.28025 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 shutdown_connections 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.751+0000 7f0bff03a740 1 -- 172.21.15.45:0/2036334415 wait complete. 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.752+0000 7f0bff03a740 20 client.28025 trim_cache size 0 max 16384 2024-03-12T05:23:02.747 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.753+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 msgr2=0x7f0be00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.753+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 0x7f0be00a40e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0be409a670 tx=0x7f0be4010d20 comp rx=0 tx=0).stop 2024-03-12T05:23:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.753+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 msgr2=0x7f0bf410be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.753+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf410be00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0be400ea20 tx=0x7f0be400eee0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 shutdown_connections 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f0be00a1c20 0x7f0be00a40e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0bf41166c0 0x7f0bf410c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0bf4108c80 0x7f0bf410c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 --2- 172.21.15.45:0/524040524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bf4108360 0x7f0bf410be00 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.754+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 >> 172.21.15.45:0/524040524 conn(0x7f0bf406c260 msgr2=0x7f0bf4116d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.755+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 shutdown_connections 2024-03-12T05:23:02.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.755+0000 7f0bfc38a640 1 -- 172.21.15.45:0/524040524 wait complete. 2024-03-12T05:23:02.769 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:02.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:23:02.817 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:02 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3089393000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 -- 172.21.15.45:0/1368991364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 msgr2=0x7fbf0010db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 --2- 172.21.15.45:0/1368991364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0010db10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009930 tx=0x7fbef402bc20 comp rx=0 tx=0).stop 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 -- 172.21.15.45:0/1368991364 shutdown_connections 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 --2- 172.21.15.45:0/1368991364 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf00107720 0x7fbf00107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 --2- 172.21.15.45:0/1368991364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf001080e0 0x7fbf001145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 --2- 172.21.15.45:0/1368991364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0010db10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.951 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 -- 172.21.15.45:0/1368991364 >> 172.21.15.45:0/1368991364 conn(0x7fbf000756c0 msgr2=0x7fbf00077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 -- 172.21.15.45:0/1368991364 shutdown_connections 2024-03-12T05:23:02.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.956+0000 7fbf063d7640 1 -- 172.21.15.45:0/1368991364 wait complete. 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 Processor -- start 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 -- start start 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 0x7fbf001a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf001080e0 0x7fbf001a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0019e050 con 0x7fbf001080e0 2024-03-12T05:23:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0019e1c0 con 0x7fbf00107720 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.957+0000 7fbf063d7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0019e380 con 0x7fbf0010d730 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbf04974640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 0x7fbf001a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbf04974640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0019da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51000/0 (socket says 172.21.15.45:51000) 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbf04974640 1 -- 172.21.15.45:0/2408155513 learned_addr learned my addr 172.21.15.45:0/2408155513 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf001080e0 0x7fbf001a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 -- 172.21.15.45:0/2408155513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 msgr2=0x7fbf0019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 -- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf001080e0 msgr2=0x7fbf001a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf001080e0 0x7fbf001a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 -- 172.21.15.45:0/2408155513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef4009560 con 0x7fbf00107720 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefffff640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 0x7fbf001a3800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbef402c160 tx=0x7fbef4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.958+0000 7fbefd7fa640 1 -- 172.21.15.45:0/2408155513 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40035f0 con 0x7fbf00107720 2024-03-12T05:23:02.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.959+0000 7fbefd7fa640 1 -- 172.21.15.45:0/2408155513 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbef4003c00 con 0x7fbf00107720 2024-03-12T05:23:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.959+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0019e600 con 0x7fbf00107720 2024-03-12T05:23:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.959+0000 7fbefd7fa640 1 -- 172.21.15.45:0/2408155513 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4005650 con 0x7fbf00107720 2024-03-12T05:23:02.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.959+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf001aa8a0 con 0x7fbf00107720 2024-03-12T05:23:02.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.962+0000 7fbefd7fa640 1 -- 172.21.15.45:0/2408155513 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbef403e660 con 0x7fbf00107720 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.962+0000 7fbf08fac740 1 Processor -- start 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.962+0000 7fbf08fac740 1 -- start start 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0d9b70 0x563f1c0e2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0e2750 0x563f1c0e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c094960 con 0x563f1c0d9b70 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c048310 con 0x563f1c0e2750 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf08fac740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c045840 con 0x563f1c0d9250 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf04974640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf04974640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51016/0 (socket says 172.21.15.45:51016) 2024-03-12T05:23:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbf04974640 1 -- 172.21.15.45:0/1277571512 learned_addr learned my addr 172.21.15.45:0/1277571512 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbefffff640 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0d9b70 0x563f1c0e2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.963+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0e2750 0x563f1c0e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbefd7fa640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 0x7fbee80a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 -- 172.21.15.45:0/1277571512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0e2750 msgr2=0x563f1c0e4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0e2750 0x563f1c0e4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 -- 172.21.15.45:0/1277571512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0d9b70 msgr2=0x563f1c0e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0d9b70 0x563f1c0e2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 -- 172.21.15.45:0/1277571512 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1c003060 con 0x563f1c0d9250 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbefd7fa640 1 -- 172.21.15.45:0/2408155513 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbef40f0260 con 0x7fbf00107720 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 0x7fbee80a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.964+0000 7fbf04974640 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009fb0 tx=0x7fbef8011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb26a9f0e260346 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.965+0000 7fbee67fc640 1 -- 172.21.15.45:0/1277571512 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8021070 con 0x563f1c0d9250 2024-03-12T05:23:02.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.965+0000 7fbee67fc640 1 -- 172.21.15.45:0/1277571512 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbef801c400 con 0x563f1c0d9250 2024-03-12T05:23:02.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.965+0000 7fbee67fc640 1 -- 172.21.15.45:0/1277571512 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef801b5e0 con 0x563f1c0d9250 2024-03-12T05:23:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- 172.21.15.45:0/1277571512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 msgr2=0x563f1c0d9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009fb0 tx=0x7fbef8011870 comp rx=0 tx=0).stop 2024-03-12T05:23:02.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf04974640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 0x7fbee80a4160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbef8020000 tx=0x7fbef801b9a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- 172.21.15.45:0/1277571512 shutdown_connections 2024-03-12T05:23:02.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0d9b70 0x563f1c0e2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0e2750 0x563f1c0e4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1277571512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0d9630 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- 172.21.15.45:0/1277571512 >> 172.21.15.45:0/1277571512 conn(0x563f1c0d3980 msgr2=0x563f1c0d3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- 172.21.15.45:0/1277571512 shutdown_connections 2024-03-12T05:23:02.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- 172.21.15.45:0/1277571512 wait complete. 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 -- start start 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.966+0000 7fbf08fac740 1 Processor -- start 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 0x563f1c0df700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbefffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 0x563f1c0df700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbefffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 0x563f1c0df700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51320/0 (socket says 172.21.15.45:51320) 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbeff7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbeff7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51026/0 (socket says 172.21.15.45:51026) 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf04974640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf04974640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51032/0 (socket says 172.21.15.45:51032) 2024-03-12T05:23:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbefffff640 1 -- 172.21.15.45:0/2543239960 learned_addr learned my addr 172.21.15.45:0/2543239960 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:02.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c041020 con 0x563f1c0e2750 2024-03-12T05:23:02.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1bd14430 con 0x563f1c0d9250 2024-03-12T05:23:02.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.967+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1bcd0e00 con 0x563f1c0d9b70 2024-03-12T05:23:02.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2543239960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 msgr2=0x563f1c0df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 0x563f1c0df700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2543239960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 msgr2=0x563f1c0e0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:02.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef8009c10 con 0x563f1c0d9b70 2024-03-12T05:23:02.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbf04974640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:02.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbf0010cc40 tx=0x7fbef00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9d95f0694adbce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbedf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.968+0000 7fbf08fac740 10 client.0 _ll_register_callbacks cb 0x563f1bfe8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.969+0000 7fbedf7fe640 1 -- 172.21.15.45:0/2543239960 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0004230 con 0x563f1c0d9b70 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.969+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f1bd407e0 con 0x563f1c0d9b70 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.969+0000 7fbedf7fe640 1 -- 172.21.15.45:0/2543239960 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbef0004390 con 0x563f1c0d9b70 2024-03-12T05:23:02.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.970+0000 7fbedf7fe640 1 -- 172.21.15.45:0/2543239960 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbef0011e90 con 0x563f1c0d9b70 2024-03-12T05:23:02.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.970+0000 7fbf08fac740 10 client.28003 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:02.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.970+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x563f1c03f6a0 con 0x563f1c0d9b70 2024-03-12T05:23:02.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.970+0000 7fbedf7fe640 1 -- 172.21.15.45:0/2543239960 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbef00035f0 con 0x563f1c0d9b70 2024-03-12T05:23:02.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 10 client.28003 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 10 client.28003 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 20 client.28003 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c0f2880 0x563f1c0f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 4 client.28003 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f1bd407e0 con 0x563f1c0f2880 2024-03-12T05:23:02.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.971+0000 7fbefffff640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c0f2880 0x563f1c0f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.972+0000 7fbefffff640 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c0f2880 0x563f1c0f4d10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66b267b9faa7db5a server_cookie=a4291ab6fc90137a in_seq=0 out_seq=0 2024-03-12T05:23:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.972+0000 7fbedf7fe640 10 client.28003 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.973+0000 7fbedf7fe640 1 -- 172.21.15.45:0/2543239960 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563f1bd407e0 con 0x563f1c0f2880 2024-03-12T05:23:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.973+0000 7fbedf7fe640 10 client.28003 handle_command_reply: tid=0 2024-03-12T05:23:02.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.973+0000 7fbf08fac740 1 client.28003 shutdown 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.974+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 msgr2=0x563f1c0dfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.974+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbf0010cc40 tx=0x7fbef00117b0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 shutdown_connections 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c0f2880 0x563f1c0f4d10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9b70 0x563f1c0dfc40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2543239960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9250 0x563f1c0df700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 >> 172.21.15.45:0/2543239960 conn(0x563f1c0d3980 msgr2=0x563f1c0d56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbedf7fe640 0 client.28003 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 shutdown_connections 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 1 -- 172.21.15.45:0/2543239960 wait complete. 2024-03-12T05:23:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:02.975+0000 7fbf08fac740 20 client.28003 trim_cache size 0 max 16384 2024-03-12T05:23:02.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:02 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3089393000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:02.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.004+0000 7fbf08fac740 1 Processor -- start 2024-03-12T05:23:02.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.004+0000 7fbf08fac740 1 -- start start 2024-03-12T05:23:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.005+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.005+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0dffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.005+0000 7fbefffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.005+0000 7fbefffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51030/0 (socket says 172.21.15.45:51030) 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbefffff640 1 -- 172.21.15.45:0/2803321427 learned_addr learned my addr 172.21.15.45:0/2803321427 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0dc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1bfc6130 con 0x563f1c0e2750 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0dffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c0d8340 con 0x563f1c0d9b70 2024-03-12T05:23:03.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c0e4b40 con 0x563f1c0d9250 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.006+0000 7fbf04974640 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0dc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.007+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2803321427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 msgr2=0x563f1c0e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.007+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.007+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2803321427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 msgr2=0x563f1c0dc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.007+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0dc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.007+0000 7fbeff7fe640 1 -- 172.21.15.45:0/2803321427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1bfdbc20 con 0x563f1c0d9b70 2024-03-12T05:23:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.008+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0dffa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x563f1c0e0cd0 tx=0x7fbef0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22008efd1d55724f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.008+0000 7fbee5ffb640 1 -- 172.21.15.45:0/2803321427 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef0033070 con 0x563f1c0d9b70 2024-03-12T05:23:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.008+0000 7fbee5ffb640 1 -- 172.21.15.45:0/2803321427 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbef0016070 con 0x563f1c0d9b70 2024-03-12T05:23:03.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.008+0000 7fbee5ffb640 1 -- 172.21.15.45:0/2803321427 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef001b470 con 0x563f1c0d9b70 2024-03-12T05:23:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.009+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 msgr2=0x563f1c0dffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.009+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0dffa0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x563f1c0e0cd0 tx=0x7fbef0014000 comp rx=0 tx=0).stop 2024-03-12T05:23:03.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 shutdown_connections 2024-03-12T05:23:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0dc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0dffa0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 --2- 172.21.15.45:0/2803321427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 >> 172.21.15.45:0/2803321427 conn(0x563f1c0d3980 msgr2=0x563f1c0ddfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 shutdown_connections 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.010+0000 7fbf08fac740 1 -- 172.21.15.45:0/2803321427 wait complete. 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.011+0000 7fbf08fac740 1 -- start start 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.011+0000 7fbf08fac740 1 Processor -- start 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0e7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbf08fac740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbf08fac740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c0e8660 con 0x563f1c0e2750 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbefffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbefffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51032/0 (socket says 172.21.15.45:51032) 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbefffff640 1 -- 172.21.15.45:0/1665025819 learned_addr learned my addr 172.21.15.45:0/1665025819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0e7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.012+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c0e87a0 con 0x563f1c0d9b70 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1c0e8960 con 0x563f1c0d9250 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbf04974640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbf08fac740 10 client.0 _ll_register_callbacks cb 0x7fbef8009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbeff7fe640 1 -- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 msgr2=0x563f1c0e9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbeff7fe640 1 -- 172.21.15.45:0/1665025819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 msgr2=0x563f1c0e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbeff7fe640 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f1c03f6a0 con 0x563f1c0d9b70 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.013+0000 7fbefffff640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbeff7fe640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0e7be0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x563f1c0d9b40 tx=0x7fbef0003b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7399f3d8b24a468d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbedf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:03.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef003e070 con 0x563f1c0d9b70 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f1c0cf0f0 con 0x563f1c0d9b70 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbef0005030 con 0x563f1c0d9b70 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbef0033070 con 0x563f1c0d9b70 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbf08fac740 10 client.28043 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.014+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563f1c0018c0 con 0x563f1c0d9b70 2024-03-12T05:23:03.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563f1c0cf0f0 con 0x563f1c0d9b70 2024-03-12T05:23:03.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbef0041030 con 0x563f1c0d9b70 2024-03-12T05:23:03.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 10 client.28043 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:03.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 10 client.28043 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:23:03.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 20 client.28043 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:03.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c14a080 0x563f1c14c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 4 client.28043 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fbef801c400 con 0x563f1c14a080 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.015+0000 7fbefffff640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c14a080 0x563f1c14c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbedf7fe640 10 client.28043 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbefffff640 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c14a080 0x563f1c14c470 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=726d3c21795a43cd server_cookie=e03bd938f1786bf9 in_seq=0 out_seq=0 2024-03-12T05:23:03.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbedf7fe640 1 -- 172.21.15.45:0/1665025819 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fbef801c400 con 0x563f1c14a080 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbedf7fe640 10 client.28043 handle_command_reply: tid=0 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 client.28043 shutdown 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 msgr2=0x563f1c0e7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0e7be0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x563f1c0d9b40 tx=0x7fbef0003b70 comp rx=0 tx=0).stop 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 shutdown_connections 2024-03-12T05:23:03.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563f1c0e2750 0x563f1c0e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x563f1c14a080 0x563f1c14c470 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f1c0d9b70 0x563f1c0e7be0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 --2- 172.21.15.45:0/1665025819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f1c0d9250 0x563f1c0e9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 >> 172.21.15.45:0/1665025819 conn(0x563f1c0d3980 msgr2=0x563f1c0d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.016+0000 7fbedf7fe640 0 client.28043 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:03.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.017+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 shutdown_connections 2024-03-12T05:23:03.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.017+0000 7fbf08fac740 1 -- 172.21.15.45:0/1665025819 wait complete. 2024-03-12T05:23:03.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.017+0000 7fbf08fac740 20 client.28043 trim_cache size 0 max 16384 2024-03-12T05:23:03.026 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 msgr2=0x7fbee80a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 0x7fbee80a4160 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbef8020000 tx=0x7fbef801b9a0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 msgr2=0x7fbf001a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 0x7fbf001a3800 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbef402c160 tx=0x7fbef4004270 comp rx=0 tx=0).stop 2024-03-12T05:23:03.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 shutdown_connections 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbee80a1ca0 0x7fbee80a4160 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf001080e0 0x7fbf001a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf0010d730 0x7fbf0019da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 --2- 172.21.15.45:0/2408155513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00107720 0x7fbf001a3800 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 >> 172.21.15.45:0/2408155513 conn(0x7fbf000756c0 msgr2=0x7fbf00075be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 shutdown_connections 2024-03-12T05:23:03.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.018+0000 7fbf063d7640 1 -- 172.21.15.45:0/2408155513 wait complete. 2024-03-12T05:23:03.031 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:03.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:23:03.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.205+0000 7fbc6e148640 1 -- 172.21.15.45:0/306945752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6810cb80 msgr2=0x7fbc6810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.205+0000 7fbc6e148640 1 --2- 172.21.15.45:0/306945752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6810cb80 0x7fbc6810cf60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009930 tx=0x7fbc5802bc30 comp rx=0 tx=0).stop 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.205+0000 7fbc6e148640 1 -- 172.21.15.45:0/306945752 shutdown_connections 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.205+0000 7fbc6e148640 1 --2- 172.21.15.45:0/306945752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc68108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.205+0000 7fbc6e148640 1 --2- 172.21.15.45:0/306945752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc68108b40 0x7fbc68114a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.206+0000 7fbc6e148640 1 --2- 172.21.15.45:0/306945752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc6810cb80 0x7fbc6810cf60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.206+0000 7fbc6e148640 1 -- 172.21.15.45:0/306945752 >> 172.21.15.45:0/306945752 conn(0x7fbc6806c260 msgr2=0x7fbc6806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.206+0000 7fbc6e148640 1 -- 172.21.15.45:0/306945752 shutdown_connections 2024-03-12T05:23:03.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.206+0000 7fbc6e148640 1 -- 172.21.15.45:0/306945752 wait complete. 2024-03-12T05:23:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.207+0000 7fbc6e148640 1 Processor -- start 2024-03-12T05:23:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.207+0000 7fbc6e148640 1 -- start start 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc681a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc68108b40 0x7fbc681a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc681a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc681a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51054/0 (socket says 172.21.15.45:51054) 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc677fe640 1 -- 172.21.15.45:0/4015149086 learned_addr learned my addr 172.21.15.45:0/4015149086 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc68108b40 0x7fbc681a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 0x7fbc681aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681ab390 con 0x7fbc68108180 2024-03-12T05:23:03.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681ab500 con 0x7fbc68108b40 2024-03-12T05:23:03.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc681ab6c0 con 0x7fbc6810cb80 2024-03-12T05:23:03.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.208+0000 7fbc67fff640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 0x7fbc681aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 -- 172.21.15.45:0/4015149086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc68108b40 msgr2=0x7fbc681a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc68108b40 0x7fbc681a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 -- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 msgr2=0x7fbc681a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc681a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009560 con 0x7fbc6810cb80 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc67fff640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 0x7fbc681aad90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbc60005fd0 tx=0x7fbc6000aac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc64ff9640 1 -- 172.21.15.45:0/4015149086 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60004d20 con 0x7fbc6810cb80 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc681ab9a0 con 0x7fbc6810cb80 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc64ff9640 1 -- 172.21.15.45:0/4015149086 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc60010040 con 0x7fbc6810cb80 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.209+0000 7fbc64ff9640 1 -- 172.21.15.45:0/4015149086 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600078b0 con 0x7fbc6810cb80 2024-03-12T05:23:03.205 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.210+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc681b3220 con 0x7fbc6810cb80 2024-03-12T05:23:03.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.212+0000 7fbc64ff9640 1 -- 172.21.15.45:0/4015149086 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbc60007a90 con 0x7fbc6810cb80 2024-03-12T05:23:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.213+0000 7fbc64ff9640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 0x7fbc500a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.213+0000 7fbc64ff9640 1 -- 172.21.15.45:0/4015149086 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc600cce30 con 0x7fbc6810cb80 2024-03-12T05:23:03.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.213+0000 7fbc677fe640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 0x7fbc500a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 Processor -- start 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 -- start start 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 0x563634971210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x563634973b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563634923960 con 0x563634971750 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636348d7310 con 0x563634968b70 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.214+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636348d4840 con 0x563634968250 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51058/0 (socket says 172.21.15.45:51058) 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 -- 172.21.15.45:0/656133582 learned_addr learned my addr 172.21.15.45:0/656133582 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 0x7fbc500a40e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006010 tx=0x7fbc58005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 0x563634971210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 -- 172.21.15.45:0/656133582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 msgr2=0x563634971210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 0x563634971210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 -- 172.21.15.45:0/656133582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 msgr2=0x563634973b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x563634973b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 -- 172.21.15.45:0/656133582 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563634892060 con 0x563634968250 2024-03-12T05:23:03.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 0x563634971210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:23:03.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.215+0000 7fbc677fe640 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbc58004af0 tx=0x7fbc5803f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bf2b9f50953e734 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.216+0000 7fbc4dffb640 1 -- 172.21.15.45:0/656133582 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5802c9b0 con 0x563634968250 2024-03-12T05:23:03.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.216+0000 7fbc4dffb640 1 -- 172.21.15.45:0/656133582 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc580044f0 con 0x563634968250 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.216+0000 7fbc4dffb640 1 -- 172.21.15.45:0/656133582 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58004dc0 con 0x563634968250 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 -- 172.21.15.45:0/656133582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 msgr2=0x563634968630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbc58004af0 tx=0x7fbc5803f000 comp rx=0 tx=0).stop 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 -- 172.21.15.45:0/656133582 shutdown_connections 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x563634973b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968b70 0x563634971210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/656133582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634968630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 -- 172.21.15.45:0/656133582 >> 172.21.15.45:0/656133582 conn(0x563634962980 msgr2=0x563634962d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 -- 172.21.15.45:0/656133582 shutdown_connections 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.217+0000 7fbc70dfa740 1 -- 172.21.15.45:0/656133582 wait complete. 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.218+0000 7fbc70dfa740 1 -- start start 2024-03-12T05:23:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.218+0000 7fbc70dfa740 1 Processor -- start 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.218+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 0x56363496ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x56363496f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636348d0020 con 0x563634971750 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636345a3430 con 0x563634968250 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363455fe00 con 0x563634968b70 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc70dfa740 10 client.0 _ll_register_callbacks cb 0x563634877cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51384/0 (socket says 172.21.15.45:51384) 2024-03-12T05:23:03.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 -- 172.21.15.45:0/3562433953 learned_addr learned my addr 172.21.15.45:0/3562433953 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc67fff640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x56363496f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 0x56363496ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 -- 172.21.15.45:0/3562433953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 msgr2=0x56363496ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 0x56363496ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 -- 172.21.15.45:0/3562433953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 msgr2=0x56363496f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x56363496f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc677fe640 1 -- 172.21.15.45:0/3562433953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc68115280 con 0x563634968250 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.219+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 0x56363496ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc677fe640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbc5802bc70 tx=0x7fbc58003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10eec8459581c79c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58004340 con 0x563634968250 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc58047040 con 0x563634968250 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636345cf7e0 con 0x563634968250 2024-03-12T05:23:03.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc58004dc0 con 0x563634968250 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.220+0000 7fbc70dfa740 10 client.28052 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5636348ce6a0 con 0x563634968250 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5636345cf7e0 con 0x563634968250 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc5803f8e0 con 0x563634968250 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 10 client.28052 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 10 client.28052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 20 client.28052 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563634992680 0x563634994aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 4 client.28052 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:03.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.221+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56363460c720 con 0x563634992680 2024-03-12T05:23:03.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:02 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3089393000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.222+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563634992680 0x563634994aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.222+0000 7fbc46ffd640 10 client.28052 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.222+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563634992680 0x563634994aa0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=446b16d413a4620f server_cookie=1d4c6c7f139cfb10 in_seq=0 out_seq=0 2024-03-12T05:23:03.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.224+0000 7fbc46ffd640 1 -- 172.21.15.45:0/3562433953 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56363460c720 con 0x563634992680 2024-03-12T05:23:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.224+0000 7fbc46ffd640 10 client.28052 handle_command_reply: tid=0 2024-03-12T05:23:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.225+0000 7fbc70dfa740 1 client.28052 shutdown 2024-03-12T05:23:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.225+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 msgr2=0x56363496e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbc5802bc70 tx=0x7fbc58003820 comp rx=0 tx=0).stop 2024-03-12T05:23:03.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 shutdown_connections 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563634992680 0x563634994aa0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634971750 0x56363496f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968b70 0x56363496ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/3562433953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634968250 0x56363496e770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 >> 172.21.15.45:0/3562433953 conn(0x563634962980 msgr2=0x5636349646d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.226+0000 7fbc46ffd640 0 client.28052 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.227+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 shutdown_connections 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.227+0000 7fbc70dfa740 1 -- 172.21.15.45:0/3562433953 wait complete. 2024-03-12T05:23:03.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.227+0000 7fbc70dfa740 20 client.28052 trim_cache size 0 max 16384 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.249+0000 7fbc70dfa740 1 Processor -- start 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 -- start start 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51078/0 (socket says 172.21.15.45:51078) 2024-03-12T05:23:03.245 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x56363496b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc66ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51080/0 (socket says 172.21.15.45:51080) 2024-03-12T05:23:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc677fe640 1 -- 172.21.15.45:0/591526766 learned_addr learned my addr 172.21.15.45:0/591526766 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563634855130 con 0x563634968b70 2024-03-12T05:23:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363460c720 con 0x563634971750 2024-03-12T05:23:03.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.250+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563634985150 con 0x563634968250 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 -- 172.21.15.45:0/591526766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 msgr2=0x563634970ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 -- 172.21.15.45:0/591526766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 msgr2=0x56363496b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x56363496b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 -- 172.21.15.45:0/591526766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56363486ac20 con 0x563634968b70 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc677fe640 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.251+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x563634995380 tx=0x7fbc5c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51b2a1e206b43d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.252+0000 7fbc4d7fa640 1 -- 172.21.15.45:0/591526766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c018040 con 0x563634968b70 2024-03-12T05:23:03.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.252+0000 7fbc4d7fa640 1 -- 172.21.15.45:0/591526766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc5c0035f0 con 0x563634968b70 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.252+0000 7fbc4d7fa640 1 -- 172.21.15.45:0/591526766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c0126b0 con 0x563634968b70 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 msgr2=0x56363496f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x563634995380 tx=0x7fbc5c00b390 comp rx=0 tx=0).stop 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 shutdown_connections 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968b70 0x56363496f010 secure :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0x563634995380 tx=0x7fbc5c00b390 comp rx=0 tx=0).stop 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x56363496b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/591526766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563634968250 0x563634970ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.253+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 >> 172.21.15.45:0/591526766 conn(0x563634962980 msgr2=0x5636349699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.254+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 shutdown_connections 2024-03-12T05:23:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.256+0000 7fbc70dfa740 1 -- 172.21.15.45:0/591526766 wait complete. 2024-03-12T05:23:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.257+0000 7fbc70dfa740 1 -- start start 2024-03-12T05:23:03.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.257+0000 7fbc70dfa740 1 Processor -- start 2024-03-12T05:23:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.257+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.257+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x563634988c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.257+0000 7fbc70dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636349e6b80 0x5636349e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636349e94b0 con 0x563634968250 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636349e95f0 con 0x563634971750 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc70dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636349e9730 con 0x5636349e6b80 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc70dfa740 10 client.0 _ll_register_callbacks cb 0x7fbc58009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc67fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636349e6b80 0x5636349e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51082/0 (socket says 172.21.15.45:51082) 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 -- 172.21.15.45:0/1676263996 learned_addr learned my addr 172.21.15.45:0/1676263996 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:03.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x563634988c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 -- 172.21.15.45:0/1676263996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636349e6b80 msgr2=0x5636349e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636349e6b80 0x5636349e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 -- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 msgr2=0x563634988c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x563634988c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.258+0000 7fbc677fe640 1 -- 172.21.15.45:0/1676263996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636348ce6a0 con 0x563634968250 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.259+0000 7fbc677fe640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009900 tx=0x7fbc580058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef35de6ab66691ca server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.259+0000 7fbc46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.259+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5803f640 con 0x563634968250 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.259+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbc5802de40 con 0x563634968250 2024-03-12T05:23:03.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.260+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56363495e0f0 con 0x563634968250 2024-03-12T05:23:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.260+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc58004350 con 0x563634968250 2024-03-12T05:23:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.260+0000 7fbc70dfa740 10 client.18576 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:03.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.260+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fbc5c004410 con 0x563634968250 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56363495e0f0 con 0x563634968250 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc58034ab0 con 0x563634968250 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 10 client.18576 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 10 client.18576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 20 client.18576 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636349eaa90 0x5636349ece80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 4 client.18576 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.261+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fbc580044f0 con 0x5636349eaa90 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.262+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636349eaa90 0x5636349ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:03.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.262+0000 7fbc46ffd640 10 client.18576 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.262+0000 7fbc66ffd640 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636349eaa90 0x5636349ece80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d433e5f3e81296 server_cookie=945154aefdda6b44 in_seq=0 out_seq=0 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.262+0000 7fbc46ffd640 1 -- 172.21.15.45:0/1676263996 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fbc580044f0 con 0x5636349eaa90 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.262+0000 7fbc46ffd640 10 client.18576 handle_command_reply: tid=0 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 client.18576 shutdown 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 msgr2=0x563634988720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009900 tx=0x7fbc580058d0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 shutdown_connections 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5636349eaa90 0x5636349ece80 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5636349e6b80 0x5636349e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563634971750 0x563634988c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 --2- 172.21.15.45:0/1676263996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563634968250 0x563634988720 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 >> 172.21.15.45:0/1676263996 conn(0x563634962980 msgr2=0x563634963d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc46ffd640 0 client.18576 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:03.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 shutdown_connections 2024-03-12T05:23:03.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.263+0000 7fbc70dfa740 1 -- 172.21.15.45:0/1676263996 wait complete. 2024-03-12T05:23:03.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.264+0000 7fbc70dfa740 20 client.18576 trim_cache size 0 max 16384 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.264+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 msgr2=0x7fbc500a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.264+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 0x7fbc500a40e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006010 tx=0x7fbc58005a20 comp rx=0 tx=0).stop 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 msgr2=0x7fbc681aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 0x7fbc681aad90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbc60005fd0 tx=0x7fbc6000aac0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 shutdown_connections 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc500a1c20 0x7fbc500a40e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc6810cb80 0x7fbc681aad90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc68108b40 0x7fbc681a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 --2- 172.21.15.45:0/4015149086 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc68108180 0x7fbc681a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 >> 172.21.15.45:0/4015149086 conn(0x7fbc6806c260 msgr2=0x7fbc68070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 shutdown_connections 2024-03-12T05:23:03.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:03.265+0000 7fbc6e148640 1 -- 172.21.15.45:0/4015149086 wait complete. 2024-03-12T05:23:03.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:03 smithi116 ceph-mon[35486]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-03-12T05:23:04.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:03 smithi045 ceph-mon[32174]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-03-12T05:23:04.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:03 smithi130 ceph-mon[35398]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-03-12T05:23:04.278 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:23:04.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:23:04.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:23:04.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6e37fe640 1 -- 172.21.15.45:0/3189459761 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d802cb70 con 0x7fe6e41057a0 2024-03-12T05:23:04.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 -- 172.21.15.45:0/3189459761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e41057a0 msgr2=0x7fe6e4105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/3189459761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e41057a0 0x7fe6e4105b80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009930 tx=0x7fe6d802bc30 comp rx=0 tx=0).stop 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 -- 172.21.15.45:0/3189459761 shutdown_connections 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/3189459761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/3189459761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e40ff790 0x7fe6e40ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/3189459761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e41057a0 0x7fe6e4105b80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.460+0000 7fe6eaef4640 1 -- 172.21.15.45:0/3189459761 >> 172.21.15.45:0/3189459761 conn(0x7fe6e40765a0 msgr2=0x7fe6e40fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.461+0000 7fe6eaef4640 1 -- 172.21.15.45:0/3189459761 shutdown_connections 2024-03-12T05:23:04.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.461+0000 7fe6eaef4640 1 -- 172.21.15.45:0/3189459761 wait complete. 2024-03-12T05:23:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.461+0000 7fe6eaef4640 1 Processor -- start 2024-03-12T05:23:04.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.461+0000 7fe6eaef4640 1 -- start start 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 0x7fe6e419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 0x7fe6e4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419a670 con 0x7fe6e4100150 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419a7e0 con 0x7fe6e41057a0 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6eaef4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419a9a0 con 0x7fe6e40ff790 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51102/0 (socket says 172.21.15.45:51102) 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6e3fff640 1 -- 172.21.15.45:0/2729955963 learned_addr learned my addr 172.21.15.45:0/2729955963 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 0x7fe6e419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.462+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 0x7fe6e4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2729955963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 msgr2=0x7fe6e4199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 0x7fe6e4199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 msgr2=0x7fe6e419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2729955963 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d8009560 con 0x7fe6e40ff790 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 0x7fe6e4199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 0x7fe6e419f590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802c170 tx=0x7fe6d80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.463+0000 7fe6e1ffb640 1 -- 172.21.15.45:0/2729955963 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d802cb70 con 0x7fe6e40ff790 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.464+0000 7fe6e1ffb640 1 -- 172.21.15.45:0/2729955963 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6d802de90 con 0x7fe6e40ff790 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.464+0000 7fe6e1ffb640 1 -- 172.21.15.45:0/2729955963 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8005450 con 0x7fe6e40ff790 2024-03-12T05:23:04.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.464+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e419ac20 con 0x7fe6e40ff790 2024-03-12T05:23:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.464+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e41a6560 con 0x7fe6e40ff790 2024-03-12T05:23:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.468+0000 7fe6e1ffb640 1 -- 172.21.15.45:0/2729955963 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe6d8004e50 con 0x7fe6e40ff790 2024-03-12T05:23:04.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.469+0000 7fe6edba6740 1 Processor -- start 2024-03-12T05:23:04.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.469+0000 7fe6edba6740 1 -- start start 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.469+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b2d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e1ffb640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 0x7fe6cc0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e9491640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e9491640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51444/0 (socket says 172.21.15.45:51444) 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e9491640 1 -- 172.21.15.45:0/2116732051 learned_addr learned my addr 172.21.15.45:0/2116732051 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b2fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a92f90 con 0x563681b24d30 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a904c0 con 0x563681b24380 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a8bca0 con 0x563681b2d8b0 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e3fff640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 0x7fe6cc0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.470+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b2d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 -- 172.21.15.45:0/2116732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 msgr2=0x563681b2fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b2fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 -- 172.21.15.45:0/2116732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 msgr2=0x563681b2d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b2d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 -- 172.21.15.45:0/2116732051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563681a4e0b0 con 0x563681b24380 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b2d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e1ffb640 1 -- 172.21.15.45:0/2729955963 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe6d80ef790 con 0x7fe6e40ff790 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400e6d0 tx=0x7fe6d400eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70fe4072c47c7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6e3fff640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 0x7fe6cc0a4230 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe6e410ca80 tx=0x7fe6dc007670 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6caffd640 1 -- 172.21.15.45:0/2116732051 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4013070 con 0x563681b24380 2024-03-12T05:23:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6caffd640 1 -- 172.21.15.45:0/2116732051 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6d400ddd0 con 0x563681b24380 2024-03-12T05:23:04.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.471+0000 7fe6caffd640 1 -- 172.21.15.45:0/2116732051 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4017d60 con 0x563681b24380 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 msgr2=0x563681b24760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400e6d0 tx=0x7fe6d400eb90 comp rx=0 tx=0).stop 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 shutdown_connections 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b2d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2116732051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b24760 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 >> 172.21.15.45:0/2116732051 conn(0x563681b1e840 msgr2=0x563681b1ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 shutdown_connections 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.473+0000 7fe6edba6740 1 -- 172.21.15.45:0/2116732051 wait complete. 2024-03-12T05:23:04.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.474+0000 7fe6edba6740 1 -- start start 2024-03-12T05:23:04.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 Processor -- start 2024-03-12T05:23:04.472 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b2ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b2b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6e8c90640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b2ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6e8c90640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51120/0 (socket says 172.21.15.45:51120) 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b2ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51460/0 (socket says 172.21.15.45:51460) 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6e3fff640 1 -- 172.21.15.45:0/2377632163 learned_addr learned my addr 172.21.15.45:0/2377632163 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a30f40 con 0x563681b2d8b0 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56368175f4b0 con 0x563681b24d30 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a11670 con 0x563681b24380 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.475+0000 7fe6edba6740 10 client.0 _ll_register_callbacks cb 0x563681a7ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:04.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b2b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 msgr2=0x563681b2ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b2ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2377632163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 msgr2=0x563681b2b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b2b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d400bdb0 con 0x563681b24380 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80ecf60 tx=0x7fe6d80dc170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccb34f9aed94b644 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80bcb40 con 0x563681b24380 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe6d80bc440 con 0x563681b24380 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.476+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563681a65f40 con 0x563681b24380 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d80c0590 con 0x563681b24380 2024-03-12T05:23:04.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 10 client.28036 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563681a8a320 con 0x563681b24380 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563681a65f40 con 0x563681b24380 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6d803c070 con 0x563681b24380 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 10 client.28036 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 10 client.28036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 20 client.28036 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b3da80 0x563681b3fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 4 client.28036 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.477+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563681a82310 con 0x563681b3da80 2024-03-12T05:23:04.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.478+0000 7fe6e3fff640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b3da80 0x563681b3fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.478+0000 7fe6c3fff640 10 client.28036 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.478+0000 7fe6e3fff640 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b3da80 0x563681b3fe70 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1218471ca76877f3 server_cookie=49840d62d30697fa in_seq=0 out_seq=0 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.479+0000 7fe6c3fff640 1 -- 172.21.15.45:0/2377632163 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563681a82310 con 0x563681b3da80 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.479+0000 7fe6c3fff640 10 client.28036 handle_command_reply: tid=0 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.479+0000 7fe6edba6740 1 client.28036 shutdown 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.479+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 msgr2=0x563681b2a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.479+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80ecf60 tx=0x7fe6d80dc170 comp rx=0 tx=0).stop 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 shutdown_connections 2024-03-12T05:23:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b3da80 0x563681b3fe70 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b2b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b2ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2377632163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2a8f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 >> 172.21.15.45:0/2377632163 conn(0x563681b1e840 msgr2=0x563681b207a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6c3fff640 0 client.28036 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 shutdown_connections 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 1 -- 172.21.15.45:0/2377632163 wait complete. 2024-03-12T05:23:04.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.480+0000 7fe6edba6740 20 client.28036 trim_cache size 0 max 16384 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 Processor -- start 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 -- start start 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b27b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b28090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681b13350 con 0x563681b2d8b0 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681b25110 con 0x563681b24d30 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6edba6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681a82310 con 0x563681b24380 2024-03-12T05:23:04.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.507+0000 7fe6e8c90640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51136/0 (socket says 172.21.15.45:51136) 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e9491640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b28090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2209407503 learned_addr learned my addr 172.21.15.45:0/2209407503 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2209407503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 msgr2=0x563681b27b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b27b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2209407503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 msgr2=0x563681b28090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b28090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e8c90640 1 -- 172.21.15.45:0/2209407503 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563681a26c00 con 0x563681b24380 2024-03-12T05:23:04.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.508+0000 7fe6e9491640 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b28090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.509+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8004c20 tx=0x7fe6d8005100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ccc630689c1a4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.509+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/2209407503 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80bbda0 con 0x563681b24380 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.509+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/2209407503 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe6d80bcb40 con 0x563681b24380 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.509+0000 7fe6ca7fc640 1 -- 172.21.15.45:0/2209407503 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80c03d0 con 0x563681b24380 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.510+0000 7fe6edba6740 1 -- 172.21.15.45:0/2209407503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 msgr2=0x563681b2d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.510+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8004c20 tx=0x7fe6d8005100 comp rx=0 tx=0).stop 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 -- 172.21.15.45:0/2209407503 shutdown_connections 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b2d8b0 0x563681b28090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24d30 0x563681b27b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 --2- 172.21.15.45:0/2209407503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b24380 0x563681b2d020 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 -- 172.21.15.45:0/2209407503 >> 172.21.15.45:0/2209407503 conn(0x563681b1e840 msgr2=0x563681b291c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 -- 172.21.15.45:0/2209407503 shutdown_connections 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.511+0000 7fe6edba6740 1 -- 172.21.15.45:0/2209407503 wait complete. 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 -- start start 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 Processor -- start 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b33d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b34250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681b95c80 con 0x563681b24d30 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681b95df0 con 0x563681b24380 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563681b95fb0 con 0x563681b2d8b0 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6edba6740 10 client.0 _ll_register_callbacks cb 0x7fe6d400bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6e9491640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b34250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6e9491640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51148/0 (socket says 172.21.15.45:51148) 2024-03-12T05:23:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6e9491640 1 -- 172.21.15.45:0/3453981831 learned_addr learned my addr 172.21.15.45:0/3453981831 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.512+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b34250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51146/0 (socket says 172.21.15.45:51146) 2024-03-12T05:23:04.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b33d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 -- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 msgr2=0x563681b33d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b33d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 -- 172.21.15.45:0/3453981831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 msgr2=0x563681b34250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b34250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 -- 172.21.15.45:0/3453981831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563681a4e0b0 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6e9491640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4019000 tx=0x7fe6d4017f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f80584c3048bf9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.513+0000 7fe6c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4013040 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe6d40054c0 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563681ab1900 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d400d920 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6edba6740 10 client.28042 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.514+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563681a8a320 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563681ab1900 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe6d4003bd0 con 0x563681b2d8b0 2024-03-12T05:23:04.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 10 client.28042 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 10 client.28042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 20 client.28042 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b9e8f0 0x563681ba0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 4 client.28042 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:23:04.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.515+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56368178b860 con 0x563681b9e8f0 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.516+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b9e8f0 0x563681ba0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.517+0000 7fe6e8c90640 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b9e8f0 0x563681ba0d10 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9fddca6ab4e093f server_cookie=8452e8492cd6d360 in_seq=0 out_seq=0 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.517+0000 7fe6c3fff640 10 client.28042 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.518+0000 7fe6c3fff640 1 -- 172.21.15.45:0/3453981831 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56368178b860 con 0x563681b9e8f0 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.518+0000 7fe6c3fff640 10 client.28042 handle_command_reply: tid=0 2024-03-12T05:23:04.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.518+0000 7fe6edba6740 1 client.28042 shutdown 2024-03-12T05:23:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 msgr2=0x563681b95740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4019000 tx=0x7fe6d4017f50 comp rx=0 tx=0).stop 2024-03-12T05:23:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 shutdown_connections 2024-03-12T05:23:04.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x563681b9e8f0 0x563681ba0d10 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563681b24d30 0x563681b34250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563681b2d8b0 0x563681b95740 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 --2- 172.21.15.45:0/3453981831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563681b24380 0x563681b33d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 >> 172.21.15.45:0/3453981831 conn(0x563681b1e840 msgr2=0x563681b2a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6c3fff640 0 client.28042 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 shutdown_connections 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 1 -- 172.21.15.45:0/3453981831 wait complete. 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.519+0000 7fe6edba6740 20 client.28042 trim_cache size 0 max 16384 2024-03-12T05:23:04.515 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:04.516 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:23:04.516 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "3767f725-08bc-40e2-bd8b-25544ddfe5af", 2024-03-12T05:23:04.516 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:23:04.516 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 msgr2=0x7fe6cc0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 0x7fe6cc0a4230 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe6e410ca80 tx=0x7fe6dc007670 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 msgr2=0x7fe6e419f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 0x7fe6e419f590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802c170 tx=0x7fe6d80045e0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 shutdown_connections 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe6cc0a1d70 0x7fe6cc0a4230 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe6e4100150 0x7fe6e419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6e41057a0 0x7fe6e4199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 --2- 172.21.15.45:0/2729955963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6e40ff790 0x7fe6e419f590 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.521+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 >> 172.21.15.45:0/2729955963 conn(0x7fe6e40765a0 msgr2=0x7fe6e4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.522+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 shutdown_connections 2024-03-12T05:23:04.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.522+0000 7fe6eaef4640 1 -- 172.21.15.45:0/2729955963 wait complete. 2024-03-12T05:23:04.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:23:04.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:23:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 -- 172.21.15.45:0/369365369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4860102a80 msgr2=0x7f486010af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 --2- 172.21.15.45:0/369365369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4860102a80 0x7f486010af60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4850009900 tx=0x7f485002bcc0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 -- 172.21.15.45:0/369365369 shutdown_connections 2024-03-12T05:23:04.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 --2- 172.21.15.45:0/369365369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f486010b530 0x7f486010d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 --2- 172.21.15.45:0/369365369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4860102a80 0x7f486010af60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 --2- 172.21.15.45:0/369365369 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102160 0x7f4860102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.731+0000 7f486732c640 1 -- 172.21.15.45:0/369365369 >> 172.21.15.45:0/369365369 conn(0x7f4860076c30 msgr2=0x7f48600fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.732+0000 7f486732c640 1 -- 172.21.15.45:0/369365369 shutdown_connections 2024-03-12T05:23:04.727 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.732+0000 7f486732c640 1 -- 172.21.15.45:0/369365369 wait complete. 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.732+0000 7f486732c640 1 Processor -- start 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.732+0000 7f486732c640 1 -- start start 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102a80 0x7f48601a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486010b530 0x7f486010a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486010a650 con 0x7f4860102a80 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486010a7c0 con 0x7f4860102160 2024-03-12T05:23:04.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f486732c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486010a980 con 0x7f486010b530 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48658c9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486010b530 0x7f486010a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51498/0 (socket says 172.21.15.45:51498) 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 -- 172.21.15.45:0/3925735734 learned_addr learned my addr 172.21.15.45:0/3925735734 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48648c7640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102a80 0x7f48601a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 -- 172.21.15.45:0/3925735734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486010b530 msgr2=0x7f486010a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486010b530 0x7f486010a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.733+0000 7f48650c8640 1 -- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102a80 msgr2=0x7f48601a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.734+0000 7f48650c8640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102a80 0x7f48601a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.734+0000 7f48650c8640 1 -- 172.21.15.45:0/3925735734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850009560 con 0x7f4860102160 2024-03-12T05:23:04.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.734+0000 7f48650c8640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f485c00d950 tx=0x7f485c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.737+0000 7f48567fc640 1 -- 172.21.15.45:0/3925735734 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c009c30 con 0x7f4860102160 2024-03-12T05:23:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.737+0000 7f48567fc640 1 -- 172.21.15.45:0/3925735734 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f485c009d90 con 0x7f4860102160 2024-03-12T05:23:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.737+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486010ac60 con 0x7f4860102160 2024-03-12T05:23:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.737+0000 7f48567fc640 1 -- 172.21.15.45:0/3925735734 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c00f6b0 con 0x7f4860102160 2024-03-12T05:23:04.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.737+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48601aa810 con 0x7f4860102160 2024-03-12T05:23:04.734 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.740+0000 7f48567fc640 1 -- 172.21.15.45:0/3925735734 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f485c00f810 con 0x7f4860102160 2024-03-12T05:23:04.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.740+0000 7f48567fc640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 0x7f484c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f48567fc640 1 -- 172.21.15.45:0/3925735734 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f485c0cd2c0 con 0x7f4860102160 2024-03-12T05:23:04.735 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 Processor -- start 2024-03-12T05:23:04.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f48648c7640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 0x7f484c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 -- start start 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc10b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc114450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc07a280 con 0x557acc10bdb0 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc0777b0 con 0x557acc114990 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.741+0000 7f4869fd8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc072f90 con 0x557acc10b490 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51508/0 (socket says 172.21.15.45:51508) 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 -- 172.21.15.45:0/3672675604 learned_addr learned my addr 172.21.15.45:0/3672675604 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48650c8640 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc10b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48648c7640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 0x7f484c0a40e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4850005ab0 tx=0x7f4850005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 -- 172.21.15.45:0/3672675604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 msgr2=0x557acc10b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc10b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 -- 172.21.15.45:0/3672675604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 msgr2=0x557acc114450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 -- 172.21.15.45:0/3672675604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557acc031570 con 0x557acc114990 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f48658c9640 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f485800afd0 tx=0x7f485800a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de8f43d7a53cb43 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f484b7fe640 1 -- 172.21.15.45:0/3672675604 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858014040 con 0x557acc114990 2024-03-12T05:23:04.738 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.742+0000 7f484b7fe640 1 -- 172.21.15.45:0/3672675604 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f485800cd10 con 0x557acc114990 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f484b7fe640 1 -- 172.21.15.45:0/3672675604 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858019d70 con 0x557acc114990 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 -- 172.21.15.45:0/3672675604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 msgr2=0x557acc116d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f485800afd0 tx=0x7f485800a580 comp rx=0 tx=0).stop 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 -- 172.21.15.45:0/3672675604 shutdown_connections 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc116d80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 --2- 172.21.15.45:0/3672675604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc10b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 -- 172.21.15.45:0/3672675604 >> 172.21.15.45:0/3672675604 conn(0x557acc105bc0 msgr2=0x557acc105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.743+0000 7f4869fd8740 1 -- 172.21.15.45:0/3672675604 shutdown_connections 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 -- 172.21.15.45:0/3672675604 wait complete. 2024-03-12T05:23:04.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 -- start start 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 Processor -- start 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc114380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc111d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.744+0000 7f48648c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f48648c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51190/0 (socket says 172.21.15.45:51190) 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f48650c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc114380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f48650c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc114380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51200/0 (socket says 172.21.15.45:51200) 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f48648c7640 1 -- 172.21.15.45:0/1742559065 learned_addr learned my addr 172.21.15.45:0/1742559065 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f48658c9640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc111d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acbd46430 con 0x557acc10bdb0 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acbd02e00 con 0x557acc114990 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.745+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acbff7fd0 con 0x557acc10b490 2024-03-12T05:23:04.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f4869fd8740 10 client.0 _ll_register_callbacks cb 0x557acc04dfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 -- 172.21.15.45:0/1742559065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 msgr2=0x557acc114380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc114380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 -- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 msgr2=0x557acc111d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc111d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485800acb0 con 0x557acc10bdb0 2024-03-12T05:23:04.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557acbd727e0 con 0x557acc10bdb0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f48648c7640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4850009230 tx=0x7f4850004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f803508d634aa38 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f4848ff9640 10 client.18603 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.746+0000 7f4848ff9640 1 -- 172.21.15.45:0/1742559065 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485002cb50 con 0x557acc10bdb0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.747+0000 7f4848ff9640 1 -- 172.21.15.45:0/1742559065 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4850046050 con 0x557acc10bdb0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.747+0000 7f4848ff9640 1 -- 172.21.15.45:0/1742559065 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f485002ccb0 con 0x557acc10bdb0 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.748+0000 7f4869fd8740 10 client.18603 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:04.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.748+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x557acc071610 con 0x557acc10bdb0 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.749+0000 7f4848ff9640 1 -- 172.21.15.45:0/1742559065 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4850004dc0 con 0x557acc10bdb0 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 10 client.18603 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 10 client.18603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 20 client.18603 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc124ad0 0x557acc126ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 4 client.18603 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:04.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.750+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557acbdaf720 con 0x557acc124ad0 2024-03-12T05:23:04.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.751+0000 7f48650c8640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc124ad0 0x557acc126ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.752+0000 7f4848ff9640 10 client.18603 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.752+0000 7f48650c8640 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc124ad0 0x557acc126ef0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfc6e7a9580e191d server_cookie=8105de464c72621f in_seq=0 out_seq=0 2024-03-12T05:23:04.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.752+0000 7f4848ff9640 1 -- 172.21.15.45:0/1742559065 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557acbdaf720 con 0x557acc124ad0 2024-03-12T05:23:04.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.752+0000 7f4848ff9640 10 client.18603 handle_command_reply: tid=0 2024-03-12T05:23:04.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 client.18603 shutdown 2024-03-12T05:23:04.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 msgr2=0x557acc111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4850009230 tx=0x7f4850004580 comp rx=0 tx=0).stop 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 shutdown_connections 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc124ad0 0x557acc126ef0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10bdb0 0x557acc111710 secure :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4850009230 tx=0x7f4850004580 comp rx=0 tx=0).stop 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4848ff9640 0 client.18603 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc114990 0x557acc111d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 --2- 172.21.15.45:0/1742559065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc10b490 0x557acc114380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.753+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 >> 172.21.15.45:0/1742559065 conn(0x557acc105bc0 msgr2=0x557acc107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.754+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 shutdown_connections 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.754+0000 7f4869fd8740 1 -- 172.21.15.45:0/1742559065 wait complete. 2024-03-12T05:23:04.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.754+0000 7f4869fd8740 20 client.18603 trim_cache size 0 max 16384 2024-03-12T05:23:04.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.784+0000 7f4869fd8740 1 Processor -- start 2024-03-12T05:23:04.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.784+0000 7f4869fd8740 1 -- start start 2024-03-12T05:23:04.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc113e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc114380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc112b00 con 0x557acc10b490 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc12c120 con 0x557acc10ffe0 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f4869fd8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc1125c0 con 0x557acc114990 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:51522/0 (socket says 172.21.15.45:51522) 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48648c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc114380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 -- 172.21.15.45:0/2667153654 learned_addr learned my addr 172.21.15.45:0/2667153654 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48650c8640 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 -- 172.21.15.45:0/2667153654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 msgr2=0x557acc114380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc114380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 -- 172.21.15.45:0/2667153654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 msgr2=0x557acc113e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc113e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48658c9640 1 -- 172.21.15.45:0/2667153654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557acc00dac0 con 0x557acc10ffe0 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.785+0000 7f48650c8640 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.786+0000 7f48658c9640 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f485800afa0 tx=0x7f48580012e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9326e86f7232a137 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.786+0000 7f484affd640 1 -- 172.21.15.45:0/2667153654 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858014030 con 0x557acc10ffe0 2024-03-12T05:23:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.786+0000 7f484affd640 1 -- 172.21.15.45:0/2667153654 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f485800ce80 con 0x557acc10ffe0 2024-03-12T05:23:04.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.786+0000 7f484affd640 1 -- 172.21.15.45:0/2667153654 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858025420 con 0x557acc10ffe0 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 -- 172.21.15.45:0/2667153654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 msgr2=0x557acc110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f485800afa0 tx=0x7f48580012e0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 -- 172.21.15.45:0/2667153654 shutdown_connections 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc110440 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc114380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 --2- 172.21.15.45:0/2667153654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc113e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 -- 172.21.15.45:0/2667153654 >> 172.21.15.45:0/2667153654 conn(0x557acc105bc0 msgr2=0x557acc1009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 -- 172.21.15.45:0/2667153654 shutdown_connections 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.787+0000 7f4869fd8740 1 -- 172.21.15.45:0/2667153654 wait complete. 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.788+0000 7f4869fd8740 1 -- start start 2024-03-12T05:23:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.788+0000 7f4869fd8740 1 Processor -- start 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc119160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc119770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f48650c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f48650c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51214/0 (socket says 172.21.15.45:51214) 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f48650c8640 1 -- 172.21.15.45:0/4144252957 learned_addr learned my addr 172.21.15.45:0/4144252957 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc119e10 con 0x557acc10b490 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc119f80 con 0x557acc10ffe0 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557acc17c000 con 0x557acc114990 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f4869fd8740 10 client.0 _ll_register_callbacks cb 0x7f4858000b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.789+0000 7f48658c9640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc119770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 -- 172.21.15.45:0/4144252957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 msgr2=0x557acc119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc119770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 -- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 msgr2=0x557acc119160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:23:04.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc119160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557acc031570 con 0x557acc10b490 2024-03-12T05:23:04.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f48650c8640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f485c009610 tx=0x7f485c09f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5f9025da64e4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:04.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.790+0000 7f4848ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.791+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c0cc040 con 0x557acc10b490 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.791+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f485c084250 con 0x557acc10b490 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.791+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f485800cd10 con 0x557acc10b490 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.792+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f485c084410 con 0x557acc10b490 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.792+0000 7f4869fd8740 10 client.18609 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:04.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.792+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557acc0336a0 con 0x557acc10b490 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f485800cd10 con 0x557acc10b490 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f485c09a590 con 0x557acc10b490 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 10 client.18609 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 10 client.18609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 20 client.18609 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc17d6c0 0x557acc17fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 4 client.18609 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:23:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557acbdfe890 con 0x557acc17d6c0 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.793+0000 7f48648c7640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc17d6c0 0x557acc17fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.794+0000 7f48648c7640 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc17d6c0 0x557acc17fae0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f0d309ef9b22eef server_cookie=275ad9a2dda1bc23 in_seq=0 out_seq=0 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.794+0000 7f4848ff9640 10 client.18609 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.794+0000 7f4848ff9640 1 -- 172.21.15.45:0/4144252957 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557acbdfe890 con 0x557acc17d6c0 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.794+0000 7f4848ff9640 10 client.18609 handle_command_reply: tid=0 2024-03-12T05:23:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.794+0000 7f4869fd8740 1 client.18609 shutdown 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 msgr2=0x557acc11bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f485c009610 tx=0x7f485c09f200 comp rx=0 tx=0).stop 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 shutdown_connections 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x557acc17d6c0 0x557acc17fae0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557acc114990 0x557acc119770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557acc10ffe0 0x557acc119160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 --2- 172.21.15.45:0/4144252957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557acc10b490 0x557acc11bdd0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 >> 172.21.15.45:0/4144252957 conn(0x557acc105bc0 msgr2=0x557acc1009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.795+0000 7f4848ff9640 0 client.18609 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.796+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 shutdown_connections 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.796+0000 7f4869fd8740 1 -- 172.21.15.45:0/4144252957 wait complete. 2024-03-12T05:23:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.796+0000 7f4869fd8740 20 client.18609 trim_cache size 0 max 16384 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "3767f725-08bc-40e2-bd8b-25544ddfe5af": { 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "3767f725-08bc-40e2-bd8b-25544ddfe5af", 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:23:04.792 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 msgr2=0x7f484c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 0x7f484c0a40e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4850005ab0 tx=0x7f4850005a20 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 msgr2=0x7f48601a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f485c00d950 tx=0x7f485c00de10 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 shutdown_connections 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f484c0a1c20 0x7f484c0a40e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4860102a80 0x7f48601a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f486010b530 0x7f486010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 --2- 172.21.15.45:0/3925735734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4860102160 0x7f48601a3770 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.797+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 >> 172.21.15.45:0/3925735734 conn(0x7f4860076c30 msgr2=0x7f48600ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:04.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.798+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 shutdown_connections 2024-03-12T05:23:04.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:04.798+0000 7f486732c640 1 -- 172.21.15.45:0/3925735734 wait complete. 2024-03-12T05:23:04.812 INFO:tasks.cephfs.filesystem:scrub status for tag:3767f725-08bc-40e2-bd8b-25544ddfe5af - {'path': '/', 'tag': '3767f725-08bc-40e2-bd8b-25544ddfe5af', 'options': 'recursive,force'} 2024-03-12T05:23:04.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:04 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:23:04.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:04 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:04.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:04 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:04.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:04 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 660.0 s, 1236.0 tps, lat 80.816 ms stddev 33.736 2024-03-12T05:23:04.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:04 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:23:04.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:04 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:04.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:04 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:04.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:04 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:05.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:04 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:23:05.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:04 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:05.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:04 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:05.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:04 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:23:05.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:05 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:23:06.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:05 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:06.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:06.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:06.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:06.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 16 MiB/s wr, 1.62k op/s 2024-03-12T05:23:06.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:05 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:07.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:07 smithi116 ceph-mon[35486]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-03-12T05:23:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:07 smithi045 ceph-mon[32174]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-03-12T05:23:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:07 smithi130 ceph-mon[35398]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-03-12T05:23:08.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:08 smithi116 ceph-mon[35486]: osdmap e101: 12 total, 12 up, 12 in 2024-03-12T05:23:09.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:08 smithi045 ceph-mon[32174]: osdmap e101: 12 total, 12 up, 12 in 2024-03-12T05:23:09.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:08 smithi130 ceph-mon[35398]: osdmap e101: 12 total, 12 up, 12 in 2024-03-12T05:23:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 665.0 s, 1158.9 tps, lat 86.179 ms stddev 57.732 2024-03-12T05:23:10.016 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:09 smithi045 ceph-mon[32174]: pgmap v384: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-03-12T05:23:10.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:09 smithi130 ceph-mon[35398]: pgmap v384: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-03-12T05:23:10.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:09 smithi116 ceph-mon[35486]: pgmap v384: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-03-12T05:23:12.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:11 smithi045 ceph-mon[32174]: pgmap v385: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:12.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:11 smithi130 ceph-mon[35398]: pgmap v385: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:12.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:11 smithi116 ceph-mon[35486]: pgmap v385: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:14.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:13 smithi045 ceph-mon[32174]: pgmap v386: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:14.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:13 smithi130 ceph-mon[35398]: pgmap v386: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:14.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:13 smithi116 ceph-mon[35486]: pgmap v386: 97 pgs: 97 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:23:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 670.0 s, 1311.7 tps, lat 76.492 ms stddev 35.804 2024-03-12T05:23:15.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:15 smithi045 ceph-mon[32174]: osdmap e102: 12 total, 12 up, 12 in 2024-03-12T05:23:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:15 smithi130 ceph-mon[35398]: osdmap e102: 12 total, 12 up, 12 in 2024-03-12T05:23:15.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:15 smithi116 ceph-mon[35486]: osdmap e102: 12 total, 12 up, 12 in 2024-03-12T05:23:16.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:16 smithi045 ceph-mon[32174]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 14 MiB/s wr, 1.53k op/s 2024-03-12T05:23:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:16 smithi130 ceph-mon[35398]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 14 MiB/s wr, 1.53k op/s 2024-03-12T05:23:16.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:16 smithi116 ceph-mon[35486]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 14 MiB/s wr, 1.53k op/s 2024-03-12T05:23:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:18 smithi045 ceph-mon[32174]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:23:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:18 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:18.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:18 smithi130 ceph-mon[35398]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:23:18.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:18.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:18.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:18.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:18 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:18 smithi116 ceph-mon[35486]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-03-12T05:23:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:18.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:18 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 675.0 s, 1297.8 tps, lat 76.818 ms stddev 22.447 2024-03-12T05:23:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:20 smithi045 ceph-mon[32174]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:23:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:20 smithi130 ceph-mon[35398]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:23:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:20 smithi116 ceph-mon[35486]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-03-12T05:23:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:22 smithi045 ceph-mon[32174]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:22.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:22 smithi130 ceph-mon[35398]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:22.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:22 smithi116 ceph-mon[35486]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:24.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:24 smithi045 ceph-mon[32174]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:24.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 680.0 s, 1429.0 tps, lat 70.002 ms stddev 19.843 2024-03-12T05:23:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:24 smithi130 ceph-mon[35398]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:24 smithi116 ceph-mon[35486]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 12 MiB/s wr, 1.29k op/s 2024-03-12T05:23:25.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:25 smithi045 ceph-mon[32174]: pgmap v393: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:23:25.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:25 smithi130 ceph-mon[35398]: pgmap v393: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:23:25.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:25 smithi116 ceph-mon[35486]: pgmap v393: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 12 MiB/s wr, 1.36k op/s 2024-03-12T05:23:26.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:26 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:23:26.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:26 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:23:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:26 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:23:28.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:23:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:28.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:23:29.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:29 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:23:29.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:29 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:29.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:29 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:23:29.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:29 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:29.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:29 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:23:29.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:29 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:23:29.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 685.0 s, 1172.8 tps, lat 85.155 ms stddev 35.965 2024-03-12T05:23:30.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:30 smithi130 ceph-mon[35398]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:23:30.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:30 smithi116 ceph-mon[35486]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:23:30.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:30 smithi045 ceph-mon[32174]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:23:32.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:32 smithi130 ceph-mon[35398]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-12T05:23:32.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:32 smithi116 ceph-mon[35486]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-12T05:23:32.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:32 smithi045 ceph-mon[32174]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-12T05:23:34.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:34 smithi130 ceph-mon[35398]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:23:34.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:34 smithi116 ceph-mon[35486]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:23:34.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:34 smithi045 ceph-mon[32174]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-03-12T05:23:34.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:23:34.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:23:34.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 690.0 s, 1223.6 tps, lat 81.921 ms stddev 46.048 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 -- 172.21.15.45:0/4120564586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100b30 msgr2=0x7fb824104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 --2- 172.21.15.45:0/4120564586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100b30 0x7fb824104b70 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb820009cd0 tx=0x7fb82002c100 comp rx=0 tx=0).stop 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 -- 172.21.15.45:0/4120564586 shutdown_connections 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 --2- 172.21.15.45:0/4120564586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100b30 0x7fb824104b70 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 --2- 172.21.15.45:0/4120564586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb824105250 0x7fb82410ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:34.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 --2- 172.21.15.45:0/4120564586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb824100180 0x7fb824100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 -- 172.21.15.45:0/4120564586 >> 172.21.15.45:0/4120564586 conn(0x7fb8240fba00 msgr2=0x7fb8240fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:34.999+0000 7fb82c264640 1 -- 172.21.15.45:0/4120564586 shutdown_connections 2024-03-12T05:23:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.000+0000 7fb82c264640 1 -- 172.21.15.45:0/4120564586 wait complete. 2024-03-12T05:23:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.000+0000 7fb82c264640 1 Processor -- start 2024-03-12T05:23:34.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 -- start start 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb824105250 0x7fb824107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241078c0 0x7fb8241a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82410b020 con 0x7fb824100180 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241aa3a0 con 0x7fb8241078c0 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82c264640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241aa560 con 0x7fb824105250 2024-03-12T05:23:34.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82a000640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47414/0 (socket says 172.21.15.45:47414) 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.001+0000 7fb82a000640 1 -- 172.21.15.45:0/635427248 learned_addr learned my addr 172.21.15.45:0/635427248 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 -- 172.21.15.45:0/635427248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb824105250 msgr2=0x7fb824107380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb824105250 0x7fb824107380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 -- 172.21.15.45:0/635427248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241078c0 msgr2=0x7fb8241a9e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241078c0 0x7fb8241a9e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:34.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 -- 172.21.15.45:0/635427248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb820009900 con 0x7fb824100180 2024-03-12T05:23:34.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.002+0000 7fb82a000640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c920 tx=0x7fb81c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:34.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.003+0000 7fb81affd640 1 -- 172.21.15.45:0/635427248 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c007c70 con 0x7fb824100180 2024-03-12T05:23:34.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.003+0000 7fb81affd640 1 -- 172.21.15.45:0/635427248 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb81c007dd0 con 0x7fb824100180 2024-03-12T05:23:34.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.003+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8241aa840 con 0x7fb824100180 2024-03-12T05:23:34.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.003+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241aad50 con 0x7fb824100180 2024-03-12T05:23:34.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.003+0000 7fb81affd640 1 -- 172.21.15.45:0/635427248 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c018780 con 0x7fb824100180 2024-03-12T05:23:35.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.006+0000 7fb81affd640 1 -- 172.21.15.45:0/635427248 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb81c0189c0 con 0x7fb824100180 2024-03-12T05:23:35.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 Processor -- start 2024-03-12T05:23:35.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 -- start start 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 0x565550815870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550820d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x565550784280 con 0x565550815490 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655507817b0 con 0x56555081e990 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb82ee37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56555077cf90 con 0x565550815db0 2024-03-12T05:23:35.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.007+0000 7fb81affd640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 0x7fb8100a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb8297ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550820d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb81affd640 1 -- 172.21.15.45:0/635427248 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb81c0cd860 con 0x7fb824100180 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb82a801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 0x565550815870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb82a801640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 0x565550815870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47416/0 (socket says 172.21.15.45:47416) 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb82a801640 1 -- 172.21.15.45:0/3767315499 learned_addr learned my addr 172.21.15.45:0/3767315499 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb82a000640 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.008+0000 7fb82a801640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 0x7fb8100a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 -- 172.21.15.45:0/3767315499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 msgr2=0x56555081e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 -- 172.21.15.45:0/3767315499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 msgr2=0x565550815870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 0x565550815870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 -- 172.21.15.45:0/3767315499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56555073b570 con 0x56555081e990 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb82a000640 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb8297ff640 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550820d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb824108600 tx=0x7fb820032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da8c5376fb5cf36 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.009+0000 7fb82a801640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 0x7fb8100a41f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8140143a0 tx=0x7fb81400b380 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.010+0000 7fb80ffff640 1 -- 172.21.15.45:0/3767315499 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820032eb0 con 0x56555081e990 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.010+0000 7fb80ffff640 1 -- 172.21.15.45:0/3767315499 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb820034bc0 con 0x56555081e990 2024-03-12T05:23:35.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.010+0000 7fb80ffff640 1 -- 172.21.15.45:0/3767315499 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820003970 con 0x56555081e990 2024-03-12T05:23:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 -- 172.21.15.45:0/3767315499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 msgr2=0x565550820d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550820d80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb824108600 tx=0x7fb820032340 comp rx=0 tx=0).stop 2024-03-12T05:23:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 -- 172.21.15.45:0/3767315499 shutdown_connections 2024-03-12T05:23:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550820d80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3767315499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815490 0x565550815870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.011+0000 7fb82ee37740 1 -- 172.21.15.45:0/3767315499 >> 172.21.15.45:0/3767315499 conn(0x56555080fbc0 msgr2=0x56555080ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.012+0000 7fb82ee37740 1 -- 172.21.15.45:0/3767315499 shutdown_connections 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.012+0000 7fb82ee37740 1 -- 172.21.15.45:0/3767315499 wait complete. 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.012+0000 7fb82ee37740 1 -- start start 2024-03-12T05:23:35.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.013+0000 7fb82ee37740 1 Processor -- start 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.013+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565550815490 0x56555081e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.013+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56555081e990 0x56555081bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51112/0 (socket says 172.21.15.45:51112) 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 -- 172.21.15.45:0/355239785 learned_addr learned my addr 172.21.15.45:0/355239785 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x565550450430 con 0x56555081e990 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56555040ce00 con 0x565550815490 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565550701fd0 con 0x565550815db0 2024-03-12T05:23:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82a000640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565550815490 0x56555081e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 10 client.0 _ll_register_callbacks cb 0x565550757fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 -- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565550815490 msgr2=0x56555081e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565550815490 0x56555081e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 -- 172.21.15.45:0/355239785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56555081e990 msgr2=0x56555081bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56555081e990 0x56555081bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb8297ff640 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82410d630 con 0x565550815db0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.014+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56555047c7e0 con 0x565550815db0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.015+0000 7fb8297ff640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb820006010 tx=0x7fb8200095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1a7fb08b007a38f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.015+0000 7fb80d7fa640 10 client.28066 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.015+0000 7fb80d7fa640 1 -- 172.21.15.45:0/355239785 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820034640 con 0x565550815db0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb80d7fa640 1 -- 172.21.15.45:0/355239785 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb8200347a0 con 0x565550815db0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb80d7fa640 1 -- 172.21.15.45:0/355239785 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb820032ec0 con 0x565550815db0 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb82ee37740 10 client.28066 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:35.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x56555077b610 con 0x565550815db0 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb80d7fa640 1 -- 172.21.15.45:0/355239785 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb820003970 con 0x565550815db0 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb82ee37740 10 client.28066 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb82ee37740 10 client.28066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.016+0000 7fb82ee37740 20 client.28066 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550830980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb82ee37740 4 client.28066 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5655504b9720 con 0x56555082e540 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb82a000640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550830980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb80d7fa640 10 client.28066 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.017+0000 7fb82a000640 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550830980 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdb88455c31112a8 server_cookie=67d971d74a1d9fea in_seq=0 out_seq=0 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.018+0000 7fb80d7fa640 1 -- 172.21.15.45:0/355239785 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5655504b9720 con 0x56555082e540 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.018+0000 7fb80d7fa640 10 client.28066 handle_command_reply: tid=0 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.018+0000 7fb82ee37740 1 client.28066 shutdown 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.018+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 msgr2=0x56555081b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.018+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb820006010 tx=0x7fb8200095b0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 shutdown_connections 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550830980 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56555081e990 0x56555081bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815db0 0x56555081b740 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 --2- 172.21.15.45:0/355239785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x565550815490 0x56555081e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb80d7fa640 0 client.28066 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:35.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 >> 172.21.15.45:0/355239785 conn(0x56555080fbc0 msgr2=0x565550811910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 shutdown_connections 2024-03-12T05:23:35.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 1 -- 172.21.15.45:0/355239785 wait complete. 2024-03-12T05:23:35.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.019+0000 7fb82ee37740 20 client.28066 trim_cache size 0 max 16384 2024-03-12T05:23:35.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 Processor -- start 2024-03-12T05:23:35.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 -- start start 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x56555081de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x56555081e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56555080b300 con 0x565550815db0 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56555082b630 con 0x56555081e990 2024-03-12T05:23:35.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82ee37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56555080f450 con 0x565550815490 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82a801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82a801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49486/0 (socket says 172.21.15.45:49486) 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82a801640 1 -- 172.21.15.45:0/3533396824 learned_addr learned my addr 172.21.15.45:0/3533396824 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.048+0000 7fb82a000640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x56555081de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb8297ff640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x56555081e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 -- 172.21.15.45:0/3533396824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 msgr2=0x56555081de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x56555081de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 -- 172.21.15.45:0/3533396824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 msgr2=0x56555081e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x56555081e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 -- 172.21.15.45:0/3533396824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565550717ac0 con 0x56555081e990 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a000640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x56555081de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb82a801640 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb81400bca0 tx=0x7fb814017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dc61f8d7d6bee30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb80f7fe640 1 -- 172.21.15.45:0/3533396824 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814005b90 con 0x56555081e990 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb80f7fe640 1 -- 172.21.15.45:0/3533396824 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb81401d040 con 0x56555081e990 2024-03-12T05:23:35.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.049+0000 7fb80f7fe640 1 -- 172.21.15.45:0/3533396824 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814021650 con 0x56555081e990 2024-03-12T05:23:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 -- 172.21.15.45:0/3533396824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 msgr2=0x56555081afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb81400bca0 tx=0x7fb814017000 comp rx=0 tx=0).stop 2024-03-12T05:23:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 -- 172.21.15.45:0/3533396824 shutdown_connections 2024-03-12T05:23:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x56555081e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x56555081afd0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 --2- 172.21.15.45:0/3533396824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x56555081de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.050+0000 7fb82ee37740 1 -- 172.21.15.45:0/3533396824 >> 172.21.15.45:0/3533396824 conn(0x56555080fbc0 msgr2=0x56555081a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.051+0000 7fb82ee37740 1 -- 172.21.15.45:0/3533396824 shutdown_connections 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.051+0000 7fb82ee37740 1 -- 172.21.15.45:0/3533396824 wait complete. 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.051+0000 7fb82ee37740 1 -- start start 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.051+0000 7fb82ee37740 1 Processor -- start 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x565550822990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x565550822ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x565550823670 con 0x565550815db0 2024-03-12T05:23:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x565550885010 con 0x56555081e990 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655508851d0 con 0x565550815490 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82ee37740 10 client.0 _ll_register_callbacks cb 0x7fb820037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82a801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82a801640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49502/0 (socket says 172.21.15.45:49502) 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82a801640 1 -- 172.21.15.45:0/183339050 learned_addr learned my addr 172.21.15.45:0/183339050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb82a000640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x565550822990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.052+0000 7fb8297ff640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x565550822ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 -- 172.21.15.45:0/183339050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 msgr2=0x565550822990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x565550822990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 -- 172.21.15.45:0/183339050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 msgr2=0x565550822ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x565550822ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 -- 172.21.15.45:0/183339050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241aa840 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb8297ff640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x565550822ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82a801640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb814014c30 tx=0x7fb814005f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d8e3791118d4d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb80d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814006630 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb814011040 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb820034bc0 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb814006870 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.053+0000 7fb82ee37740 10 client.28094 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fb81400ba60 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb820034bc0 con 0x56555081e990 2024-03-12T05:23:35.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb81402b400 con 0x56555081e990 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 10 client.28094 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 10 client.28094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 20 client.28094 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550888a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 4 client.28094 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565550508890 con 0x56555082e540 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.054+0000 7fb82a000640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550888a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.055+0000 7fb82a000640 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550888a70 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e594ce6f85668d server_cookie=60c23ddb5dce9dcc in_seq=0 out_seq=0 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.055+0000 7fb80d7fa640 10 client.28094 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.055+0000 7fb80d7fa640 1 -- 172.21.15.45:0/183339050 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x565550508890 con 0x56555082e540 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.055+0000 7fb80d7fa640 10 client.28094 handle_command_reply: tid=0 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.055+0000 7fb82ee37740 1 client.28094 shutdown 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 msgr2=0x565550884aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb814014c30 tx=0x7fb814005f50 comp rx=0 tx=0).stop 2024-03-12T05:23:35.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 shutdown_connections 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56555082e540 0x565550888a70 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x565550815db0 0x565550822ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56555081e990 0x565550884aa0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 --2- 172.21.15.45:0/183339050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565550815490 0x565550822990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 >> 172.21.15.45:0/183339050 conn(0x56555080fbc0 msgr2=0x56555081afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.056+0000 7fb80d7fa640 0 client.28094 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:35.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.057+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 shutdown_connections 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.057+0000 7fb82ee37740 1 -- 172.21.15.45:0/183339050 wait complete. 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.058+0000 7fb82ee37740 20 client.28094 trim_cache size 0 max 16384 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:23:35.057 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.058+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 msgr2=0x7fb8100a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.058+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 0x7fb8100a41f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8140143a0 tx=0x7fb81400b380 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.058+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 msgr2=0x7fb824106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.058+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c920 tx=0x7fb81c00cde0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 shutdown_connections 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb8100a1d30 0x7fb8100a41f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8241078c0 0x7fb8241a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb824105250 0x7fb824107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 --2- 172.21.15.45:0/635427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824100180 0x7fb824106e40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 >> 172.21.15.45:0/635427248 conn(0x7fb8240fba00 msgr2=0x7fb8240fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 shutdown_connections 2024-03-12T05:23:35.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.059+0000 7fb82c264640 1 -- 172.21.15.45:0/635427248 wait complete. 2024-03-12T05:23:35.072 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:23:35.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:23:35.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 -- 172.21.15.45:0/2585900754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd324103f50 msgr2=0x7fd324107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2585900754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd324103f50 0x7fd324107f90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009860 tx=0x7fd30c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 -- 172.21.15.45:0/2585900754 shutdown_connections 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2585900754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd324110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2585900754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd324103f50 0x7fd324107f90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2585900754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3241035a0 0x7fd324103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.247+0000 7fd328f8a640 1 -- 172.21.15.45:0/2585900754 >> 172.21.15.45:0/2585900754 conn(0x7fd3240fedc0 msgr2=0x7fd3241011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.248+0000 7fd328f8a640 1 -- 172.21.15.45:0/2585900754 shutdown_connections 2024-03-12T05:23:35.243 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.248+0000 7fd328f8a640 1 -- 172.21.15.45:0/2585900754 wait complete. 2024-03-12T05:23:35.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.249+0000 7fd328f8a640 1 Processor -- start 2024-03-12T05:23:35.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.249+0000 7fd328f8a640 1 -- start start 2024-03-12T05:23:35.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.249+0000 7fd328f8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 0x7fd32419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd328f8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd328f8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 0x7fd324199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd32259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 0x7fd32419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd32259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 0x7fd32419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51140/0 (socket says 172.21.15.45:51140) 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47454/0 (socket says 172.21.15.45:47454) 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd31bfff640 1 -- 172.21.15.45:0/2601392942 learned_addr learned my addr 172.21.15.45:0/2601392942 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32419a2b0 con 0x7fd324108670 2024-03-12T05:23:35.246 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32419a420 con 0x7fd324199800 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32419a590 con 0x7fd3241035a0 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.250+0000 7fd322d9d640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 0x7fd324199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd322d9d640 1 -- 172.21.15.45:0/2601392942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 msgr2=0x7fd32419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd322d9d640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 0x7fd32419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd322d9d640 1 -- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 msgr2=0x7fd32419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd322d9d640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd322d9d640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c009490 con 0x7fd324199800 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.251+0000 7fd31bfff640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:35.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd322d9d640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 0x7fd324199cb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009fd0 tx=0x7fd31c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00ce20 con 0x7fd324199800 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd31c005020 con 0x7fd324199800 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c010690 con 0x7fd324199800 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3241a64f0 con 0x7fd324199800 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.252+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3241a6980 con 0x7fd324199800 2024-03-12T05:23:35.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.254+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd324104a70 con 0x7fd324199800 2024-03-12T05:23:35.249 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.255+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd31c005190 con 0x7fd324199800 2024-03-12T05:23:35.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.255+0000 7fd31b7fe640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 0x7fd3100a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.256+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd31c014070 con 0x7fd324199800 2024-03-12T05:23:35.250 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.256+0000 7fd32259c640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 0x7fd3100a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.256+0000 7fd32259c640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 0x7fd3100a41b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd314005fd0 tx=0x7fd314009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.260+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd31c096cd0 con 0x7fd324199800 2024-03-12T05:23:35.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.678+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd32410b8a0 con 0x7fd324199800 2024-03-12T05:23:35.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.679+0000 7fd31b7fe640 1 -- 172.21.15.45:0/2601392942 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fd31c09a390 con 0x7fd324199800 2024-03-12T05:23:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:23:35.674 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:23:35.674 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:23:35.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.682+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 msgr2=0x7fd3100a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.682+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 0x7fd3100a41b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd314005fd0 tx=0x7fd314009260 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.682+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 msgr2=0x7fd324199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.682+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 0x7fd324199cb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009fd0 tx=0x7fd31c00ef10 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 shutdown_connections 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd3100a1cf0 0x7fd3100a41b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd324108670 0x7fd32419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd324199800 0x7fd324199cb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 --2- 172.21.15.45:0/2601392942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3241035a0 0x7fd32419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 >> 172.21.15.45:0/2601392942 conn(0x7fd3240fedc0 msgr2=0x7fd3241001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 shutdown_connections 2024-03-12T05:23:35.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.683+0000 7fd328f8a640 1 -- 172.21.15.45:0/2601392942 wait complete. 2024-03-12T05:23:35.695 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:35.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 -- 172.21.15.45:0/4073423372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04981142d0 msgr2=0x7f04981166c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 --2- 172.21.15.45:0/4073423372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04981142d0 0x7f04981166c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0490009be0 tx=0x7f049002bef0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 -- 172.21.15.45:0/4073423372 shutdown_connections 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 --2- 172.21.15.45:0/4073423372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04981142d0 0x7f04981166c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 --2- 172.21.15.45:0/4073423372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108c80 0x7f0498113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 --2- 172.21.15.45:0/4073423372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0498108360 0x7f0498108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.877+0000 7f049fa55640 1 -- 172.21.15.45:0/4073423372 >> 172.21.15.45:0/4073423372 conn(0x7f049806c260 msgr2=0x7f049806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.878+0000 7f049fa55640 1 -- 172.21.15.45:0/4073423372 shutdown_connections 2024-03-12T05:23:35.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.878+0000 7f049fa55640 1 -- 172.21.15.45:0/4073423372 wait complete. 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 Processor -- start 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 -- start start 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108360 0x7f049819fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 0x7f04981a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 0x7f04981a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04981a4d40 con 0x7f0498108c80 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04981a4eb0 con 0x7f04981142d0 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049fa55640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04981a5070 con 0x7f0498108360 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 0x7f04981a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 0x7f04981a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47482/0 (socket says 172.21.15.45:47482) 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 0x7f04981a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049cff0640 1 -- 172.21.15.45:0/268822329 learned_addr learned my addr 172.21.15.45:0/268822329 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049d7f1640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108360 0x7f049819fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 -- 172.21.15.45:0/268822329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108360 msgr2=0x7f049819fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108360 0x7f049819fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 -- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 msgr2=0x7f04981a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 0x7f04981a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.879+0000 7f049dff2640 1 -- 172.21.15.45:0/268822329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0490009840 con 0x7f04981142d0 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.880+0000 7f049dff2640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 0x7f04981a4740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0490036040 tx=0x7f0490033d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.880+0000 7f048e7fc640 1 -- 172.21.15.45:0/268822329 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490003b80 con 0x7f04981142d0 2024-03-12T05:23:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.881+0000 7f048e7fc640 1 -- 172.21.15.45:0/268822329 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0490003ce0 con 0x7f04981142d0 2024-03-12T05:23:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.881+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04981a52f0 con 0x7f04981142d0 2024-03-12T05:23:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.881+0000 7f048e7fc640 1 -- 172.21.15.45:0/268822329 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490034670 con 0x7f04981142d0 2024-03-12T05:23:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.881+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04981acbd0 con 0x7f04981142d0 2024-03-12T05:23:35.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.885+0000 7f048e7fc640 1 -- 172.21.15.45:0/268822329 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0490004040 con 0x7f04981142d0 2024-03-12T05:23:35.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.885+0000 7f04a262e740 1 Processor -- start 2024-03-12T05:23:35.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 -- start start 2024-03-12T05:23:35.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b2a7960 con 0x561e9b2fa2a0 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b25b310 con 0x561e9b2e6a50 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f04a262e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b258840 con 0x561e9b2f98f0 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f049d7f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f049cff0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f049cff0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2f9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51170/0 (socket says 172.21.15.45:51170) 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f049cff0640 1 -- 172.21.15.45:0/2656484570 learned_addr learned my addr 172.21.15.45:0/2656484570 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.886+0000 7f049cff0640 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f048e7fc640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 0x7f04840a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 -- 172.21.15.45:0/2656484570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 msgr2=0x561e9b2f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 -- 172.21.15.45:0/2656484570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 msgr2=0x561e9b2e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2e6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 -- 172.21.15.45:0/2656484570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e9b216060 con 0x561e9b2fa2a0 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049dff2640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 0x7f04840a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049cff0640 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:35.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f048e7fc640 1 -- 172.21.15.45:0/268822329 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f04900f0390 con 0x7f04981142d0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.887+0000 7f049d7f1640 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2e6430 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0494009c30 tx=0x7f0494011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce55f959a3a8460 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.888+0000 7f049dff2640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 0x7f04840a4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f04900bcfd0 tx=0x7f0490009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.888+0000 7f04837fe640 1 -- 172.21.15.45:0/2656484570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494004050 con 0x561e9b2fa2a0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.888+0000 7f04837fe640 1 -- 172.21.15.45:0/2656484570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f04940035f0 con 0x561e9b2fa2a0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.888+0000 7f04837fe640 1 -- 172.21.15.45:0/2656484570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494019720 con 0x561e9b2fa2a0 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 -- 172.21.15.45:0/2656484570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 msgr2=0x561e9b2e6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2e6430 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0494009c30 tx=0x7f0494011810 comp rx=0 tx=0).stop 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 -- 172.21.15.45:0/2656484570 shutdown_connections 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2e6430 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 --2- 172.21.15.45:0/2656484570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 -- 172.21.15.45:0/2656484570 >> 172.21.15.45:0/2656484570 conn(0x561e9b2e8ef0 msgr2=0x561e9b2e92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 -- 172.21.15.45:0/2656484570 shutdown_connections 2024-03-12T05:23:35.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.889+0000 7f04a262e740 1 -- 172.21.15.45:0/2656484570 wait complete. 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.890+0000 7f04a262e740 1 -- start start 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.890+0000 7f04a262e740 1 Processor -- start 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.890+0000 7f04a262e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b254020 con 0x561e9b2fa2a0 2024-03-12T05:23:35.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9af27430 con 0x561e9b2e6a50 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9aee3e00 con 0x561e9b2f98f0 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f04a262e740 10 client.0 _ll_register_callbacks cb 0x561e9b1fbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47496/0 (socket says 172.21.15.45:47496) 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049cff0640 1 -- 172.21.15.45:0/331073973 learned_addr learned my addr 172.21.15.45:0/331073973 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049cff0640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 -- 172.21.15.45:0/331073973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 msgr2=0x561e9b2fd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2fd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 -- 172.21.15.45:0/331073973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 msgr2=0x561e9b2fe360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049d7f1640 1 -- 172.21.15.45:0/331073973 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0494009890 con 0x561e9b2f98f0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.891+0000 7f049cff0640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.892+0000 7f049d7f1640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fde20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0494006010 tx=0x7f0494005e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37186793ea0f6898 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.892+0000 7f0480ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.892+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494004460 con 0x561e9b2f98f0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.892+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e9af537e0 con 0x561e9b2f98f0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.893+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0494005040 con 0x561e9b2f98f0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.893+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0494005240 con 0x561e9b2f98f0 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.893+0000 7f04a262e740 10 client.28090 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:35.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.893+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561e9b2526a0 con 0x561e9b2f98f0 2024-03-12T05:23:35.891 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.893+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x561e9af537e0 con 0x561e9b2f98f0 2024-03-12T05:23:35.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f04940194a0 con 0x561e9b2f98f0 2024-03-12T05:23:35.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 10 client.28090 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 10 client.28090 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 20 client.28090 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b30e2c0 0x561e9b3106e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 4 client.28090 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e9af90720 con 0x561e9b30e2c0 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.894+0000 7f049cff0640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b30e2c0 0x561e9b3106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.895+0000 7f0480ff9640 10 client.28090 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.895+0000 7f049cff0640 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b30e2c0 0x561e9b3106e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=13d038c6c73e9480 server_cookie=ae7e6b71a188e37c in_seq=0 out_seq=0 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f0480ff9640 1 -- 172.21.15.45:0/331073973 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561e9af90720 con 0x561e9b30e2c0 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f0480ff9640 10 client.28090 handle_command_reply: tid=0 2024-03-12T05:23:35.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 client.28090 shutdown 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 msgr2=0x561e9b2fde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fde20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0494006010 tx=0x7f0494005e70 comp rx=0 tx=0).stop 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 shutdown_connections 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b30e2c0 0x561e9b3106e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2fa2a0 0x561e9b2fe360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fde20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 --2- 172.21.15.45:0/331073973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2e6a50 0x561e9b2fd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f0480ff9640 0 client.28090 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 >> 172.21.15.45:0/331073973 conn(0x561e9b2e8ef0 msgr2=0x561e9b2e7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 shutdown_connections 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.896+0000 7f04a262e740 1 -- 172.21.15.45:0/331073973 wait complete. 2024-03-12T05:23:35.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.897+0000 7f04a262e740 20 client.28090 trim_cache size 0 max 16384 2024-03-12T05:23:35.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.919+0000 7f04a262e740 1 Processor -- start 2024-03-12T05:23:35.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.919+0000 7f04a262e740 1 -- start start 2024-03-12T05:23:35.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.920+0000 7f04a262e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.920+0000 7f04a262e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.920+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.920+0000 7f049cff0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47500/0 (socket says 172.21.15.45:47500) 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.920+0000 7f049cff0640 1 -- 172.21.15.45:0/608702977 learned_addr learned my addr 172.21.15.45:0/608702977 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f04a262e740 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b304e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049d7f1640 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b1d9130 con 0x561e9b2e6a50 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b30e040 con 0x561e9b2fa2a0 2024-03-12T05:23:35.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b2e25c0 con 0x561e9b2f98f0 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049cff0640 1 -- 172.21.15.45:0/608702977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 msgr2=0x561e9b2fe180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049cff0640 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fe180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049cff0640 1 -- 172.21.15.45:0/608702977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 msgr2=0x561e9b304e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049cff0640 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b304e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.921+0000 7f049cff0640 1 -- 172.21.15.45:0/608702977 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e9b1eec20 con 0x561e9b2e6a50 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.922+0000 7f049cff0640 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f048800b9c0 tx=0x7f048800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7609dae429ddf4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.922+0000 7f0482ffd640 1 -- 172.21.15.45:0/608702977 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048800be00 con 0x561e9b2e6a50 2024-03-12T05:23:35.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.922+0000 7f0482ffd640 1 -- 172.21.15.45:0/608702977 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0488025070 con 0x561e9b2e6a50 2024-03-12T05:23:35.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.922+0000 7f0482ffd640 1 -- 172.21.15.45:0/608702977 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488012c90 con 0x561e9b2e6a50 2024-03-12T05:23:35.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.923+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 msgr2=0x561e9b300010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.923+0000 7f04a262e740 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f048800b9c0 tx=0x7f048800b4a0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.924+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 shutdown_connections 2024-03-12T05:23:35.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.924+0000 7f04a262e740 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b304e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.924+0000 7f04a262e740 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2f98f0 0x561e9b2fe180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.924+0000 7f04a262e740 1 --2- 172.21.15.45:0/608702977 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2e6a50 0x561e9b300010 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.925+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 >> 172.21.15.45:0/608702977 conn(0x561e9b2e8ef0 msgr2=0x561e9b2f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.925+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 shutdown_connections 2024-03-12T05:23:35.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.925+0000 7f04a262e740 1 -- 172.21.15.45:0/608702977 wait complete. 2024-03-12T05:23:35.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.925+0000 7f04a262e740 1 -- start start 2024-03-12T05:23:35.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.926+0000 7f04a262e740 1 Processor -- start 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.926+0000 7f04a262e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2e6a50 0x561e9b304010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f04a262e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f04a262e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b3647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f049d7f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f049d7f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47508/0 (socket says 172.21.15.45:47508) 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f049d7f1640 1 -- 172.21.15.45:0/3271753548 learned_addr learned my addr 172.21.15.45:0/3271753548 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b364d10 con 0x561e9b2f98f0 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b364e50 con 0x561e9b2fa2a0 2024-03-12T05:23:35.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.927+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e9b364f90 con 0x561e9b2e6a50 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049cff0640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2e6a50 0x561e9b304010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049d7f1640 1 -- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2e6a50 msgr2=0x561e9b304010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049d7f1640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2e6a50 0x561e9b304010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049d7f1640 1 -- 172.21.15.45:0/3271753548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 msgr2=0x561e9b3647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049d7f1640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b3647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f049d7f1640 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048801a040 con 0x561e9b2f98f0 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.928+0000 7f04a262e740 10 client.0 _ll_register_callbacks cb 0x7f04940097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:35.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.929+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e9b2e9870 con 0x561e9b2f98f0 2024-03-12T05:23:35.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.929+0000 7f049d7f1640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0494019800 tx=0x7f0494019c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d4185a35c8c890 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:35.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f0480ff9640 10 client.18648 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:35.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f0480ff9640 1 -- 172.21.15.45:0/3271753548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494003c70 con 0x561e9b2f98f0 2024-03-12T05:23:35.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f0480ff9640 1 -- 172.21.15.45:0/3271753548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f04940052d0 con 0x561e9b2f98f0 2024-03-12T05:23:35.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f0480ff9640 1 -- 172.21.15.45:0/3271753548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04940054d0 con 0x561e9b2f98f0 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f04a262e740 10 client.18648 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.930+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x561e9b2148c0 con 0x561e9b2f98f0 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f0480ff9640 1 -- 172.21.15.45:0/3271753548 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0494041470 con 0x561e9b2f98f0 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 10 client.18648 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 10 client.18648 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 20 client.18648 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b366200 0x561e9b3685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 4 client.18648 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f04940035f0 con 0x561e9b366200 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.931+0000 7f049cff0640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b366200 0x561e9b3685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.932+0000 7f049cff0640 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b366200 0x561e9b3685f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=61bf877643b0369e server_cookie=fe33bd8a74f6806b in_seq=0 out_seq=0 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.932+0000 7f0480ff9640 10 client.18648 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.932+0000 7f0480ff9640 1 -- 172.21.15.45:0/3271753548 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f04940035f0 con 0x561e9b366200 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.932+0000 7f0480ff9640 10 client.18648 handle_command_reply: tid=0 2024-03-12T05:23:35.927 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.932+0000 7f04a262e740 1 client.18648 shutdown 2024-03-12T05:23:35.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.933+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 msgr2=0x561e9b304550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.933+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0494019800 tx=0x7f0494019c60 comp rx=0 tx=0).stop 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 shutdown_connections 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x561e9b366200 0x561e9b3685f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561e9b2f98f0 0x561e9b304550 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561e9b2fa2a0 0x561e9b3647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f0480ff9640 0 client.18648 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 --2- 172.21.15.45:0/3271753548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e9b2e6a50 0x561e9b304010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 >> 172.21.15.45:0/3271753548 conn(0x561e9b2e8ef0 msgr2=0x561e9b2f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 shutdown_connections 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 1 -- 172.21.15.45:0/3271753548 wait complete. 2024-03-12T05:23:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.934+0000 7f04a262e740 20 client.18648 trim_cache size 0 max 16384 2024-03-12T05:23:35.930 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 msgr2=0x7f04840a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 0x7f04840a4230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f04900bcfd0 tx=0x7f0490009500 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 msgr2=0x7f04981a4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 0x7f04981a4740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0490036040 tx=0x7f0490033d80 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 shutdown_connections 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f04840a1d70 0x7f04840a4230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498108c80 0x7f04981a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04981142d0 0x7f04981a4740 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 --2- 172.21.15.45:0/268822329 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0498108360 0x7f049819fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 >> 172.21.15.45:0/268822329 conn(0x7f049806c260 msgr2=0x7f049806fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.935+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 shutdown_connections 2024-03-12T05:23:35.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:35.936+0000 7f049fa55640 1 -- 172.21.15.45:0/268822329 wait complete. 2024-03-12T05:23:35.947 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:35.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.134+0000 7fd14759e640 1 -- 172.21.15.45:0/121096375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400a4de0 msgr2=0x7fd1400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.134+0000 7fd14759e640 1 --2- 172.21.15.45:0/121096375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400a4de0 0x7fd1400a51c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd148066e50 tx=0x7fd14808eaf0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 -- 172.21.15.45:0/121096375 shutdown_connections 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 --2- 172.21.15.45:0/121096375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a5700 0x7fd1400b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 --2- 172.21.15.45:0/121096375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400b7a70 0x7fd1400b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 --2- 172.21.15.45:0/121096375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400a4de0 0x7fd1400a51c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 -- 172.21.15.45:0/121096375 >> 172.21.15.45:0/121096375 conn(0x7fd14001a640 msgr2=0x7fd14001aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 -- 172.21.15.45:0/121096375 shutdown_connections 2024-03-12T05:23:36.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.135+0000 7fd14759e640 1 -- 172.21.15.45:0/121096375 wait complete. 2024-03-12T05:23:36.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 Processor -- start 2024-03-12T05:23:36.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 -- start start 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 0x7fd140149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 0x7fd140149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd140143dc0 con 0x7fd1400a4de0 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd140143f30 con 0x7fd1400b7a70 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14759e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1401440f0 con 0x7fd1400a5700 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 0x7fd140149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49580/0 (socket says 172.21.15.45:49580) 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 0x7fd140149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51208/0 (socket says 172.21.15.45:51208) 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd146d9d640 1 -- 172.21.15.45:0/1343586854 learned_addr learned my addr 172.21.15.45:0/1343586854 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.136+0000 7fd14659c640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 0x7fd140149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 -- 172.21.15.45:0/1343586854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 msgr2=0x7fd140149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 0x7fd140149ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 -- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 msgr2=0x7fd140149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 0x7fd140149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 -- 172.21.15.45:0/1343586854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14804f750 con 0x7fd1400b7a70 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd14659c640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 0x7fd140149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.137+0000 7fd146d9d640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd13800e970 tx=0x7fd13800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.138+0000 7fd1377fe640 1 -- 172.21.15.45:0/1343586854 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13800ccc0 con 0x7fd1400b7a70 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.138+0000 7fd1377fe640 1 -- 172.21.15.45:0/1343586854 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd13800ce20 con 0x7fd1400b7a70 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.138+0000 7fd1377fe640 1 -- 172.21.15.45:0/1343586854 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138010430 con 0x7fd1400b7a70 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.138+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1401443d0 con 0x7fd1400b7a70 2024-03-12T05:23:36.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.138+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1401504f0 con 0x7fd1400b7a70 2024-03-12T05:23:36.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.141+0000 7fd1377fe640 1 -- 172.21.15.45:0/1343586854 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd138010590 con 0x7fd1400b7a70 2024-03-12T05:23:36.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 Processor -- start 2024-03-12T05:23:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 -- start start 2024-03-12T05:23:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9d80 0x55d722c02420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722c02960 0x55d722c04d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722bb4960 con 0x55d722bf9460 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722b68310 con 0x55d722bf9d80 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722b65840 con 0x55d722c02960 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47524/0 (socket says 172.21.15.45:47524) 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd145d9b640 1 -- 172.21.15.45:0/2951298249 learned_addr learned my addr 172.21.15.45:0/2951298249 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.142+0000 7fd14659c640 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9d80 0x55d722c02420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722c02960 0x55d722c04d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 -- 172.21.15.45:0/2951298249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722c02960 msgr2=0x55d722c04d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722c02960 0x55d722c04d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 -- 172.21.15.45:0/2951298249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9d80 msgr2=0x55d722c02420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9d80 0x55d722c02420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.143+0000 7fd145d9b640 1 -- 172.21.15.45:0/2951298249 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d722b23060 con 0x55d722bf9460 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd145d9b640 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0145b0 tx=0x7fd13c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2319316409c8083 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd1377fe640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 0x7fd1300a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd134ff9640 1 -- 172.21.15.45:0/2951298249 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c004030 con 0x55d722bf9460 2024-03-12T05:23:36.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd14659c640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 0x7fd1300a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd134ff9640 1 -- 172.21.15.45:0/2951298249 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd13c005020 con 0x55d722bf9460 2024-03-12T05:23:36.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.144+0000 7fd134ff9640 1 -- 172.21.15.45:0/2951298249 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c004ab0 con 0x55d722bf9460 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd1377fe640 1 -- 172.21.15.45:0/1343586854 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd138014070 con 0x7fd1400b7a70 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 -- 172.21.15.45:0/2951298249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 msgr2=0x55d722bf9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0145b0 tx=0x7fd13c00b960 comp rx=0 tx=0).stop 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 -- 172.21.15.45:0/2951298249 shutdown_connections 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722c02960 0x55d722c04d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9d80 0x55d722c02420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/2951298249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9460 0x55d722bf9840 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.145+0000 7fd14f9b8740 1 -- 172.21.15.45:0/2951298249 >> 172.21.15.45:0/2951298249 conn(0x55d722bf3b90 msgr2=0x55d722bf3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.146+0000 7fd14659c640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 0x7fd1300a4160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd148066cc0 tx=0x7fd148067020 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.146+0000 7fd14f9b8740 1 -- 172.21.15.45:0/2951298249 shutdown_connections 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.146+0000 7fd14f9b8740 1 -- 172.21.15.45:0/2951298249 wait complete. 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.146+0000 7fd14f9b8740 1 -- start start 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.146+0000 7fd14f9b8740 1 Processor -- start 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 0x55d722bff980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9d80 0x55d722bffec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 0x55d722c00400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 0x55d722bff980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 0x55d722bff980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51226/0 (socket says 172.21.15.45:51226) 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14659c640 1 -- 172.21.15.45:0/134145658 learned_addr learned my addr 172.21.15.45:0/134145658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd145d9b640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9d80 0x55d722bffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722b61020 con 0x55d722bf9d80 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd146d9d640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 0x55d722c00400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722834430 con 0x55d722c02960 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7227f0e00 con 0x55d722bf9460 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd14f9b8740 10 client.0 _ll_register_callbacks cb 0x55d722b08cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd146d9d640 1 -- 172.21.15.45:0/134145658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 msgr2=0x55d722bff980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd146d9d640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 0x55d722bff980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.147+0000 7fd146d9d640 1 -- 172.21.15.45:0/134145658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9d80 msgr2=0x55d722bffec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd146d9d640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9d80 0x55d722bffec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd146d9d640 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c00be00 con 0x55d722c02960 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd146d9d640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 0x55d722c00400 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd1380cb110 tx=0x7fd138004780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa69d8628c82e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd12dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13809e740 con 0x55d722c02960 2024-03-12T05:23:36.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd1380a2d90 con 0x55d722c02960 2024-03-12T05:23:36.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d7228607e0 con 0x55d722c02960 2024-03-12T05:23:36.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1380cb390 con 0x55d722c02960 2024-03-12T05:23:36.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd14f9b8740 10 client.28124 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.148+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d722b5f6a0 con 0x55d722c02960 2024-03-12T05:23:36.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd1380cb4f0 con 0x55d722c02960 2024-03-12T05:23:36.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd13801f060 con 0x55d722c02960 2024-03-12T05:23:36.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 10 client.28124 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 10 client.28124 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:23:36.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 20 client.28124 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c15310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 4 client.28124 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.149+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7228607e0 con 0x55d722c12e90 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.150+0000 7fd14659c640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c15310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.150+0000 7fd12dffb640 10 client.28124 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.150+0000 7fd14659c640 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c15310 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3b216db432cdd3a1 server_cookie=d9442a8baf7d5fd2 in_seq=0 out_seq=0 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.151+0000 7fd12dffb640 1 -- 172.21.15.45:0/134145658 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d7228607e0 con 0x55d722c12e90 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.151+0000 7fd12dffb640 10 client.28124 handle_command_reply: tid=0 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.151+0000 7fd14f9b8740 1 client.28124 shutdown 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 msgr2=0x55d722c00400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 0x55d722c00400 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd1380cb110 tx=0x7fd138004780 comp rx=0 tx=0).stop 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 shutdown_connections 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c15310 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722bf9d80 0x55d722bffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722c02960 0x55d722c00400 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd12dffb640 0 client.28124 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/134145658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9460 0x55d722bff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.152+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 >> 172.21.15.45:0/134145658 conn(0x55d722bf3b90 msgr2=0x55d722bf58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.153+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 shutdown_connections 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.153+0000 7fd14f9b8740 1 -- 172.21.15.45:0/134145658 wait complete. 2024-03-12T05:23:36.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.153+0000 7fd14f9b8740 20 client.28124 trim_cache size 0 max 16384 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 Processor -- start 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 -- start start 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c09860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c09da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722bef340 con 0x55d722c02960 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722ae6130 con 0x55d722bf9460 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722c05630 con 0x55d722bf9d80 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c09da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49622/0 (socket says 172.21.15.45:49622) 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd14659c640 1 -- 172.21.15.45:0/1193314208 learned_addr learned my addr 172.21.15.45:0/1193314208 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c09da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47538/0 (socket says 172.21.15.45:47538) 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.183+0000 7fd145d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c09860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 -- 172.21.15.45:0/1193314208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 msgr2=0x55d722c09860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c09860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 -- 172.21.15.45:0/1193314208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 msgr2=0x55d722c09da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c09da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 -- 172.21.15.45:0/1193314208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d722afbc20 con 0x55d722bf9460 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd145d9b640 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c09860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:36.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.184+0000 7fd14659c640 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd1480626c0 tx=0x7fd1480a5000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22bdbf6d23cec371 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.188+0000 7fd12ffff640 1 -- 172.21.15.45:0/1193314208 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1480a0040 con 0x55d722bf9460 2024-03-12T05:23:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.189+0000 7fd12ffff640 1 -- 172.21.15.45:0/1193314208 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd1480a1040 con 0x55d722bf9460 2024-03-12T05:23:36.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.189+0000 7fd12ffff640 1 -- 172.21.15.45:0/1193314208 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1480916e0 con 0x55d722bf9460 2024-03-12T05:23:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.189+0000 7fd14f9b8740 1 -- 172.21.15.45:0/1193314208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 msgr2=0x55d722bff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.189+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd1480626c0 tx=0x7fd1480a5000 comp rx=0 tx=0).stop 2024-03-12T05:23:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 -- 172.21.15.45:0/1193314208 shutdown_connections 2024-03-12T05:23:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c09da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c09860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/1193314208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722bff260 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 -- 172.21.15.45:0/1193314208 >> 172.21.15.45:0/1193314208 conn(0x55d722bf3b90 msgr2=0x55d722bf3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 -- 172.21.15.45:0/1193314208 shutdown_connections 2024-03-12T05:23:36.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 -- 172.21.15.45:0/1193314208 wait complete. 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 -- start start 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.190+0000 7fd14f9b8740 1 Processor -- start 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722c09a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c07db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722c08830 con 0x55d722c02960 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722c089a0 con 0x55d722bf9460 2024-03-12T05:23:36.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d722c08b60 con 0x55d722bf9d80 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14f9b8740 10 client.0 _ll_register_callbacks cb 0x7fd13c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd146d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47550/0 (socket says 172.21.15.45:47550) 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd146d9d640 1 -- 172.21.15.45:0/3507223726 learned_addr learned my addr 172.21.15.45:0/3507223726 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd14659c640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722c09a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.191+0000 7fd145d9b640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c07db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 -- 172.21.15.45:0/3507223726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 msgr2=0x55d722c07db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c07db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 -- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 msgr2=0x55d722c09a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722c09a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 -- 172.21.15.45:0/3507223726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1400baa40 con 0x55d722c02960 2024-03-12T05:23:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd146d9d640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd138096f70 tx=0x7fd1380cb640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e63a77447d54f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd12dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:23:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.192+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1380048b0 con 0x55d722c02960 2024-03-12T05:23:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.193+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd1380a2d90 con 0x55d722c02960 2024-03-12T05:23:36.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.193+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd13c005020 con 0x55d722c02960 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.193+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd138099f50 con 0x55d722c02960 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.193+0000 7fd14f9b8740 10 client.18666 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.193+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d722b218c0 con 0x55d722c02960 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd13c005020 con 0x55d722c02960 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd13801f070 con 0x55d722c02960 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 10 client.18666 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 10 client.18666 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:23:36.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 20 client.18666 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c6c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 4 client.18666 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.194+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d72289d720 con 0x55d722c12e90 2024-03-12T05:23:36.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.195+0000 7fd14659c640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c6c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.196+0000 7fd12dffb640 10 client.18666 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:36.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.197+0000 7fd14659c640 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c6c400 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=acfa73eba9b9c332 server_cookie=ec8cdeccf6bf1309 in_seq=0 out_seq=0 2024-03-12T05:23:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.197+0000 7fd12dffb640 1 -- 172.21.15.45:0/3507223726 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d72289d720 con 0x55d722c12e90 2024-03-12T05:23:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.197+0000 7fd12dffb640 10 client.18666 handle_command_reply: tid=0 2024-03-12T05:23:36.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.197+0000 7fd14f9b8740 1 client.18666 shutdown 2024-03-12T05:23:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.198+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 msgr2=0x55d722c082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.198+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd138096f70 tx=0x7fd1380cb640 comp rx=0 tx=0).stop 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 shutdown_connections 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55d722c12e90 0x55d722c6c400 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d722c02960 0x55d722c082f0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d722bf9d80 0x55d722c07db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 --2- 172.21.15.45:0/3507223726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d722bf9460 0x55d722c09a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 >> 172.21.15.45:0/3507223726 conn(0x55d722bf3b90 msgr2=0x55d722bf4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd12dffb640 0 client.18666 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 shutdown_connections 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 1 -- 172.21.15.45:0/3507223726 wait complete. 2024-03-12T05:23:36.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.199+0000 7fd14f9b8740 20 client.18666 trim_cache size 0 max 16384 2024-03-12T05:23:36.195 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.200+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 msgr2=0x7fd1300a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.200+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 0x7fd1300a4160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd148066cc0 tx=0x7fd148067020 comp rx=0 tx=0).stop 2024-03-12T05:23:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.200+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 msgr2=0x7fd140143630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.200+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd13800e970 tx=0x7fd13800ee30 comp rx=0 tx=0).stop 2024-03-12T05:23:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 shutdown_connections 2024-03-12T05:23:36.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd1300a1ca0 0x7fd1300a4160 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1400b7a70 0x7fd140143630 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd1400a5700 0x7fd140149ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 --2- 172.21.15.45:0/1343586854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1400a4de0 0x7fd140149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.201+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 >> 172.21.15.45:0/1343586854 conn(0x7fd14001a640 msgr2=0x7fd1400b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.202+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 shutdown_connections 2024-03-12T05:23:36.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.202+0000 7fd14759e640 1 -- 172.21.15.45:0/1343586854 wait complete. 2024-03-12T05:23:36.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:36 smithi045 ceph-mon[32174]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:23:36.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:36 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2601392942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:36.215 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:36.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 -- 172.21.15.45:0/2399411900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 msgr2=0x7f316410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 --2- 172.21.15.45:0/2399411900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316410db50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3154009890 tx=0x7f315402bc00 comp rx=0 tx=0).stop 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 -- 172.21.15.45:0/2399411900 shutdown_connections 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 --2- 172.21.15.45:0/2399411900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164108980 0x7f31641145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 --2- 172.21.15.45:0/2399411900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164107fc0 0x7f3164108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 --2- 172.21.15.45:0/2399411900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316410db50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.403+0000 7f316919a640 1 -- 172.21.15.45:0/2399411900 >> 172.21.15.45:0/2399411900 conn(0x7f3164076d30 msgr2=0x7f3164079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.405+0000 7f316919a640 1 -- 172.21.15.45:0/2399411900 shutdown_connections 2024-03-12T05:23:36.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.405+0000 7f316919a640 1 -- 172.21.15.45:0/2399411900 wait complete. 2024-03-12T05:23:36.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.406+0000 7f316919a640 1 Processor -- start 2024-03-12T05:23:36.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.406+0000 7f316919a640 1 -- start start 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 0x7f31641a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316419e140 con 0x7f3164107fc0 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f3162d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f3162d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47564/0 (socket says 172.21.15.45:47564) 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f3162d9d640 1 -- 172.21.15.45:0/3101002925 learned_addr learned my addr 172.21.15.45:0/3101002925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316419e2b0 con 0x7f316410d770 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316259c640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 0x7f31641a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316419e470 con 0x7f3164108980 2024-03-12T05:23:36.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.407+0000 7f316359e640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 -- 172.21.15.45:0/3101002925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 msgr2=0x7f316419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 -- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 msgr2=0x7f31641a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 -- 172.21.15.45:0/3101002925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009560 con 0x7f3164108980 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f3162d9d640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f316259c640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 0x7f31641a3e80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f315800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f314bfff640 1 -- 172.21.15.45:0/3101002925 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315800cd00 con 0x7f3164108980 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f314bfff640 1 -- 172.21.15.45:0/3101002925 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31580044d0 con 0x7f3164108980 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.408+0000 7f314bfff640 1 -- 172.21.15.45:0/3101002925 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158010430 con 0x7f3164108980 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.409+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3154009490 con 0x7f3164108980 2024-03-12T05:23:36.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.409+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31641aa800 con 0x7f3164108980 2024-03-12T05:23:36.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.413+0000 7f316bd75740 1 Processor -- start 2024-03-12T05:23:36.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.413+0000 7f316bd75740 1 -- start start 2024-03-12T05:23:36.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f316bd75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f316bd75740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5f1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f314bfff640 1 -- 172.21.15.45:0/3101002925 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f31580035f0 con 0x7f3164108980 2024-03-12T05:23:36.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49660/0 (socket says 172.21.15.45:49660) 2024-03-12T05:23:36.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f316359e640 1 -- 172.21.15.45:0/2730771434 learned_addr learned my addr 172.21.15.45:0/2730771434 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.414+0000 7f3162d9d640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5f1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316bd75740 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5f1750 0x55868e5f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5a3960 con 0x55868e5e8b70 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e557310 con 0x55868e5e8250 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f314bfff640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 0x7f314c0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316259c640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5f1750 0x55868e5f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e554840 con 0x55868e5f1750 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f314bfff640 1 -- 172.21.15.45:0/3101002925 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3158014070 con 0x7f3164108980 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.415+0000 7f316359e640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 0x7f314c0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 -- 172.21.15.45:0/2730771434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5f1750 msgr2=0x55868e5f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5f1750 0x55868e5f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 -- 172.21.15.45:0/2730771434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 msgr2=0x55868e5e8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 -- 172.21.15.45:0/2730771434 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55868e512060 con 0x55868e5e8b70 2024-03-12T05:23:36.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f316359e640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.416+0000 7f3162d9d640 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5f1210 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f315402c210 tx=0x7f315402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30790799f1e0b589 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.417+0000 7f316359e640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 0x7f314c0a41b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f315c0049d0 tx=0x7f315c004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.417+0000 7f31497fa640 1 -- 172.21.15.45:0/2730771434 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315402c7a0 con 0x55868e5e8b70 2024-03-12T05:23:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.417+0000 7f31497fa640 1 -- 172.21.15.45:0/2730771434 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31540044f0 con 0x55868e5e8b70 2024-03-12T05:23:36.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.417+0000 7f31497fa640 1 -- 172.21.15.45:0/2730771434 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154003780 con 0x55868e5e8b70 2024-03-12T05:23:36.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 msgr2=0x55868e5f1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5f1210 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f315402c210 tx=0x7f315402cd30 comp rx=0 tx=0).stop 2024-03-12T05:23:36.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 shutdown_connections 2024-03-12T05:23:36.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5f1210 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5f1750 0x55868e5f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 --2- 172.21.15.45:0/2730771434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8250 0x55868e5e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.418+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 >> 172.21.15.45:0/2730771434 conn(0x55868e5e2980 msgr2=0x55868e5e2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.419+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 shutdown_connections 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.419+0000 7f316bd75740 1 -- 172.21.15.45:0/2730771434 wait complete. 2024-03-12T05:23:36.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.419+0000 7f316bd75740 1 -- start start 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.419+0000 7f316bd75740 1 Processor -- start 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5eece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e550020 con 0x55868e5e8b70 2024-03-12T05:23:36.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e223430 con 0x55868e5f1750 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e1dfe00 con 0x55868e5e8250 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49670/0 (socket says 172.21.15.45:49670) 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 -- 172.21.15.45:0/3649958496 learned_addr learned my addr 172.21.15.45:0/3649958496 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316bd75740 10 client.0 _ll_register_callbacks cb 0x55868e4f7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 -- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 msgr2=0x55868e5ee7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5ee7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 -- 172.21.15.45:0/3649958496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 msgr2=0x55868e5eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.420+0000 7f316359e640 1 -- 172.21.15.45:0/3649958496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3164114ec0 con 0x55868e5f1750 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.421+0000 7f316359e640 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f315c0094b0 tx=0x7f315c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=867348efdba7f92f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.421+0000 7f3146ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:36.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.421+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c02b460 con 0x55868e5f1750 2024-03-12T05:23:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.421+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f315c02b5c0 con 0x55868e5f1750 2024-03-12T05:23:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.421+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55868e24f7e0 con 0x55868e5f1750 2024-03-12T05:23:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.422+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f315c02b780 con 0x55868e5f1750 2024-03-12T05:23:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.422+0000 7f316bd75740 10 client.28142 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.422+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55868e54e6a0 con 0x55868e5f1750 2024-03-12T05:23:36.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.422+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f315c022900 con 0x55868e5f1750 2024-03-12T05:23:36.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.423+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f315c035580 con 0x55868e5f1750 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.423+0000 7f316bd75740 10 client.28142 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.423+0000 7f316bd75740 10 client.28142 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.424+0000 7f316bd75740 20 client.28142 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.424+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e601a80 0x55868e603f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.424+0000 7f316bd75740 4 client.28142 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.424+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55868e24f7e0 con 0x55868e601a80 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.424+0000 7f3162d9d640 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e601a80 0x55868e603f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.425+0000 7f3162d9d640 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e601a80 0x55868e603f40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c20ea90798e2af6 server_cookie=3ff47d9d18f4acee in_seq=0 out_seq=0 2024-03-12T05:23:36.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.425+0000 7f3146ff5640 10 client.28142 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f3146ff5640 1 -- 172.21.15.45:0/3649958496 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55868e24f7e0 con 0x55868e601a80 2024-03-12T05:23:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f3146ff5640 10 client.28142 handle_command_reply: tid=0 2024-03-12T05:23:36.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 client.28142 shutdown 2024-03-12T05:23:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 msgr2=0x55868e5ef220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f315c0094b0 tx=0x7f315c004a40 comp rx=0 tx=0).stop 2024-03-12T05:23:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 shutdown_connections 2024-03-12T05:23:36.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e601a80 0x55868e603f40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8b70 0x55868e5eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5ef220 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f316bd75740 1 --2- 172.21.15.45:0/3649958496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.426+0000 7f3146ff5640 0 client.28142 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.427+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 >> 172.21.15.45:0/3649958496 conn(0x55868e5e2980 msgr2=0x55868e5e46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.427+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 shutdown_connections 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.427+0000 7f316bd75740 1 -- 172.21.15.45:0/3649958496 wait complete. 2024-03-12T05:23:36.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.427+0000 7f316bd75740 20 client.28142 trim_cache size 0 max 16384 2024-03-12T05:23:36.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 Processor -- start 2024-03-12T05:23:36.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 -- start start 2024-03-12T05:23:36.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5f1750 0x55868e5ed070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e4d5130 con 0x55868e5f1750 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5de2b0 con 0x55868e5e8b70 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316bd75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5f3ed0 con 0x55868e5e8250 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.453+0000 7f316259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49672/0 (socket says 172.21.15.45:49672) 2024-03-12T05:23:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f3162d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 -- 172.21.15.45:0/2794552750 learned_addr learned my addr 172.21.15.45:0/2794552750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316359e640 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5f1750 0x55868e5ed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 -- 172.21.15.45:0/2794552750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 msgr2=0x55868e5f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 -- 172.21.15.45:0/2794552750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5f1750 msgr2=0x55868e5ed070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5f1750 0x55868e5ed070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 -- 172.21.15.45:0/2794552750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55868e4eac20 con 0x55868e5e8b70 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f316259c640 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3158003a50 tx=0x7f315806c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4252160dad049307 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.454+0000 7f3148ff9640 1 -- 172.21.15.45:0/2794552750 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315800ce60 con 0x55868e5e8b70 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.455+0000 7f3148ff9640 1 -- 172.21.15.45:0/2794552750 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f315809ad50 con 0x55868e5e8b70 2024-03-12T05:23:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.455+0000 7f3148ff9640 1 -- 172.21.15.45:0/2794552750 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158096cc0 con 0x55868e5e8b70 2024-03-12T05:23:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 -- 172.21.15.45:0/2794552750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 msgr2=0x55868e6088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3158003a50 tx=0x7f315806c8d0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 -- 172.21.15.45:0/2794552750 shutdown_connections 2024-03-12T05:23:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5f1750 0x55868e5ed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5e8b70 0x55868e6088e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 --2- 172.21.15.45:0/2794552750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8250 0x55868e5f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 -- 172.21.15.45:0/2794552750 >> 172.21.15.45:0/2794552750 conn(0x55868e5e2980 msgr2=0x55868e5e9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 -- 172.21.15.45:0/2794552750 shutdown_connections 2024-03-12T05:23:36.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.456+0000 7f316bd75740 1 -- 172.21.15.45:0/2794552750 wait complete. 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.457+0000 7f316bd75740 1 -- start start 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.457+0000 7f316bd75740 1 Processor -- start 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 0x55868e5f8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5f75e0 con 0x55868e5e8250 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5f7750 con 0x55868e5f1750 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55868e5f7910 con 0x55868e5e8b70 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316bd75740 10 client.0 _ll_register_callbacks cb 0x7f315c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51304/0 (socket says 172.21.15.45:51304) 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 -- 172.21.15.45:0/1692110117 learned_addr learned my addr 172.21.15.45:0/1692110117 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316359e640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f3162d9d640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 0x55868e5f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 -- 172.21.15.45:0/1692110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 msgr2=0x55868e5f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 -- 172.21.15.45:0/1692110117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 msgr2=0x55868e5f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 0x55868e5f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316259c640 1 -- 172.21.15.45:0/1692110117 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31641aa800 con 0x55868e5e8b70 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f3162d9d640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 0x55868e5f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:36.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.458+0000 7f316359e640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f316259c640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3158097020 tx=0x7f3158096ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5361a74b12d718ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f3146ff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580ba2c0 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31540044f0 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f315809e7a0 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.459+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31580b3170 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 10 client.28129 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55868e5108c0 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f31540044f0 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f315801f060 con 0x55868e5e8b70 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 10 client.28129 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 10 client.28129 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 20 client.28129 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e659020 0x55868e65b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 4 client.28129 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.460+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55868e28c720 con 0x55868e659020 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f3162d9d640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e659020 0x55868e65b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f3146ff5640 10 client.28129 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f3162d9d640 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e659020 0x55868e65b440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9320db52d7bd4174 server_cookie=27459d78f9b51b8f in_seq=0 out_seq=0 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f3146ff5640 1 -- 172.21.15.45:0/1692110117 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55868e28c720 con 0x55868e659020 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f3146ff5640 10 client.28129 handle_command_reply: tid=0 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.461+0000 7f316bd75740 1 client.28129 shutdown 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 msgr2=0x55868e5f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3158097020 tx=0x7f3158096ef0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 shutdown_connections 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55868e659020 0x55868e65b440 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55868e5f1750 0x55868e5f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55868e5e8b70 0x55868e5f6b60 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 --2- 172.21.15.45:0/1692110117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55868e5e8250 0x55868e5f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 >> 172.21.15.45:0/1692110117 conn(0x55868e5e2980 msgr2=0x55868e6088e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f3146ff5640 0 client.28129 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 shutdown_connections 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.462+0000 7f316bd75740 1 -- 172.21.15.45:0/1692110117 wait complete. 2024-03-12T05:23:36.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.463+0000 7f316bd75740 20 client.28129 trim_cache size 0 max 16384 2024-03-12T05:23:36.460 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 msgr2=0x7f314c0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 0x7f314c0a41b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f315c0049d0 tx=0x7f315c004ab0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 msgr2=0x7f31641a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 0x7f31641a3e80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f315800ee50 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 shutdown_connections 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f314c0a1cf0 0x7f314c0a41b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f316410d770 0x7f316419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3164108980 0x7f31641a3e80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 --2- 172.21.15.45:0/3101002925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3164107fc0 0x7f31641a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 >> 172.21.15.45:0/3101002925 conn(0x7f3164076d30 msgr2=0x7f3164079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 shutdown_connections 2024-03-12T05:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.464+0000 7f316919a640 1 -- 172.21.15.45:0/3101002925 wait complete. 2024-03-12T05:23:36.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:36 smithi130 ceph-mon[35398]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:23:36.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:36 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2601392942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:36.479 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:36.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:23:36.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:36 smithi116 ceph-mon[35486]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 15 MiB/s wr, 1.69k op/s 2024-03-12T05:23:36.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:36 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2601392942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:23:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.655+0000 7f220f9c7640 1 -- 172.21.15.45:0/1291413969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 msgr2=0x7f2208114e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1291413969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f2208114e60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2200009ca0 tx=0x7f220002c100 comp rx=0 tx=0).stop 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 -- 172.21.15.45:0/1291413969 shutdown_connections 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1291413969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f2208114e60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1291413969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 0x7f220810d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1291413969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 0x7f220810c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 -- 172.21.15.45:0/1291413969 >> 172.21.15.45:0/1291413969 conn(0x7f220806c6c0 msgr2=0x7f220806cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.656+0000 7f220f9c7640 1 -- 172.21.15.45:0/1291413969 shutdown_connections 2024-03-12T05:23:36.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.657+0000 7f220f9c7640 1 -- 172.21.15.45:0/1291413969 wait complete. 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.657+0000 7f220f9c7640 1 Processor -- start 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 -- start start 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 0x7f220819fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 0x7f22081a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22081a4da0 con 0x7f2208112a70 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22081a4f10 con 0x7f220810cd10 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220f9c7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22081a50d0 con 0x7f2208108a10 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220df64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220df64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47626/0 (socket says 172.21.15.45:47626) 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220df64640 1 -- 172.21.15.45:0/1716552017 learned_addr learned my addr 172.21.15.45:0/1716552017 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220d763640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 0x7f220819fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.658+0000 7f220cf62640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 0x7f22081a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 -- 172.21.15.45:0/1716552017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 msgr2=0x7f220819fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 0x7f220819fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 -- 172.21.15.45:0/1716552017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 msgr2=0x7f22081a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 0x7f22081a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 -- 172.21.15.45:0/1716552017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200009900 con 0x7f2208112a70 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f220df64640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2200037040 tx=0x7f2200034460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f21fe7fc640 1 -- 172.21.15.45:0/1716552017 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000325f0 con 0x7f2208112a70 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f21fe7fc640 1 -- 172.21.15.45:0/1716552017 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2200003970 con 0x7f2208112a70 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.659+0000 7f21fe7fc640 1 -- 172.21.15.45:0/1716552017 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000044e0 con 0x7f2208112a70 2024-03-12T05:23:36.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.660+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22081a5350 con 0x7f2208112a70 2024-03-12T05:23:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.660+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22081159a0 con 0x7f2208112a70 2024-03-12T05:23:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.663+0000 7f21fe7fc640 1 -- 172.21.15.45:0/1716552017 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f220003b040 con 0x7f2208112a70 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 Processor -- start 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 -- start start 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf0c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf17b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdec7960 con 0x556afdf0c250 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afde7b310 con 0x556afdf0cb70 2024-03-12T05:23:36.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.665+0000 7f2212599740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afde78840 con 0x556afdf15750 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f21fe7fc640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 0x7f21f40a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49704/0 (socket says 172.21.15.45:49704) 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 -- 172.21.15.45:0/2323868403 learned_addr learned my addr 172.21.15.45:0/2323868403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220cf62640 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf0c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220df64640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 0x7f21f40a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220cf62640 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf17b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 -- 172.21.15.45:0/2323868403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 msgr2=0x556afdf17b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf17b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 -- 172.21.15.45:0/2323868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 msgr2=0x556afdf0c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf0c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220d763640 1 -- 172.21.15.45:0/2323868403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556afde36060 con 0x556afdf0cb70 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.666+0000 7f220df64640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 0x7f21f40a41f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f22000b8b90 tx=0x7f22000b82a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.667+0000 7f220d763640 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f220400ad00 tx=0x7f220400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=180198474162f572 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.667+0000 7f21fe7fc640 1 -- 172.21.15.45:0/1716552017 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f22000efa20 con 0x7f2208112a70 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.667+0000 7f21f37fe640 1 -- 172.21.15.45:0/2323868403 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220400a800 con 0x556afdf0cb70 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.667+0000 7f21f37fe640 1 -- 172.21.15.45:0/2323868403 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2204018af0 con 0x556afdf0cb70 2024-03-12T05:23:36.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.667+0000 7f21f37fe640 1 -- 172.21.15.45:0/2323868403 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204010c50 con 0x556afdf0cb70 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 -- 172.21.15.45:0/2323868403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 msgr2=0x556afdf15210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f220400ad00 tx=0x7f220400f790 comp rx=0 tx=0).stop 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 -- 172.21.15.45:0/2323868403 shutdown_connections 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf17b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf15210 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 --2- 172.21.15.45:0/2323868403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf0c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 -- 172.21.15.45:0/2323868403 >> 172.21.15.45:0/2323868403 conn(0x556afdf06980 msgr2=0x556afdf06d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 -- 172.21.15.45:0/2323868403 shutdown_connections 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.668+0000 7f2212599740 1 -- 172.21.15.45:0/2323868403 wait complete. 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.669+0000 7f2212599740 1 -- start start 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.669+0000 7f2212599740 1 Processor -- start 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0cb70 0x556afdf12d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afde74020 con 0x556afdf0cb70 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdb47430 con 0x556afdf0c250 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdb03e00 con 0x556afdf15750 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.670+0000 7f2212599740 10 client.0 _ll_register_callbacks cb 0x556afde1bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49714/0 (socket says 172.21.15.45:49714) 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 -- 172.21.15.45:0/163238343 learned_addr learned my addr 172.21.15.45:0/163238343 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220d763640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0cb70 0x556afdf12d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.667 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 -- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 msgr2=0x556afdf13290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf13290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 -- 172.21.15.45:0/163238343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0cb70 msgr2=0x556afdf12d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0cb70 0x556afdf12d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 -- 172.21.15.45:0/163238343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220400a960 con 0x556afdf0c250 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.671+0000 7f220cf62640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f21f800aa00 tx=0x7f21f800a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d56010e5d680f9e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.672+0000 7f21f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.672+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8004050 con 0x556afdf0c250 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.672+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f21f80041b0 con 0x556afdf0c250 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.672+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556afdb737e0 con 0x556afdf0c250 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.673+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f8022080 con 0x556afdf0c250 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.673+0000 7f2212599740 10 client.28160 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.668 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.673+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x556afde726a0 con 0x556afdf0c250 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.674+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556afdb737e0 con 0x556afdf0c250 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f21f8003d60 con 0x556afdf0c250 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 10 client.28160 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 10 client.28160 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 20 client.28160 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf25b90 0x556afdf27fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 4 client.28160 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556afdbb0720 con 0x556afdf25b90 2024-03-12T05:23:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.675+0000 7f220d763640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf25b90 0x556afdf27fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.670 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.676+0000 7f220d763640 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf25b90 0x556afdf27fa0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=194619efad89aaf4 server_cookie=ab04874340891efa in_seq=0 out_seq=0 2024-03-12T05:23:36.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.676+0000 7f21f0ff9640 10 client.28160 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f21f0ff9640 1 -- 172.21.15.45:0/163238343 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556afdbb0720 con 0x556afdf25b90 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f21f0ff9640 10 client.28160 handle_command_reply: tid=0 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 client.28160 shutdown 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 msgr2=0x556afdf12810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f21f800aa00 tx=0x7f21f800a730 comp rx=0 tx=0).stop 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 shutdown_connections 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0cb70 0x556afdf12d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf25b90 0x556afdf27fa0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 --2- 172.21.15.45:0/163238343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0c250 0x556afdf12810 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 >> 172.21.15.45:0/163238343 conn(0x556afdf06980 msgr2=0x556afdf086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.677+0000 7f21f0ff9640 0 client.28160 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.678+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 shutdown_connections 2024-03-12T05:23:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.678+0000 7f2212599740 1 -- 172.21.15.45:0/163238343 wait complete. 2024-03-12T05:23:36.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.678+0000 7f2212599740 20 client.28160 trim_cache size 0 max 16384 2024-03-12T05:23:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.705+0000 7f2212599740 1 Processor -- start 2024-03-12T05:23:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f2212599740 1 -- start start 2024-03-12T05:23:36.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f2212599740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf14f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f2212599740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf12d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f2212599740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf12d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f220cf62640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf14f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf12d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49718/0 (socket says 172.21.15.45:49718) 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f220d763640 1 -- 172.21.15.45:0/4239810548 learned_addr learned my addr 172.21.15.45:0/4239810548 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.706+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afddf9130 con 0x556afdf0c250 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdf0b9f0 con 0x556afdf0cb70 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdf0beb0 con 0x556afdf15750 2024-03-12T05:23:36.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 -- 172.21.15.45:0/4239810548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 msgr2=0x556afdf132c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf132c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 -- 172.21.15.45:0/4239810548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 msgr2=0x556afdf12d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf12d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 -- 172.21.15.45:0/4239810548 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556afde0ec20 con 0x556afdf0c250 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.707+0000 7f220cf62640 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf14f40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21f80058c0 tx=0x7f21f8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a14d784147d4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.708+0000 7f21f2ffd640 1 -- 172.21.15.45:0/4239810548 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f800e740 con 0x556afdf0c250 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.708+0000 7f21f2ffd640 1 -- 172.21.15.45:0/4239810548 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f21f800a2b0 con 0x556afdf0c250 2024-03-12T05:23:36.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.708+0000 7f21f2ffd640 1 -- 172.21.15.45:0/4239810548 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f800aee0 con 0x556afdf0c250 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 msgr2=0x556afdf14f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf14f40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21f80058c0 tx=0x7f21f8005850 comp rx=0 tx=0).stop 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 shutdown_connections 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf12d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 --2- 172.21.15.45:0/4239810548 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf14f40 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 >> 172.21.15.45:0/4239810548 conn(0x556afdf06980 msgr2=0x556afdf0db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 shutdown_connections 2024-03-12T05:23:36.704 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.709+0000 7f2212599740 1 -- 172.21.15.45:0/4239810548 wait complete. 2024-03-12T05:23:36.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.710+0000 7f2212599740 1 -- start start 2024-03-12T05:23:36.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.710+0000 7f2212599740 1 Processor -- start 2024-03-12T05:23:36.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf1a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf7b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdf7b5d0 con 0x556afdf0c250 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdf7b710 con 0x556afdf0cb70 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556afdf7b850 con 0x556afdf15750 2024-03-12T05:23:36.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.711+0000 7f2212599740 10 client.0 _ll_register_callbacks cb 0x7f220400a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49724/0 (socket says 172.21.15.45:49724) 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 -- 172.21.15.45:0/311426149 learned_addr learned my addr 172.21.15.45:0/311426149 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220cf62640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf7b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 -- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 msgr2=0x556afdf7b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf7b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 -- 172.21.15.45:0/311426149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 msgr2=0x556afdf1a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf1a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.712+0000 7f220d763640 1 -- 172.21.15.45:0/311426149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556afde726a0 con 0x556afdf0cb70 2024-03-12T05:23:36.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f220d763640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2204000c00 tx=0x7f220400a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6006c109b6bd53fd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f21f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204028070 con 0x556afdf0cb70 2024-03-12T05:23:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2204018850 con 0x556afdf0cb70 2024-03-12T05:23:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556afdf28370 con 0x556afdf0cb70 2024-03-12T05:23:36.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2204005040 con 0x556afdf0cb70 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f2212599740 10 client.28166 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.713+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x556afde348c0 con 0x556afdf0cb70 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x556afdf28370 con 0x556afdf0cb70 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2204010bc0 con 0x556afdf0cb70 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 10 client.28166 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 10 client.28166 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 20 client.28166 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf7cfc0 0x556afdf7f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 4 client.28166 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f2204018af0 con 0x556afdf7cfc0 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.714+0000 7f220cf62640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf7cfc0 0x556afdf7f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.715+0000 7f21f0ff9640 10 client.28166 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f220cf62640 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf7cfc0 0x556afdf7f3e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41c258c3f59afd0f server_cookie=96b42c09c2a03f05 in_seq=0 out_seq=0 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f21f0ff9640 1 -- 172.21.15.45:0/311426149 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f2204018af0 con 0x556afdf7cfc0 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f21f0ff9640 10 client.28166 handle_command_reply: tid=0 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f2212599740 1 client.28166 shutdown 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 msgr2=0x556afdf1ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.716+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2204000c00 tx=0x7f220400a360 comp rx=0 tx=0).stop 2024-03-12T05:23:36.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 shutdown_connections 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x556afdf7cfc0 0x556afdf7f3e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556afdf15750 0x556afdf7b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556afdf0cb70 0x556afdf1ad10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 --2- 172.21.15.45:0/311426149 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556afdf0c250 0x556afdf1a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f21f0ff9640 0 client.28166 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 >> 172.21.15.45:0/311426149 conn(0x556afdf06980 msgr2=0x556afdf110c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 shutdown_connections 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 1 -- 172.21.15.45:0/311426149 wait complete. 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.717+0000 7f2212599740 20 client.28166 trim_cache size 0 max 16384 2024-03-12T05:23:36.713 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 msgr2=0x7f21f40a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 0x7f21f40a41f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f22000b8b90 tx=0x7f22000b82a0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 msgr2=0x7f22081a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2200037040 tx=0x7f2200034460 comp rx=0 tx=0).stop 2024-03-12T05:23:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 shutdown_connections 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f21f40a1d30 0x7f21f40a41f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2208112a70 0x7f22081a47a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f220810cd10 0x7f22081a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 --2- 172.21.15.45:0/1716552017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2208108a10 0x7f220819fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 >> 172.21.15.45:0/1716552017 conn(0x7f220806c6c0 msgr2=0x7f22081139e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 shutdown_connections 2024-03-12T05:23:36.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.718+0000 7f220f9c7640 1 -- 172.21.15.45:0/1716552017 wait complete. 2024-03-12T05:23:36.730 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:23:36.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 -- 172.21.15.45:0/3601804957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073550 msgr2=0x7fb464073930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 --2- 172.21.15.45:0/3601804957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073550 0x7fb464073930 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb460009ca0 tx=0x7fb46002c180 comp rx=0 tx=0).stop 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 -- 172.21.15.45:0/3601804957 shutdown_connections 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 --2- 172.21.15.45:0/3601804957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 --2- 172.21.15.45:0/3601804957 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073e70 0x7fb46410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 --2- 172.21.15.45:0/3601804957 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073550 0x7fb464073930 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 -- 172.21.15.45:0/3601804957 >> 172.21.15.45:0/3601804957 conn(0x7fb4640fcbb0 msgr2=0x7fb4640fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 -- 172.21.15.45:0/3601804957 shutdown_connections 2024-03-12T05:23:36.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.912+0000 7fb46bf63640 1 -- 172.21.15.45:0/3601804957 wait complete. 2024-03-12T05:23:36.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.913+0000 7fb46bf63640 1 Processor -- start 2024-03-12T05:23:36.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.913+0000 7fb46bf63640 1 -- start start 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.914+0000 7fb46bf63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073550 0x7fb46410d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.914+0000 7fb46bf63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.914+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47678/0 (socket says 172.21.15.45:47678) 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb4694fe640 1 -- 172.21.15.45:0/1467961772 learned_addr learned my addr 172.21.15.45:0/1467961772 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.914+0000 7fb469cff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073550 0x7fb46410d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46410b040 con 0x7fb464073e70 2024-03-12T05:23:36.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46410b1b0 con 0x7fb46410d9c0 2024-03-12T05:23:36.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.915+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4641aa220 con 0x7fb464073550 2024-03-12T05:23:36.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb46a500640 1 -- 172.21.15.45:0/1467961772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073550 msgr2=0x7fb46410d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073550 0x7fb46410d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb46a500640 1 -- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 msgr2=0x7fb46410a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.911 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb46a500640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460009900 con 0x7fb46410d9c0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.916+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410a8e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00e940 tx=0x7fb45c00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb45affd640 1 -- 172.21.15.45:0/1467961772 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c009620 con 0x7fb46410d9c0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb45affd640 1 -- 172.21.15.45:0/1467961772 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb45c0044d0 con 0x7fb46410d9c0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb45affd640 1 -- 172.21.15.45:0/1467961772 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c004a00 con 0x7fb46410d9c0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4641aa4d0 con 0x7fb46410d9c0 2024-03-12T05:23:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.917+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4641aa990 con 0x7fb46410d9c0 2024-03-12T05:23:36.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.921+0000 7fb45affd640 1 -- 172.21.15.45:0/1467961772 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb45c00ccc0 con 0x7fb46410d9c0 2024-03-12T05:23:36.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.921+0000 7fb46ec0b740 1 Processor -- start 2024-03-12T05:23:36.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.921+0000 7fb46ec0b740 1 -- start start 2024-03-12T05:23:36.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162250 0x5607ff162630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 0x5607ff16db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb45affd640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 0x7fb4500a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb469cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb469cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49754/0 (socket says 172.21.15.45:49754) 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb469cff640 1 -- 172.21.15.45:0/1566446139 learned_addr learned my addr 172.21.15.45:0/1566446139 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb45affd640 1 -- 172.21.15.45:0/1467961772 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb45c014070 con 0x7fb46410d9c0 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff11d960 con 0x5607ff162250 2024-03-12T05:23:36.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162250 0x5607ff162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff0d1310 con 0x5607ff162b70 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff0ce840 con 0x5607ff16b750 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.922+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 0x7fb4500a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 0x5607ff16db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 -- 172.21.15.45:0/1566446139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 msgr2=0x5607ff16db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 0x5607ff16db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 -- 172.21.15.45:0/1566446139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162250 msgr2=0x5607ff162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162250 0x5607ff162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 -- 172.21.15.45:0/1566446139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ff08c060 con 0x5607ff162b70 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 0x5607ff16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.923+0000 7fb469cff640 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb46002c6c0 tx=0x7fb460005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2999b829afe75cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.924+0000 7fb44ffff640 1 -- 172.21.15.45:0/1566446139 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460034dc0 con 0x5607ff162b70 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.924+0000 7fb44ffff640 1 -- 172.21.15.45:0/1566446139 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb460003970 con 0x5607ff162b70 2024-03-12T05:23:36.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.924+0000 7fb44ffff640 1 -- 172.21.15.45:0/1566446139 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460032a80 con 0x5607ff162b70 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 msgr2=0x5607ff16b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46a500640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 0x7fb4500a41b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0cab60 tx=0x7fb45c00ef30 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb46002c6c0 tx=0x7fb460005f70 comp rx=0 tx=0).stop 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 shutdown_connections 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff16b750 0x5607ff16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff16b210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.920 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1566446139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162250 0x5607ff162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 >> 172.21.15.45:0/1566446139 conn(0x5607ff15c980 msgr2=0x5607ff15cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 shutdown_connections 2024-03-12T05:23:36.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1566446139 wait complete. 2024-03-12T05:23:36.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 -- start start 2024-03-12T05:23:36.921 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.925+0000 7fb46ec0b740 1 Processor -- start 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.926+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.926+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff168cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff1691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51380/0 (socket says 172.21.15.45:51380) 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb469cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff168cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb469cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff168cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49760/0 (socket says 172.21.15.45:49760) 2024-03-12T05:23:36.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb4694fe640 1 -- 172.21.15.45:0/1508213104 learned_addr learned my addr 172.21.15.45:0/1508213104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff0ca020 con 0x5607ff16b750 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fed9d430 con 0x5607ff162b70 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.927+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607fed59e00 con 0x5607ff162250 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff1691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 -- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 msgr2=0x5607ff168cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff168cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.923 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 -- 172.21.15.45:0/1508213104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 msgr2=0x5607ff1691f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff1691f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.928+0000 7fb4694fe640 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464110aa0 con 0x5607ff162250 2024-03-12T05:23:36.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.929+0000 7fb46ec0b740 10 client.0 _ll_register_callbacks cb 0x5607ff071cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.924 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.929+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607fedc97e0 con 0x5607ff162250 2024-03-12T05:23:36.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.930+0000 7fb4694fe640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb454012fc0 tx=0x7fb454005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11a62e63a3c6a704 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.930+0000 7fb44d7fa640 10 client.28150 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.930+0000 7fb44d7fa640 1 -- 172.21.15.45:0/1508213104 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454016070 con 0x5607ff162250 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.930+0000 7fb44d7fa640 1 -- 172.21.15.45:0/1508213104 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb454012110 con 0x5607ff162250 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.931+0000 7fb44d7fa640 1 -- 172.21.15.45:0/1508213104 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb454012310 con 0x5607ff162250 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.931+0000 7fb46ec0b740 10 client.28150 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.926 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.931+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5607ff0c86a0 con 0x5607ff162250 2024-03-12T05:23:36.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.933+0000 7fb44d7fa640 1 -- 172.21.15.45:0/1508213104 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb45400e5c0 con 0x5607ff162250 2024-03-12T05:23:36.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.933+0000 7fb46ec0b740 10 client.28150 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.928 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.933+0000 7fb46ec0b740 10 client.28150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.934+0000 7fb46ec0b740 20 client.28150 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.934+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff17b8b0 0x5607ff17dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.934+0000 7fb46ec0b740 4 client.28150 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.934+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607fee06720 con 0x5607ff17b8b0 2024-03-12T05:23:36.929 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.934+0000 7fb469cff640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff17b8b0 0x5607ff17dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.935+0000 7fb44d7fa640 10 client.28150 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:36.930 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.935+0000 7fb469cff640 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff17b8b0 0x5607ff17dd50 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e55cc5ea461bb10b server_cookie=5dcf06e98e632477 in_seq=0 out_seq=0 2024-03-12T05:23:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.936+0000 7fb44d7fa640 1 -- 172.21.15.45:0/1508213104 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607fee06720 con 0x5607ff17b8b0 2024-03-12T05:23:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.936+0000 7fb44d7fa640 10 client.28150 handle_command_reply: tid=0 2024-03-12T05:23:36.931 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.936+0000 7fb46ec0b740 1 client.28150 shutdown 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 msgr2=0x5607ff168770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb454012fc0 tx=0x7fb454005f90 comp rx=0 tx=0).stop 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 shutdown_connections 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff17b8b0 0x5607ff17dd50 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff1691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb44d7fa640 0 client.28150 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162b70 0x5607ff168cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/1508213104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162250 0x5607ff168770 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 >> 172.21.15.45:0/1508213104 conn(0x5607ff15c980 msgr2=0x5607ff15e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 shutdown_connections 2024-03-12T05:23:36.932 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 1 -- 172.21.15.45:0/1508213104 wait complete. 2024-03-12T05:23:36.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.937+0000 7fb46ec0b740 20 client.28150 trim_cache size 0 max 16384 2024-03-12T05:23:36.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 Processor -- start 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 -- start start 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff16aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162b70 0x5607ff1659f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff165f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff04f130 con 0x5607ff16b750 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff1582b0 con 0x5607ff162250 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb46ec0b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff15dbb0 con 0x5607ff162b70 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff16aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff16aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49776/0 (socket says 172.21.15.45:49776) 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb4694fe640 1 -- 172.21.15.45:0/3979414387 learned_addr learned my addr 172.21.15.45:0/3979414387 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff165f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.959+0000 7fb469cff640 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162b70 0x5607ff1659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 -- 172.21.15.45:0/3979414387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162b70 msgr2=0x5607ff1659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162b70 0x5607ff1659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 -- 172.21.15.45:0/3979414387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 msgr2=0x5607ff16aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff16aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 -- 172.21.15.45:0/3979414387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ff064c20 con 0x5607ff16b750 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff165f30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb454009fd0 tx=0x7fb454017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd93a6b5fb00b5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb44f7fe640 1 -- 172.21.15.45:0/3979414387 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454016070 con 0x5607ff16b750 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb44f7fe640 1 -- 172.21.15.45:0/3979414387 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5607fedc97e0 con 0x5607ff16b750 2024-03-12T05:23:36.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.960+0000 7fb44f7fe640 1 -- 172.21.15.45:0/3979414387 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400e5c0 con 0x5607ff16b750 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.961+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3979414387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 msgr2=0x5607ff165f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.961+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff165f30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb454009fd0 tx=0x7fb454017000 comp rx=0 tx=0).stop 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.961+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3979414387 shutdown_connections 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.961+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff16b750 0x5607ff165f30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.961+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff162b70 0x5607ff1659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3979414387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff16aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3979414387 >> 172.21.15.45:0/3979414387 conn(0x5607ff15c980 msgr2=0x5607ff167040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3979414387 shutdown_connections 2024-03-12T05:23:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3979414387 wait complete. 2024-03-12T05:23:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 -- start start 2024-03-12T05:23:36.958 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.962+0000 7fb46ec0b740 1 Processor -- start 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.963+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.963+0000 7fb46ec0b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162b70 0x5607ff171e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.963+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.963+0000 7fb4694fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49792/0 (socket says 172.21.15.45:49792) 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb469cff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162b70 0x5607ff171e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 -- 172.21.15.45:0/3752616841 learned_addr learned my addr 172.21.15.45:0/3752616841 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff17b8b0 0x5607ff1cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff1cc3c0 con 0x5607ff162b70 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff1cc500 con 0x5607ff162250 2024-03-12T05:23:36.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ff1cc6c0 con 0x5607ff17b8b0 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb46ec0b740 10 client.0 _ll_register_callbacks cb 0x7fb45c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff17b8b0 0x5607ff1cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 -- 172.21.15.45:0/3752616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff17b8b0 msgr2=0x5607ff1cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff17b8b0 0x5607ff1cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 -- 172.21.15.45:0/3752616841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162b70 msgr2=0x5607ff171e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162b70 0x5607ff171e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.964+0000 7fb4694fe640 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607ff0c86a0 con 0x5607ff162250 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb460003970 con 0x5607ff162250 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb4694fe640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb454009550 tx=0x7fb454009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c320ec3e407414a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:36.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb44d7fa640 10 client.28184 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb44d7fa640 1 -- 172.21.15.45:0/3752616841 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454016040 con 0x5607ff162250 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb44d7fa640 1 -- 172.21.15.45:0/3752616841 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb454000e40 con 0x5607ff162250 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb44d7fa640 1 -- 172.21.15.45:0/3752616841 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb454004030 con 0x5607ff162250 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb46ec0b740 10 client.28184 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.965+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5607ff08a8c0 con 0x5607ff162250 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb44d7fa640 1 -- 172.21.15.45:0/3752616841 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb454004190 con 0x5607ff162250 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 10 client.28184 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 10 client.28184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 20 client.28184 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff1d3aa0 0x5607ff1d5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 4 client.28184 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5607fee55890 con 0x5607ff1d3aa0 2024-03-12T05:23:36.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.966+0000 7fb469cff640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff1d3aa0 0x5607ff1d5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb44d7fa640 10 client.28184 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb469cff640 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff1d3aa0 0x5607ff1d5ec0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6a6bb539e11afc6 server_cookie=56b94fe3e5742376 in_seq=0 out_seq=0 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb44d7fa640 1 -- 172.21.15.45:0/3752616841 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5607fee55890 con 0x5607ff1d3aa0 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb44d7fa640 10 client.28184 handle_command_reply: tid=0 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb46ec0b740 1 client.28184 shutdown 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 msgr2=0x5607ff171940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb454009550 tx=0x7fb454009780 comp rx=0 tx=0).stop 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 shutdown_connections 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.967+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5607ff1d3aa0 0x5607ff1d5ec0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5607ff162b70 0x5607ff171e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb44d7fa640 0 client.28184 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5607ff17b8b0 0x5607ff1cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 --2- 172.21.15.45:0/3752616841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607ff162250 0x5607ff171940 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 >> 172.21.15.45:0/3752616841 conn(0x5607ff15c980 msgr2=0x5607ff15ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 shutdown_connections 2024-03-12T05:23:36.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 1 -- 172.21.15.45:0/3752616841 wait complete. 2024-03-12T05:23:36.965 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.968+0000 7fb46ec0b740 20 client.28184 trim_cache size 0 max 16384 2024-03-12T05:23:36.965 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 msgr2=0x7fb4500a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 0x7fb4500a41b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb45c0cab60 tx=0x7fb45c00ef30 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 msgr2=0x7fb46410a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410a8e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00e940 tx=0x7fb45c00ee00 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 shutdown_connections 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb4500a1cf0 0x7fb4500a41b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb464073e70 0x7fb46410a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb46410d9c0 0x7fb46410a8e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 --2- 172.21.15.45:0/1467961772 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb464073550 0x7fb46410d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:36.966 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 >> 172.21.15.45:0/1467961772 conn(0x7fb4640fcbb0 msgr2=0x7fb4640fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:36.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 shutdown_connections 2024-03-12T05:23:36.967 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:36.969+0000 7fb46bf63640 1 -- 172.21.15.45:0/1467961772 wait complete. 2024-03-12T05:23:37.982 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:23:37.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:23:37.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.168+0000 7f13bdcc0640 1 -- 172.21.15.45:0/4175701742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 msgr2=0x7f13b81145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.168+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/4175701742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b81145c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009c10 tx=0x7f13b002bf80 comp rx=0 tx=0).stop 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.169+0000 7f13bdcc0640 1 -- 172.21.15.45:0/4175701742 shutdown_connections 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.169+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/4175701742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.169+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/4175701742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b81145c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.169+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/4175701742 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.169+0000 7f13bdcc0640 1 -- 172.21.15.45:0/4175701742 >> 172.21.15.45:0/4175701742 conn(0x7f13b8076d30 msgr2=0x7f13b8079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.170+0000 7f13bdcc0640 1 -- 172.21.15.45:0/4175701742 shutdown_connections 2024-03-12T05:23:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.170+0000 7f13bdcc0640 1 -- 172.21.15.45:0/4175701742 wait complete. 2024-03-12T05:23:38.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 Processor -- start 2024-03-12T05:23:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 -- start start 2024-03-12T05:23:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8083720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b8083c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b807da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b807e1a0 con 0x7f13b8107fc0 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b807e310 con 0x7f13b8108980 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.171+0000 7f13bdcc0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b807e4d0 con 0x7f13b810d770 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8083720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8083720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37218/0 (socket says 172.21.15.45:37218) 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b77fe640 1 -- 172.21.15.45:0/234996857 learned_addr learned my addr 172.21.15.45:0/234996857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b807da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b8083c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 -- 172.21.15.45:0/234996857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 msgr2=0x7f13b8083c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b8083c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 -- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 msgr2=0x7f13b8083720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8083720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 -- 172.21.15.45:0/234996857 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0009840 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.172+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b807da10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036040 tx=0x7f13b0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.173+0000 7f13b4ff9640 1 -- 172.21.15.45:0/234996857 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b00045f0 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.173+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b81b2d90 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.173+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b81b3160 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.173+0000 7f13b4ff9640 1 -- 172.21.15.45:0/234996857 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f13b0004750 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.173+0000 7f13b4ff9640 1 -- 172.21.15.45:0/234996857 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b003e560 con 0x7f13b810d770 2024-03-12T05:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 Processor -- start 2024-03-12T05:23:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13b4ff9640 1 -- 172.21.15.45:0/234996857 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f13b003e6c0 con 0x7f13b810d770 2024-03-12T05:23:38.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 -- start start 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b7052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b707d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b66af90 con 0x56508b7058b0 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b6684c0 con 0x56508b6fc380 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.176+0000 7f13c096e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b663ca0 con 0x56508b6fcd30 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49298/0 (socket says 172.21.15.45:49298) 2024-03-12T05:23:38.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 -- 172.21.15.45:0/4266912067 learned_addr learned my addr 172.21.15.45:0/4266912067 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b707d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b77fe640 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b7052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b4ff9640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 0x7f13a00a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 -- 172.21.15.45:0/4266912067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 msgr2=0x56508b7052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b7052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 -- 172.21.15.45:0/4266912067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 msgr2=0x56508b707d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b707d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 -- 172.21.15.45:0/4266912067 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508b6260b0 con 0x56508b6fc380 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b4ff9640 1 -- 172.21.15.45:0/234996857 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f13b00eff80 con 0x7f13b810d770 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 0x7f13a00a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.177+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0058c0 tx=0x7f13ac005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be0a7f2ace1e938e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.178+0000 7f139dffb640 1 -- 172.21.15.45:0/4266912067 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004050 con 0x56508b6fc380 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.178+0000 7f139dffb640 1 -- 172.21.15.45:0/4266912067 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f13ac005020 con 0x56508b6fc380 2024-03-12T05:23:38.174 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.178+0000 7f139dffb640 1 -- 172.21.15.45:0/4266912067 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004b00 con 0x56508b6fc380 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 -- 172.21.15.45:0/4266912067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 msgr2=0x56508b6fc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0058c0 tx=0x7f13ac005850 comp rx=0 tx=0).stop 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13b7fff640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 0x7f13a00a4160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f13b002c4c0 tx=0x7f13b003ef40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 -- 172.21.15.45:0/4266912067 shutdown_connections 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b707d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b7052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 --2- 172.21.15.45:0/4266912067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fc380 0x56508b6fc760 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.179+0000 7f13c096e740 1 -- 172.21.15.45:0/4266912067 >> 172.21.15.45:0/4266912067 conn(0x56508b6f6840 msgr2=0x56508b6f6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.180+0000 7f13c096e740 1 -- 172.21.15.45:0/4266912067 shutdown_connections 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.180+0000 7f13c096e740 1 -- 172.21.15.45:0/4266912067 wait complete. 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.180+0000 7f13c096e740 1 -- start start 2024-03-12T05:23:38.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 Processor -- start 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 0x56508b702de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51686/0 (socket says 172.21.15.45:51686) 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 0x56508b702de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 0x56508b702de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49302/0 (socket says 172.21.15.45:49302) 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13b6ffd640 1 -- 172.21.15.45:0/1780973421 learned_addr learned my addr 172.21.15.45:0/1780973421 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b703320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b608f40 con 0x56508b7058b0 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b3374b0 con 0x56508b6fcd30 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.181+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b5e9670 con 0x56508b6fc380 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b703320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 -- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 msgr2=0x56508b702de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 0x56508b702de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 -- 172.21.15.45:0/1780973421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 msgr2=0x56508b703320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b703320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac019040 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13c096e740 10 client.0 _ll_register_callbacks cb 0x56508b656f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56508b63df40 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00a010 tx=0x7f13ac0056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf0f2b269d840e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.182+0000 7f1396ffd640 10 client.28168 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f1396ffd640 1 -- 172.21.15.45:0/1780973421 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004880 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f1396ffd640 1 -- 172.21.15.45:0/1780973421 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f13ac0049e0 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f1396ffd640 1 -- 172.21.15.45:0/1780973421 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13ac004ba0 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f13c096e740 10 client.28168 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x56508b662320 con 0x56508b6fc380 2024-03-12T05:23:38.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f1396ffd640 1 -- 172.21.15.45:0/1780973421 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f13ac01d070 con 0x56508b6fc380 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f13c096e740 10 client.28168 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f13c096e740 10 client.28168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.183+0000 7f13c096e740 20 client.28168 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.184+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b715b60 0x56508b717fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.184+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b715b60 0x56508b717fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.184+0000 7f13c096e740 4 client.28168 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:38.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.184+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56508b65a310 con 0x56508b715b60 2024-03-12T05:23:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.185+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b715b60 0x56508b717fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9523b1f1e062aa7a server_cookie=63b4afec48424cbc in_seq=0 out_seq=0 2024-03-12T05:23:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.185+0000 7f1396ffd640 10 client.28168 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.185+0000 7f1396ffd640 1 -- 172.21.15.45:0/1780973421 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56508b65a310 con 0x56508b715b60 2024-03-12T05:23:38.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.185+0000 7f1396ffd640 10 client.28168 handle_command_reply: tid=0 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 client.28168 shutdown 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 msgr2=0x56508b7028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00a010 tx=0x7f13ac0056f0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 shutdown_connections 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b715b60 0x56508b717fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b7058b0 0x56508b703320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b6fcd30 0x56508b702de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 --2- 172.21.15.45:0/1780973421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fc380 0x56508b7028a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f1396ffd640 0 client.28168 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.186+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 >> 172.21.15.45:0/1780973421 conn(0x56508b6f6840 msgr2=0x56508b6f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.187+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 shutdown_connections 2024-03-12T05:23:38.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.187+0000 7f13c096e740 1 -- 172.21.15.45:0/1780973421 wait complete. 2024-03-12T05:23:38.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.187+0000 7f13c096e740 20 client.28168 trim_cache size 0 max 16384 2024-03-12T05:23:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.214+0000 7f13c096e740 1 Processor -- start 2024-03-12T05:23:38.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.214+0000 7f13c096e740 1 -- start start 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b704fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b7058b0 0x56508b6ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b6f2180 con 0x56508b6fc380 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b708700 con 0x56508b7058b0 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13c096e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b7083f0 con 0x56508b6fcd30 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b704fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51688/0 (socket says 172.21.15.45:51688) 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13b6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b704fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37242/0 (socket says 172.21.15.45:37242) 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.215+0000 7f13b77fe640 1 -- 172.21.15.45:0/1307171725 learned_addr learned my addr 172.21.15.45:0/1307171725 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 -- 172.21.15.45:0/1307171725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b7058b0 msgr2=0x56508b6ffb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b7058b0 0x56508b6ffb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 -- 172.21.15.45:0/1307171725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 msgr2=0x56508b704fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b704fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 -- 172.21.15.45:0/1307171725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508b5fec00 con 0x56508b6fcd30 2024-03-12T05:23:38.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.216+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x56508b6fd130 tx=0x7f13a8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd6e24d89ec6b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.218+0000 7f139d7fa640 1 -- 172.21.15.45:0/1307171725 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8017040 con 0x56508b6fcd30 2024-03-12T05:23:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.218+0000 7f139d7fa640 1 -- 172.21.15.45:0/1307171725 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f13a8018040 con 0x56508b6fcd30 2024-03-12T05:23:38.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.218+0000 7f139d7fa640 1 -- 172.21.15.45:0/1307171725 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8004c00 con 0x56508b6fcd30 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 -- 172.21.15.45:0/1307171725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 msgr2=0x56508b703140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x56508b6fd130 tx=0x7f13a8005f90 comp rx=0 tx=0).stop 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 -- 172.21.15.45:0/1307171725 shutdown_connections 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b7058b0 0x56508b6ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b6fcd30 0x56508b703140 secure :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0x56508b6fd130 tx=0x7f13a8005f90 comp rx=0 tx=0).stop 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 --2- 172.21.15.45:0/1307171725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b704fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 -- 172.21.15.45:0/1307171725 >> 172.21.15.45:0/1307171725 conn(0x56508b6f6840 msgr2=0x56508b701170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 -- 172.21.15.45:0/1307171725 shutdown_connections 2024-03-12T05:23:38.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.219+0000 7f13c096e740 1 -- 172.21.15.45:0/1307171725 wait complete. 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 -- start start 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 Processor -- start 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b766490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 0x56508b70bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b70c4c0 con 0x56508b6fc380 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b70c630 con 0x56508b70bab0 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508b70c7f0 con 0x56508b7058b0 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.220+0000 7f13c096e740 10 client.0 _ll_register_callbacks cb 0x7f13ac009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51694/0 (socket says 172.21.15.45:51694) 2024-03-12T05:23:38.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 -- 172.21.15.45:0/1913644520 learned_addr learned my addr 172.21.15.45:0/1913644520 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 0x56508b70bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b766490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 -- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 msgr2=0x56508b70bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 0x56508b70bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 -- 172.21.15.45:0/1913644520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 msgr2=0x56508b766490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b766490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b77fe640 1 -- 172.21.15.45:0/1913644520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a8009bf0 con 0x56508b7058b0 2024-03-12T05:23:38.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.221+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 0x56508b70bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f13b77fe640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f13a8000c00 tx=0x7f13a80044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2df0c9257e8907c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f1396ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8005220 con 0x56508b7058b0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f13a8005380 con 0x56508b7058b0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56508b689900 con 0x56508b7058b0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13a8005580 con 0x56508b7058b0 2024-03-12T05:23:38.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f13c096e740 10 client.28174 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.222+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56508b624910 con 0x56508b7058b0 2024-03-12T05:23:38.219 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x56508b689900 con 0x56508b7058b0 2024-03-12T05:23:38.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f13a801f360 con 0x56508b7058b0 2024-03-12T05:23:38.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 10 client.28174 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:38.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 10 client.28174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:38.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 20 client.28174 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:38.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b76e610 0x56508b770a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 4 client.28174 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:23:38.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56508b363860 con 0x56508b76e610 2024-03-12T05:23:38.222 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.223+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b76e610 0x56508b770a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.224+0000 7f13b6ffd640 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b76e610 0x56508b770a30 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666a4785fd53ae7 server_cookie=6931761237b90b59 in_seq=0 out_seq=0 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.224+0000 7f1396ffd640 10 client.28174 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.227+0000 7f1396ffd640 1 -- 172.21.15.45:0/1913644520 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56508b363860 con 0x56508b76e610 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.227+0000 7f1396ffd640 10 client.28174 handle_command_reply: tid=0 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.227+0000 7f13c096e740 1 client.28174 shutdown 2024-03-12T05:23:38.223 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 msgr2=0x56508b7669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f13a8000c00 tx=0x7f13a80044d0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 shutdown_connections 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x56508b76e610 0x56508b770a30 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56508b70bab0 0x56508b70bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56508b7058b0 0x56508b7669d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 --2- 172.21.15.45:0/1913644520 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56508b6fc380 0x56508b766490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 >> 172.21.15.45:0/1913644520 conn(0x56508b6f6840 msgr2=0x56508b6f5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f1396ffd640 0 client.28174 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 shutdown_connections 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 1 -- 172.21.15.45:0/1913644520 wait complete. 2024-03-12T05:23:38.224 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.228+0000 7f13c096e740 20 client.28174 trim_cache size 0 max 16384 2024-03-12T05:23:38.225 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:38.225 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:23:38.225 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "e2bf173b-a131-464a-8a5d-f45ed06a9abc", 2024-03-12T05:23:38.225 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:23:38.225 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:38.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.233+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 msgr2=0x7f13a00a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.233+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 0x7f13a00a4160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f13b002c4c0 tx=0x7f13b003ef40 comp rx=0 tx=0).stop 2024-03-12T05:23:38.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.233+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 msgr2=0x7f13b807da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.233+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b807da10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f13b0036040 tx=0x7f13b0004270 comp rx=0 tx=0).stop 2024-03-12T05:23:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 shutdown_connections 2024-03-12T05:23:38.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f13a00a1ca0 0x7f13a00a4160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13b810d770 0x7f13b807da10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b8108980 0x7f13b8083c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 --2- 172.21.15.45:0/234996857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13b8107fc0 0x7f13b8083720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.234+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 >> 172.21.15.45:0/234996857 conn(0x7f13b8076d30 msgr2=0x7f13b80796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.235+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 shutdown_connections 2024-03-12T05:23:38.231 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.237+0000 7f13bdcc0640 1 -- 172.21.15.45:0/234996857 wait complete. 2024-03-12T05:23:38.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:23:38.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:23:38.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:38 smithi045 ceph-mon[32174]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 9.4 MiB/s wr, 1.08k op/s 2024-03-12T05:23:38.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.455+0000 7f45127fc640 1 -- 172.21.15.45:0/2184247239 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504032340 con 0x7f45141023a0 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 -- 172.21.15.45:0/2184247239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45141023a0 msgr2=0x7f4514102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 --2- 172.21.15.45:0/2184247239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45141023a0 0x7f4514102780 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4504009860 tx=0x7f450402bc00 comp rx=0 tx=0).stop 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 -- 172.21.15.45:0/2184247239 shutdown_connections 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 --2- 172.21.15.45:0/2184247239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f451410b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 --2- 172.21.15.45:0/2184247239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f451410b6e0 0x7f451410db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 --2- 172.21.15.45:0/2184247239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45141023a0 0x7f4514102780 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.456+0000 7f451a08d640 1 -- 172.21.15.45:0/2184247239 >> 172.21.15.45:0/2184247239 conn(0x7f4514077230 msgr2=0x7f4514077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.457+0000 7f451a08d640 1 -- 172.21.15.45:0/2184247239 shutdown_connections 2024-03-12T05:23:38.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.457+0000 7f451a08d640 1 -- 172.21.15.45:0/2184247239 wait complete. 2024-03-12T05:23:38.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.457+0000 7f451a08d640 1 Processor -- start 2024-03-12T05:23:38.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 -- start start 2024-03-12T05:23:38.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f4514100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 0x7f4514100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514101f40 con 0x7f4514102cc0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141aa090 con 0x7f45141023a0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f451a08d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141aa1d0 con 0x7f451410b6e0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f45137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f4514100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f45137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:49332/0 (socket says 172.21.15.45:49332) 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f45137fe640 1 -- 172.21.15.45:0/1050053991 learned_addr learned my addr 172.21.15.45:0/1050053991 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.458+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 0x7f4514100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 -- 172.21.15.45:0/1050053991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 msgr2=0x7f45140ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 -- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 msgr2=0x7f4514100530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f4514100530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 -- 172.21.15.45:0/1050053991 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504009490 con 0x7f451410b6e0 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f45137fe640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f4514100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 0x7f4514100a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f450c009fd0 tx=0x7f450c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4510ff9640 1 -- 172.21.15.45:0/1050053991 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c00cd60 con 0x7f451410b6e0 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4510ff9640 1 -- 172.21.15.45:0/1050053991 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f450c005020 con 0x7f451410b6e0 2024-03-12T05:23:38.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f4510ff9640 1 -- 172.21.15.45:0/1050053991 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c010690 con 0x7f451410b6e0 2024-03-12T05:23:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.459+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45141aa4b0 con 0x7f451410b6e0 2024-03-12T05:23:38.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.460+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45141aa970 con 0x7f451410b6e0 2024-03-12T05:23:38.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.463+0000 7f4510ff9640 1 -- 172.21.15.45:0/1050053991 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f450c0107f0 con 0x7f451410b6e0 2024-03-12T05:23:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.463+0000 7f451cc5a740 1 Processor -- start 2024-03-12T05:23:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.463+0000 7f451cc5a740 1 -- start start 2024-03-12T05:23:38.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.463+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b4570f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b4571aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f451cc5a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4567e280 con 0x564b4570f2b0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f451cc5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4567b7b0 con 0x564b457187b0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f451cc5a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b45676f90 con 0x564b4570fbd0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4510ff9640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 0x7f44fc0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4510ff9640 1 -- 172.21.15.45:0/1050053991 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f450c014070 con 0x7f451410b6e0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 0x7f44fc0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51720/0 (socket says 172.21.15.45:51720) 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 -- 172.21.15.45:0/1013660459 learned_addr learned my addr 172.21.15.45:0/1013660459 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b4571aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b4570f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 -- 172.21.15.45:0/1013660459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 msgr2=0x564b4571aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b4571aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 -- 172.21.15.45:0/1013660459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 msgr2=0x564b4570f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b4570f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f45137fe640 1 -- 172.21.15.45:0/1013660459 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b45635570 con 0x564b4570fbd0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.464+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b4571aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.465+0000 7f4513fff640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 0x7f44fc0a4160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f450c096e90 tx=0x7f450c005f70 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.465+0000 7f45137fe640 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4504009980 tx=0x7f45040056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36872175692f87b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.465+0000 7f44f9ffb640 1 -- 172.21.15.45:0/1013660459 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504004440 con 0x564b4570fbd0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.465+0000 7f44f9ffb640 1 -- 172.21.15.45:0/1013660459 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45040348f0 con 0x564b4570fbd0 2024-03-12T05:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.465+0000 7f44f9ffb640 1 -- 172.21.15.45:0/1013660459 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504003850 con 0x564b4570fbd0 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 -- 172.21.15.45:0/1013660459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 msgr2=0x564b45718270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4504009980 tx=0x7f45040056d0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 -- 172.21.15.45:0/1013660459 shutdown_connections 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b4571aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b45718270 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1013660459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b4570f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 -- 172.21.15.45:0/1013660459 >> 172.21.15.45:0/1013660459 conn(0x564b45709990 msgr2=0x564b45709d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 -- 172.21.15.45:0/1013660459 shutdown_connections 2024-03-12T05:23:38.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.466+0000 7f451cc5a740 1 -- 172.21.15.45:0/1013660459 wait complete. 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 -- start start 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 Processor -- start 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570fbd0 0x564b45715580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b457187b0 0x564b45715cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4534a430 con 0x564b457187b0 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b45306e00 con 0x564b4570fbd0 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b455fbfd0 con 0x564b4570f2b0 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f451cc5a740 10 client.0 _ll_register_callbacks cb 0x564b45651fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.467+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:51734/0 (socket says 172.21.15.45:51734) 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 -- 172.21.15.45:0/2054368979 learned_addr learned my addr 172.21.15.45:0/2054368979 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b457187b0 0x564b45715cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f45137fe640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570fbd0 0x564b45715580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 -- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570fbd0 msgr2=0x564b45715580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570fbd0 0x564b45715580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 -- 172.21.15.45:0/2054368979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b457187b0 msgr2=0x564b45715cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b457187b0 0x564b45715cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 -- 172.21.15.45:0/2054368979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451410e800 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f4512ffd640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4508005fd0 tx=0x7f4508013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a97fa4bd5044f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f44f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508016070 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f450800fe60 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.468+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b453767e0 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4508004250 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 10 client.28186 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564b45675610 con 0x564b4570f2b0 2024-03-12T05:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x564b453767e0 con 0x564b4570f2b0 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4508021450 con 0x564b4570f2b0 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 10 client.28186 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 10 client.28186 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 20 client.28186 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457288f0 0x564b4572ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 4 client.28186 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:23:38.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.469+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b453b3720 con 0x564b457288f0 2024-03-12T05:23:38.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:38 smithi130 ceph-mon[35398]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 9.4 MiB/s wr, 1.08k op/s 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.470+0000 7f45137fe640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457288f0 0x564b4572ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.470+0000 7f44f2ffd640 10 client.28186 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.471+0000 7f45137fe640 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457288f0 0x564b4572ad00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c8886d73c833a5 server_cookie=e6668b3912e6ecf in_seq=0 out_seq=0 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f44f2ffd640 1 -- 172.21.15.45:0/2054368979 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564b453b3720 con 0x564b457288f0 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f44f2ffd640 10 client.28186 handle_command_reply: tid=0 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 client.28186 shutdown 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 msgr2=0x564b457181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4508005fd0 tx=0x7f4508013040 comp rx=0 tx=0).stop 2024-03-12T05:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 shutdown_connections 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457288f0 0x564b4572ad00 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b457187b0 0x564b45715cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570fbd0 0x564b45715580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 --2- 172.21.15.45:0/2054368979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570f2b0 0x564b457181f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 >> 172.21.15.45:0/2054368979 conn(0x564b45709990 msgr2=0x564b4570b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.472+0000 7f44f2ffd640 0 client.28186 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.473+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 shutdown_connections 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.473+0000 7f451cc5a740 1 -- 172.21.15.45:0/2054368979 wait complete. 2024-03-12T05:23:38.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.473+0000 7f451cc5a740 20 client.28186 trim_cache size 0 max 16384 2024-03-12T05:23:38.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:38 smithi116 ceph-mon[35486]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 9.4 MiB/s wr, 1.08k op/s 2024-03-12T05:23:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 Processor -- start 2024-03-12T05:23:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 -- start start 2024-03-12T05:23:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b457181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b45712800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b457166f0 con 0x564b4570f2b0 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4572fce0 con 0x564b457187b0 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f451cc5a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b453b3750 con 0x564b4570fbd0 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37292/0 (socket says 172.21.15.45:37292) 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f4512ffd640 1 -- 172.21.15.45:0/4285387798 learned_addr learned my addr 172.21.15.45:0/4285387798 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.503+0000 7f45137fe640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b457181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f4512ffd640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b45712800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 -- 172.21.15.45:0/4285387798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 msgr2=0x564b45712800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b45712800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 -- 172.21.15.45:0/4285387798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 msgr2=0x564b45717cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 -- 172.21.15.45:0/4285387798 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b45611ac0 con 0x564b4570fbd0 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f4512ffd640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.504+0000 7f45137fe640 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b457181f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4504005e00 tx=0x7f4504041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f164d2ca6287e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.505+0000 7f44f97fa640 1 -- 172.21.15.45:0/4285387798 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450402c7a0 con 0x564b4570fbd0 2024-03-12T05:23:38.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.505+0000 7f44f97fa640 1 -- 172.21.15.45:0/4285387798 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f450402c990 con 0x564b4570fbd0 2024-03-12T05:23:38.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.505+0000 7f44f97fa640 1 -- 172.21.15.45:0/4285387798 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504056460 con 0x564b4570fbd0 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 -- 172.21.15.45:0/4285387798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 msgr2=0x564b457181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b457181f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4504005e00 tx=0x7f4504041040 comp rx=0 tx=0).stop 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 -- 172.21.15.45:0/4285387798 shutdown_connections 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b457187b0 0x564b45712800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b4570fbd0 0x564b457181f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 --2- 172.21.15.45:0/4285387798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570f2b0 0x564b45717cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 -- 172.21.15.45:0/4285387798 >> 172.21.15.45:0/4285387798 conn(0x564b45709990 msgr2=0x564b45713e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 -- 172.21.15.45:0/4285387798 shutdown_connections 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.506+0000 7f451cc5a740 1 -- 172.21.15.45:0/4285387798 wait complete. 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.507+0000 7f451cc5a740 1 -- start start 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.507+0000 7f451cc5a740 1 Processor -- start 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.507+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 0x564b4571c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.507+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 0x564b4571c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f451cc5a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b457187b0 0x564b4571cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f451cc5a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4577ee20 con 0x564b4570fbd0 2024-03-12T05:23:38.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f45137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 0x564b4571c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 0x564b4571c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f45137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 0x564b4571c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37308/0 (socket says 172.21.15.45:37308) 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f45137fe640 1 -- 172.21.15.45:0/1116346756 learned_addr learned my addr 172.21.15.45:0/1116346756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f451cc5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4577ef90 con 0x564b4570f2b0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b4577f150 con 0x564b457187b0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f451cc5a740 10 client.0 _ll_register_callbacks cb 0x7f4508005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b457187b0 0x564b4571cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 -- 172.21.15.45:0/1116346756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b457187b0 msgr2=0x564b4571cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b457187b0 0x564b4571cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 -- 172.21.15.45:0/1116346756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 msgr2=0x564b4571c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 0x564b4571c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.508+0000 7f4512ffd640 1 -- 172.21.15.45:0/1116346756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b45675610 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.509+0000 7f4512ffd640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 0x564b4571c190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4508013010 tx=0x7f4508009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ffb69d405f2456 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.509+0000 7f44f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.509+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508016030 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.509+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4508000e40 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.509+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45040348f0 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4508004040 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 10 client.28217 fetch_fsmap learned FSMap version 33 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564b45635570 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f45040348f0 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f45080047a0 con 0x564b4570f2b0 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 10 client.28217 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 10 client.28217 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:23:38.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 20 client.28217 populate_metadata read hostname 'smithi045' 2024-03-12T05:23:38.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457807a0 0x564b45782bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:23:38.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 4 client.28217 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:23:38.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.510+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564b45402890 con 0x564b457807a0 2024-03-12T05:23:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.511+0000 7f45137fe640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457807a0 0x564b45782bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:23:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.511+0000 7f45137fe640 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457807a0 0x564b45782bc0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4feee05fd7df5457 server_cookie=26b528f5e4542277 in_seq=0 out_seq=0 2024-03-12T05:23:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.511+0000 7f44f2ffd640 10 client.28217 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.511+0000 7f44f2ffd640 1 -- 172.21.15.45:0/1116346756 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564b45402890 con 0x564b457807a0 2024-03-12T05:23:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.511+0000 7f44f2ffd640 10 client.28217 handle_command_reply: tid=0 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 client.28217 shutdown 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 msgr2=0x564b4571c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 0x564b4571c190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4508013010 tx=0x7f4508009fb0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 shutdown_connections 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564b457807a0 0x564b45782bc0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564b4570fbd0 0x564b4571c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b457187b0 0x564b4571cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 --2- 172.21.15.45:0/1116346756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564b4570f2b0 0x564b4571c190 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 >> 172.21.15.45:0/1116346756 conn(0x564b45709990 msgr2=0x564b45714ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f44f2ffd640 0 client.28217 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 shutdown_connections 2024-03-12T05:23:38.509 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 1 -- 172.21.15.45:0/1116346756 wait complete. 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.512+0000 7f451cc5a740 20 client.28217 trim_cache size 0 max 16384 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "e2bf173b-a131-464a-8a5d-f45ed06a9abc": { 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "e2bf173b-a131-464a-8a5d-f45ed06a9abc", 2024-03-12T05:23:38.510 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:23:38.511 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:23:38.511 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:23:38.511 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:23:38.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.513+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 msgr2=0x7f44fc0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.513+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 0x7f44fc0a4160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f450c096e90 tx=0x7f450c005f70 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.513+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 msgr2=0x7f4514100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.513+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 0x7f4514100a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f450c009fd0 tx=0x7f450c00ee50 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 shutdown_connections 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f44fc0a1ca0 0x7f44fc0a4160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4514102cc0 0x7f4514100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f451410b6e0 0x7f4514100a70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 --2- 172.21.15.45:0/1050053991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45141023a0 0x7f45140ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 >> 172.21.15.45:0/1050053991 conn(0x7f4514077230 msgr2=0x7f45140fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 shutdown_connections 2024-03-12T05:23:38.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:23:38.514+0000 7f451a08d640 1 -- 172.21.15.45:0/1050053991 wait complete. 2024-03-12T05:23:38.526 INFO:tasks.cephfs.filesystem:scrub status for tag:e2bf173b-a131-464a-8a5d-f45ed06a9abc - {'path': '/', 'tag': 'e2bf173b-a131-464a-8a5d-f45ed06a9abc', 'options': 'recursive,force'} 2024-03-12T05:23:39.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:23:39.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:39 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.492 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:39 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:23:39.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:39 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:23:39.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 695.0 s, 1343.6 tps, lat 74.311 ms stddev 23.308 2024-03-12T05:23:40.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:40 smithi130 ceph-mon[35398]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:23:40.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:40 smithi116 ceph-mon[35486]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:23:40.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:40 smithi045 ceph-mon[32174]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:23:42.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:42 smithi130 ceph-mon[35398]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:23:42.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:42 smithi116 ceph-mon[35486]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:23:42.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:42 smithi045 ceph-mon[32174]: pgmap v401: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-03-12T05:23:44.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:44 smithi130 ceph-mon[35398]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 9.5 MiB/s wr, 1.05k op/s 2024-03-12T05:23:44.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:44 smithi116 ceph-mon[35486]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 9.5 MiB/s wr, 1.05k op/s 2024-03-12T05:23:44.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:44 smithi045 ceph-mon[32174]: pgmap v402: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 9.5 MiB/s wr, 1.05k op/s 2024-03-12T05:23:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 700.0 s, 1352.0 tps, lat 73.939 ms stddev 27.474 2024-03-12T05:23:46.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:46 smithi130 ceph-mon[35398]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:23:46.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:46 smithi116 ceph-mon[35486]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:23:46.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:46 smithi045 ceph-mon[32174]: pgmap v403: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 14 MiB/s wr, 1.61k op/s 2024-03-12T05:23:48.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:48 smithi130 ceph-mon[35398]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 9.6 MiB/s wr, 1.06k op/s 2024-03-12T05:23:48.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:48 smithi116 ceph-mon[35486]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 9.6 MiB/s wr, 1.06k op/s 2024-03-12T05:23:48.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:48 smithi045 ceph-mon[32174]: pgmap v404: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 9.6 MiB/s wr, 1.06k op/s 2024-03-12T05:23:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 705.0 s, 1358.2 tps, lat 73.693 ms stddev 25.601 2024-03-12T05:23:50.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:50 smithi130 ceph-mon[35398]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:23:50.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:50 smithi116 ceph-mon[35486]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:23:50.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:50 smithi045 ceph-mon[32174]: pgmap v405: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-03-12T05:23:52.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:52 smithi130 ceph-mon[35398]: pgmap v406: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:23:52.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:52 smithi116 ceph-mon[35486]: pgmap v406: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:23:52.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:52 smithi045 ceph-mon[32174]: pgmap v406: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 13 MiB/s wr, 1.48k op/s 2024-03-12T05:23:53.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:53.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:53.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:53 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:53.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:53.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:53.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:53 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:53.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:53.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:53.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:53 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:54.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:54 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:23:54.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:54 smithi116 ceph-mon[35486]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:23:54.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:54 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:23:54.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:54 smithi045 ceph-mon[32174]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:23:54.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:54 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:23:54.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:54 smithi130 ceph-mon[35398]: pgmap v407: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 9.6 MiB/s wr, 1.11k op/s 2024-03-12T05:23:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 710.0 s, 1308.6 tps, lat 76.514 ms stddev 34.075 2024-03-12T05:23:56.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:56 smithi116 ceph-mon[35486]: pgmap v408: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:23:56.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:56 smithi045 ceph-mon[32174]: pgmap v408: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:23:56.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:56 smithi130 ceph-mon[35398]: pgmap v408: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 14 MiB/s wr, 1.65k op/s 2024-03-12T05:23:58.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:23:58 smithi116 ceph-mon[35486]: pgmap v409: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 9.6 MiB/s wr, 1.09k op/s 2024-03-12T05:23:58.616 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:23:58 smithi045 ceph-mon[32174]: pgmap v409: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 9.6 MiB/s wr, 1.09k op/s 2024-03-12T05:23:58.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:23:58 smithi130 ceph-mon[35398]: pgmap v409: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 9.6 MiB/s wr, 1.09k op/s 2024-03-12T05:23:59.950 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 715.0 s, 1411.6 tps, lat 70.640 ms stddev 27.259 2024-03-12T05:24:00.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:00 smithi116 ceph-mon[35486]: pgmap v410: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:24:00.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:00 smithi045 ceph-mon[32174]: pgmap v410: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:24:00.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:00 smithi130 ceph-mon[35398]: pgmap v410: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:24:02.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:02 smithi116 ceph-mon[35486]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:02.517 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:02 smithi130 ceph-mon[35398]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:02.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:02 smithi045 ceph-mon[32174]: pgmap v411: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:04.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:04 smithi116 ceph-mon[35486]: pgmap v412: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 9.8 MiB/s wr, 1.12k op/s 2024-03-12T05:24:04.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:04 smithi045 ceph-mon[32174]: pgmap v412: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 9.8 MiB/s wr, 1.12k op/s 2024-03-12T05:24:04.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:04 smithi130 ceph-mon[35398]: pgmap v412: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 9.8 MiB/s wr, 1.12k op/s 2024-03-12T05:24:04.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 720.0 s, 1267.1 tps, lat 79.152 ms stddev 41.090 2024-03-12T05:24:06.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:06 smithi116 ceph-mon[35486]: pgmap v413: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:24:06.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:06 smithi045 ceph-mon[32174]: pgmap v413: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:24:06.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:06 smithi130 ceph-mon[35398]: pgmap v413: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 14 MiB/s wr, 1.64k op/s 2024-03-12T05:24:08.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:08 smithi116 ceph-mon[35486]: pgmap v414: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 9.4 MiB/s wr, 1.10k op/s 2024-03-12T05:24:08.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:24:08.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:24:08.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:08 smithi045 ceph-mon[32174]: pgmap v414: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 9.4 MiB/s wr, 1.10k op/s 2024-03-12T05:24:08.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:08 smithi130 ceph-mon[35398]: pgmap v414: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 9.4 MiB/s wr, 1.10k op/s 2024-03-12T05:24:08.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.750+0000 7f2e600d0640 1 -- 172.21.15.45:0/650260349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810d260 msgr2=0x7f2e5810d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.750+0000 7f2e600d0640 1 --2- 172.21.15.45:0/650260349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810d260 0x7f2e5810d6e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2e5401a940 tx=0x7f2e5403cc80 comp rx=0 tx=0).stop 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.751+0000 7f2e600d0640 1 -- 172.21.15.45:0/650260349 shutdown_connections 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.751+0000 7f2e600d0640 1 --2- 172.21.15.45:0/650260349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810d260 0x7f2e5810d6e0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.751+0000 7f2e600d0640 1 --2- 172.21.15.45:0/650260349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5810de10 0x7f2e5811c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.751+0000 7f2e600d0640 1 --2- 172.21.15.45:0/650260349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e5810fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.751+0000 7f2e600d0640 1 -- 172.21.15.45:0/650260349 >> 172.21.15.45:0/650260349 conn(0x7f2e5806c060 msgr2=0x7f2e5806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.757+0000 7f2e600d0640 1 -- 172.21.15.45:0/650260349 shutdown_connections 2024-03-12T05:24:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.760+0000 7f2e600d0640 1 -- 172.21.15.45:0/650260349 wait complete. 2024-03-12T05:24:08.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.761+0000 7f2e600d0640 1 Processor -- start 2024-03-12T05:24:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 -- start start 2024-03-12T05:24:08.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5810d260 0x7f2e58118380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810de10 0x7f2e581188c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581a39e0 con 0x7f2e5810de10 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581a3b50 con 0x7f2e5810d260 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.765+0000 7f2e600d0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581a3d10 con 0x7f2e5810f790 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45452/0 (socket says 172.21.15.45:45452) 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 -- 172.21.15.45:0/3144159970 learned_addr learned my addr 172.21.15.45:0/3144159970 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 -- 172.21.15.45:0/3144159970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5810d260 msgr2=0x7f2e58118380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5810d260 0x7f2e58118380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 -- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810de10 msgr2=0x7f2e581188c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810de10 0x7f2e581188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.766+0000 7f2e5e66d640 1 -- 172.21.15.45:0/3144159970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5401a570 con 0x7f2e5810f790 2024-03-12T05:24:08.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.767+0000 7f2e5e66d640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000cad0 tx=0x7f2e5000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.767+0000 7f2e4effd640 1 -- 172.21.15.45:0/3144159970 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000ae20 con 0x7f2e5810f790 2024-03-12T05:24:08.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.767+0000 7f2e4effd640 1 -- 172.21.15.45:0/3144159970 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e50005020 con 0x7f2e5810f790 2024-03-12T05:24:08.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.767+0000 7f2e4effd640 1 -- 172.21.15.45:0/3144159970 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000f6b0 con 0x7f2e5810f790 2024-03-12T05:24:08.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.768+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e581a3ff0 con 0x7f2e5810f790 2024-03-12T05:24:08.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.769+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e581aa670 con 0x7f2e5810f790 2024-03-12T05:24:08.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.772+0000 7f2e62d77740 1 Processor -- start 2024-03-12T05:24:08.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.776+0000 7f2e4effd640 1 -- 172.21.15.45:0/3144159970 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2e50005190 con 0x7f2e5810f790 2024-03-12T05:24:08.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.776+0000 7f2e62d77740 1 -- start start 2024-03-12T05:24:08.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.776+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee27814490 0x55ee27814870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.776+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.777+0000 7f2e4effd640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 0x7f2e440a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.777+0000 7f2e4effd640 1 -- 172.21.15.45:0/3144159970 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f2e500cdc30 con 0x7f2e5810f790 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2781d990 0x55ee2781fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e62d77740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee27783280 con 0x55ee27814db0 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e62d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee277807b0 con 0x55ee2781d990 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e62d77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee2777bf90 con 0x55ee27814490 2024-03-12T05:24:08.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e5de6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e5de6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58784/0 (socket says 172.21.15.45:58784) 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.778+0000 7f2e5de6c640 1 -- 172.21.15.45:0/866091102 learned_addr learned my addr 172.21.15.45:0/866091102 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5de6c640 1 -- 172.21.15.45:0/866091102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee27814490 msgr2=0x55ee27814870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee27814490 0x55ee27814870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5de6c640 1 -- 172.21.15.45:0/866091102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2781d990 msgr2=0x55ee2781fd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2781d990 0x55ee2781fd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5de6c640 1 -- 172.21.15.45:0/866091102 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee2773a570 con 0x55ee27814db0 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.779+0000 7f2e5e66d640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 0x7f2e440a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.780+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800a980 tx=0x7f2e4800ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9acdc83a0f32a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.781+0000 7f2e5e66d640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 0x7f2e440a4270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000caa0 tx=0x7f2e50005440 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.782+0000 7f2e43fff640 1 -- 172.21.15.45:0/866091102 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4800cd80 con 0x55ee27814db0 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.782+0000 7f2e43fff640 1 -- 172.21.15.45:0/866091102 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e480054d0 con 0x55ee27814db0 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.782+0000 7f2e43fff640 1 -- 172.21.15.45:0/866091102 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4800f450 con 0x55ee27814db0 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.783+0000 7f2e62d77740 1 -- 172.21.15.45:0/866091102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 msgr2=0x55ee2781d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.783+0000 7f2e62d77740 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800a980 tx=0x7f2e4800ae40 comp rx=0 tx=0).stop 2024-03-12T05:24:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 -- 172.21.15.45:0/866091102 shutdown_connections 2024-03-12T05:24:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781d450 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee2781d990 0x55ee2781fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 --2- 172.21.15.45:0/866091102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee27814490 0x55ee27814870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 -- 172.21.15.45:0/866091102 >> 172.21.15.45:0/866091102 conn(0x55ee2780ebc0 msgr2=0x55ee2780efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 -- 172.21.15.45:0/866091102 shutdown_connections 2024-03-12T05:24:08.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.784+0000 7f2e62d77740 1 -- 172.21.15.45:0/866091102 wait complete. 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 -- start start 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 Processor -- start 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee2744f430 con 0x55ee27814db0 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee2740be00 con 0x55ee27814490 2024-03-12T05:24:08.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee27700fd0 con 0x55ee2781d990 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.785+0000 7f2e62d77740 10 client.0 _ll_register_callbacks cb 0x55ee27756fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42328/0 (socket says 172.21.15.45:42328) 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 -- 172.21.15.45:0/1429688838 learned_addr learned my addr 172.21.15.45:0/1429688838 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 -- 172.21.15.45:0/1429688838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 msgr2=0x55ee2781ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 -- 172.21.15.45:0/1429688838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 msgr2=0x55ee2781a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 -- 172.21.15.45:0/1429688838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4800a5b0 con 0x55ee27814490 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.786+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2e5403d970 tx=0x7f2e5403d1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3926b0cfd1aa4a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.787+0000 7f2e417fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.787+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54004260 con 0x55ee27814490 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.787+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f2e540043c0 con 0x55ee27814490 2024-03-12T05:24:08.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.787+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee2747b7e0 con 0x55ee27814490 2024-03-12T05:24:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.788+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e5403d9d0 con 0x55ee27814490 2024-03-12T05:24:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.788+0000 7f2e62d77740 10 client.28220 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.788+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ee2777a610 con 0x55ee27814490 2024-03-12T05:24:08.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e5405e020 con 0x55ee27814490 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2e5403f5f0 con 0x55ee27814490 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 10 client.28220 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 10 client.28220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 20 client.28220 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee2784fcc0 0x55ee278520e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 4 client.28220 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.791+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee274b8720 con 0x55ee2784fcc0 2024-03-12T05:24:08.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.792+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee2784fcc0 0x55ee278520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.812+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee2784fcc0 0x55ee278520e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5577d71407b4c60c server_cookie=dc8a40ea7f3235fe in_seq=0 out_seq=0 2024-03-12T05:24:08.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.813+0000 7f2e417fa640 10 client.28220 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:08.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.814+0000 7f2e417fa640 1 -- 172.21.15.45:0/1429688838 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ee274b8720 con 0x55ee2784fcc0 2024-03-12T05:24:08.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.814+0000 7f2e417fa640 10 client.28220 handle_command_reply: tid=0 2024-03-12T05:24:08.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.814+0000 7f2e62d77740 1 client.28220 shutdown 2024-03-12T05:24:08.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.815+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 msgr2=0x55ee2781d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.815+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2e5403d970 tx=0x7f2e5403d1a0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 shutdown_connections 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee2784fcc0 0x55ee278520e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814db0 0x55ee2781a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1429688838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814490 0x55ee2781d3c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 >> 172.21.15.45:0/1429688838 conn(0x55ee2780ebc0 msgr2=0x55ee27810910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.816+0000 7f2e417fa640 0 client.28220 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:08.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.822+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 shutdown_connections 2024-03-12T05:24:08.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.822+0000 7f2e62d77740 1 -- 172.21.15.45:0/1429688838 wait complete. 2024-03-12T05:24:08.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.822+0000 7f2e62d77740 20 client.28220 trim_cache size 0 max 16384 2024-03-12T05:24:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.859+0000 7f2e62d77740 1 Processor -- start 2024-03-12T05:24:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.859+0000 7f2e62d77740 1 -- start start 2024-03-12T05:24:08.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.859+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee27856a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.860+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee2781ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.860+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.860+0000 7f2e62d77740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee274b8720 con 0x55ee27814490 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.860+0000 7f2e62d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee27809010 con 0x55ee27814db0 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.860+0000 7f2e62d77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee2782a680 con 0x55ee2781d990 2024-03-12T05:24:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.861+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.861+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee27856a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.861+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee27856a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58808/0 (socket says 172.21.15.45:58808) 2024-03-12T05:24:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.861+0000 7f2e5d66b640 1 -- 172.21.15.45:0/1163937721 learned_addr learned my addr 172.21.15.45:0/1163937721 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:08.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.864+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee2781ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.865+0000 7f2e5de6c640 1 -- 172.21.15.45:0/1163937721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 msgr2=0x55ee2781d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.865+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.865+0000 7f2e5de6c640 1 -- 172.21.15.45:0/1163937721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 msgr2=0x55ee27856a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.865+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee27856a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.865+0000 7f2e5de6c640 1 -- 172.21.15.45:0/1163937721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee27716ac0 con 0x55ee27814db0 2024-03-12T05:24:08.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.866+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee2781ce80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x55ee2781bd20 tx=0x7f2e4800a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18038fefc3afc5a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.867+0000 7f2e437fe640 1 -- 172.21.15.45:0/1163937721 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4801d040 con 0x55ee27814db0 2024-03-12T05:24:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.867+0000 7f2e437fe640 1 -- 172.21.15.45:0/1163937721 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e4801f040 con 0x55ee27814db0 2024-03-12T05:24:08.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.867+0000 7f2e437fe640 1 -- 172.21.15.45:0/1163937721 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4800fc50 con 0x55ee27814db0 2024-03-12T05:24:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.869+0000 7f2e62d77740 1 -- 172.21.15.45:0/1163937721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 msgr2=0x55ee2781ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.869+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee2781ce80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x55ee2781bd20 tx=0x7f2e4800a340 comp rx=0 tx=0).stop 2024-03-12T05:24:08.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.873+0000 7f2e62d77740 1 -- 172.21.15.45:0/1163937721 shutdown_connections 2024-03-12T05:24:08.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.873+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee2781d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.873+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee2781ce80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.873+0000 7f2e62d77740 1 --2- 172.21.15.45:0/1163937721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee27856a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.873+0000 7f2e62d77740 1 -- 172.21.15.45:0/1163937721 >> 172.21.15.45:0/1163937721 conn(0x55ee2780ebc0 msgr2=0x55ee2780efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.874+0000 7f2e62d77740 1 -- 172.21.15.45:0/1163937721 shutdown_connections 2024-03-12T05:24:08.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.875+0000 7f2e62d77740 1 -- 172.21.15.45:0/1163937721 wait complete. 2024-03-12T05:24:08.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.875+0000 7f2e62d77740 1 -- start start 2024-03-12T05:24:08.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.875+0000 7f2e62d77740 1 Processor -- start 2024-03-12T05:24:08.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee278215b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee27821af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee27822050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee278a6380 con 0x55ee27814490 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee278a64f0 con 0x55ee27814db0 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee278a66b0 con 0x55ee2781d990 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.876+0000 7f2e62d77740 10 client.0 _ll_register_callbacks cb 0x7f2e48005ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.877+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee278215b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.877+0000 7f2e5d66b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee278215b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58822/0 (socket says 172.21.15.45:58822) 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.877+0000 7f2e5d66b640 1 -- 172.21.15.45:0/2914302513 learned_addr learned my addr 172.21.15.45:0/2914302513 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:08.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.877+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee27822050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.879+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee27821af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 -- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 msgr2=0x55ee27821af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee27821af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 -- 172.21.15.45:0/2914302513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 msgr2=0x55ee278215b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee278215b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 -- 172.21.15.45:0/2914302513 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee2773a570 con 0x55ee2781d990 2024-03-12T05:24:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e5d66b640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee27822050 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2e54004900 tx=0x7f2e5403d1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d5fa891d124fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.880+0000 7f2e417fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.881+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54004470 con 0x55ee2781d990 2024-03-12T05:24:08.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.881+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e480054d0 con 0x55ee2781d990 2024-03-12T05:24:08.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.881+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f2e54000e40 con 0x55ee2781d990 2024-03-12T05:24:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.885+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e54005530 con 0x55ee2781d990 2024-03-12T05:24:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.885+0000 7f2e62d77740 10 client.28204 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:08.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.885+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ee2773c6a0 con 0x55ee2781d990 2024-03-12T05:24:08.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2e480054d0 con 0x55ee2781d990 2024-03-12T05:24:08.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2e5403f350 con 0x55ee2781d990 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 10 client.28204 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 10 client.28204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 20 client.28204 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee278a7c10 0x55ee278aa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 4 client.28204 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.888+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ee27507890 con 0x55ee278a7c10 2024-03-12T05:24:08.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.889+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee278a7c10 0x55ee278aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:08.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.893+0000 7f2e5de6c640 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee278a7c10 0x55ee278aa030 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26cb3a55359906a9 server_cookie=76aa9d2d9bc1799e in_seq=0 out_seq=0 2024-03-12T05:24:08.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.895+0000 7f2e417fa640 10 client.28204 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:08.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.899+0000 7f2e417fa640 1 -- 172.21.15.45:0/2914302513 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ee27507890 con 0x55ee278a7c10 2024-03-12T05:24:08.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.899+0000 7f2e417fa640 10 client.28204 handle_command_reply: tid=0 2024-03-12T05:24:08.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.899+0000 7f2e62d77740 1 client.28204 shutdown 2024-03-12T05:24:08.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.899+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 msgr2=0x55ee27822050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.899+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee27822050 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2e54004900 tx=0x7f2e5403d1a0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 shutdown_connections 2024-03-12T05:24:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ee278a7c10 0x55ee278aa030 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ee2781d990 0x55ee27822050 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee27814db0 0x55ee27821af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 --2- 172.21.15.45:0/2914302513 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ee27814490 0x55ee278215b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.900+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 >> 172.21.15.45:0/2914302513 conn(0x55ee2780ebc0 msgr2=0x55ee27856a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.901+0000 7f2e417fa640 0 client.28204 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:08.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.902+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 shutdown_connections 2024-03-12T05:24:08.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.903+0000 7f2e62d77740 1 -- 172.21.15.45:0/2914302513 wait complete. 2024-03-12T05:24:08.903 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:08.903 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:24:08.903 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:24:08.903 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:08.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.907+0000 7f2e62d77740 20 client.28204 trim_cache size 0 max 16384 2024-03-12T05:24:08.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.910+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 msgr2=0x7f2e440a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.910+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 0x7f2e440a4270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000caa0 tx=0x7f2e50005440 comp rx=0 tx=0).stop 2024-03-12T05:24:08.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.910+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 msgr2=0x7f2e581a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:08.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.910+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000cad0 tx=0x7f2e5000cf90 comp rx=0 tx=0).stop 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 shutdown_connections 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2e440a1db0 0x7f2e440a4270 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e5810de10 0x7f2e581188c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e5810f790 0x7f2e581a33e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 --2- 172.21.15.45:0/3144159970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5810d260 0x7f2e58118380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:08.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.914+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 >> 172.21.15.45:0/3144159970 conn(0x7f2e5806c060 msgr2=0x7f2e58070460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:08.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.915+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 shutdown_connections 2024-03-12T05:24:08.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:08.916+0000 7f2e600d0640 1 -- 172.21.15.45:0/3144159970 wait complete. 2024-03-12T05:24:08.932 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:24:08.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 -- 172.21.15.45:0/39057113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4103ad0 msgr2=0x7f61a4103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 --2- 172.21.15.45:0/39057113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4103ad0 0x7f61a4103eb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6198009890 tx=0x7f619802bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 -- 172.21.15.45:0/39057113 shutdown_connections 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 --2- 172.21.15.45:0/39057113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 0x7f61a4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 --2- 172.21.15.45:0/39057113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4104480 0x7f61a4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 --2- 172.21.15.45:0/39057113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4103ad0 0x7f61a4103eb0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 -- 172.21.15.45:0/39057113 >> 172.21.15.45:0/39057113 conn(0x7f61a4076d50 msgr2=0x7f61a4079170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 -- 172.21.15.45:0/39057113 shutdown_connections 2024-03-12T05:24:09.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.138+0000 7f61a9120640 1 -- 172.21.15.45:0/39057113 wait complete. 2024-03-12T05:24:09.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.140+0000 7f61a9120640 1 Processor -- start 2024-03-12T05:24:09.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.140+0000 7f61a9120640 1 -- start start 2024-03-12T05:24:09.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a9120640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a9120640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 0x7f61a41960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42376/0 (socket says 172.21.15.45:42376) 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a2d9d640 1 -- 172.21.15.45:0/3595631121 learned_addr learned my addr 172.21.15.45:0/3595631121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a259c640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 0x7f61a41960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4199ee0 0x7f61a419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.141+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a419a8d0 con 0x7f61a4199ee0 2024-03-12T05:24:09.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a419aa40 con 0x7f61a4103ad0 2024-03-12T05:24:09.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a419abb0 con 0x7f61a4108680 2024-03-12T05:24:09.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a359e640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4199ee0 0x7f61a419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a2d9d640 1 -- 172.21.15.45:0/3595631121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 msgr2=0x7f61a41960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a2d9d640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 0x7f61a41960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a2d9d640 1 -- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4199ee0 msgr2=0x7f61a419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a2d9d640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4199ee0 0x7f61a419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.142+0000 7f61a2d9d640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198009490 con 0x7f61a4103ad0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f61a2d9d640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6198037040 tx=0x7f6198004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61980348d0 con 0x7f61a4103ad0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a41a2120 con 0x7f61a4103ad0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f619802ce90 con 0x7f61a4103ad0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198032cb0 con 0x7f61a4103ad0 2024-03-12T05:24:09.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.143+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a41a24e0 con 0x7f61a4103ad0 2024-03-12T05:24:09.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.145+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a40683c0 con 0x7f61a4103ad0 2024-03-12T05:24:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.148+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f619803b040 con 0x7f61a4103ad0 2024-03-12T05:24:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.148+0000 7f618bfff640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 0x7f618c0a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.148+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f61980ef4b0 con 0x7f61a4103ad0 2024-03-12T05:24:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.149+0000 7f61a259c640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 0x7f618c0a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.150+0000 7f61a259c640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 0x7f618c0a41a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f61a407d9d0 tx=0x7f6194009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.154+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61980b8400 con 0x7f61a4103ad0 2024-03-12T05:24:09.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.564+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61a41969d0 con 0x7f61a4103ad0 2024-03-12T05:24:09.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.565+0000 7f618bfff640 1 -- 172.21.15.45:0/3595631121 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f61980bbac0 con 0x7f61a4103ad0 2024-03-12T05:24:09.559 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:24:09.559 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:24:09.560 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:24:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.568+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 msgr2=0x7f618c0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.568+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 0x7f618c0a41a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f61a407d9d0 tx=0x7f6194009260 comp rx=0 tx=0).stop 2024-03-12T05:24:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 msgr2=0x7f61a4195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6198037040 tx=0x7f6198004530 comp rx=0 tx=0).stop 2024-03-12T05:24:09.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 shutdown_connections 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f618c0a1ce0 0x7f618c0a41a0 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f61a407d9d0 tx=0x7f6194009260 comp rx=0 tx=0).stop 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61a4199ee0 0x7f61a419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f61a4108680 0x7f61a41960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 --2- 172.21.15.45:0/3595631121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61a4103ad0 0x7f61a4195ba0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.569+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 >> 172.21.15.45:0/3595631121 conn(0x7f61a4076d50 msgr2=0x7f61a4106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.570+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 shutdown_connections 2024-03-12T05:24:09.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.570+0000 7f61a9120640 1 -- 172.21.15.45:0/3595631121 wait complete. 2024-03-12T05:24:09.581 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:09 smithi045 ceph-mon[32174]: osdmap e103: 12 total, 12 up, 12 in 2024-03-12T05:24:09.584 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:09.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:24:09.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:09 smithi130 ceph-mon[35398]: osdmap e103: 12 total, 12 up, 12 in 2024-03-12T05:24:09.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:09 smithi116 ceph-mon[35486]: osdmap e103: 12 total, 12 up, 12 in 2024-03-12T05:24:09.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.763+0000 7fa9a2587640 1 -- 172.21.15.45:0/111895057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10f6a0 msgr2=0x7fa99c10fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.763+0000 7fa9a2587640 1 --2- 172.21.15.45:0/111895057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10f6a0 0x7fa99c10fa80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009860 tx=0x7fa98c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.765+0000 7fa9a2587640 1 -- 172.21.15.45:0/111895057 shutdown_connections 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.765+0000 7fa9a2587640 1 --2- 172.21.15.45:0/111895057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa99c10dce0 0x7fa99c11c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.765+0000 7fa9a2587640 1 --2- 172.21.15.45:0/111895057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10d260 0x7fa99c10d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.765+0000 7fa9a2587640 1 --2- 172.21.15.45:0/111895057 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10f6a0 0x7fa99c10fa80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.765+0000 7fa9a2587640 1 -- 172.21.15.45:0/111895057 >> 172.21.15.45:0/111895057 conn(0x7fa99c06c030 msgr2=0x7fa99c06c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.766+0000 7fa9a2587640 1 -- 172.21.15.45:0/111895057 shutdown_connections 2024-03-12T05:24:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.766+0000 7fa9a2587640 1 -- 172.21.15.45:0/111895057 wait complete. 2024-03-12T05:24:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 Processor -- start 2024-03-12T05:24:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 -- start start 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10d260 0x7fa99c1a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa99c10dce0 0x7fa99c1a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1ab120 con 0x7fa99c10d260 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1ab290 con 0x7fa99c10dce0 2024-03-12T05:24:09.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a2587640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c1ab450 con 0x7fa99c10f6a0 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a1d86640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.767+0000 7fa9a1d86640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45538/0 (socket says 172.21.15.45:45538) 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 -- 172.21.15.45:0/4164455884 learned_addr learned my addr 172.21.15.45:0/4164455884 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1585640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10d260 0x7fa99c1a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 -- 172.21.15.45:0/4164455884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa99c10dce0 msgr2=0x7fa99c1a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa99c10dce0 0x7fa99c1a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 -- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10d260 msgr2=0x7fa99c1a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10d260 0x7fa99c1a6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 -- 172.21.15.45:0/4164455884 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998009490 con 0x7fa99c10f6a0 2024-03-12T05:24:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.768+0000 7fa9a1d86640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa994009fd0 tx=0x7fa99400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.769+0000 7fa9927fc640 1 -- 172.21.15.45:0/4164455884 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994004b20 con 0x7fa99c10f6a0 2024-03-12T05:24:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.769+0000 7fa9927fc640 1 -- 172.21.15.45:0/4164455884 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa994003e20 con 0x7fa99c10f6a0 2024-03-12T05:24:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.769+0000 7fa9927fc640 1 -- 172.21.15.45:0/4164455884 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994004780 con 0x7fa99c10f6a0 2024-03-12T05:24:09.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.769+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c009490 con 0x7fa99c10f6a0 2024-03-12T05:24:09.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.769+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c1b2dd0 con 0x7fa99c10f6a0 2024-03-12T05:24:09.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.772+0000 7fa9927fc640 1 -- 172.21.15.45:0/4164455884 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa99400a040 con 0x7fa99c10f6a0 2024-03-12T05:24:09.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 Processor -- start 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 -- start start 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d81b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d826d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d7d6960 con 0x55643d824960 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d78a310 con 0x55643d81bd80 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d787840 con 0x55643d81b460 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42406/0 (socket says 172.21.15.45:42406) 2024-03-12T05:24:09.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9a1585640 1 -- 172.21.15.45:0/3184682442 learned_addr learned my addr 172.21.15.45:0/3184682442 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.773+0000 7fa9927fc640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 0x7fa9880a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d81b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 -- 172.21.15.45:0/3184682442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 msgr2=0x55643d81b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d81b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 -- 172.21.15.45:0/3184682442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 msgr2=0x55643d826d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d826d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9927fc640 1 -- 172.21.15.45:0/4164455884 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa9940cd820 con 0x7fa99c10f6a0 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 -- 172.21.15.45:0/3184682442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55643d745060 con 0x55643d81bd80 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d81b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1585640 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c035c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5041b94a54d207f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.774+0000 7fa9a1d86640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 0x7fa9880a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.775+0000 7fa9877fe640 1 -- 172.21.15.45:0/3184682442 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c004040 con 0x55643d81bd80 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.775+0000 7fa9877fe640 1 -- 172.21.15.45:0/3184682442 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa98c02ce90 con 0x55643d81bd80 2024-03-12T05:24:09.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.775+0000 7fa9877fe640 1 -- 172.21.15.45:0/3184682442 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c005560 con 0x55643d81bd80 2024-03-12T05:24:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a1d86640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 0x7fa9880a4230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa99400ffd0 tx=0x7fa994003f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.774 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 -- 172.21.15.45:0/3184682442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 msgr2=0x55643d824420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c035c10 comp rx=0 tx=0).stop 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 -- 172.21.15.45:0/3184682442 shutdown_connections 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d826d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d824420 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/3184682442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d81b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 -- 172.21.15.45:0/3184682442 >> 172.21.15.45:0/3184682442 conn(0x55643d815b90 msgr2=0x55643d815f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 -- 172.21.15.45:0/3184682442 shutdown_connections 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.776+0000 7fa9a63c1740 1 -- 172.21.15.45:0/3184682442 wait complete. 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 -- start start 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 Processor -- start 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d821920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d821920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42414/0 (socket says 172.21.15.45:42414) 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a1585640 1 -- 172.21.15.45:0/831157698 learned_addr learned my addr 172.21.15.45:0/831157698 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.777+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d783020 con 0x55643d824960 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d456430 con 0x55643d81bd80 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d412e00 con 0x55643d81b460 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a63c1740 10 client.0 _ll_register_callbacks cb 0x55643d72acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:09.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a1585640 1 -- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 msgr2=0x55643d821920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a1585640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d821920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a1585640 1 -- 172.21.15.45:0/831157698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 msgr2=0x55643d8223a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a1585640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8223a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a1585640 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c11d450 con 0x55643d81bd80 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.778+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.779+0000 7fa9a1585640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa98c02bbb0 tx=0x7fa98c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f281958768fd5a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.779+0000 7fa984ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.779+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c004460 con 0x55643d81bd80 2024-03-12T05:24:09.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.779+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa98c034650 con 0x55643d81bd80 2024-03-12T05:24:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.780+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55643d4827e0 con 0x55643d81bd80 2024-03-12T05:24:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.780+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa98c034810 con 0x55643d81bd80 2024-03-12T05:24:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.780+0000 7fa9a63c1740 10 client.28238 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:09.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.780+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55643d7816a0 con 0x55643d81bd80 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55643d4827e0 con 0x55643d81bd80 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa98c059070 con 0x55643d81bd80 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 10 client.28238 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 10 client.28238 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 20 client.28238 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d8347f0 0x55643d836be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 4 client.28238 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.781+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55643d4bf720 con 0x55643d8347f0 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.782+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d8347f0 0x55643d836be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.783+0000 7fa984ff9640 10 client.28238 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:09.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.783+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d8347f0 0x55643d836be0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=820dee4838f809c2 server_cookie=10a24b15db564913 in_seq=0 out_seq=0 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.783+0000 7fa984ff9640 1 -- 172.21.15.45:0/831157698 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55643d4bf720 con 0x55643d8347f0 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.783+0000 7fa984ff9640 10 client.28238 handle_command_reply: tid=0 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.783+0000 7fa9a63c1740 1 client.28238 shutdown 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 msgr2=0x55643d821e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa98c02bbb0 tx=0x7fa98c004270 comp rx=0 tx=0).stop 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 shutdown_connections 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d8347f0 0x55643d836be0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa984ff9640 0 client.28238 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d821e60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/831157698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d821920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 >> 172.21.15.45:0/831157698 conn(0x55643d815b90 msgr2=0x55643d8178e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 shutdown_connections 2024-03-12T05:24:09.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 1 -- 172.21.15.45:0/831157698 wait complete. 2024-03-12T05:24:09.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.784+0000 7fa9a63c1740 20 client.28238 trim_cache size 0 max 16384 2024-03-12T05:24:09.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.806+0000 7fa9a63c1740 1 Processor -- start 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.806+0000 7fa9a63c1740 1 -- start start 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d824050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8221c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d708130 con 0x55643d824960 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d826db0 con 0x55643d81bd80 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d81c750 con 0x55643d81b460 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8221c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d824050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d824050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45566/0 (socket says 172.21.15.45:45566) 2024-03-12T05:24:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42420/0 (socket says 172.21.15.45:42420) 2024-03-12T05:24:09.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a0d84640 1 -- 172.21.15.45:0/1184946198 learned_addr learned my addr 172.21.15.45:0/1184946198 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 -- 172.21.15.45:0/1184946198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 msgr2=0x55643d824050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d824050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 -- 172.21.15.45:0/1184946198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 msgr2=0x55643d8221c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8221c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.807+0000 7fa9a1585640 1 -- 172.21.15.45:0/1184946198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55643d71dc20 con 0x55643d81bd80 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.808+0000 7fa9a1585640 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0fe3d6c42ef099b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.808+0000 7fa986ffd640 1 -- 172.21.15.45:0/1184946198 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0035f0 con 0x55643d81bd80 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.808+0000 7fa986ffd640 1 -- 172.21.15.45:0/1184946198 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa98c003ae0 con 0x55643d81bd80 2024-03-12T05:24:09.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.808+0000 7fa986ffd640 1 -- 172.21.15.45:0/1184946198 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c035e00 con 0x55643d81bd80 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 -- 172.21.15.45:0/1184946198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 msgr2=0x55643d820900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c005d40 comp rx=0 tx=0).stop 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 -- 172.21.15.45:0/1184946198 shutdown_connections 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d824960 0x55643d8221c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d81bd80 0x55643d820900 secure :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa98c000c00 tx=0x7fa98c005d40 comp rx=0 tx=0).stop 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.809+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/1184946198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d81b460 0x55643d824050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.810+0000 7fa9a63c1740 1 -- 172.21.15.45:0/1184946198 >> 172.21.15.45:0/1184946198 conn(0x55643d815b90 msgr2=0x55643d8201f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.812+0000 7fa9a63c1740 1 -- 172.21.15.45:0/1184946198 shutdown_connections 2024-03-12T05:24:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.812+0000 7fa9a63c1740 1 -- 172.21.15.45:0/1184946198 wait complete. 2024-03-12T05:24:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 -- start start 2024-03-12T05:24:09.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 Processor -- start 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 0x55643d82ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d8299e0 0x55643d829e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d82a3a0 con 0x55643d81b460 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d82a510 con 0x55643d8299e0 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55643d82a6d0 con 0x55643d824960 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a63c1740 10 client.0 _ll_register_callbacks cb 0x7fa98c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:09.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.813+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58904/0 (socket says 172.21.15.45:58904) 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 -- 172.21.15.45:0/4226171187 learned_addr learned my addr 172.21.15.45:0/4226171187 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a1585640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 0x55643d82ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d8299e0 0x55643d829e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 -- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 msgr2=0x55643d82ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 0x55643d82ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 -- 172.21.15.45:0/4226171187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d8299e0 msgr2=0x55643d829e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d8299e0 0x55643d829e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 -- 172.21.15.45:0/4226171187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55643d7816a0 con 0x55643d81b460 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a1585640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 0x55643d82ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa9a0d84640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa99c072250 tx=0x7fa998013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6846bc1e56ee905 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:09.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.814+0000 7fa984ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:09.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.815+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998004030 con 0x55643d81b460 2024-03-12T05:24:09.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.816+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fa998016040 con 0x55643d81b460 2024-03-12T05:24:09.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.816+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa98c02ce90 con 0x55643d81b460 2024-03-12T05:24:09.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.816+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa998004190 con 0x55643d81b460 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 10 client.18780 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55643d7438c0 con 0x55643d81b460 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa98c02ce90 con 0x55643d81b460 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa998026020 con 0x55643d81b460 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 10 client.18780 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 10 client.18780 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 20 client.18780 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.817+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d88c2b0 0x55643d88e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.818+0000 7fa9a63c1740 4 client.18780 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:09.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.818+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55643d50e890 con 0x55643d88c2b0 2024-03-12T05:24:09.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.818+0000 7fa9a1585640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d88c2b0 0x55643d88e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:09.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.819+0000 7fa984ff9640 10 client.18780 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:09.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa9a1585640 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d88c2b0 0x55643d88e6d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=426f65a535381b70 server_cookie=8dabdadc64ae761a in_seq=0 out_seq=0 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa984ff9640 1 -- 172.21.15.45:0/4226171187 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55643d50e890 con 0x55643d88c2b0 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa984ff9640 10 client.18780 handle_command_reply: tid=0 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa9a63c1740 1 client.18780 shutdown 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 msgr2=0x55643d82b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.820+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa99c072250 tx=0x7fa998013040 comp rx=0 tx=0).stop 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 shutdown_connections 2024-03-12T05:24:09.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55643d88c2b0 0x55643d88e6d0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643d8299e0 0x55643d829e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55643d824960 0x55643d82ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 --2- 172.21.15.45:0/4226171187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55643d81b460 0x55643d82b500 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 >> 172.21.15.45:0/4226171187 conn(0x55643d815b90 msgr2=0x55643d814440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa984ff9640 0 client.18780 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:09.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 shutdown_connections 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 1 -- 172.21.15.45:0/4226171187 wait complete. 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.821+0000 7fa9a63c1740 20 client.18780 trim_cache size 0 max 16384 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.823+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 msgr2=0x7fa9880a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.823+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 0x7fa9880a4230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa99400ffd0 tx=0x7fa994003f90 comp rx=0 tx=0).stop 2024-03-12T05:24:09.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.823+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 msgr2=0x7fa99c1aab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.823+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa994009fd0 tx=0x7fa99400c800 comp rx=0 tx=0).stop 2024-03-12T05:24:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 shutdown_connections 2024-03-12T05:24:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9880a1d70 0x7fa9880a4230 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa99c10f6a0 0x7fa99c1aab20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa99c10dce0 0x7fa99c1a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 --2- 172.21.15.45:0/4164455884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa99c10d260 0x7fa99c1a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:09.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.824+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 >> 172.21.15.45:0/4164455884 conn(0x7fa99c06c030 msgr2=0x7fa99c0706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:09.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.826+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 shutdown_connections 2024-03-12T05:24:09.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:09.827+0000 7fa9a2587640 1 -- 172.21.15.45:0/4164455884 wait complete. 2024-03-12T05:24:09.842 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:09.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:24:09.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 725.0 s, 1086.6 tps, lat 91.482 ms stddev 67.587 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.018+0000 7f675e7b5640 1 -- 172.21.15.45:0/84295317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810f9a0 msgr2=0x7f675810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.018+0000 7f675e7b5640 1 --2- 172.21.15.45:0/84295317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810f9a0 0x7f675810d490 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6750009be0 tx=0x7f675002bef0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 -- 172.21.15.45:0/84295317 shutdown_connections 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 --2- 172.21.15.45:0/84295317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675810d9d0 0x7f675810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 --2- 172.21.15.45:0/84295317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f675811a960 0x7f675811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 --2- 172.21.15.45:0/84295317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810f9a0 0x7f675810d490 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 -- 172.21.15.45:0/84295317 >> 172.21.15.45:0/84295317 conn(0x7f675806c260 msgr2=0x7f675806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 -- 172.21.15.45:0/84295317 shutdown_connections 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.019+0000 7f675e7b5640 1 -- 172.21.15.45:0/84295317 wait complete. 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.020+0000 7f675e7b5640 1 Processor -- start 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.020+0000 7f675e7b5640 1 -- start start 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810d9d0 0x7f67581a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f675810f9a0 0x7f67581a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67581ab3e0 con 0x7f675811a960 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67581ab550 con 0x7f675810d9d0 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675e7b5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67581ab710 con 0x7f675810f9a0 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58922/0 (socket says 172.21.15.45:58922) 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 -- 172.21.15.45:0/774961485 learned_addr learned my addr 172.21.15.45:0/774961485 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f6757fff640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810d9d0 0x7f67581a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 -- 172.21.15.45:0/774961485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f675810f9a0 msgr2=0x7f67581a6ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f675810f9a0 0x7f67581a6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 -- 172.21.15.45:0/774961485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810d9d0 msgr2=0x7f67581a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810d9d0 0x7f67581a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.021+0000 7f675cd52640 1 -- 172.21.15.45:0/774961485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009560 con 0x7f675811a960 2024-03-12T05:24:10.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.022+0000 7f675cd52640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f674800d750 tx=0x7f674800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.022+0000 7f67557fa640 1 -- 172.21.15.45:0/774961485 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748009bd0 con 0x7f675811a960 2024-03-12T05:24:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.022+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6750009840 con 0x7f675811a960 2024-03-12T05:24:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.022+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67581b3090 con 0x7f675811a960 2024-03-12T05:24:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.023+0000 7f67557fa640 1 -- 172.21.15.45:0/774961485 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f674800de60 con 0x7f675811a960 2024-03-12T05:24:10.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.023+0000 7f67557fa640 1 -- 172.21.15.45:0/774961485 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748009bd0 con 0x7f675811a960 2024-03-12T05:24:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.026+0000 7f6761381740 1 Processor -- start 2024-03-12T05:24:10.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.026+0000 7f67557fa640 1 -- 172.21.15.45:0/774961485 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6748004ad0 con 0x7f675811a960 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 -- start start 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791da250 0x557e791da630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791e3750 0x557e791e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e79195960 con 0x557e791dab70 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e79149310 con 0x557e791da250 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.027+0000 7f6761381740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e79146840 con 0x557e791e3750 2024-03-12T05:24:10.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f6757fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f67557fa640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 0x7f67400a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f6757fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58926/0 (socket says 172.21.15.45:58926) 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f6757fff640 1 -- 172.21.15.45:0/811533591 learned_addr learned my addr 172.21.15.45:0/811533591 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f675cd52640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 0x7f67400a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.028+0000 7f67557fa640 1 -- 172.21.15.45:0/774961485 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f67480cce80 con 0x7f675811a960 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 -- 172.21.15.45:0/811533591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791e3750 msgr2=0x557e791e5b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:10.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791e3750 0x557e791e5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 -- 172.21.15.45:0/811533591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791da250 msgr2=0x557e791da630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791da250 0x557e791da630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 -- 172.21.15.45:0/811533591 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e79104060 con 0x557e791dab70 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.029+0000 7f6757fff640 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6750004af0 tx=0x7f675002c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637a841c7306f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.030+0000 7f673e7fc640 1 -- 172.21.15.45:0/811533591 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500335e0 con 0x557e791dab70 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.030+0000 7f673e7fc640 1 -- 172.21.15.45:0/811533591 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6750033740 con 0x557e791dab70 2024-03-12T05:24:10.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.030+0000 7f673e7fc640 1 -- 172.21.15.45:0/811533591 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67500042a0 con 0x557e791dab70 2024-03-12T05:24:10.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.031+0000 7f6761381740 1 -- 172.21.15.45:0/811533591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 msgr2=0x557e791e3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.031+0000 7f6761381740 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f6750004af0 tx=0x7f675002c910 comp rx=0 tx=0).stop 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.031+0000 7f675cd52640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 0x7f67400a4120 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f67480a2c20 tx=0x7f6748096d40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 -- 172.21.15.45:0/811533591 shutdown_connections 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e791e3210 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791e3750 0x557e791e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 --2- 172.21.15.45:0/811533591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791da250 0x557e791da630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 -- 172.21.15.45:0/811533591 >> 172.21.15.45:0/811533591 conn(0x557e791d4980 msgr2=0x557e791d4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 -- 172.21.15.45:0/811533591 shutdown_connections 2024-03-12T05:24:10.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 -- 172.21.15.45:0/811533591 wait complete. 2024-03-12T05:24:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 -- start start 2024-03-12T05:24:10.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.032+0000 7f6761381740 1 Processor -- start 2024-03-12T05:24:10.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e79142020 con 0x557e791da250 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e78e15430 con 0x557e791e3750 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e78dd1e00 con 0x557e791dab70 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6761381740 10 client.0 _ll_register_callbacks cb 0x557e790e9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45612/0 (socket says 172.21.15.45:45612) 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 -- 172.21.15.45:0/3600853623 learned_addr learned my addr 172.21.15.45:0/3600853623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f67577fe640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f67577fe640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 -- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 msgr2=0x557e791e1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 -- 172.21.15.45:0/3600853623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 msgr2=0x557e791e07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f6757fff640 1 -- 172.21.15.45:0/3600853623 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675811d680 con 0x557e791dab70 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.033+0000 7f67577fe640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:10.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f6757fff640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6750005f00 tx=0x7f6750033a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=631d1182065e437d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f67377fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750003f50 con 0x557e791dab70 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f675004f030 con 0x557e791dab70 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e78e417e0 con 0x557e791dab70 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67500040b0 con 0x557e791dab70 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f6761381740 10 client.28228 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.034+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557e791406a0 con 0x557e791dab70 2024-03-12T05:24:10.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.035+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67500046c0 con 0x557e791dab70 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f675003c040 con 0x557e791dab70 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 10 client.28228 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 10 client.28228 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 20 client.28228 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e791f3780 0x557e791f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 4 client.28228 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.036+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e78e417e0 con 0x557e791f3780 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.037+0000 7f67577fe640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e791f3780 0x557e791f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.037+0000 7f67577fe640 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e791f3780 0x557e791f5ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b58ee0c81856193c server_cookie=9454e65dd26f14de in_seq=0 out_seq=0 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.037+0000 7f67377fe640 10 client.28228 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.038+0000 7f67377fe640 1 -- 172.21.15.45:0/3600853623 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557e78e417e0 con 0x557e791f3780 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.038+0000 7f67377fe640 10 client.28228 handle_command_reply: tid=0 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.038+0000 7f6761381740 1 client.28228 shutdown 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.038+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 msgr2=0x557e791e0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.038+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6750005f00 tx=0x7f6750033a10 comp rx=0 tx=0).stop 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 shutdown_connections 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e791f3780 0x557e791f5ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e0d10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f67377fe640 0 client.28228 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 --2- 172.21.15.45:0/3600853623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 >> 172.21.15.45:0/3600853623 conn(0x557e791d4980 msgr2=0x557e791d66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 shutdown_connections 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 1 -- 172.21.15.45:0/3600853623 wait complete. 2024-03-12T05:24:10.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.039+0000 7f6761381740 20 client.28228 trim_cache size 0 max 16384 2024-03-12T05:24:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 Processor -- start 2024-03-12T05:24:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 -- start start 2024-03-12T05:24:10.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791dda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e790c7130 con 0x557e791da250 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e791d5980 con 0x557e791e3750 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6761381740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e791d02b0 con 0x557e791dab70 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f67577fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f67577fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58936/0 (socket says 172.21.15.45:58936) 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f67577fe640 1 -- 172.21.15.45:0/1413267233 learned_addr learned my addr 172.21.15.45:0/1413267233 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.060+0000 7f6757fff640 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 -- 172.21.15.45:0/1413267233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 msgr2=0x557e791e1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 -- 172.21.15.45:0/1413267233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 msgr2=0x557e791dda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791dda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 -- 172.21.15.45:0/1413267233 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e790dcc20 con 0x557e791da250 2024-03-12T05:24:10.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.061+0000 7f67577fe640 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f674c013110 tx=0x7f674c009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff96518acd318b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.062+0000 7f673dffb640 1 -- 172.21.15.45:0/1413267233 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c018040 con 0x557e791da250 2024-03-12T05:24:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.062+0000 7f673dffb640 1 -- 172.21.15.45:0/1413267233 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f674c015580 con 0x557e791da250 2024-03-12T05:24:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.062+0000 7f673dffb640 1 -- 172.21.15.45:0/1413267233 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c01ca90 con 0x557e791da250 2024-03-12T05:24:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 -- 172.21.15.45:0/1413267233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 msgr2=0x557e791e2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f674c013110 tx=0x7f674c009b10 comp rx=0 tx=0).stop 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 -- 172.21.15.45:0/1413267233 shutdown_connections 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791dda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791dab70 0x557e791e1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 --2- 172.21.15.45:0/1413267233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791da250 0x557e791e2f00 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 -- 172.21.15.45:0/1413267233 >> 172.21.15.45:0/1413267233 conn(0x557e791d4980 msgr2=0x557e791dbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 -- 172.21.15.45:0/1413267233 shutdown_connections 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.063+0000 7f6761381740 1 -- 172.21.15.45:0/1413267233 wait complete. 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 -- start start 2024-03-12T05:24:10.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 Processor -- start 2024-03-12T05:24:10.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 0x557e792440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e791e9f60 con 0x557e791dab70 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e791ea0d0 con 0x557e791e3750 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e791ea290 con 0x557e791da250 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.064+0000 7f6761381740 10 client.0 _ll_register_callbacks cb 0x7f674c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58952/0 (socket says 172.21.15.45:58952) 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 -- 172.21.15.45:0/248633318 learned_addr learned my addr 172.21.15.45:0/248633318 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f67577fe640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 0x557e792440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f67577fe640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 -- 172.21.15.45:0/248633318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 msgr2=0x557e792440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 0x557e792440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 -- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 msgr2=0x557e791e99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f6757fff640 1 -- 172.21.15.45:0/248633318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c013270 con 0x557e791dab70 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.065+0000 7f67577fe640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 0x557e792440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.066+0000 7f6757fff640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x557e791e1da0 tx=0x7f6750009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dec3eb7aa63c08a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.066+0000 7f67377fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.066+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750063070 con 0x557e791dab70 2024-03-12T05:24:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.066+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f675004f040 con 0x557e791dab70 2024-03-12T05:24:10.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.066+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6750033740 con 0x557e791dab70 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.067+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6750003970 con 0x557e791dab70 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.067+0000 7f6761381740 10 client.18798 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.067+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557e791028c0 con 0x557e791dab70 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.067+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6750033740 con 0x557e791dab70 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.067+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6750066030 con 0x557e791dab70 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 10 client.18798 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 10 client.18798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 20 client.18798 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e7925cc90 0x557e7925f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 4 client.18798 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557e78e7e720 con 0x557e7925cc90 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.068+0000 7f67577fe640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e7925cc90 0x557e7925f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.069+0000 7f67577fe640 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e7925cc90 0x557e7925f0b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6218f4daa0844387 server_cookie=b687d118c34504a6 in_seq=0 out_seq=0 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.069+0000 7f67377fe640 10 client.18798 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.069+0000 7f67377fe640 1 -- 172.21.15.45:0/248633318 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557e78e7e720 con 0x557e7925cc90 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.069+0000 7f67377fe640 10 client.18798 handle_command_reply: tid=0 2024-03-12T05:24:10.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.069+0000 7f6761381740 1 client.18798 shutdown 2024-03-12T05:24:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 msgr2=0x557e79244630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x557e791e1da0 tx=0x7f6750009ba0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 shutdown_connections 2024-03-12T05:24:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x557e7925cc90 0x557e7925f0b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557e791dab70 0x557e79244630 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557e791e3750 0x557e791e99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 --2- 172.21.15.45:0/248633318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557e791da250 0x557e792440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 >> 172.21.15.45:0/248633318 conn(0x557e791d4980 msgr2=0x557e791d5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f67377fe640 0 client.18798 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 shutdown_connections 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 1 -- 172.21.15.45:0/248633318 wait complete. 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.070+0000 7f6761381740 20 client.18798 trim_cache size 0 max 16384 2024-03-12T05:24:10.067 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 msgr2=0x7f67400a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 0x7f67400a4120 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f67480a2c20 tx=0x7f6748096d40 comp rx=0 tx=0).stop 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 msgr2=0x7f67581aade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f674800d750 tx=0x7f674800dc10 comp rx=0 tx=0).stop 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 shutdown_connections 2024-03-12T05:24:10.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67400a1c60 0x7f67400a4120 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f675811a960 0x7f67581aade0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f675810f9a0 0x7f67581a6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 --2- 172.21.15.45:0/774961485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f675810d9d0 0x7f67581a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.071+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 >> 172.21.15.45:0/774961485 conn(0x7f675806c260 msgr2=0x7f6758071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.072+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 shutdown_connections 2024-03-12T05:24:10.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.072+0000 7f675e7b5640 1 -- 172.21.15.45:0/774961485 wait complete. 2024-03-12T05:24:10.083 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:10.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:24:10.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.271+0000 7fb08d59a640 1 -- 172.21.15.45:0/1436981920 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08402cb70 con 0x7fb090108180 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.271+0000 7fb094dca640 1 -- 172.21.15.45:0/1436981920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb090108180 msgr2=0x7fb090108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.271+0000 7fb094dca640 1 --2- 172.21.15.45:0/1436981920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb090108180 0x7fb090108600 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb084009900 tx=0x7fb08402bc30 comp rx=0 tx=0).stop 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.271+0000 7fb094dca640 1 -- 172.21.15.45:0/1436981920 shutdown_connections 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 --2- 172.21.15.45:0/1436981920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010fd00 0x7fb09010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 --2- 172.21.15.45:0/1436981920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb090108180 0x7fb090108600 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 --2- 172.21.15.45:0/1436981920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb09010f890 0x7fb09010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 -- 172.21.15.45:0/1436981920 >> 172.21.15.45:0/1436981920 conn(0x7fb09006c680 msgr2=0x7fb0900720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 -- 172.21.15.45:0/1436981920 shutdown_connections 2024-03-12T05:24:10.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.272+0000 7fb094dca640 1 -- 172.21.15.45:0/1436981920 wait complete. 2024-03-12T05:24:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 Processor -- start 2024-03-12T05:24:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 -- start start 2024-03-12T05:24:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 0x7fb0901a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010f890 0x7fb0901a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901ab370 con 0x7fb09010f890 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901ab4e0 con 0x7fb09010fd00 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.273+0000 7fb094dca640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0901ab6a0 con 0x7fb090108180 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 0x7fb0901a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42498/0 (socket says 172.21.15.45:42498) 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 -- 172.21.15.45:0/153680276 learned_addr learned my addr 172.21.15.45:0/153680276 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010f890 0x7fb0901a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 -- 172.21.15.45:0/153680276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 msgr2=0x7fb0901a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 0x7fb0901a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 -- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010f890 msgr2=0x7fb0901a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010f890 0x7fb0901a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 -- 172.21.15.45:0/153680276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084009560 con 0x7fb09010fd00 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08e59c640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 0x7fb0901a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.274+0000 7fb08ed9d640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb08402c1c0 tx=0x7fb084004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.275+0000 7fb07f7fe640 1 -- 172.21.15.45:0/153680276 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0840043c0 con 0x7fb09010fd00 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.275+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0901ab920 con 0x7fb09010fd00 2024-03-12T05:24:10.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.275+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0901b3200 con 0x7fb09010fd00 2024-03-12T05:24:10.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.275+0000 7fb07f7fe640 1 -- 172.21.15.45:0/153680276 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb084003aa0 con 0x7fb09010fd00 2024-03-12T05:24:10.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.276+0000 7fb07f7fe640 1 -- 172.21.15.45:0/153680276 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084005460 con 0x7fb09010fd00 2024-03-12T05:24:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.279+0000 7fb07f7fe640 1 -- 172.21.15.45:0/153680276 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb08403e450 con 0x7fb09010fd00 2024-03-12T05:24:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.279+0000 7fb09799d740 1 Processor -- start 2024-03-12T05:24:10.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.279+0000 7fb09799d740 1 -- start start 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c6f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c71b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9c21960 con 0x5610f9c6f750 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9bd5310 con 0x5610f9c66250 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb09799d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9bd2840 con 0x5610f9c66b70 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42506/0 (socket says 172.21.15.45:42506) 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 -- 172.21.15.45:0/3489754108 learned_addr learned my addr 172.21.15.45:0/3489754108 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c71b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 -- 172.21.15.45:0/3489754108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 msgr2=0x5610f9c6f210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c6f210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 -- 172.21.15.45:0/3489754108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 msgr2=0x5610f9c71b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c71b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.280+0000 7fb08dd9b640 1 -- 172.21.15.45:0/3489754108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610f9b90060 con 0x5610f9c66250 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.281+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb0800058c0 tx=0x7fb080005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29788629a97eb255 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.281+0000 7fb07f7fe640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 0x7fb0780a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.281+0000 7fb07cff9640 1 -- 172.21.15.45:0/3489754108 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004030 con 0x5610f9c66250 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.281+0000 7fb07cff9640 1 -- 172.21.15.45:0/3489754108 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb080005020 con 0x5610f9c66250 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.282+0000 7fb07f7fe640 1 -- 172.21.15.45:0/153680276 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fb08402de40 con 0x7fb09010fd00 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.282+0000 7fb07cff9640 1 -- 172.21.15.45:0/3489754108 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004c60 con 0x5610f9c66250 2024-03-12T05:24:10.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.282+0000 7fb08e59c640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 0x7fb0780a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- 172.21.15.45:0/3489754108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 msgr2=0x5610f9c66630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb08e59c640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 0x7fb0780a4270 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5610f9c65d20 tx=0x7fb08800a430 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb0800058c0 tx=0x7fb080005850 comp rx=0 tx=0).stop 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- 172.21.15.45:0/3489754108 shutdown_connections 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c71b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c6f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 --2- 172.21.15.45:0/3489754108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66250 0x5610f9c66630 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- 172.21.15.45:0/3489754108 >> 172.21.15.45:0/3489754108 conn(0x5610f9c60980 msgr2=0x5610f9c60d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- 172.21.15.45:0/3489754108 shutdown_connections 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- 172.21.15.45:0/3489754108 wait complete. 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 -- start start 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.283+0000 7fb09799d740 1 Processor -- start 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 0x5610f9c6cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c6d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9bce020 con 0x5610f9c66b70 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f98a1430 con 0x5610f9c6f750 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f985de00 con 0x5610f9c66250 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb09799d740 10 client.0 _ll_register_callbacks cb 0x5610f9b75cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c6d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c6d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42514/0 (socket says 172.21.15.45:42514) 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45656/0 (socket says 172.21.15.45:45656) 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08ed9d640 1 -- 172.21.15.45:0/240385473 learned_addr learned my addr 172.21.15.45:0/240385473 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.284+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 0x5610f9c6cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 -- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 msgr2=0x5610f9c6c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 -- 172.21.15.45:0/240385473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 msgr2=0x5610f9c6d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c6d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 -- 172.21.15.45:0/240385473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb080019040 con 0x5610f9c66b70 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 0x5610f9c6cc50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb080009800 tx=0x7fb080005580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7c9814fdd713f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.285+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004bf0 con 0x5610f9c66b70 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.286+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fb080004d50 con 0x5610f9c66b70 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.286+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610f98cd7e0 con 0x5610f9c66b70 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.287+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb080004040 con 0x5610f9c66b70 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.287+0000 7fb09799d740 10 client.18810 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.287+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5610f9bcc6a0 con 0x5610f9c66b70 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.287+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5610f98cd7e0 con 0x5610f9c66b70 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb08002f070 con 0x5610f9c66b70 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 10 client.18810 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 10 client.18810 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 20 client.18810 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9c7fa60 0x5610f9c81e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 4 client.18810 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.288+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610f990a720 con 0x5610f9c7fa60 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.289+0000 7fb08e59c640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9c7fa60 0x5610f9c81e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.289+0000 7fb075ffb640 10 client.18810 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:10.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.289+0000 7fb08e59c640 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9c7fa60 0x5610f9c81e80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b88cf16e784c1fef server_cookie=3fb30513055936c7 in_seq=0 out_seq=0 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.290+0000 7fb075ffb640 1 -- 172.21.15.45:0/240385473 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5610f990a720 con 0x5610f9c7fa60 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.290+0000 7fb075ffb640 10 client.18810 handle_command_reply: tid=0 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.290+0000 7fb09799d740 1 client.18810 shutdown 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 msgr2=0x5610f9c6cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 0x5610f9c6cc50 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb080009800 tx=0x7fb080005580 comp rx=0 tx=0).stop 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 shutdown_connections 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9c7fa60 0x5610f9c81e80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66b70 0x5610f9c6cc50 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c6d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 --2- 172.21.15.45:0/240385473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 >> 172.21.15.45:0/240385473 conn(0x5610f9c60980 msgr2=0x5610f9c626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb075ffb640 0 client.18810 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 shutdown_connections 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 1 -- 172.21.15.45:0/240385473 wait complete. 2024-03-12T05:24:10.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.291+0000 7fb09799d740 20 client.18810 trim_cache size 0 max 16384 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 Processor -- start 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 -- start start 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66b70 0x5610f9c867b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9b53130 con 0x5610f9c6f750 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9c7c690 con 0x5610f9c66b70 2024-03-12T05:24:10.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb09799d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9c71b40 con 0x5610f9c66250 2024-03-12T05:24:10.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c679f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58988/0 (socket says 172.21.15.45:58988) 2024-03-12T05:24:10.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb08ed9d640 1 -- 172.21.15.45:0/3973653408 learned_addr learned my addr 172.21.15.45:0/3973653408 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.316+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66b70 0x5610f9c867b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 -- 172.21.15.45:0/3973653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66b70 msgr2=0x5610f9c867b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66b70 0x5610f9c867b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 -- 172.21.15.45:0/3973653408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 msgr2=0x5610f9c679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.318+0000 7fb08e59c640 1 -- 172.21.15.45:0/3973653408 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610f9b68c20 con 0x5610f9c66250 2024-03-12T05:24:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.319+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6ee40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x5610f9c82730 tx=0x7fb088015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=400a02e2a2600542 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.319+0000 7fb077fff640 1 -- 172.21.15.45:0/3973653408 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088027de0 con 0x5610f9c66250 2024-03-12T05:24:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.320+0000 7fb077fff640 1 -- 172.21.15.45:0/3973653408 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb088027890 con 0x5610f9c66250 2024-03-12T05:24:10.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.320+0000 7fb077fff640 1 -- 172.21.15.45:0/3973653408 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08801fe80 con 0x5610f9c66250 2024-03-12T05:24:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 -- 172.21.15.45:0/3973653408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 msgr2=0x5610f9c6ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6ee40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x5610f9c82730 tx=0x7fb088015000 comp rx=0 tx=0).stop 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 -- 172.21.15.45:0/3973653408 shutdown_connections 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c6f750 0x5610f9c679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c66b70 0x5610f9c867b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 --2- 172.21.15.45:0/3973653408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66250 0x5610f9c6ee40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 -- 172.21.15.45:0/3973653408 >> 172.21.15.45:0/3973653408 conn(0x5610f9c60980 msgr2=0x5610f9c5ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 -- 172.21.15.45:0/3973653408 shutdown_connections 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.321+0000 7fb09799d740 1 -- 172.21.15.45:0/3973653408 wait complete. 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.322+0000 7fb09799d740 1 -- start start 2024-03-12T05:24:10.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.322+0000 7fb09799d740 1 Processor -- start 2024-03-12T05:24:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.322+0000 7fb09799d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 0x5610f9c75050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c75590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c75ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9c76fa0 con 0x5610f9c66250 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb08dd9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c75590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb08ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c75ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb08dd9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c75590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45676/0 (socket says 172.21.15.45:45676) 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb08dd9b640 1 -- 172.21.15.45:0/3336216081 learned_addr learned my addr 172.21.15.45:0/3336216081 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9cd5f40 con 0x5610f9c6f750 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610f9cd6080 con 0x5610f9c66b70 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb09799d740 10 client.0 _ll_register_callbacks cb 0x7fb080009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.323+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 0x5610f9c75050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 -- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 msgr2=0x5610f9c75590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c75590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 -- 172.21.15.45:0/3336216081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 msgr2=0x5610f9c75ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c75ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08800a9b0 con 0x5610f9c66250 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb08e59c640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 0x5610f9c75050 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb08801fa20 tx=0x7fb088027610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba9760fd668074e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.324+0000 7fb075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:10.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08801fe90 con 0x5610f9c66250 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fb088035070 con 0x5610f9c66250 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb080005020 con 0x5610f9c66250 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb088027e80 con 0x5610f9c66250 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb09799d740 10 client.18816 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.325+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5610f9b8e8c0 con 0x5610f9c66250 2024-03-12T05:24:10.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb088046080 con 0x5610f9c66250 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb088040030 con 0x5610f9c66250 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 10 client.18816 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 10 client.18816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 20 client.18816 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9cd77a0 0x5610f9cd9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 4 client.18816 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5610f9959890 con 0x5610f9cd77a0 2024-03-12T05:24:10.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.326+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9cd77a0 0x5610f9cd9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.328+0000 7fb08dd9b640 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9cd77a0 0x5610f9cd9bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38b02ee18aebaf21 server_cookie=47f24023c2da0528 in_seq=0 out_seq=0 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.328+0000 7fb075ffb640 10 client.18816 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.328+0000 7fb075ffb640 1 -- 172.21.15.45:0/3336216081 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5610f9959890 con 0x5610f9cd77a0 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.328+0000 7fb075ffb640 10 client.18816 handle_command_reply: tid=0 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.328+0000 7fb09799d740 1 client.18816 shutdown 2024-03-12T05:24:10.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 msgr2=0x5610f9c75050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 0x5610f9c75050 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb08801fa20 tx=0x7fb088027610 comp rx=0 tx=0).stop 2024-03-12T05:24:10.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 shutdown_connections 2024-03-12T05:24:10.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5610f9cd77a0 0x5610f9cd9bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5610f9c6f750 0x5610f9c75ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610f9c66b70 0x5610f9c75590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 --2- 172.21.15.45:0/3336216081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5610f9c66250 0x5610f9c75050 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 >> 172.21.15.45:0/3336216081 conn(0x5610f9c60980 msgr2=0x5610f9c679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb075ffb640 0 client.18816 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.329+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 shutdown_connections 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.330+0000 7fb09799d740 1 -- 172.21.15.45:0/3336216081 wait complete. 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.330+0000 7fb09799d740 20 client.18816 trim_cache size 0 max 16384 2024-03-12T05:24:10.325 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:10.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 msgr2=0x7fb0780a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 0x7fb0780a4270 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5610f9c65d20 tx=0x7fb08800a430 comp rx=0 tx=0).stop 2024-03-12T05:24:10.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 msgr2=0x7fb0901aad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb08402c1c0 tx=0x7fb084004040 comp rx=0 tx=0).stop 2024-03-12T05:24:10.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 shutdown_connections 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb0780a1db0 0x7fb0780a4270 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb09010f890 0x7fb0901a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb09010fd00 0x7fb0901aad70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 --2- 172.21.15.45:0/153680276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb090108180 0x7fb0901a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 >> 172.21.15.45:0/153680276 conn(0x7fb09006c680 msgr2=0x7fb09010b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.331+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 shutdown_connections 2024-03-12T05:24:10.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.332+0000 7fb094dca640 1 -- 172.21.15.45:0/153680276 wait complete. 2024-03-12T05:24:10.348 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:10.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:24:10.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:10 smithi045 ceph-mon[32174]: pgmap v416: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:24:10.389 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:10 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3595631121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:10.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.534+0000 7f424e7fc640 1 -- 172.21.15.45:0/603710674 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424402cb70 con 0x7f425010d610 2024-03-12T05:24:10.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.535+0000 7f42561d7640 1 -- 172.21.15.45:0/603710674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 msgr2=0x7f425010d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.535+0000 7f42561d7640 1 --2- 172.21.15.45:0/603710674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425010d9f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4244009930 tx=0x7f424402bc30 comp rx=0 tx=0).stop 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 -- 172.21.15.45:0/603710674 shutdown_connections 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 --2- 172.21.15.45:0/603710674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107580 0x7f4250107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 --2- 172.21.15.45:0/603710674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4250107f40 0x7f42501143d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 --2- 172.21.15.45:0/603710674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425010d9f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 -- 172.21.15.45:0/603710674 >> 172.21.15.45:0/603710674 conn(0x7f4250076d40 msgr2=0x7f4250079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 -- 172.21.15.45:0/603710674 shutdown_connections 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.536+0000 7f42561d7640 1 -- 172.21.15.45:0/603710674 wait complete. 2024-03-12T05:24:10.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.537+0000 7f42561d7640 1 Processor -- start 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 -- start start 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4250107580 0x7f42500816f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107f40 0x7f4250081c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425007b0e0 con 0x7f4250107f40 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425007b250 con 0x7f425010d610 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f42561d7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425007b410 con 0x7f4250107580 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f424ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f424ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42546/0 (socket says 172.21.15.45:42546) 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f424ffff640 1 -- 172.21.15.45:0/3928048295 learned_addr learned my addr 172.21.15.45:0/3928048295 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.538+0000 7f424effd640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107f40 0x7f4250081c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 -- 172.21.15.45:0/3928048295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4250107580 msgr2=0x7f42500816f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4250107580 0x7f42500816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 -- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107f40 msgr2=0x7f4250081c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107f40 0x7f4250081c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 -- 172.21.15.45:0/3928048295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4244009560 con 0x7f425010d610 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424ffff640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f424800ff50 tx=0x7f424800d760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f424cff9640 1 -- 172.21.15.45:0/3928048295 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424800aa50 con 0x7f425010d610 2024-03-12T05:24:10.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425007f210 con 0x7f425010d610 2024-03-12T05:24:10.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.539+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425007f720 con 0x7f425010d610 2024-03-12T05:24:10.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.540+0000 7f424cff9640 1 -- 172.21.15.45:0/3928048295 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424800cc50 con 0x7f425010d610 2024-03-12T05:24:10.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.540+0000 7f424cff9640 1 -- 172.21.15.45:0/3928048295 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4248016da0 con 0x7f425010d610 2024-03-12T05:24:10.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.543+0000 7f424cff9640 1 -- 172.21.15.45:0/3928048295 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4248009070 con 0x7f425010d610 2024-03-12T05:24:10.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 Processor -- start 2024-03-12T05:24:10.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 -- start start 2024-03-12T05:24:10.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e63840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 0x562804e6c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e6c960 0x562804e6ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e1e960 con 0x562804e6c960 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804dd2310 con 0x562804e63460 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f4258dad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804dcf840 con 0x562804e63d80 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f424cff9640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 0x7f42400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f424f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 0x562804e6c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f424effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e63840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f424f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 0x562804e6c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45700/0 (socket says 172.21.15.45:45700) 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.544+0000 7f424f7fe640 1 -- 172.21.15.45:0/2675948418 learned_addr learned my addr 172.21.15.45:0/2675948418 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e6c960 0x562804e6ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424ffff640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 0x7f42400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 -- 172.21.15.45:0/2675948418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 msgr2=0x562804e6c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 0x562804e6c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 -- 172.21.15.45:0/2675948418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e6c960 msgr2=0x562804e6ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e6c960 0x562804e6ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424effd640 1 -- 172.21.15.45:0/2675948418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562804d8d060 con 0x562804e63460 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.545+0000 7f424cff9640 1 -- 172.21.15.45:0/3928048295 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f42480cdef0 con 0x7f425010d610 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.546+0000 7f424effd640 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e63840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f423c005d00 tx=0x7f423c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d690b92b949a35a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.546+0000 7f4239ffb640 1 -- 172.21.15.45:0/2675948418 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c012d20 con 0x562804e63460 2024-03-12T05:24:10.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.546+0000 7f424ffff640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 0x7f42400a40e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4248099990 tx=0x7f4248065d60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.546+0000 7f4239ffb640 1 -- 172.21.15.45:0/2675948418 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f423c012e80 con 0x562804e63460 2024-03-12T05:24:10.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.546+0000 7f4239ffb640 1 -- 172.21.15.45:0/2675948418 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c004a00 con 0x562804e63460 2024-03-12T05:24:10.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 -- 172.21.15.45:0/2675948418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 msgr2=0x562804e63840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e63840 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f423c005d00 tx=0x7f423c005c90 comp rx=0 tx=0).stop 2024-03-12T05:24:10.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 -- 172.21.15.45:0/2675948418 shutdown_connections 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e6c960 0x562804e6ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e63d80 0x562804e6c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 --2- 172.21.15.45:0/2675948418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e63840 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 -- 172.21.15.45:0/2675948418 >> 172.21.15.45:0/2675948418 conn(0x562804e5db90 msgr2=0x562804e5df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 -- 172.21.15.45:0/2675948418 shutdown_connections 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.547+0000 7f4258dad740 1 -- 172.21.15.45:0/2675948418 wait complete. 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.548+0000 7f4258dad740 1 -- start start 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.548+0000 7f4258dad740 1 Processor -- start 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f4258dad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f4258dad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63d80 0x562804e69ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f4258dad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e6a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f4258dad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804dcb020 con 0x562804e63460 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f424effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f424effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59032/0 (socket says 172.21.15.45:59032) 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f424effd640 1 -- 172.21.15.45:0/105159726 learned_addr learned my addr 172.21.15.45:0/105159726 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.549+0000 7f424f7fe640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63d80 0x562804e69ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.550+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804a9e430 con 0x562804e63d80 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.550+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804a5ae00 con 0x562804e6c960 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.550+0000 7f4258dad740 10 client.0 _ll_register_callbacks cb 0x562804d72cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.550+0000 7f424effd640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e6a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 -- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 msgr2=0x562804e6a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e6a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 -- 172.21.15.45:0/105159726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63d80 msgr2=0x562804e69ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63d80 0x562804e69ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c01a040 con 0x562804e63460 2024-03-12T05:24:10.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f424effd640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f423c00b6b0 tx=0x7f423c00bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d7085ab76f5670 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f422effd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:10.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c004100 con 0x562804e63460 2024-03-12T05:24:10.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.551+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f423c004260 con 0x562804e63460 2024-03-12T05:24:10.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.552+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562804b07720 con 0x562804e63460 2024-03-12T05:24:10.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.552+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f423c004420 con 0x562804e63460 2024-03-12T05:24:10.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.552+0000 7f4258dad740 10 client.18828 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.552+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562804dc96a0 con 0x562804e63460 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562804b07720 con 0x562804e63460 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f423c0046c0 con 0x562804e63460 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 10 client.18828 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 10 client.18828 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 20 client.18828 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804e7cc40 0x562804e7f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 4 client.18828 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.553+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562804b56890 con 0x562804e7cc40 2024-03-12T05:24:10.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.554+0000 7f424f7fe640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804e7cc40 0x562804e7f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.555+0000 7f422effd640 10 client.18828 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.555+0000 7f424f7fe640 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804e7cc40 0x562804e7f060 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f9df392b74e6ca4 server_cookie=cbd84b8e9b332446 in_seq=0 out_seq=0 2024-03-12T05:24:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.556+0000 7f422effd640 1 -- 172.21.15.45:0/105159726 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562804b56890 con 0x562804e7cc40 2024-03-12T05:24:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.556+0000 7f422effd640 10 client.18828 handle_command_reply: tid=0 2024-03-12T05:24:10.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.556+0000 7f4258dad740 1 client.18828 shutdown 2024-03-12T05:24:10.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.556+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 msgr2=0x562804e69980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.556+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f423c00b6b0 tx=0x7f423c00bfb0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 shutdown_connections 2024-03-12T05:24:10.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804e7cc40 0x562804e7f060 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e6a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63d80 0x562804e69ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 --2- 172.21.15.45:0/105159726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63460 0x562804e69980 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 >> 172.21.15.45:0/105159726 conn(0x562804e5db90 msgr2=0x562804e5f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f422effd640 0 client.18828 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 shutdown_connections 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 1 -- 172.21.15.45:0/105159726 wait complete. 2024-03-12T05:24:10.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.557+0000 7f4258dad740 20 client.18828 trim_cache size 0 max 16384 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 Processor -- start 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 -- start start 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e6a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e66c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804d50130 con 0x562804e63d80 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e6f630 con 0x562804e63460 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f4258dad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e5eb90 con 0x562804e6c960 2024-03-12T05:24:10.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f424effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f424effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42572/0 (socket says 172.21.15.45:42572) 2024-03-12T05:24:10.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f424effd640 1 -- 172.21.15.45:0/1768028073 learned_addr learned my addr 172.21.15.45:0/1768028073 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.580+0000 7f424f7fe640 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e6a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 -- 172.21.15.45:0/1768028073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 msgr2=0x562804e66c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e66c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 -- 172.21.15.45:0/1768028073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 msgr2=0x562804e6a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e6a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 -- 172.21.15.45:0/1768028073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562804d65c20 con 0x562804e63460 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f424effd640 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f423c0049f0 tx=0x7f423c00b570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eb1d0d79d45aa0c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f42397fa640 1 -- 172.21.15.45:0/1768028073 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c00bb50 con 0x562804e63460 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f42397fa640 1 -- 172.21.15.45:0/1768028073 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f423c026070 con 0x562804e63460 2024-03-12T05:24:10.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.581+0000 7f42397fa640 1 -- 172.21.15.45:0/1768028073 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c005390 con 0x562804e63460 2024-03-12T05:24:10.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.582+0000 7f4258dad740 1 -- 172.21.15.45:0/1768028073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 msgr2=0x562804e6c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.582+0000 7f4258dad740 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f423c0049f0 tx=0x7f423c00b570 comp rx=0 tx=0).stop 2024-03-12T05:24:10.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 -- 172.21.15.45:0/1768028073 shutdown_connections 2024-03-12T05:24:10.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e6a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e66c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 --2- 172.21.15.45:0/1768028073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e6c0b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 -- 172.21.15.45:0/1768028073 >> 172.21.15.45:0/1768028073 conn(0x562804e5db90 msgr2=0x562804e682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 -- 172.21.15.45:0/1768028073 shutdown_connections 2024-03-12T05:24:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 -- 172.21.15.45:0/1768028073 wait complete. 2024-03-12T05:24:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.583+0000 7f4258dad740 1 -- start start 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.584+0000 7f4258dad740 1 Processor -- start 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.585+0000 7f4258dad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.585+0000 7f4258dad740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e73d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.585+0000 7f424effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.585+0000 7f424effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42574/0 (socket says 172.21.15.45:42574) 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.585+0000 7f424effd640 1 -- 172.21.15.45:0/3608365880 learned_addr learned my addr 172.21.15.45:0/3608365880 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e71e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e73d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e72590 con 0x562804e63d80 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e72700 con 0x562804e63460 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562804e728c0 con 0x562804e6c960 2024-03-12T05:24:10.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f4258dad740 10 client.0 _ll_register_callbacks cb 0x7f423c00ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424effd640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e71e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 -- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 msgr2=0x562804e71e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e71e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 -- 172.21.15.45:0/3608365880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 msgr2=0x562804e73840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424f7fe640 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562804dc96a0 con 0x562804e63d80 2024-03-12T05:24:10.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.586+0000 7f424effd640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.587+0000 7f424f7fe640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e73d80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4244009900 tx=0x7f4244004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a5ea0756b8e667 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.587+0000 7f422effd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.587+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424402c980 con 0x562804e63d80 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.587+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f424402de40 con 0x562804e63d80 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.588+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562804aca7e0 con 0x562804e63d80 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.588+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f424402cae0 con 0x562804e63d80 2024-03-12T05:24:10.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.588+0000 7f4258dad740 10 client.18834 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.588+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562804d8b8c0 con 0x562804e63d80 2024-03-12T05:24:10.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f424405a080 con 0x562804e63d80 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4244054030 con 0x562804e63d80 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 10 client.18834 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 10 client.18834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 20 client.18834 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804ed44f0 0x562804ed6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 4 client.18834 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.589+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562804aca7e0 con 0x562804ed44f0 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f424effd640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804ed44f0 0x562804ed6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f424effd640 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804ed44f0 0x562804ed6910 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=875cdb59a5188ac3 server_cookie=cd966530e344b689 in_seq=0 out_seq=0 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f422effd640 10 client.18834 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:10.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f422effd640 1 -- 172.21.15.45:0/3608365880 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562804aca7e0 con 0x562804ed44f0 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f422effd640 10 client.18834 handle_command_reply: tid=0 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.590+0000 7f4258dad740 1 client.18834 shutdown 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 msgr2=0x562804e73d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e73d80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4244009900 tx=0x7f4244004a90 comp rx=0 tx=0).stop 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 shutdown_connections 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562804e63d80 0x562804e73d80 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x562804ed44f0 0x562804ed6910 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562804e6c960 0x562804e71e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 --2- 172.21.15.45:0/3608365880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562804e63460 0x562804e73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 >> 172.21.15.45:0/3608365880 conn(0x562804e5db90 msgr2=0x562804e68760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f422effd640 0 client.18834 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 shutdown_connections 2024-03-12T05:24:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 1 -- 172.21.15.45:0/3608365880 wait complete. 2024-03-12T05:24:10.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.591+0000 7f4258dad740 20 client.18834 trim_cache size 0 max 16384 2024-03-12T05:24:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 msgr2=0x7f42400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 0x7f42400a40e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4248099990 tx=0x7f4248065d60 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 msgr2=0x7f425007a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f424800ff50 tx=0x7f424800d760 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 shutdown_connections 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f42400a1c20 0x7f42400a40e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4250107f40 0x7f4250081c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f425010d610 0x7f425007a950 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 --2- 172.21.15.45:0/3928048295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4250107580 0x7f42500816f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.592+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 >> 172.21.15.45:0/3928048295 conn(0x7f4250076d40 msgr2=0x7f42500786c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.593+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 shutdown_connections 2024-03-12T05:24:10.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.593+0000 7f42561d7640 1 -- 172.21.15.45:0/3928048295 wait complete. 2024-03-12T05:24:10.605 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:10.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:24:10.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:10 smithi130 ceph-mon[35398]: pgmap v416: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:24:10.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:10 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3595631121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:10.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:10 smithi116 ceph-mon[35486]: pgmap v416: 97 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 87 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:24:10.742 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:10 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3595631121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.829+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/3623633394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8103590 msgr2=0x7fe2f8103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.829+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/3623633394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8103590 0x7fe2f8103970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009930 tx=0x7fe2e802bc40 comp rx=0 tx=0).stop 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/3623633394 shutdown_connections 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/3623633394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8108660 0x7fe2f8110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/3623633394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8103f40 0x7fe2f8107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/3623633394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8103590 0x7fe2f8103970 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/3623633394 >> 172.21.15.45:0/3623633394 conn(0x7fe2f80fedf0 msgr2=0x7fe2f8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/3623633394 shutdown_connections 2024-03-12T05:24:10.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.830+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/3623633394 wait complete. 2024-03-12T05:24:10.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.831+0000 7fe2fe7c7640 1 Processor -- start 2024-03-12T05:24:10.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.832+0000 7fe2fe7c7640 1 -- start start 2024-03-12T05:24:10.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 0x7fe2f819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42598/0 (socket says 172.21.15.45:42598) 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 0x7fe2f8199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819a300 con 0x7fe2f8108660 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819a470 con 0x7fe2f8103590 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fe7c7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f819a630 con 0x7fe2f8199910 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 0x7fe2f819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2f7fff640 1 -- 172.21.15.45:0/1255834655 learned_addr learned my addr 172.21.15.45:0/1255834655 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.833+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 0x7fe2f8199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 -- 172.21.15.45:0/1255834655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 msgr2=0x7fe2f819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 -- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 msgr2=0x7fe2f819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 0x7fe2f819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 -- 172.21.15.45:0/1255834655 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0009840 con 0x7fe2f8199910 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 0x7fe2f819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 0x7fe2f8199dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000ec50 tx=0x7fe2f000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.834+0000 7fe2f57fa640 1 -- 172.21.15.45:0/1255834655 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0003b70 con 0x7fe2f8199910 2024-03-12T05:24:10.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.835+0000 7fe2f57fa640 1 -- 172.21.15.45:0/1255834655 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe2f000ce70 con 0x7fe2f8199910 2024-03-12T05:24:10.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.835+0000 7fe2f57fa640 1 -- 172.21.15.45:0/1255834655 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0010670 con 0x7fe2f8199910 2024-03-12T05:24:10.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.835+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e8009560 con 0x7fe2f8199910 2024-03-12T05:24:10.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.835+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81a6950 con 0x7fe2f8199910 2024-03-12T05:24:10.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.838+0000 7fe2f57fa640 1 -- 172.21.15.45:0/1255834655 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe2f0010890 con 0x7fe2f8199910 2024-03-12T05:24:10.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.839+0000 7fe2f57fa640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 0x7fe2e00a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.839+0000 7fe2f57fa640 1 -- 172.21.15.45:0/1255834655 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fe2f0014070 con 0x7fe2f8199910 2024-03-12T05:24:10.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.839+0000 7fe301397740 1 Processor -- start 2024-03-12T05:24:10.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.839+0000 7fe301397740 1 -- start start 2024-03-12T05:24:10.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 0x7fe2e00a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a38440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a4c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5159f3960 con 0x55f515a38060 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5159a7310 con 0x55f515a49f60 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe301397740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5159a4840 con 0x55f515a38a10 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45738/0 (socket says 172.21.15.45:45738) 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2f77fe640 1 -- 172.21.15.45:0/81445209 learned_addr learned my addr 172.21.15.45:0/81445209 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a4c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.840+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a38440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 -- 172.21.15.45:0/81445209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 msgr2=0x55f515a4c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a4c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 -- 172.21.15.45:0/81445209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 msgr2=0x55f515a38440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a38440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 -- 172.21.15.45:0/81445209 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f515962060 con 0x55f515a38a10 2024-03-12T05:24:10.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a4c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:10.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a38440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009f00 tx=0x7fe2ec011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbaa02337e4e552 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.841+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 0x7fe2e00a4230 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000c00 tx=0x7fe2e80058b0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.842+0000 7fe2de7fc640 1 -- 172.21.15.45:0/81445209 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec004230 con 0x55f515a38a10 2024-03-12T05:24:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.842+0000 7fe2de7fc640 1 -- 172.21.15.45:0/81445209 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe2ec004390 con 0x55f515a38a10 2024-03-12T05:24:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.842+0000 7fe2de7fc640 1 -- 172.21.15.45:0/81445209 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec004e50 con 0x55f515a38a10 2024-03-12T05:24:10.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 -- 172.21.15.45:0/81445209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 msgr2=0x55f515a49a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009f00 tx=0x7fe2ec011810 comp rx=0 tx=0).stop 2024-03-12T05:24:10.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 -- 172.21.15.45:0/81445209 shutdown_connections 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a4c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a49a20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 --2- 172.21.15.45:0/81445209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a38440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 -- 172.21.15.45:0/81445209 >> 172.21.15.45:0/81445209 conn(0x55f515a32980 msgr2=0x55f515a32d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 -- 172.21.15.45:0/81445209 shutdown_connections 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.843+0000 7fe301397740 1 -- 172.21.15.45:0/81445209 wait complete. 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 -- start start 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 Processor -- start 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a47020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a47aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5159a0020 con 0x55f515a38060 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515673430 con 0x55f515a49f60 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f51562fe00 con 0x55f515a38a10 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.844+0000 7fe301397740 10 client.0 _ll_register_callbacks cb 0x55f515947cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45754/0 (socket says 172.21.15.45:45754) 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1834068914 learned_addr learned my addr 172.21.15.45:0/1834068914 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a47aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a47020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 msgr2=0x55f515a47aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a47aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1834068914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 msgr2=0x55f515a47020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a47020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1834068914 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec009c00 con 0x55f515a38a10 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.845+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a47aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00fd00 tx=0x7fe2ec00fd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddb038d8c20aed9d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe2d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec019430 con 0x55f515a38a10 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fe2ec019590 con 0x55f515a38a10 2024-03-12T05:24:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f51569f7e0 con 0x55f515a38a10 2024-03-12T05:24:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2ec019750 con 0x55f515a38a10 2024-03-12T05:24:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe301397740 10 client.28270 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.846+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f51599e6a0 con 0x55f515a38a10 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f51569f7e0 con 0x55f515a38a10 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe2ec023990 con 0x55f515a38a10 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 10 client.28270 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 10 client.28270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 20 client.28270 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515a4fa80 0x55f515a51ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 4 client.28270 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5156dc720 con 0x55f515a4fa80 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.847+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515a4fa80 0x55f515a51ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.849+0000 7fe2d77fe640 10 client.28270 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.849+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515a4fa80 0x55f515a51ec0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c0987cf5b41c513 server_cookie=6f35bbbca49b744e in_seq=0 out_seq=0 2024-03-12T05:24:10.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.849+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1834068914 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5156dc720 con 0x55f515a4fa80 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.849+0000 7fe2d77fe640 10 client.28270 handle_command_reply: tid=0 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.849+0000 7fe301397740 1 client.28270 shutdown 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 msgr2=0x55f515a47560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00fd00 tx=0x7fe2ec00fd30 comp rx=0 tx=0).stop 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 shutdown_connections 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515a4fa80 0x55f515a51ec0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a47aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a47560 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 --2- 172.21.15.45:0/1834068914 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a47020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 >> 172.21.15.45:0/1834068914 conn(0x55f515a32980 msgr2=0x55f515a346d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe2d77fe640 0 client.28270 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 shutdown_connections 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 1 -- 172.21.15.45:0/1834068914 wait complete. 2024-03-12T05:24:10.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.850+0000 7fe301397740 20 client.28270 trim_cache size 0 max 16384 2024-03-12T05:24:10.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.874+0000 7fe301397740 1 Processor -- start 2024-03-12T05:24:10.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.875+0000 7fe301397740 1 -- start start 2024-03-12T05:24:10.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.875+0000 7fe301397740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a49750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.875+0000 7fe301397740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.870 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.875+0000 7fe301397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a49750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.871 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45758/0 (socket says 172.21.15.45:45758) 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe2f77fe640 1 -- 172.21.15.45:0/814339001 learned_addr learned my addr 172.21.15.45:0/814339001 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515925130 con 0x55f515a38060 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515a2df70 con 0x55f515a49f60 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.876+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515a333d0 con 0x55f515a38a10 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 -- 172.21.15.45:0/814339001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 msgr2=0x55f515a442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 -- 172.21.15.45:0/814339001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 msgr2=0x55f515a49750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a49750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 -- 172.21.15.45:0/814339001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f51593ac20 con 0x55f515a38a10 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.877+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009370 tx=0x7fe2ec0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fd7e5098e9e7056 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.878+0000 7fe2ddffb640 1 -- 172.21.15.45:0/814339001 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec039070 con 0x55f515a38a10 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.878+0000 7fe2ddffb640 1 -- 172.21.15.45:0/814339001 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe2ec018420 con 0x55f515a38a10 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.878+0000 7fe2ddffb640 1 -- 172.21.15.45:0/814339001 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0238c0 con 0x55f515a38a10 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 msgr2=0x55f515a46020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009370 tx=0x7fe2ec0096c0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 shutdown_connections 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a49f60 0x55f515a442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a38a10 0x55f515a46020 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 --2- 172.21.15.45:0/814339001 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a49750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 >> 172.21.15.45:0/814339001 conn(0x55f515a32980 msgr2=0x55f515a45910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.879+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 shutdown_connections 2024-03-12T05:24:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.880+0000 7fe301397740 1 -- 172.21.15.45:0/814339001 wait complete. 2024-03-12T05:24:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.880+0000 7fe301397740 1 -- start start 2024-03-12T05:24:10.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 Processor -- start 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a3ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a49f60 0x55f515a3c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515a3da10 con 0x55f515a38060 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515aca100 con 0x55f515a38a10 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe301397740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f515aca270 con 0x55f515a49f60 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe2f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42632/0 (socket says 172.21.15.45:42632) 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a3ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1439959219 learned_addr learned my addr 172.21.15.45:0/1439959219 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.881+0000 7fe2fcd64640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a49f60 0x55f515a3c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1439959219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a49f60 msgr2=0x55f515a3c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a49f60 0x55f515a3c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1439959219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 msgr2=0x55f515a3ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a3ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 -- 172.21.15.45:0/1439959219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f51599e6a0 con 0x55f515a38a10 2024-03-12T05:24:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2f77fe640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec023670 tx=0x7fe2ec0236a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f38a293e990eb383 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe301397740 10 client.0 _ll_register_callbacks cb 0x7fe2ec009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:10.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1439959219 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec039070 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f515a46e50 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.882+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1439959219 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fe2ec00fce0 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1439959219 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2ec005ab0 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 10 client.28310 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55f5159608c0 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1439959219 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe2ec0239d0 con 0x55f515a38a10 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 10 client.28310 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 10 client.28310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:10.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 20 client.28310 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515acb360 0x55f515acd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.883+0000 7fe301397740 4 client.28310 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f515a46e50 con 0x55f515acb360 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515acb360 0x55f515acd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe2f7fff640 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515acb360 0x55f515acd780 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f71aa41a1368b6 server_cookie=48415ab3d44f6319 in_seq=0 out_seq=0 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe2d77fe640 10 client.28310 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe2d77fe640 1 -- 172.21.15.45:0/1439959219 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f515a46e50 con 0x55f515acb360 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe2d77fe640 10 client.28310 handle_command_reply: tid=0 2024-03-12T05:24:10.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.884+0000 7fe301397740 1 client.28310 shutdown 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 msgr2=0x55f515a3bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec023670 tx=0x7fe2ec0236a0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 shutdown_connections 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f515acb360 0x55f515acd780 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f515a49f60 0x55f515a3c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe2d77fe640 0 client.28310 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f515a38a10 0x55f515a3bfb0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 --2- 172.21.15.45:0/1439959219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f515a38060 0x55f515a3ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.886+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 >> 172.21.15.45:0/1439959219 conn(0x55f515a32980 msgr2=0x55f515a461a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.887+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 shutdown_connections 2024-03-12T05:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.887+0000 7fe301397740 1 -- 172.21.15.45:0/1439959219 wait complete. 2024-03-12T05:24:10.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.887+0000 7fe301397740 20 client.28310 trim_cache size 0 max 16384 2024-03-12T05:24:10.884 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 msgr2=0x7fe2e00a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 0x7fe2e00a4230 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000c00 tx=0x7fe2e80058b0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 msgr2=0x7fe2f8199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 0x7fe2f8199dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000ec50 tx=0x7fe2f000c6a0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 shutdown_connections 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe2e00a1d70 0x7fe2e00a4230 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f8108660 0x7fe2f819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2f8199910 0x7fe2f8199dc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 --2- 172.21.15.45:0/1255834655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2f8103590 0x7fe2f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 >> 172.21.15.45:0/1255834655 conn(0x7fe2f80fedf0 msgr2=0x7fe2f80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:10.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 shutdown_connections 2024-03-12T05:24:10.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:10.888+0000 7fe2fe7c7640 1 -- 172.21.15.45:0/1255834655 wait complete. 2024-03-12T05:24:11.905 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:24:11.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:24:11.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:24:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.087+0000 7fdc90e81640 1 -- 172.21.15.45:0/2197589507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c106100 msgr2=0x7fdc8c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.087+0000 7fdc90e81640 1 --2- 172.21.15.45:0/2197589507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c106100 0x7fdc8c1064e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009be0 tx=0x7fdc7c02bf80 comp rx=0 tx=0).stop 2024-03-12T05:24:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.090+0000 7fdc90e81640 1 -- 172.21.15.45:0/2197589507 shutdown_connections 2024-03-12T05:24:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.090+0000 7fdc90e81640 1 --2- 172.21.15.45:0/2197589507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c100040 0x7fdc8c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.090+0000 7fdc90e81640 1 --2- 172.21.15.45:0/2197589507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 0x7fdc8c10cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.090+0000 7fdc90e81640 1 --2- 172.21.15.45:0/2197589507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c106100 0x7fdc8c1064e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.091+0000 7fdc90e81640 1 -- 172.21.15.45:0/2197589507 >> 172.21.15.45:0/2197589507 conn(0x7fdc8c077110 msgr2=0x7fdc8c077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.092+0000 7fdc90e81640 1 -- 172.21.15.45:0/2197589507 shutdown_connections 2024-03-12T05:24:12.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.092+0000 7fdc90e81640 1 -- 172.21.15.45:0/2197589507 wait complete. 2024-03-12T05:24:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.093+0000 7fdc90e81640 1 Processor -- start 2024-03-12T05:24:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.093+0000 7fdc90e81640 1 -- start start 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c100040 0x7fdc8c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 0x7fdc8c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1991b0 con 0x7fdc8c106100 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c199320 con 0x7fdc8c1009e0 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc90e81640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1a3a90 con 0x7fdc8c100040 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc8ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc8ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59126/0 (socket says 172.21.15.45:59126) 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/4199462889 learned_addr learned my addr 172.21.15.45:0/4199462889 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 0x7fdc8c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.094+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c100040 0x7fdc8c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/4199462889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c100040 msgr2=0x7fdc8c198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c100040 0x7fdc8c198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/4199462889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 msgr2=0x7fdc8c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 0x7fdc8c1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/4199462889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74009560 con 0x7fdc8c106100 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400e970 tx=0x7fdc8400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.095+0000 7fdc837fe640 1 -- 172.21.15.45:0/4199462889 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84009720 con 0x7fdc8c106100 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.096+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c009840 con 0x7fdc8c106100 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.096+0000 7fdc837fe640 1 -- 172.21.15.45:0/4199462889 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc840054d0 con 0x7fdc8c106100 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.096+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1a40f0 con 0x7fdc8c106100 2024-03-12T05:24:12.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.096+0000 7fdc837fe640 1 -- 172.21.15.45:0/4199462889 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84010430 con 0x7fdc8c106100 2024-03-12T05:24:12.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.098+0000 7fdc837fe640 1 -- 172.21.15.45:0/4199462889 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fdc84010590 con 0x7fdc8c106100 2024-03-12T05:24:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.099+0000 7fdc837fe640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 0x7fdc780a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.099+0000 7fdc837fe640 1 -- 172.21.15.45:0/4199462889 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fdc84014070 con 0x7fdc8c106100 2024-03-12T05:24:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.099+0000 7fdc93a52740 1 Processor -- start 2024-03-12T05:24:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 0x7fdc780a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 -- start start 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027560760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a02756bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0274cef90 con 0x55a027560d30 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0274cc4c0 con 0x55a0275698b0 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc93a52740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0274c7ca0 con 0x55a027560380 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59138/0 (socket says 172.21.15.45:59138) 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc89d9b640 1 -- 172.21.15.45:0/2396706556 learned_addr learned my addr 172.21.15.45:0/2396706556 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.100+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027560760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a02756bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/2396706556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 msgr2=0x55a027560760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027560760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/2396706556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 msgr2=0x55a0275692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/2396706556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02748a0b0 con 0x55a0275698b0 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a02756bd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdc840cc060 tx=0x7fdc8409b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baae24e6933b6ace server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.101+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 0x7fdc780a4230 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c004af0 tx=0x7fdc7c000f40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.102+0000 7fdc80ff9640 1 -- 172.21.15.45:0/2396706556 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8409b540 con 0x55a0275698b0 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.102+0000 7fdc80ff9640 1 -- 172.21.15.45:0/2396706556 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc8409f690 con 0x55a0275698b0 2024-03-12T05:24:12.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.102+0000 7fdc80ff9640 1 -- 172.21.15.45:0/2396706556 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc840cc2e0 con 0x55a0275698b0 2024-03-12T05:24:12.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 -- 172.21.15.45:0/2396706556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 msgr2=0x55a02756bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a02756bd00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdc840cc060 tx=0x7fdc8409b390 comp rx=0 tx=0).stop 2024-03-12T05:24:12.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 -- 172.21.15.45:0/2396706556 shutdown_connections 2024-03-12T05:24:12.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a02756bd00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 --2- 172.21.15.45:0/2396706556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027560760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 -- 172.21.15.45:0/2396706556 >> 172.21.15.45:0/2396706556 conn(0x55a02755a840 msgr2=0x55a02755ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 -- 172.21.15.45:0/2396706556 shutdown_connections 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.103+0000 7fdc93a52740 1 -- 172.21.15.45:0/2396706556 wait complete. 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.104+0000 7fdc93a52740 1 -- start start 2024-03-12T05:24:12.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.104+0000 7fdc93a52740 1 Processor -- start 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.104+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027566810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.104+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a027566d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027567290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc93a52740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02746cf40 con 0x55a027560d30 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc93a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02719b4b0 con 0x55a0275698b0 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc93a52740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02744d670 con 0x55a027560380 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc93a52740 10 client.0 _ll_register_callbacks cb 0x55a0274baf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a027566d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027567290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a027566d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59140/0 (socket says 172.21.15.45:59140) 2024-03-12T05:24:12.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc89d9b640 1 -- 172.21.15.45:0/3511815741 learned_addr learned my addr 172.21.15.45:0/3511815741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027566810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/3511815741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 msgr2=0x55a027566810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027566810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/3511815741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 msgr2=0x55a027566d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a027566d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8ad9d640 1 -- 172.21.15.45:0/3511815741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1a40f0 con 0x55a0275698b0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.105+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027566810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027567290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400c6a0 tx=0x7fdc8409bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdb9f84f81d7d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8409b540 con 0x55a0275698b0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fdc840cc2e0 con 0x55a0275698b0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0274a1f40 con 0x55a0275698b0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc840cc4e0 con 0x55a0275698b0 2024-03-12T05:24:12.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc93a52740 10 client.28322 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:12.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.106+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a0274c6320 con 0x55a0275698b0 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a0274a1f40 con 0x55a0275698b0 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdc8401f060 con 0x55a0275698b0 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 10 client.28322 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 10 client.28322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 20 client.28322 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a0275823e0 0x55a027584800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 4 client.28322 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.107+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0274be310 con 0x55a0275823e0 2024-03-12T05:24:12.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.108+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a0275823e0 0x55a027584800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.110+0000 7fdc71ffb640 10 client.28322 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.110+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a0275823e0 0x55a027584800 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86a4b8a93657612 server_cookie=6af08e3802544c5 in_seq=0 out_seq=0 2024-03-12T05:24:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.110+0000 7fdc71ffb640 1 -- 172.21.15.45:0/3511815741 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0274be310 con 0x55a0275823e0 2024-03-12T05:24:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.110+0000 7fdc71ffb640 10 client.28322 handle_command_reply: tid=0 2024-03-12T05:24:12.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.110+0000 7fdc93a52740 1 client.28322 shutdown 2024-03-12T05:24:12.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.111+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 msgr2=0x55a027567290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027567290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400c6a0 tx=0x7fdc8409bd00 comp rx=0 tx=0).stop 2024-03-12T05:24:12.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 shutdown_connections 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a0275823e0 0x55a027584800 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc71ffb640 0 client.28322 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a027566d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027567290 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 --2- 172.21.15.45:0/3511815741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027566810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 >> 172.21.15.45:0/3511815741 conn(0x55a02755a840 msgr2=0x55a02755c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 shutdown_connections 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 1 -- 172.21.15.45:0/3511815741 wait complete. 2024-03-12T05:24:12.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.112+0000 7fdc93a52740 20 client.28322 trim_cache size 0 max 16384 2024-03-12T05:24:12.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.143+0000 7fdc93a52740 1 Processor -- start 2024-03-12T05:24:12.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.145+0000 7fdc93a52740 1 -- start start 2024-03-12T05:24:12.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.146+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027568f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.146+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.146+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027563a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.146+0000 7fdc93a52740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02755b410 con 0x55a027560d30 2024-03-12T05:24:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc93a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a027567b60 con 0x55a0275698b0 2024-03-12T05:24:12.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc8ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027563a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59146/0 (socket says 172.21.15.45:59146) 2024-03-12T05:24:12.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 -- 172.21.15.45:0/1310399593 learned_addr learned my addr 172.21.15.45:0/1310399593 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02755bc80 con 0x55a027560380 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027568f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 -- 172.21.15.45:0/1310399593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 msgr2=0x55a027568f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027568f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 -- 172.21.15.45:0/1310399593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 msgr2=0x55a027563a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.147+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027563a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc89d9b640 1 -- 172.21.15.45:0/1310399593 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a027462c00 con 0x55a027560d30 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027568f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009fb0 tx=0x7fdc74009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3702699a260ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027563a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:12.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc73fff640 1 -- 172.21.15.45:0/1310399593 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74017050 con 0x55a027560d30 2024-03-12T05:24:12.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc73fff640 1 -- 172.21.15.45:0/1310399593 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdc74018040 con 0x55a027560d30 2024-03-12T05:24:12.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.148+0000 7fdc73fff640 1 -- 172.21.15.45:0/1310399593 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74010670 con 0x55a027560d30 2024-03-12T05:24:12.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.149+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 msgr2=0x55a0275670b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.149+0000 7fdc93a52740 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009fb0 tx=0x7fdc74009260 comp rx=0 tx=0).stop 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 shutdown_connections 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560d30 0x55a0275670b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a027563a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 --2- 172.21.15.45:0/1310399593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560380 0x55a027568f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 >> 172.21.15.45:0/1310399593 conn(0x55a02755a840 msgr2=0x55a0275650e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 shutdown_connections 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 -- 172.21.15.45:0/1310399593 wait complete. 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 -- start start 2024-03-12T05:24:12.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.150+0000 7fdc93a52740 1 Processor -- start 2024-03-12T05:24:12.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560380 0x55a02756f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a0275d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a027570630 con 0x55a027560380 2024-03-12T05:24:12.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0275d8df0 con 0x55a0275698b0 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a027578840 con 0x55a027560d30 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc93a52740 10 client.0 _ll_register_callbacks cb 0x7fdc84005640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45826/0 (socket says 172.21.15.45:45826) 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 -- 172.21.15.45:0/347451613 learned_addr learned my addr 172.21.15.45:0/347451613 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a0275d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 -- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 msgr2=0x55a0275d88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a0275d88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 -- 172.21.15.45:0/347451613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560380 msgr2=0x55a02756f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560380 0x55a02756f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.151+0000 7fdc89d9b640 1 -- 172.21.15.45:0/347451613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc74009c60 con 0x55a027560d30 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.152+0000 7fdc8ad9d640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a0275d88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.152+0000 7fdc89d9b640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009530 tx=0x7fdc7400f6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c997f6df0ac26e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.152+0000 7fdc71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:12.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.152+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74004020 con 0x55a027560d30 2024-03-12T05:24:12.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0274ed900 con 0x55a027560d30 2024-03-12T05:24:12.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fdc74004180 con 0x55a027560d30 2024-03-12T05:24:12.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc74017020 con 0x55a027560d30 2024-03-12T05:24:12.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc93a52740 10 client.28291 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:12.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a027488910 con 0x55a027560d30 2024-03-12T05:24:12.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.153+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a0274ed900 con 0x55a027560d30 2024-03-12T05:24:12.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdc74004a70 con 0x55a027560d30 2024-03-12T05:24:12.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 10 client.28291 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:12.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 10 client.28291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 20 client.28291 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a027579f00 0x55a02757c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 4 client.28291 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.154+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a0271c7860 con 0x55a027579f00 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.155+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a027579f00 0x55a02757c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.155+0000 7fdc8a59c640 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a027579f00 0x55a02757c320 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e06c07872717fa5c server_cookie=a460972ef14b7663 in_seq=0 out_seq=0 2024-03-12T05:24:12.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.155+0000 7fdc71ffb640 10 client.28291 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc71ffb640 1 -- 172.21.15.45:0/347451613 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a0271c7860 con 0x55a027579f00 2024-03-12T05:24:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc71ffb640 10 client.28291 handle_command_reply: tid=0 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 client.28291 shutdown 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 msgr2=0x55a02756ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009530 tx=0x7fdc7400f6b0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 shutdown_connections 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55a027579f00 0x55a02757c320 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a0275698b0 0x55a0275d88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a027560d30 0x55a02756ff30 secure :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009530 tx=0x7fdc7400f6b0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 --2- 172.21.15.45:0/347451613 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a027560380 0x55a02756f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 >> 172.21.15.45:0/347451613 conn(0x55a02755a840 msgr2=0x55a02755be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.157+0000 7fdc71ffb640 0 client.28291 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.158+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 shutdown_connections 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.158+0000 7fdc93a52740 1 -- 172.21.15.45:0/347451613 wait complete. 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.158+0000 7fdc93a52740 20 client.28291 trim_cache size 0 max 16384 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "293e7fdf-a0d5-4662-8e8e-c063c9f51cd6", 2024-03-12T05:24:12.153 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 msgr2=0x7fdc780a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 0x7fdc780a4230 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c004af0 tx=0x7fdc7c000f40 comp rx=0 tx=0).stop 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 msgr2=0x7fdc8c1a3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400e970 tx=0x7fdc8400ed70 comp rx=0 tx=0).stop 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 shutdown_connections 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdc780a1d70 0x7fdc780a4230 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc8c106100 0x7fdc8c1a3500 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc8c1009e0 0x7fdc8c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 --2- 172.21.15.45:0/4199462889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdc8c100040 0x7fdc8c198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 >> 172.21.15.45:0/4199462889 conn(0x7fdc8c077110 msgr2=0x7fdc8c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 shutdown_connections 2024-03-12T05:24:12.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.159+0000 7fdc90e81640 1 -- 172.21.15.45:0/4199462889 wait complete. 2024-03-12T05:24:12.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:24:12.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.370+0000 7f85dbfc7640 1 -- 172.21.15.45:0/3806466476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4108660 msgr2=0x7f85d4110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.370+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/3806466476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4108660 0x7f85d4110230 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009c10 tx=0x7f85cc02bef0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.371+0000 7f85dbfc7640 1 -- 172.21.15.45:0/3806466476 shutdown_connections 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.371+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/3806466476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4108660 0x7f85d4110230 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.371+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/3806466476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4103f40 0x7f85d4107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.371+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/3806466476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d4103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.371+0000 7f85dbfc7640 1 -- 172.21.15.45:0/3806466476 >> 172.21.15.45:0/3806466476 conn(0x7f85d40fee70 msgr2=0x7f85d4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.372+0000 7f85dbfc7640 1 -- 172.21.15.45:0/3806466476 shutdown_connections 2024-03-12T05:24:12.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.372+0000 7f85dbfc7640 1 -- 172.21.15.45:0/3806466476 wait complete. 2024-03-12T05:24:12.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.373+0000 7f85dbfc7640 1 Processor -- start 2024-03-12T05:24:12.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85dbfc7640 1 -- start start 2024-03-12T05:24:12.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85dbfc7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85dbfc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45842/0 (socket says 172.21.15.45:45842) 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.374+0000 7f85d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42700/0 (socket says 172.21.15.45:42700) 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85d9562640 1 -- 172.21.15.45:0/1001842991 learned_addr learned my addr 172.21.15.45:0/1001842991 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4199740 0x7f85d4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419a210 con 0x7f85d4199740 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419a380 con 0x7f85d4108660 2024-03-12T05:24:12.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.375+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419a540 con 0x7f85d4103590 2024-03-12T05:24:12.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85da564640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4199740 0x7f85d4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85d9562640 1 -- 172.21.15.45:0/1001842991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 msgr2=0x7f85d419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85d9562640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85d9562640 1 -- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4199740 msgr2=0x7f85d4199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85d9562640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4199740 0x7f85d4199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.376+0000 7f85d9562640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c4009560 con 0x7f85d4108660 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85d9562640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e9a0 tx=0x7f85c400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85caffd640 1 -- 172.21.15.45:0/1001842991 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c400cd80 con 0x7f85d4108660 2024-03-12T05:24:12.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85caffd640 1 -- 172.21.15.45:0/1001842991 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85c40044d0 con 0x7f85d4108660 2024-03-12T05:24:12.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85caffd640 1 -- 172.21.15.45:0/1001842991 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4010690 con 0x7f85d4108660 2024-03-12T05:24:12.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85cc009840 con 0x7f85d4108660 2024-03-12T05:24:12.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.377+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d41a6790 con 0x7f85d4108660 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.381+0000 7f85deb97740 1 Processor -- start 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 -- start start 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08efac0 0x55d5e08efea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e0856280 con 0x55d5e08efac0 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e08537b0 con 0x55d5e08f8fc0 2024-03-12T05:24:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85deb97740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e084ef90 con 0x55d5e08f03e0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85caffd640 1 -- 172.21.15.45:0/1001842991 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f85c4003e40 con 0x7f85d4108660 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42710/0 (socket says 172.21.15.45:42710) 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85d9562640 1 -- 172.21.15.45:0/2051943850 learned_addr learned my addr 172.21.15.45:0/2051943850 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.382+0000 7f85d9d63640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85da564640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08efac0 0x55d5e08efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 -- 172.21.15.45:0/2051943850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 msgr2=0x55d5e08f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 -- 172.21.15.45:0/2051943850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08efac0 msgr2=0x55d5e08efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08efac0 0x55d5e08efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 -- 172.21.15.45:0/2051943850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5e080d570 con 0x55d5e08f8fc0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9d63640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85d9562640 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cb1e0 tx=0x7f85c40d3e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=434d08cb5fc5b74f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85caffd640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 0x7f85c00a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.383+0000 7f85bffff640 1 -- 172.21.15.45:0/2051943850 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4096a20 con 0x55d5e08f8fc0 2024-03-12T05:24:12.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.384+0000 7f85caffd640 1 -- 172.21.15.45:0/1001842991 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f85c4014070 con 0x7f85d4108660 2024-03-12T05:24:12.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.384+0000 7f85bffff640 1 -- 172.21.15.45:0/2051943850 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85c40d3620 con 0x55d5e08f8fc0 2024-03-12T05:24:12.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.384+0000 7f85bffff640 1 -- 172.21.15.45:0/2051943850 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c409a780 con 0x55d5e08f8fc0 2024-03-12T05:24:12.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.384+0000 7f85da564640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 0x7f85c00a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 -- 172.21.15.45:0/2051943850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 msgr2=0x55d5e08fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cb1e0 tx=0x7f85c40d3e40 comp rx=0 tx=0).stop 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 -- 172.21.15.45:0/2051943850 shutdown_connections 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fb3b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 --2- 172.21.15.45:0/2051943850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08efac0 0x55d5e08efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.385+0000 7f85deb97740 1 -- 172.21.15.45:0/2051943850 >> 172.21.15.45:0/2051943850 conn(0x55d5e08ea1a0 msgr2=0x55d5e08ea580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.386+0000 7f85deb97740 1 -- 172.21.15.45:0/2051943850 shutdown_connections 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.386+0000 7f85deb97740 1 -- 172.21.15.45:0/2051943850 wait complete. 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.386+0000 7f85da564640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 0x7f85c00a4160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006010 tx=0x7f85cc009280 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.387+0000 7f85deb97740 1 -- start start 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.387+0000 7f85deb97740 1 Processor -- start 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e0522430 con 0x55d5e08f8fc0 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e04dee00 con 0x55d5e08efac0 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85d9562640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85d9562640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:45854/0 (socket says 172.21.15.45:45854) 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85d9562640 1 -- 172.21.15.45:0/1371277756 learned_addr learned my addr 172.21.15.45:0/1371277756 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e07d3fd0 con 0x55d5e08f03e0 2024-03-12T05:24:12.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.388+0000 7f85deb97740 10 client.0 _ll_register_callbacks cb 0x55d5e0829fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:12.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85d9d63640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 -- 172.21.15.45:0/1371277756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 msgr2=0x55d5e08f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 -- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 msgr2=0x55d5e08f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.389+0000 7f85da564640 1 -- 172.21.15.45:0/1371277756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d41a6790 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.390+0000 7f85da564640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f68c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f85cc02c430 tx=0x7f85cc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2dc9f37f955dfc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.390+0000 7f85bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.390+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc0040c0 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.390+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f85cc045070 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.390+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5e058b720 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.391+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85cc004220 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.391+0000 7f85deb97740 10 client.18876 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.391+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d5e084d610 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.391+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d5e058b720 con 0x55d5e08f8fc0 2024-03-12T05:24:12.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f85cc052020 con 0x55d5e08f8fc0 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 10 client.18876 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 10 client.18876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 20 client.18876 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0909220 0x55d5e090b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 4 client.18876 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5e05da890 con 0x55d5e0909220 2024-03-12T05:24:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.392+0000 7f85d9d63640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0909220 0x55d5e090b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.393+0000 7f85d9d63640 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0909220 0x55d5e090b610 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c5080f3217b0cc server_cookie=2d1fbfb1ca45ef64 in_seq=0 out_seq=0 2024-03-12T05:24:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.393+0000 7f85bd7fa640 10 client.18876 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.394+0000 7f85bd7fa640 1 -- 172.21.15.45:0/1371277756 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d5e05da890 con 0x55d5e0909220 2024-03-12T05:24:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.394+0000 7f85bd7fa640 10 client.18876 handle_command_reply: tid=0 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.394+0000 7f85deb97740 1 client.18876 shutdown 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.394+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 msgr2=0x55d5e08f68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.394+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f68c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f85cc02c430 tx=0x7f85cc009500 comp rx=0 tx=0).stop 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 shutdown_connections 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0909220 0x55d5e090b610 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f68c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 --2- 172.21.15.45:0/1371277756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 >> 172.21.15.45:0/1371277756 conn(0x55d5e08ea1a0 msgr2=0x55d5e08ebf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85bd7fa640 0 client.18876 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 shutdown_connections 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 1 -- 172.21.15.45:0/1371277756 wait complete. 2024-03-12T05:24:12.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.395+0000 7f85deb97740 20 client.18876 trim_cache size 0 max 16384 2024-03-12T05:24:12.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.422+0000 7f85deb97740 1 Processor -- start 2024-03-12T05:24:12.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.422+0000 7f85deb97740 1 -- start start 2024-03-12T05:24:12.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85deb97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85deb97740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85deb97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42728/0 (socket says 172.21.15.45:42728) 2024-03-12T05:24:12.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.423+0000 7f85d9d63640 1 -- 172.21.15.45:0/3565665781 learned_addr learned my addr 172.21.15.45:0/3565665781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.424+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e05da890 con 0x55d5e08f8fc0 2024-03-12T05:24:12.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.424+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e058b720 con 0x55d5e08efac0 2024-03-12T05:24:12.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.424+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e08dd320 con 0x55d5e08f03e0 2024-03-12T05:24:12.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85da564640 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85d9d63640 1 -- 172.21.15.45:0/3565665781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 msgr2=0x55d5e08f66e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85d9d63640 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f66e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85d9d63640 1 -- 172.21.15.45:0/3565665781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 msgr2=0x55d5e08f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85d9d63640 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.425+0000 7f85d9d63640 1 -- 172.21.15.45:0/3565665781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5e07e9ac0 con 0x55d5e08efac0 2024-03-12T05:24:12.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.426+0000 7f85d9d63640 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009bc0 tx=0x7f85d0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8896ccc2a68e318 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.427+0000 7f85bf7fe640 1 -- 172.21.15.45:0/3565665781 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0004220 con 0x55d5e08efac0 2024-03-12T05:24:12.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.427+0000 7f85bf7fe640 1 -- 172.21.15.45:0/3565665781 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85d0018040 con 0x55d5e08efac0 2024-03-12T05:24:12.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.427+0000 7f85bf7fe640 1 -- 172.21.15.45:0/3565665781 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0003ab0 con 0x55d5e08efac0 2024-03-12T05:24:12.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 msgr2=0x55d5e08f8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009bc0 tx=0x7f85d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 shutdown_connections 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08f03e0 0x55d5e08f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 --2- 172.21.15.45:0/3565665781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08efac0 0x55d5e08f8570 secure :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009bc0 tx=0x7f85d0005f90 comp rx=0 tx=0).stop 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 >> 172.21.15.45:0/3565665781 conn(0x55d5e08ea1a0 msgr2=0x55d5e08f4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.431+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 shutdown_connections 2024-03-12T05:24:12.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.432+0000 7f85deb97740 1 -- 172.21.15.45:0/3565665781 wait complete. 2024-03-12T05:24:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.433+0000 7f85deb97740 1 -- start start 2024-03-12T05:24:12.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.433+0000 7f85deb97740 1 Processor -- start 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.434+0000 7f85deb97740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.434+0000 7f85deb97740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.434+0000 7f85deb97740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 0x55d5e095f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.434+0000 7f85deb97740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e095fc30 con 0x55d5e08f8fc0 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85deb97740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e095fda0 con 0x55d5e08f03e0 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42736/0 (socket says 172.21.15.45:42736) 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5e095ff60 con 0x55d5e08ff490 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85d9d63640 1 -- 172.21.15.45:0/54747571 learned_addr learned my addr 172.21.15.45:0/54747571 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 0x55d5e095f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85deb97740 10 client.0 _ll_register_callbacks cb 0x7f85c40d0f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:12.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85d9562640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 -- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 msgr2=0x55d5e08fea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 -- 172.21.15.45:0/54747571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 msgr2=0x55d5e08fef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 -- 172.21.15.45:0/54747571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d0009e70 con 0x55d5e08ff490 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85d9d63640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85da564640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 0x55d5e095f6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009310 tx=0x7f85cc034ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eafdb445c0996fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:12.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.435+0000 7f85bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc05c840 con 0x55d5e08ff490 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f85cc045040 con 0x55d5e08ff490 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5e054e7e0 con 0x55d5e08ff490 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85cc005b70 con 0x55d5e08ff490 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85deb97740 10 client.28306 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:12.432 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.436+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d5e080f6a0 con 0x55d5e08ff490 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f85cc070080 con 0x55d5e08ff490 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f85cc06a030 con 0x55d5e08ff490 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 10 client.28306 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 10 client.28306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 20 client.28306 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0961310 0x55d5e0963730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 4 client.28306 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d5e054e7e0 con 0x55d5e0961310 2024-03-12T05:24:12.433 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.437+0000 7f85d9d63640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0961310 0x55d5e0963730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.438+0000 7f85d9d63640 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0961310 0x55d5e0963730 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3551818c1edad075 server_cookie=6f677c64a8931d11 in_seq=0 out_seq=0 2024-03-12T05:24:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.438+0000 7f85bd7fa640 10 client.28306 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.438+0000 7f85bd7fa640 1 -- 172.21.15.45:0/54747571 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d5e054e7e0 con 0x55d5e0961310 2024-03-12T05:24:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.439+0000 7f85bd7fa640 10 client.28306 handle_command_reply: tid=0 2024-03-12T05:24:12.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.439+0000 7f85deb97740 1 client.28306 shutdown 2024-03-12T05:24:12.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 msgr2=0x55d5e095f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.437 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 0x55d5e095f6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009310 tx=0x7f85cc034ef0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 shutdown_connections 2024-03-12T05:24:12.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d5e0961310 0x55d5e0963730 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.438 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d5e08f8fc0 0x55d5e08fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85bd7fa640 0 client.28306 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d5e08ff490 0x55d5e095f6f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 --2- 172.21.15.45:0/54747571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d5e08f03e0 0x55d5e08fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.443+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 >> 172.21.15.45:0/54747571 conn(0x55d5e08ea1a0 msgr2=0x55d5e08ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.444+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 shutdown_connections 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.444+0000 7f85deb97740 1 -- 172.21.15.45:0/54747571 wait complete. 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.444+0000 7f85deb97740 20 client.28306 trim_cache size 0 max 16384 2024-03-12T05:24:12.439 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "293e7fdf-a0d5-4662-8e8e-c063c9f51cd6": { 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "293e7fdf-a0d5-4662-8e8e-c063c9f51cd6", 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:24:12.440 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 msgr2=0x7f85c00a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 0x7f85c00a4160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006010 tx=0x7f85cc009280 comp rx=0 tx=0).stop 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 msgr2=0x7f85d419fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e9a0 tx=0x7f85c400ee60 comp rx=0 tx=0).stop 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 shutdown_connections 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f85c00a1ca0 0x7f85c00a4160 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85d4199740 0x7f85d4199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85d4108660 0x7f85d419fb70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.442 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 --2- 172.21.15.45:0/1001842991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85d4103590 0x7f85d419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:12.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.445+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 >> 172.21.15.45:0/1001842991 conn(0x7f85d40fee70 msgr2=0x7f85d41002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:12.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.446+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 shutdown_connections 2024-03-12T05:24:12.443 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:12.446+0000 7f85dbfc7640 1 -- 172.21.15.45:0/1001842991 wait complete. 2024-03-12T05:24:12.458 INFO:tasks.cephfs.filesystem:scrub status for tag:293e7fdf-a0d5-4662-8e8e-c063c9f51cd6 - {'path': '/', 'tag': '293e7fdf-a0d5-4662-8e8e-c063c9f51cd6', 'options': 'recursive,force'} 2024-03-12T05:24:12.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:12 smithi045 ceph-mon[32174]: pgmap v417: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:12.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:12 smithi130 ceph-mon[35398]: pgmap v417: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:12.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:12 smithi116 ceph-mon[35486]: pgmap v417: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:13 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:24:13.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:13.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:13.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:13.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:13.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.717 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:13 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:13.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:13 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:14.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:14 smithi045 ceph-mon[32174]: pgmap v418: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:14.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:14 smithi130 ceph-mon[35398]: pgmap v418: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:14.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:14 smithi116 ceph-mon[35486]: pgmap v418: 97 pgs: 21 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 64 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 1.21k op/s 2024-03-12T05:24:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 730.0 s, 1113.6 tps, lat 90.051 ms stddev 29.241 2024-03-12T05:24:16.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:16 smithi045 ceph-mon[32174]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:16.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:16 smithi130 ceph-mon[35398]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:16.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:16 smithi116 ceph-mon[35486]: pgmap v419: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:18.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:18 smithi045 ceph-mon[32174]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:18.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:18 smithi130 ceph-mon[35398]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:18.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:18 smithi116 ceph-mon[35486]: pgmap v420: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 9.5 MiB/s wr, 1.09k op/s 2024-03-12T05:24:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 735.0 s, 1320.5 tps, lat 75.689 ms stddev 20.902 2024-03-12T05:24:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:20 smithi045 ceph-mon[32174]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:24:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:20 smithi045 ceph-mon[32174]: osdmap e104: 12 total, 12 up, 12 in 2024-03-12T05:24:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:20 smithi130 ceph-mon[35398]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:24:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:20 smithi130 ceph-mon[35398]: osdmap e104: 12 total, 12 up, 12 in 2024-03-12T05:24:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:20 smithi116 ceph-mon[35486]: pgmap v421: 97 pgs: 97 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 10 MiB/s wr, 1.18k op/s 2024-03-12T05:24:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:20 smithi116 ceph-mon[35486]: osdmap e104: 12 total, 12 up, 12 in 2024-03-12T05:24:22.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:22 smithi045 ceph-mon[32174]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:22.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:22 smithi130 ceph-mon[35398]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:22.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:22 smithi116 ceph-mon[35486]: pgmap v423: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:23.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:23.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:23.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:23.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:23 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:23.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:23.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:23.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:23.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:23 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:23.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:23.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:23.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:23.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:23 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:24.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:24 smithi045 ceph-mon[32174]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:24.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 740.0 s, 1226.1 tps, lat 81.560 ms stddev 29.470 2024-03-12T05:24:24.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:24 smithi130 ceph-mon[35398]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:24.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:24 smithi116 ceph-mon[35486]: pgmap v424: 97 pgs: 97 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 9.9 MiB/s wr, 1.14k op/s 2024-03-12T05:24:25.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:25 smithi045 ceph-mon[32174]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:25.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:25 smithi130 ceph-mon[35398]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:25.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:25 smithi116 ceph-mon[35486]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:27.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:27 smithi116 ceph-mon[35486]: pgmap v426: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:28.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:27 smithi045 ceph-mon[32174]: pgmap v426: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:27 smithi130 ceph-mon[35398]: pgmap v426: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-03-12T05:24:28.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:28 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:24:29.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:28 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:24:29.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:28 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:24:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 745.0 s, 1176.0 tps, lat 85.188 ms stddev 30.716 2024-03-12T05:24:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:29 smithi045 ceph-mon[32174]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:24:30.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:29 smithi130 ceph-mon[35398]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:24:30.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:29 smithi116 ceph-mon[35486]: pgmap v427: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 12 MiB/s wr, 1.35k op/s 2024-03-12T05:24:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:30 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:24:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:30 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:24:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:30 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:24:31.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:30 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:24:31.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:30 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:24:31.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:30 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:24:31.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:30 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:24:31.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:30 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:24:31.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:30 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:24:32.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:31 smithi045 ceph-mon[32174]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:24:32.116 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:31 smithi130 ceph-mon[35398]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:24:32.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:31 smithi116 ceph-mon[35486]: pgmap v428: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 9.5 MiB/s wr, 1.10k op/s 2024-03-12T05:24:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:33 smithi045 ceph-mon[32174]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:24:34.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:33 smithi130 ceph-mon[35398]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:24:34.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:33 smithi116 ceph-mon[35486]: pgmap v429: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.8 MiB/s wr, 1.02k op/s 2024-03-12T05:24:34.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 750.0 s, 1174.4 tps, lat 84.943 ms stddev 39.574 2024-03-12T05:24:36.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:35 smithi045 ceph-mon[32174]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:36.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:35 smithi130 ceph-mon[35398]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:36.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:35 smithi116 ceph-mon[35486]: pgmap v430: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 1.53k op/s 2024-03-12T05:24:38.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:37 smithi045 ceph-mon[32174]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 8.7 MiB/s wr, 988 op/s 2024-03-12T05:24:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:37 smithi130 ceph-mon[35398]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 8.7 MiB/s wr, 988 op/s 2024-03-12T05:24:38.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:37 smithi116 ceph-mon[35486]: pgmap v431: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 8.7 MiB/s wr, 988 op/s 2024-03-12T05:24:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 755.0 s, 1236.6 tps, lat 80.636 ms stddev 71.677 2024-03-12T05:24:40.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:39 smithi045 ceph-mon[32174]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 10 MiB/s wr, 1.12k op/s 2024-03-12T05:24:40.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:39 smithi130 ceph-mon[35398]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 10 MiB/s wr, 1.12k op/s 2024-03-12T05:24:40.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:39 smithi116 ceph-mon[35486]: pgmap v432: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 10 MiB/s wr, 1.12k op/s 2024-03-12T05:24:42.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:41 smithi045 ceph-mon[32174]: pgmap v433: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:24:42.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:41 smithi130 ceph-mon[35398]: pgmap v433: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:24:42.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:41 smithi116 ceph-mon[35486]: pgmap v433: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-03-12T05:24:42.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:24:42.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 -- 172.21.15.45:0/385300519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c106120 msgr2=0x7f1f2c069750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 --2- 172.21.15.45:0/385300519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c106120 0x7f1f2c069750 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009890 tx=0x7f1f2002bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 -- 172.21.15.45:0/385300519 shutdown_connections 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 --2- 172.21.15.45:0/385300519 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c103b80 0x7f1f2c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 --2- 172.21.15.45:0/385300519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c106590 0x7f1f2c069c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 --2- 172.21.15.45:0/385300519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c106120 0x7f1f2c069750 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 -- 172.21.15.45:0/385300519 >> 172.21.15.45:0/385300519 conn(0x7f1f2c077060 msgr2=0x7f1f2c077470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.638+0000 7f1f31eae640 1 -- 172.21.15.45:0/385300519 shutdown_connections 2024-03-12T05:24:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.639+0000 7f1f31eae640 1 -- 172.21.15.45:0/385300519 wait complete. 2024-03-12T05:24:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 Processor -- start 2024-03-12T05:24:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 -- start start 2024-03-12T05:24:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c103b80 0x7f1f2c1a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c106120 0x7f1f2c1a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c19e380 con 0x7f1f2c106120 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c19e4f0 con 0x7f1f2c19d990 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.640+0000 7f1f31eae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c19e6b0 con 0x7f1f2c103b80 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40022/0 (socket says 172.21.15.45:40022) 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 -- 172.21.15.45:0/2180921144 learned_addr learned my addr 172.21.15.45:0/2180921144 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2affd640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c106120 0x7f1f2c1a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c103b80 0x7f1f2c1a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 -- 172.21.15.45:0/2180921144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c103b80 msgr2=0x7f1f2c1a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c103b80 0x7f1f2c1a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 -- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c106120 msgr2=0x7f1f2c1a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c106120 0x7f1f2c1a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.641+0000 7f1f2bfff640 1 -- 172.21.15.45:0/2180921144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20009490 con 0x7f1f2c19d990 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e990 tx=0x7f1f1c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f28ff9640 1 -- 172.21.15.45:0/2180921144 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c00cbe0 con 0x7f1f2c19d990 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f28ff9640 1 -- 172.21.15.45:0/2180921144 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f1c00cd40 con 0x7f1f2c19d990 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f28ff9640 1 -- 172.21.15.45:0/2180921144 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c01a6f0 con 0x7f1f2c19d990 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c1aa6a0 con 0x7f1f2c19d990 2024-03-12T05:24:42.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.642+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c1aab60 con 0x7f1f2c19d990 2024-03-12T05:24:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.646+0000 7f1f34722740 1 Processor -- start 2024-03-12T05:24:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.646+0000 7f1f28ff9640 1 -- 172.21.15.45:0/2180921144 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1f1c0233f0 con 0x7f1f2c19d990 2024-03-12T05:24:42.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 -- start start 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b742b0 0x556b31b74690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31ae3280 con 0x556b31b7d7b0 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31ae07b0 con 0x556b31b742b0 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f34722740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31adbf90 con 0x556b31b74bd0 2024-03-12T05:24:42.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41572/0 (socket says 172.21.15.45:41572) 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3312265293 learned_addr learned my addr 172.21.15.45:0/3312265293 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f2affd640 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b742b0 0x556b31b74690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.647+0000 7f1f2affd640 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3312265293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b742b0 msgr2=0x556b31b74690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b742b0 0x556b31b74690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3312265293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 msgr2=0x556b31b7fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3312265293 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b31a9a570 con 0x556b31b74bd0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005d00 tx=0x7f1f200323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c02800e0040de7d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f11ffb640 1 -- 172.21.15.45:0/3312265293 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20034440 con 0x556b31b74bd0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f11ffb640 1 -- 172.21.15.45:0/3312265293 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f2002ce90 con 0x556b31b74bd0 2024-03-12T05:24:42.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.648+0000 7f1f11ffb640 1 -- 172.21.15.45:0/3312265293 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200053c0 con 0x556b31b74bd0 2024-03-12T05:24:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.649+0000 7f1f34722740 1 -- 172.21.15.45:0/3312265293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 msgr2=0x556b31b7d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.649+0000 7f1f34722740 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005d00 tx=0x7f1f200323d0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f34722740 1 -- 172.21.15.45:0/3312265293 shutdown_connections 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f34722740 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f34722740 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b74bd0 0x556b31b7d270 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f34722740 1 --2- 172.21.15.45:0/3312265293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b742b0 0x556b31b74690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f34722740 1 -- 172.21.15.45:0/3312265293 >> 172.21.15.45:0/3312265293 conn(0x556b31b6e990 msgr2=0x556b31b6ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.650+0000 7f1f28ff9640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 0x7f1f140a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.651+0000 7f1f2affd640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 0x7f1f140a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.651+0000 7f1f28ff9640 1 -- 172.21.15.45:0/2180921144 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1f1c0cdaf0 con 0x7f1f2c19d990 2024-03-12T05:24:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.652+0000 7f1f2affd640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 0x7f1f140a4120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556b31b6fb70 tx=0x7f1f24013040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.652+0000 7f1f34722740 1 -- 172.21.15.45:0/3312265293 shutdown_connections 2024-03-12T05:24:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.652+0000 7f1f34722740 1 -- 172.21.15.45:0/3312265293 wait complete. 2024-03-12T05:24:42.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.652+0000 7f1f34722740 1 -- start start 2024-03-12T05:24:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 Processor -- start 2024-03-12T05:24:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b7a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b317af430 con 0x556b31b7d7b0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b3176be00 con 0x556b31b74bd0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.653+0000 7f1f34722740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31a60fd0 con 0x556b31b742b0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f34722740 10 client.0 _ll_register_callbacks cb 0x556b31ab6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41574/0 (socket says 172.21.15.45:41574) 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3728698807 learned_addr learned my addr 172.21.15.45:0/3728698807 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2affd640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b7a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 msgr2=0x556b31b7a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b7a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3728698807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 msgr2=0x556b31b7ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/3728698807 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c1048f0 con 0x556b31b742b0 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.654+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005b60 tx=0x7f1f20035f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b64703cf877d10a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f0affd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20005550 con 0x556b31b742b0 2024-03-12T05:24:42.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1f2002c510 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b317db7e0 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.655+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f2002c6d0 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f34722740 10 client.28318 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x556b31ada610 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f20057030 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1f20004de0 con 0x556b31b742b0 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f34722740 10 client.28318 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f34722740 10 client.28318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.656+0000 7f1f34722740 20 client.28318 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.657+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31b8d780 0x556b31b8fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.657+0000 7f1f34722740 4 client.28318 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.657+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b31818720 con 0x556b31b8d780 2024-03-12T05:24:42.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.657+0000 7f1f2affd640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31b8d780 0x556b31b8fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.658+0000 7f1f0affd640 10 client.28318 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.658+0000 7f1f2affd640 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31b8d780 0x556b31b8fba0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e98ab901fec94245 server_cookie=581c20fddd04e466 in_seq=0 out_seq=0 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f0affd640 1 -- 172.21.15.45:0/3728698807 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556b31818720 con 0x556b31b8d780 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f0affd640 10 client.28318 handle_command_reply: tid=0 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 client.28318 shutdown 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 msgr2=0x556b31b7d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1f20005b60 tx=0x7f1f20035f80 comp rx=0 tx=0).stop 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 shutdown_connections 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31b8d780 0x556b31b8fba0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b7d7b0 0x556b31b7ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b7a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 --2- 172.21.15.45:0/3728698807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b742b0 0x556b31b7d1c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 >> 172.21.15.45:0/3728698807 conn(0x556b31b6e990 msgr2=0x556b31b70730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f0affd640 0 client.28318 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 shutdown_connections 2024-03-12T05:24:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.659+0000 7f1f34722740 1 -- 172.21.15.45:0/3728698807 wait complete. 2024-03-12T05:24:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.660+0000 7f1f34722740 20 client.28318 trim_cache size 0 max 16384 2024-03-12T05:24:42.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.692+0000 7f1f34722740 1 Processor -- start 2024-03-12T05:24:42.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.692+0000 7f1f34722740 1 -- start start 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 0x556b31b758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b94a80 0x556b31b7cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b73f10 con 0x556b31b74bd0 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b8a490 con 0x556b31b7d7b0 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f34722740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b6f9e0 con 0x556b31b94a80 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f2affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f2affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40042/0 (socket says 172.21.15.45:40042) 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f2affd640 1 -- 172.21.15.45:0/2452928332 learned_addr learned my addr 172.21.15.45:0/2452928332 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.693+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 0x556b31b758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b94a80 0x556b31b7cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/2452928332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b94a80 msgr2=0x556b31b7cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b94a80 0x556b31b7cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/2452928332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 msgr2=0x556b31b94540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/2452928332 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b31a76ac0 con 0x556b31b74bd0 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2affd640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:42.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.694+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 0x556b31b758b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1f20048000 tx=0x7f1f20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b63effb1103dd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.695+0000 7f1f117fa640 1 -- 172.21.15.45:0/2452928332 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20005c90 con 0x556b31b74bd0 2024-03-12T05:24:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.695+0000 7f1f117fa640 1 -- 172.21.15.45:0/2452928332 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1f20057070 con 0x556b31b74bd0 2024-03-12T05:24:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.695+0000 7f1f117fa640 1 -- 172.21.15.45:0/2452928332 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20004030 con 0x556b31b74bd0 2024-03-12T05:24:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 -- 172.21.15.45:0/2452928332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 msgr2=0x556b31b758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 0x556b31b758b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1f20048000 tx=0x7f1f20000bc0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 -- 172.21.15.45:0/2452928332 shutdown_connections 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b94a80 0x556b31b7cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b7d7b0 0x556b31b94540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 --2- 172.21.15.45:0/2452928332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b74bd0 0x556b31b758b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 -- 172.21.15.45:0/2452928332 >> 172.21.15.45:0/2452928332 conn(0x556b31b6e990 msgr2=0x556b31b6ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 -- 172.21.15.45:0/2452928332 shutdown_connections 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.696+0000 7f1f34722740 1 -- 172.21.15.45:0/2452928332 wait complete. 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.697+0000 7f1f34722740 1 -- start start 2024-03-12T05:24:42.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.697+0000 7f1f34722740 1 Processor -- start 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.697+0000 7f1f34722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b846a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.697+0000 7f1f34722740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 0x556b31b84be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b846a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b846a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40056/0 (socket says 172.21.15.45:40056) 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2b7fe640 1 -- 172.21.15.45:0/666851429 learned_addr learned my addr 172.21.15.45:0/666851429 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2affd640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 0x556b31b84be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 0x556b31b80c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b81350 con 0x556b31b94a80 2024-03-12T05:24:42.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b814c0 con 0x556b31b74bd0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b31b81680 con 0x556b31b7d7b0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 0x556b31b80c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2affd640 1 -- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 msgr2=0x556b31b846a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.698+0000 7f1f2affd640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b846a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f2affd640 1 -- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 msgr2=0x556b31b80c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f2affd640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 0x556b31b80c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f34722740 10 client.0 _ll_register_callbacks cb 0x7f1f20037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f2affd640 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f24019040 con 0x556b31b7d7b0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f2002ce90 con 0x556b31b7d7b0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f2bfff640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 0x556b31b80c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f2affd640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 0x556b31b84be0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0783d0 tx=0x7f1f24004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37efa589ccfc3491 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f0affd640 10 client.28324 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f0affd640 1 -- 172.21.15.45:0/666851429 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24016070 con 0x556b31b7d7b0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f0affd640 1 -- 172.21.15.45:0/666851429 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1f24010d70 con 0x556b31b7d7b0 2024-03-12T05:24:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.699+0000 7f1f0affd640 1 -- 172.21.15.45:0/666851429 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f24004b20 con 0x556b31b7d7b0 2024-03-12T05:24:42.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 10 client.28324 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:42.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x556b31ada610 con 0x556b31b7d7b0 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f0affd640 1 -- 172.21.15.45:0/666851429 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1f24004c80 con 0x556b31b7d7b0 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 10 client.28324 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 10 client.28324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 20 client.28324 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.700+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31be4ee0 0x556b31be7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.701+0000 7f1f34722740 4 client.28324 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.701+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556b31867890 con 0x556b31be4ee0 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.701+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31be4ee0 0x556b31be7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f2b7fe640 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31be4ee0 0x556b31be7300 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8492cfe172fb53b server_cookie=19d0ec9ddf799bce in_seq=0 out_seq=0 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f0affd640 10 client.28324 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f0affd640 1 -- 172.21.15.45:0/666851429 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556b31867890 con 0x556b31be4ee0 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f0affd640 10 client.28324 handle_command_reply: tid=0 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 client.28324 shutdown 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 msgr2=0x556b31b84be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 0x556b31b84be0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0783d0 tx=0x7f1f24004ab0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 shutdown_connections 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x556b31be4ee0 0x556b31be7300 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x556b31b94a80 0x556b31b80c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b31b7d7b0 0x556b31b84be0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 --2- 172.21.15.45:0/666851429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556b31b74bd0 0x556b31b846a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 >> 172.21.15.45:0/666851429 conn(0x556b31b6e990 msgr2=0x556b31b758b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.702+0000 7f1f0affd640 0 client.28324 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.703+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 shutdown_connections 2024-03-12T05:24:42.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.703+0000 7f1f34722740 1 -- 172.21.15.45:0/666851429 wait complete. 2024-03-12T05:24:42.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.703+0000 7f1f34722740 20 client.28324 trim_cache size 0 max 16384 2024-03-12T05:24:42.700 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:24:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:24:42.700 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 msgr2=0x7f1f140a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 0x7f1f140a4120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556b31b6fb70 tx=0x7f1f24013040 comp rx=0 tx=0).stop 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 msgr2=0x7f1f2c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00e990 tx=0x7f1f1c00ee50 comp rx=0 tx=0).stop 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 shutdown_connections 2024-03-12T05:24:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1f140a1c60 0x7f1f140a4120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.704+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1f2c106120 0x7f1f2c1a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.705+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f2c19d990 0x7f1f2c19de40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.705+0000 7f1f31eae640 1 --2- 172.21.15.45:0/2180921144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f2c103b80 0x7f1f2c1a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.705+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 >> 172.21.15.45:0/2180921144 conn(0x7f1f2c077060 msgr2=0x7f1f2c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.705+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 shutdown_connections 2024-03-12T05:24:42.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.705+0000 7f1f31eae640 1 -- 172.21.15.45:0/2180921144 wait complete. 2024-03-12T05:24:42.719 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:24:42.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.896+0000 7f5ee63b6640 1 -- 172.21.15.45:0/1723693072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 msgr2=0x7f5ee010dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.896+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/1723693072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee010dc60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009ca0 tx=0x7f5ed802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 -- 172.21.15.45:0/1723693072 shutdown_connections 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/1723693072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee010b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/1723693072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee010dc60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/1723693072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee01027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 -- 172.21.15.45:0/1723693072 >> 172.21.15.45:0/1723693072 conn(0x7f5ee0077180 msgr2=0x7f5ee0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 -- 172.21.15.45:0/1723693072 shutdown_connections 2024-03-12T05:24:42.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.897+0000 7f5ee63b6640 1 -- 172.21.15.45:0/1723693072 wait complete. 2024-03-12T05:24:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5ee63b6640 1 Processor -- start 2024-03-12T05:24:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5ee63b6640 1 -- start start 2024-03-12T05:24:42.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5ee63b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5ee63b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5edffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.898+0000 7f5edffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40080/0 (socket says 172.21.15.45:40080) 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5edf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5edf7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59808/0 (socket says 172.21.15.45:59808) 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5edf7fe640 1 -- 172.21.15.45:0/978032980 learned_addr learned my addr 172.21.15.45:0/978032980 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee0100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01a5ae0 con 0x7f5ee0102d10 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01a5c50 con 0x7f5ee01023f0 2024-03-12T05:24:42.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee01a5e10 con 0x7f5ee010b7c0 2024-03-12T05:24:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.899+0000 7f5ee4953640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee0100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edf7fe640 1 -- 172.21.15.45:0/978032980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 msgr2=0x7f5ee0100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edf7fe640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee0100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edf7fe640 1 -- 172.21.15.45:0/978032980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 msgr2=0x7f5ee0100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edf7fe640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edf7fe640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed8009900 con 0x7f5ee0102d10 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.900+0000 7f5edffff640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.901+0000 7f5edf7fe640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000e9f0 tx=0x7f5ed000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.901+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0009700 con 0x7f5ee0102d10 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.901+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5ed000cd80 con 0x7f5ee0102d10 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.901+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed0017800 con 0x7f5ee0102d10 2024-03-12T05:24:42.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.901+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee01a60f0 con 0x7f5ee0102d10 2024-03-12T05:24:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.902+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee0197640 con 0x7f5ee0102d10 2024-03-12T05:24:42.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.904+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0067f70 con 0x7f5ee0102d10 2024-03-12T05:24:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.906+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5ed00179a0 con 0x7f5ee0102d10 2024-03-12T05:24:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.907+0000 7f5edd7fa640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 0x7f5ec80a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.907+0000 7f5edffff640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 0x7f5ec80a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.907+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5ed0014070 con 0x7f5ee0102d10 2024-03-12T05:24:42.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.908+0000 7f5edffff640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 0x7f5ec80a41f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009c20 tx=0x7f5ed4005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:42.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:42.915+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ed0096b10 con 0x7f5ee0102d10 2024-03-12T05:24:43.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.327+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ee01068b0 con 0x7f5ee0102d10 2024-03-12T05:24:43.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.328+0000 7f5edd7fa640 1 -- 172.21.15.45:0/978032980 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f5ed009a1d0 con 0x7f5ee0102d10 2024-03-12T05:24:43.323 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:24:43.323 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:24:43.324 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 msgr2=0x7f5ec80a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 0x7f5ec80a41f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009c20 tx=0x7f5ed4005f90 comp rx=0 tx=0).stop 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 msgr2=0x7f5ee0100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000e9f0 tx=0x7f5ed000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 shutdown_connections 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5ec80a1d30 0x7f5ec80a41f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee0102d10 0x7f5ee0100580 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.331+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ee010b7c0 0x7f5ee0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.332+0000 7f5ee63b6640 1 --2- 172.21.15.45:0/978032980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee01023f0 0x7f5ee0100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.332+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 >> 172.21.15.45:0/978032980 conn(0x7f5ee0077180 msgr2=0x7f5ee00fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.332+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 shutdown_connections 2024-03-12T05:24:43.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.332+0000 7f5ee63b6640 1 -- 172.21.15.45:0/978032980 wait complete. 2024-03-12T05:24:43.346 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:43.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2498823003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd80688c0 msgr2=0x7fdcd8068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2498823003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd80688c0 0x7fdcd8068d20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009890 tx=0x7fdccc02bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2498823003 shutdown_connections 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2498823003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd80688c0 0x7fdcd8068d20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2498823003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd8103c40 0x7fdcd81040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2498823003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 0x7fdcd8068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2498823003 >> 172.21.15.45:0/2498823003 conn(0x7fdcd8077180 msgr2=0x7fdcd8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2498823003 shutdown_connections 2024-03-12T05:24:43.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.529+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2498823003 wait complete. 2024-03-12T05:24:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.530+0000 7fdcdd2f4640 1 Processor -- start 2024-03-12T05:24:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 -- start start 2024-03-12T05:24:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 0x7fdcd819e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 0x7fdcd81a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd81a2c40 con 0x7fdcd8103c40 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd81a2db0 con 0x7fdcd80688c0 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcdd2f4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd81a2f70 con 0x7fdcd81060d0 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 0x7fdcd819e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40088/0 (socket says 172.21.15.45:40088) 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/2053199782 learned_addr learned my addr 172.21.15.45:0/2053199782 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 0x7fdcd819e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59826/0 (socket says 172.21.15.45:59826) 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.531+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 0x7fdcd81a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/2053199782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 msgr2=0x7fdcd81a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 0x7fdcd81a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 msgr2=0x7fdcd819e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 0x7fdcd819e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/2053199782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd0009840 con 0x7fdcd80688c0 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800e9c0 tx=0x7fdcc800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.532+0000 7fdcbffff640 1 -- 172.21.15.45:0/2053199782 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800ccc0 con 0x7fdcd80688c0 2024-03-12T05:24:43.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.533+0000 7fdcbffff640 1 -- 172.21.15.45:0/2053199782 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdcc800ce20 con 0x7fdcd80688c0 2024-03-12T05:24:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.533+0000 7fdcbffff640 1 -- 172.21.15.45:0/2053199782 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004e70 con 0x7fdcd80688c0 2024-03-12T05:24:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.533+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc009490 con 0x7fdcd80688c0 2024-03-12T05:24:43.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.533+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81aa8f0 con 0x7fdcd80688c0 2024-03-12T05:24:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.537+0000 7fdcbffff640 1 -- 172.21.15.45:0/2053199782 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fdcc8004fd0 con 0x7fdcd80688c0 2024-03-12T05:24:43.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.537+0000 7fdcdffa2740 1 Processor -- start 2024-03-12T05:24:43.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.537+0000 7fdcdffa2740 1 -- start start 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.537+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa14630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.537+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa1d750 0x555e2fa1fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f9cf960 con 0x555e2fa14250 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f983310 con 0x555e2fa1d750 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f980840 con 0x555e2fa14b70 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcbffff640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 0x7fdcc00a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41646/0 (socket says 172.21.15.45:41646) 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/3278767463 learned_addr learned my addr 172.21.15.45:0/3278767463 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa1d750 0x555e2fa1fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.533 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa14630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcbffff640 1 -- 172.21.15.45:0/2053199782 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdcc8014070 con 0x7fdcd80688c0 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/3278767463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa1d750 msgr2=0x555e2fa1fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa1d750 0x555e2fa1fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/3278767463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 msgr2=0x555e2fa14630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa14630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.538+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/3278767463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e2f93e060 con 0x555e2fa14b70 2024-03-12T05:24:43.535 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.539+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 0x7fdcc00a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.539+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80c5a30 tx=0x7fdcc809a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35465719e4e83576 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 0x7fdcc00a4120 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0005fd0 tx=0x7fdcd0004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcbd7fa640 1 -- 172.21.15.45:0/3278767463 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80046c0 con 0x555e2fa14b70 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcbd7fa640 1 -- 172.21.15.45:0/3278767463 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdcc8083e90 con 0x555e2fa14b70 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcbd7fa640 1 -- 172.21.15.45:0/3278767463 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8096d80 con 0x555e2fa14b70 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcdffa2740 1 -- 172.21.15.45:0/3278767463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 msgr2=0x555e2fa1d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.540+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80c5a30 tx=0x7fdcc809a1a0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 -- 172.21.15.45:0/3278767463 shutdown_connections 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa1d750 0x555e2fa1fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1d210 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/3278767463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa14630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 -- 172.21.15.45:0/3278767463 >> 172.21.15.45:0/3278767463 conn(0x555e2fa0e980 msgr2=0x555e2fa0ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 -- 172.21.15.45:0/3278767463 shutdown_connections 2024-03-12T05:24:43.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 -- 172.21.15.45:0/3278767463 wait complete. 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 -- start start 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.541+0000 7fdcdffa2740 1 Processor -- start 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa1ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa1b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f97c020 con 0x555e2fa1d750 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f64f430 con 0x555e2fa14b70 2024-03-12T05:24:43.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f60be00 con 0x555e2fa14250 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41660/0 (socket says 172.21.15.45:41660) 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcd67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa1ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/4280943005 learned_addr learned my addr 172.21.15.45:0/4280943005 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.542+0000 7fdcdffa2740 10 client.0 _ll_register_callbacks cb 0x555e2f923cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa1b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/4280943005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 msgr2=0x555e2fa1ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa1ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/4280943005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 msgr2=0x555e2fa1b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa1b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/4280943005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81aa8f0 con 0x555e2fa14250 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80970e0 tx=0x7fdcc80094c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6d488061b1081f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80046c0 con 0x555e2fa14250 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e2f67b7e0 con 0x555e2fa14250 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.543+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdcc809a710 con 0x555e2fa14250 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcc809ae10 con 0x555e2fa14250 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 10 client.28342 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:43.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555e2f97a6a0 con 0x555e2fa14250 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80b7030 con 0x555e2fa14250 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdcc80ba0b0 con 0x555e2fa14250 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 10 client.28342 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 10 client.28342 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 20 client.28342 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.544+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa40ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.545+0000 7fdcdffa2740 4 client.28342 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.545+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e2f6b8720 con 0x555e2fa3e7b0 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.545+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa40ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.545+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa40ba0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b728bc894a43a550 server_cookie=6ee73707600078ac in_seq=0 out_seq=0 2024-03-12T05:24:43.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.545+0000 7fdcbaff5640 10 client.28342 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcbaff5640 1 -- 172.21.15.45:0/4280943005 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e2f6b8720 con 0x555e2fa3e7b0 2024-03-12T05:24:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcbaff5640 10 client.28342 handle_command_reply: tid=0 2024-03-12T05:24:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcdffa2740 1 client.28342 shutdown 2024-03-12T05:24:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 msgr2=0x555e2fa1a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80970e0 tx=0x7fdcc80094c0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.546+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 shutdown_connections 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa40ba0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa1b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcbaff5640 0 client.28342 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa1ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/4280943005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14250 0x555e2fa1a7e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 >> 172.21.15.45:0/4280943005 conn(0x555e2fa0e980 msgr2=0x555e2fa106d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 shutdown_connections 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 1 -- 172.21.15.45:0/4280943005 wait complete. 2024-03-12T05:24:43.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.547+0000 7fdcdffa2740 20 client.28342 trim_cache size 0 max 16384 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.566+0000 7fdcdffa2740 1 Processor -- start 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.566+0000 7fdcdffa2740 1 -- start start 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.566+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa190b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2f901130 con 0x555e2fa1d750 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2fa13c90 con 0x555e2fa14250 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2fa3a5f0 con 0x555e2fa14b70 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40124/0 (socket says 172.21.15.45:40124) 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/697465704 learned_addr learned my addr 172.21.15.45:0/697465704 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/697465704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 msgr2=0x555e2fa1b080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/697465704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 msgr2=0x555e2fa190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa190b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.567+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/697465704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e2f916c20 con 0x555e2fa14250 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.568+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80058c0 tx=0x7fdcc80cb730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d88069924cc79b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.568+0000 7fdcbcff9640 1 -- 172.21.15.45:0/697465704 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc809e860 con 0x555e2fa14250 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.568+0000 7fdcbcff9640 1 -- 172.21.15.45:0/697465704 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80b7070 con 0x555e2fa14250 2024-03-12T05:24:43.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.568+0000 7fdcbcff9640 1 -- 172.21.15.45:0/697465704 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc809a710 con 0x555e2fa14250 2024-03-12T05:24:43.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- 172.21.15.45:0/697465704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 msgr2=0x555e2fa1cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80058c0 tx=0x7fdcc80cb730 comp rx=0 tx=0).stop 2024-03-12T05:24:43.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- 172.21.15.45:0/697465704 shutdown_connections 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa1d750 0x555e2fa190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa14b70 0x555e2fa1b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/697465704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14250 0x555e2fa1cf10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- 172.21.15.45:0/697465704 >> 172.21.15.45:0/697465704 conn(0x555e2fa0e980 msgr2=0x555e2fa454a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- 172.21.15.45:0/697465704 shutdown_connections 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- 172.21.15.45:0/697465704 wait complete. 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.569+0000 7fdcdffa2740 1 -- start start 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 Processor -- start 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa336a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa33be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 0x555e2fa93c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2fa34340 con 0x555e2fa14250 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2fa94160 con 0x555e2fa14b70 2024-03-12T05:24:43.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2fa94320 con 0x555e2fa1d750 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcdffa2740 10 client.0 _ll_register_callbacks cb 0x7fdcc809a5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcd77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 0x555e2fa93c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa336a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59856/0 (socket says 172.21.15.45:59856) 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.570+0000 7fdcd6ffd640 1 -- 172.21.15.45:0/933838537 learned_addr learned my addr 172.21.15.45:0/933838537 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa33be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 -- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 msgr2=0x555e2fa93c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 0x555e2fa93c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 -- 172.21.15.45:0/933838537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 msgr2=0x555e2fa336a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa336a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd67fc640 1 -- 172.21.15.45:0/933838537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e2f97a6a0 con 0x555e2fa14b70 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.571+0000 7fdcd77fe640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 0x555e2fa93c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.572+0000 7fdcd67fc640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa33be0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdccc037040 tx=0x7fdccc034c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f6a9bbf02bb362 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.572+0000 7fdcbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.574+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc004240 con 0x555e2fa14b70 2024-03-12T05:24:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.574+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fdccc04a030 con 0x555e2fa14b70 2024-03-12T05:24:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.574+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcc8083e90 con 0x555e2fa14b70 2024-03-12T05:24:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.575+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdccc045e50 con 0x555e2fa14b70 2024-03-12T05:24:43.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.575+0000 7fdcdffa2740 10 client.28385 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.575+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555e2f93c8c0 con 0x555e2fa14b70 2024-03-12T05:24:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.575+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fdccc053030 con 0x555e2fa14b70 2024-03-12T05:24:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fdccc045670 con 0x555e2fa14b70 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 10 client.28385 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 10 client.28385 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 20 client.28385 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa97bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 4 client.28385 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.576+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e2f707890 con 0x555e2fa3e7b0 2024-03-12T05:24:43.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.577+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa97bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.577+0000 7fdcd6ffd640 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa97bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f674e5d40f69ba38 server_cookie=c5c201cd7104093e in_seq=0 out_seq=0 2024-03-12T05:24:43.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.577+0000 7fdcbaff5640 10 client.28385 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcbaff5640 1 -- 172.21.15.45:0/933838537 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e2f707890 con 0x555e2fa3e7b0 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcbaff5640 10 client.28385 handle_command_reply: tid=0 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 client.28385 shutdown 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 msgr2=0x555e2fa33be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa33be0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdccc037040 tx=0x7fdccc034c20 comp rx=0 tx=0).stop 2024-03-12T05:24:43.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 shutdown_connections 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555e2fa3e7b0 0x555e2fa97bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e2fa1d750 0x555e2fa93c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e2fa14b70 0x555e2fa33be0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 --2- 172.21.15.45:0/933838537 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555e2fa14250 0x555e2fa336a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 >> 172.21.15.45:0/933838537 conn(0x555e2fa0e980 msgr2=0x555e2fa19070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcbaff5640 0 client.28385 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 shutdown_connections 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.578+0000 7fdcdffa2740 1 -- 172.21.15.45:0/933838537 wait complete. 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.579+0000 7fdcdffa2740 20 client.28385 trim_cache size 0 max 16384 2024-03-12T05:24:43.574 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:43.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.579+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 msgr2=0x7fdcc00a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.579+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 0x7fdcc00a4120 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0005fd0 tx=0x7fdcd0004ab0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.579+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 msgr2=0x7fdcd819dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.579+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800e9c0 tx=0x7fdcc800edc0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 shutdown_connections 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fdcc00a1c60 0x7fdcc00a4120 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcd8103c40 0x7fdcd819e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcd81060d0 0x7fdcd81a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 --2- 172.21.15.45:0/2053199782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd80688c0 0x7fdcd819dd80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 >> 172.21.15.45:0/2053199782 conn(0x7fdcd8077180 msgr2=0x7fdcd80fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 shutdown_connections 2024-03-12T05:24:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.580+0000 7fdcdd2f4640 1 -- 172.21.15.45:0/2053199782 wait complete. 2024-03-12T05:24:43.595 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:43.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:24:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 -- 172.21.15.45:0/1576727229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 msgr2=0x7f39a0075a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 --2- 172.21.15.45:0/1576727229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0075a90 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3990009890 tx=0x7f399002bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 -- 172.21.15.45:0/1576727229 shutdown_connections 2024-03-12T05:24:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 --2- 172.21.15.45:0/1576727229 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 0x7f39a0111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.781 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 --2- 172.21.15.45:0/1576727229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 0x7f39a01143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 --2- 172.21.15.45:0/1576727229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0075a90 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 -- 172.21.15.45:0/1576727229 >> 172.21.15.45:0/1576727229 conn(0x7f39a00fee30 msgr2=0x7f39a0101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 -- 172.21.15.45:0/1576727229 shutdown_connections 2024-03-12T05:24:43.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.786+0000 7f39a493e640 1 -- 172.21.15.45:0/1576727229 wait complete. 2024-03-12T05:24:43.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 Processor -- start 2024-03-12T05:24:43.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 -- start start 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 0x7f39a006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 0x7f39a006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a006d4d0 con 0x7f39a0075fd0 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a006d640 con 0x7f39a00756b0 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.787+0000 7f39a493e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a006d800 con 0x7f39a0111fc0 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40154/0 (socket says 172.21.15.45:40154) 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 -- 172.21.15.45:0/2944235084 learned_addr learned my addr 172.21.15.45:0/2944235084 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399ed9d640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 0x7f39a006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f3995d9b640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 0x7f39a006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 -- 172.21.15.45:0/2944235084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 msgr2=0x7f39a006cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 0x7f39a006cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 -- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 msgr2=0x7f39a006ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 0x7f39a006ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.788+0000 7f399e59c640 1 -- 172.21.15.45:0/2944235084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3990009490 con 0x7f39a00756b0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f399e59c640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f3990004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f3997fff640 1 -- 172.21.15.45:0/2944235084 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39900348d0 con 0x7f39a00756b0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f3997fff640 1 -- 172.21.15.45:0/2944235084 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f399002ce90 con 0x7f39a00756b0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f3997fff640 1 -- 172.21.15.45:0/2944235084 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990032dc0 con 0x7f39a00756b0 2024-03-12T05:24:43.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a01b2db0 con 0x7f39a00756b0 2024-03-12T05:24:43.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.789+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a01b3180 con 0x7f39a00756b0 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.793+0000 7f3997fff640 1 -- 172.21.15.45:0/2944235084 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f399003b040 con 0x7f39a00756b0 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.793+0000 7f39a75ee740 1 Processor -- start 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f3997fff640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 0x7f398c0a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 -- start start 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f3997fff640 1 -- 172.21.15.45:0/2944235084 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f39900ef470 con 0x7f39a00756b0 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060db70 0x55fab0616210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab0618b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab05c8960 con 0x55fab060db70 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab057c310 con 0x55fab0616750 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f39a75ee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab0579840 con 0x55fab060d250 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f399e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f399e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41700/0 (socket says 172.21.15.45:41700) 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.794+0000 7f399e59c640 1 -- 172.21.15.45:0/3782767486 learned_addr learned my addr 172.21.15.45:0/3782767486 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399e59c640 1 -- 172.21.15.45:0/3782767486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 msgr2=0x55fab0618b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399e59c640 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab0618b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399e59c640 1 -- 172.21.15.45:0/3782767486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060db70 msgr2=0x55fab0616210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399e59c640 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060db70 0x55fab0616210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399e59c640 1 -- 172.21.15.45:0/3782767486 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab0537060 con 0x55fab060d250 2024-03-12T05:24:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.795+0000 7f399ed9d640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 0x7f398c0a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.796+0000 7f399ed9d640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 0x7f398c0a4270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55fab0608b10 tx=0x7f3998010040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.796+0000 7f399e59c640 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3990037040 tx=0x7f39900bba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e992108f42b5b80e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.796+0000 7f3994d99640 1 -- 172.21.15.45:0/3782767486 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39900c0270 con 0x55fab060d250 2024-03-12T05:24:43.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.796+0000 7f3994d99640 1 -- 172.21.15.45:0/3782767486 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f39900c48c0 con 0x55fab060d250 2024-03-12T05:24:43.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.797+0000 7f39a75ee740 1 -- 172.21.15.45:0/3782767486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 msgr2=0x55fab060d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.797+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3990037040 tx=0x7f39900bba80 comp rx=0 tx=0).stop 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 -- 172.21.15.45:0/3782767486 shutdown_connections 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060db70 0x55fab0616210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab0618b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3782767486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060d250 0x55fab060d630 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 -- 172.21.15.45:0/3782767486 >> 172.21.15.45:0/3782767486 conn(0x55fab0607980 msgr2=0x55fab0607d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 -- 172.21.15.45:0/3782767486 shutdown_connections 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 -- 172.21.15.45:0/3782767486 wait complete. 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 -- start start 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.798+0000 7f39a75ee740 1 Processor -- start 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab06136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 0x55fab0613c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab0575020 con 0x55fab060d250 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab0248430 con 0x55fab060db70 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab0204e00 con 0x55fab0616750 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f39a75ee740 10 client.0 _ll_register_callbacks cb 0x55fab051ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f399ed9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f399ed9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41716/0 (socket says 172.21.15.45:41716) 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f399ed9d640 1 -- 172.21.15.45:0/3776754989 learned_addr learned my addr 172.21.15.45:0/3776754989 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f3995d9b640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 0x55fab0613c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.799+0000 7f399e59c640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab06136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 -- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 msgr2=0x55fab0613c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 0x55fab0613c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 -- 172.21.15.45:0/3776754989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 msgr2=0x55fab06136d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab06136d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 -- 172.21.15.45:0/3776754989 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a01b3180 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f3995d9b640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 0x55fab0613c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f399ed9d640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3998019e40 tx=0x7f3998004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=382c7b32ffe772a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f3985ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998013040 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f39980235f0 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fab02747e0 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.800+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39980237b0 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 10 client.28354 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fab05736a0 con 0x55fab0616750 2024-03-12T05:24:43.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fab02747e0 con 0x55fab0616750 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3998022070 con 0x55fab0616750 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 10 client.28354 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 10 client.28354 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 20 client.28354 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab0626910 0x55fab0628d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 4 client.28354 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.801+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fab02b1720 con 0x55fab0626910 2024-03-12T05:24:43.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.802+0000 7f399e59c640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab0626910 0x55fab0628d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.802+0000 7f399e59c640 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab0626910 0x55fab0628d30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f153d968aa811b62 server_cookie=5f468711a182a988 in_seq=0 out_seq=0 2024-03-12T05:24:43.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.802+0000 7f3985ffb640 10 client.28354 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.803+0000 7f3985ffb640 1 -- 172.21.15.45:0/3776754989 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fab02b1720 con 0x55fab0626910 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.803+0000 7f3985ffb640 10 client.28354 handle_command_reply: tid=0 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.803+0000 7f39a75ee740 1 client.28354 shutdown 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.803+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 msgr2=0x55fab0614150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.803+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3998019e40 tx=0x7f3998004a40 comp rx=0 tx=0).stop 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 shutdown_connections 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab0626910 0x55fab0628d30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab0616750 0x55fab0614150 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060db70 0x55fab0613c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f3985ffb640 0 client.28354 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 --2- 172.21.15.45:0/3776754989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab06136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 >> 172.21.15.45:0/3776754989 conn(0x55fab0607980 msgr2=0x55fab06096d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 shutdown_connections 2024-03-12T05:24:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 1 -- 172.21.15.45:0/3776754989 wait complete. 2024-03-12T05:24:43.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.804+0000 7f39a75ee740 20 client.28354 trim_cache size 0 max 16384 2024-03-12T05:24:43.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.826+0000 7f39a75ee740 1 Processor -- start 2024-03-12T05:24:43.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.826+0000 7f39a75ee740 1 -- start start 2024-03-12T05:24:43.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab0613f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab061d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab04fa130 con 0x55fab060d250 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab060c340 con 0x55fab0616750 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f39a75ee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab0608980 con 0x55fab060db70 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f3995d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f3995d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41722/0 (socket says 172.21.15.45:41722) 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f3995d9b640 1 -- 172.21.15.45:0/580253515 learned_addr learned my addr 172.21.15.45:0/580253515 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f399e59c640 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab0613f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.827+0000 7f399ed9d640 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab061d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 -- 172.21.15.45:0/580253515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 msgr2=0x55fab061d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab061d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 -- 172.21.15.45:0/580253515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 msgr2=0x55fab0613f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab0613f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 -- 172.21.15.45:0/580253515 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab050fc20 con 0x55fab060db70 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.828+0000 7f3995d9b640 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3988009e80 tx=0x7f3988012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9fdd740889b6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.829+0000 7f3987fff640 1 -- 172.21.15.45:0/580253515 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3988017070 con 0x55fab060db70 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.829+0000 7f3987fff640 1 -- 172.21.15.45:0/580253515 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f39880035f0 con 0x55fab060db70 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.829+0000 7f3987fff640 1 -- 172.21.15.45:0/580253515 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398800f6a0 con 0x55fab060db70 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- 172.21.15.45:0/580253515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 msgr2=0x55fab06094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3988009e80 tx=0x7f3988012820 comp rx=0 tx=0).stop 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- 172.21.15.45:0/580253515 shutdown_connections 2024-03-12T05:24:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab0616750 0x55fab061d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab06094c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 --2- 172.21.15.45:0/580253515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab060d250 0x55fab0613f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- 172.21.15.45:0/580253515 >> 172.21.15.45:0/580253515 conn(0x55fab0607980 msgr2=0x55fab0608db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- 172.21.15.45:0/580253515 shutdown_connections 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- 172.21.15.45:0/580253515 wait complete. 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.830+0000 7f39a75ee740 1 -- start start 2024-03-12T05:24:43.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 Processor -- start 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab061c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab0616750 0x55fab061ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab067cf70 con 0x55fab0616750 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab067d0e0 con 0x55fab060d250 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab068e2b0 con 0x55fab060db70 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f39a75ee740 10 client.0 _ll_register_callbacks cb 0x7f3990005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f399e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f399e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40192/0 (socket says 172.21.15.45:40192) 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f399e59c640 1 -- 172.21.15.45:0/1289182386 learned_addr learned my addr 172.21.15.45:0/1289182386 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f3995d9b640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab061c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.831+0000 7f399ed9d640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab0616750 0x55fab061ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 -- 172.21.15.45:0/1289182386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 msgr2=0x55fab061c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab061c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 -- 172.21.15.45:0/1289182386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab0616750 msgr2=0x55fab061ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab0616750 0x55fab061ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 -- 172.21.15.45:0/1289182386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3988009b60 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f399e59c640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f39900ef3e0 tx=0x7f39900bc2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0a38e14c0ff9a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f3985ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39900bc4e0 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39900c48c0 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f39900c4430 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39900bc640 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f39a75ee740 10 client.28400 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.832+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fab05358c0 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f39900c48c0 con 0x55fab060d250 2024-03-12T05:24:43.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f399004e070 con 0x55fab060d250 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 10 client.28400 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 10 client.28400 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 20 client.28400 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab068f620 0x55fab0691a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 4 client.28400 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fab0300890 con 0x55fab068f620 2024-03-12T05:24:43.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.833+0000 7f3995d9b640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab068f620 0x55fab0691a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:43.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f3995d9b640 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab068f620 0x55fab0691a10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b976ae7dd79bb5da server_cookie=bcd38cbb562d6f94 in_seq=0 out_seq=0 2024-03-12T05:24:43.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f3985ffb640 10 client.28400 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:43.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f3985ffb640 1 -- 172.21.15.45:0/1289182386 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fab0300890 con 0x55fab068f620 2024-03-12T05:24:43.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f3985ffb640 10 client.28400 handle_command_reply: tid=0 2024-03-12T05:24:43.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 client.28400 shutdown 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 msgr2=0x55fab061bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f39900ef3e0 tx=0x7f39900bc2a0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 shutdown_connections 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55fab068f620 0x55fab0691a10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fab0616750 0x55fab061ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fab060db70 0x55fab061c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 --2- 172.21.15.45:0/1289182386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fab060d250 0x55fab061bfb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 >> 172.21.15.45:0/1289182386 conn(0x55fab0607980 msgr2=0x55fab0609640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.834+0000 7f3985ffb640 0 client.28400 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.835+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 shutdown_connections 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.835+0000 7f39a75ee740 1 -- 172.21.15.45:0/1289182386 wait complete. 2024-03-12T05:24:43.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.835+0000 7f39a75ee740 20 client.28400 trim_cache size 0 max 16384 2024-03-12T05:24:43.833 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 msgr2=0x7f398c0a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 0x7f398c0a4270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55fab0608b10 tx=0x7f3998010040 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 msgr2=0x7f39a0072990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f3990004530 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 shutdown_connections 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f398c0a1db0 0x7f398c0a4270 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39a0075fd0 0x7f39a006ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39a0111fc0 0x7f39a006cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 --2- 172.21.15.45:0/2944235084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a00756b0 0x7f39a0072990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 >> 172.21.15.45:0/2944235084 conn(0x7f39a00fee30 msgr2=0x7f39a0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.836+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 shutdown_connections 2024-03-12T05:24:43.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:43.837+0000 7f39a493e640 1 -- 172.21.15.45:0/2944235084 wait complete. 2024-03-12T05:24:43.851 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:43.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:24:43.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:43 smithi045 ceph-mon[32174]: pgmap v434: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 9.5 MiB/s wr, 993 op/s 2024-03-12T05:24:43.895 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:43 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/978032980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 -- 172.21.15.45:0/3174928596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394410d730 msgr2=0x7f394410db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 --2- 172.21.15.45:0/3174928596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394410d730 0x7f394410db10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3938009860 tx=0x7f393802bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 -- 172.21.15.45:0/3174928596 shutdown_connections 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 --2- 172.21.15.45:0/3174928596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3944107720 0x7f3944107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 --2- 172.21.15.45:0/3174928596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 0x7f39441145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 --2- 172.21.15.45:0/3174928596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394410d730 0x7f394410db10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.041+0000 7f3948e8a640 1 -- 172.21.15.45:0/3174928596 >> 172.21.15.45:0/3174928596 conn(0x7f39440756c0 msgr2=0x7f3944077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.042+0000 7f3948e8a640 1 -- 172.21.15.45:0/3174928596 shutdown_connections 2024-03-12T05:24:44.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.042+0000 7f3948e8a640 1 -- 172.21.15.45:0/3174928596 wait complete. 2024-03-12T05:24:44.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.043+0000 7f3948e8a640 1 Processor -- start 2024-03-12T05:24:44.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.043+0000 7f3948e8a640 1 -- start start 2024-03-12T05:24:44.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 0x7f394419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f394410d730 0x7f39441a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f394259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f394259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40218/0 (socket says 172.21.15.45:40218) 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39441a2c00 con 0x7f394410d730 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39441a2d70 con 0x7f3944107720 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f3948e8a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39441a2f30 con 0x7f39441080e0 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.044+0000 7f394259c640 1 -- 172.21.15.45:0/462984246 learned_addr learned my addr 172.21.15.45:0/462984246 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f3942d9d640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f394410d730 0x7f39441a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f3941d9b640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 0x7f394419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 -- 172.21.15.45:0/462984246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 msgr2=0x7f394419e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 0x7f394419e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 -- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f394410d730 msgr2=0x7f39441a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f394410d730 0x7f39441a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 -- 172.21.15.45:0/462984246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938009490 con 0x7f3944107720 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.045+0000 7f394259c640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3938037040 tx=0x7f393802c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.046+0000 7f39337fe640 1 -- 172.21.15.45:0/462984246 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938034440 con 0x7f3944107720 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.046+0000 7f39337fe640 1 -- 172.21.15.45:0/462984246 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f393802ce90 con 0x7f3944107720 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.046+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39441a31b0 con 0x7f3944107720 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.046+0000 7f39337fe640 1 -- 172.21.15.45:0/462984246 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938034440 con 0x7f3944107720 2024-03-12T05:24:44.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.046+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39441a3480 con 0x7f3944107720 2024-03-12T05:24:44.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.051+0000 7f394ba57740 1 Processor -- start 2024-03-12T05:24:44.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.051+0000 7f394ba57740 1 -- start start 2024-03-12T05:24:44.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771250 0x557cf5771630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf577a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf572c960 con 0x557cf5771b70 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf56e0310 con 0x557cf577a750 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f394ba57740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf56dd840 con 0x557cf5771250 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3941d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3941d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40220/0 (socket says 172.21.15.45:40220) 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3941d9b640 1 -- 172.21.15.45:0/1001430393 learned_addr learned my addr 172.21.15.45:0/1001430393 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3942d9d640 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771250 0x557cf5771630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3941d9b640 1 -- 172.21.15.45:0/1001430393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771250 msgr2=0x557cf5771630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.052+0000 7f3941d9b640 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771250 0x557cf5771630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3941d9b640 1 -- 172.21.15.45:0/1001430393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 msgr2=0x557cf577a210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3941d9b640 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf577a210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3941d9b640 1 -- 172.21.15.45:0/1001430393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cf569b060 con 0x557cf577a750 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3941d9b640 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3934009bf0 tx=0x7f39340117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf105489363032a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3930ff9640 1 -- 172.21.15.45:0/1001430393 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934012450 con 0x557cf577a750 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3930ff9640 1 -- 172.21.15.45:0/1001430393 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3934012a60 con 0x557cf577a750 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.053+0000 7f3930ff9640 1 -- 172.21.15.45:0/1001430393 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393400fa40 con 0x557cf577a750 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f39337fe640 1 -- 172.21.15.45:0/462984246 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f393803b040 con 0x7f3944107720 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 -- 172.21.15.45:0/1001430393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 msgr2=0x557cf577cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3934009bf0 tx=0x7f39340117b0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 -- 172.21.15.45:0/1001430393 shutdown_connections 2024-03-12T05:24:44.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf577a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf577cb40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 --2- 172.21.15.45:0/1001430393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771250 0x557cf5771630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.054+0000 7f394ba57740 1 -- 172.21.15.45:0/1001430393 >> 172.21.15.45:0/1001430393 conn(0x557cf576b980 msgr2=0x557cf576bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f394ba57740 1 -- 172.21.15.45:0/1001430393 shutdown_connections 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f394ba57740 1 -- 172.21.15.45:0/1001430393 wait complete. 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f39337fe640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 0x7f392c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f39337fe640 1 -- 172.21.15.45:0/462984246 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f39380eeda0 con 0x7f3944107720 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f394ba57740 1 -- start start 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f3941d9b640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 0x7f392c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.055+0000 7f394ba57740 1 Processor -- start 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771b70 0x557cf5777cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf57781f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf56d9020 con 0x557cf5771250 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf53ac430 con 0x557cf577a750 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5368e00 con 0x557cf5771b70 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394ba57740 10 client.0 _ll_register_callbacks cb 0x557cf5680cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59964/0 (socket says 172.21.15.45:59964) 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f394259c640 1 -- 172.21.15.45:0/2598073531 learned_addr learned my addr 172.21.15.45:0/2598073531 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f3942d9d640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf57781f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f3942d9d640 1 -- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771b70 msgr2=0x557cf5777cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f3942d9d640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771b70 0x557cf5777cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.056+0000 7f3942d9d640 1 -- 172.21.15.45:0/2598073531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 msgr2=0x557cf5777770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3942d9d640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3942d9d640 1 -- 172.21.15.45:0/2598073531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3934009850 con 0x557cf577a750 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f394259c640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:44.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3941d9b640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 0x7f392c0a4010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x557cf576cb10 tx=0x7f39340097b0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3942d9d640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf57781f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f393c003f00 tx=0x7f393c004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34adc17ba43cd988 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3929ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c004d20 con 0x557cf577a750 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.057+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f393c012860 con 0x557cf577a750 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.058+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557cf53d87e0 con 0x557cf577a750 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.058+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f393c004e80 con 0x557cf577a750 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.058+0000 7f394ba57740 10 client.28412 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.058+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557cf56d76a0 con 0x557cf577a750 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.058+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x557cf53d87e0 con 0x557cf577a750 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f393c005800 con 0x557cf577a750 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 10 client.28412 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 10 client.28412 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 20 client.28412 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf579ba50 0x557cf579de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 4 client.28412 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:44.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.059+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cf5415720 con 0x557cf579ba50 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.060+0000 7f394259c640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf579ba50 0x557cf579de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.062+0000 7f394259c640 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf579ba50 0x557cf579de70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d662e87c959255e2 server_cookie=a09cff604412a3b0 in_seq=0 out_seq=0 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.063+0000 7f3929ffb640 10 client.28412 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.063+0000 7f3929ffb640 1 -- 172.21.15.45:0/2598073531 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557cf5415720 con 0x557cf579ba50 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.063+0000 7f3929ffb640 10 client.28412 handle_command_reply: tid=0 2024-03-12T05:24:44.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.063+0000 7f394ba57740 1 client.28412 shutdown 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.064+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 msgr2=0x557cf57781f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.064+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf57781f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f393c003f00 tx=0x7f393c004410 comp rx=0 tx=0).stop 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 shutdown_connections 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf579ba50 0x557cf579de70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf57781f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5771b70 0x557cf5777cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 --2- 172.21.15.45:0/2598073531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771250 0x557cf5777770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 >> 172.21.15.45:0/2598073531 conn(0x557cf576b980 msgr2=0x557cf576d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f3929ffb640 0 client.28412 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 shutdown_connections 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 1 -- 172.21.15.45:0/2598073531 wait complete. 2024-03-12T05:24:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.065+0000 7f394ba57740 20 client.28412 trim_cache size 0 max 16384 2024-03-12T05:24:44.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 Processor -- start 2024-03-12T05:24:44.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 -- start start 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf5771250 0x557cf5778010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5772ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf565e130 con 0x557cf5771b70 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5798700 con 0x557cf5771250 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.088+0000 7f394ba57740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf576c980 con 0x557cf577a750 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41746/0 (socket says 172.21.15.45:41746) 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 -- 172.21.15.45:0/977580432 learned_addr learned my addr 172.21.15.45:0/977580432 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 -- 172.21.15.45:0/977580432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf5771250 msgr2=0x557cf5778010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf5771250 0x557cf5778010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 -- 172.21.15.45:0/977580432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 msgr2=0x557cf5772ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:44.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5772ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 -- 172.21.15.45:0/977580432 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cf5673c20 con 0x557cf577a750 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.089+0000 7f3942d9d640 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c0099c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f702bfcdd27256 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.090+0000 7f392bfff640 1 -- 172.21.15.45:0/977580432 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c0246e0 con 0x557cf577a750 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.090+0000 7f392bfff640 1 -- 172.21.15.45:0/977580432 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f393c024840 con 0x557cf577a750 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.090+0000 7f392bfff640 1 -- 172.21.15.45:0/977580432 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c010680 con 0x557cf577a750 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 -- 172.21.15.45:0/977580432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 msgr2=0x557cf57749f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f393c000f80 tx=0x7f393c0099c0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 -- 172.21.15.45:0/977580432 shutdown_connections 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5772ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf577a750 0x557cf57749f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 --2- 172.21.15.45:0/977580432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf5771250 0x557cf5778010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 -- 172.21.15.45:0/977580432 >> 172.21.15.45:0/977580432 conn(0x557cf576b980 msgr2=0x557cf57a27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.091+0000 7f394ba57740 1 -- 172.21.15.45:0/977580432 shutdown_connections 2024-03-12T05:24:44.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 -- 172.21.15.45:0/977580432 wait complete. 2024-03-12T05:24:44.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 -- start start 2024-03-12T05:24:44.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 Processor -- start 2024-03-12T05:24:44.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5780010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf5780550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5781e70 con 0x557cf5771b70 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf5781fe0 con 0x557cf577a750 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.092+0000 7f394ba57740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cf57f24f0 con 0x557cf5780a90 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f394ba57740 10 client.0 _ll_register_callbacks cb 0x7f3934009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41750/0 (socket says 172.21.15.45:41750) 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 -- 172.21.15.45:0/1057575302 learned_addr learned my addr 172.21.15.45:0/1057575302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f394259c640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5780010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3941d9b640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf5780550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 -- 172.21.15.45:0/1057575302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 msgr2=0x557cf5780550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf5780550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 -- 172.21.15.45:0/1057575302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 msgr2=0x557cf5780010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5780010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.093+0000 7f3942d9d640 1 -- 172.21.15.45:0/1057575302 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cf56d76a0 con 0x557cf5780a90 2024-03-12T05:24:44.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.094+0000 7f3942d9d640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f393c009600 tx=0x7f393c003ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a428c8ce52986d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.098+0000 7f3929ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.098+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c034070 con 0x557cf5780a90 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.098+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3934012a60 con 0x557cf5780a90 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.098+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f393c029030 con 0x557cf5780a90 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.098+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f393c0246e0 con 0x557cf5780a90 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 10 client.28375 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557cf56998c0 con 0x557cf5780a90 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3934012a60 con 0x557cf5780a90 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f393c010460 con 0x557cf5780a90 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 10 client.28375 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 10 client.28375 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 20 client.28375 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf57f3990 0x557cf57f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 4 client.28375 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:44.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.099+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557cf5464890 con 0x557cf57f3990 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.100+0000 7f394259c640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf57f3990 0x557cf57f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f3929ffb640 10 client.28375 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394259c640 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf57f3990 0x557cf57f5d80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c33e0a14af1ac02f server_cookie=6a83e4a548223c32 in_seq=0 out_seq=0 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f3929ffb640 1 -- 172.21.15.45:0/1057575302 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557cf5464890 con 0x557cf57f3990 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f3929ffb640 10 client.28375 handle_command_reply: tid=0 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 client.28375 shutdown 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 msgr2=0x557cf57f1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f393c009600 tx=0x7f393c003ab0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 shutdown_connections 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x557cf57f3990 0x557cf57f5d80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cf5780a90 0x557cf57f1f60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557cf577a750 0x557cf5780550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f3929ffb640 0 client.28375 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 --2- 172.21.15.45:0/1057575302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x557cf5771b70 0x557cf5780010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 >> 172.21.15.45:0/1057575302 conn(0x557cf576b980 msgr2=0x557cf5775c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 shutdown_connections 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.101+0000 7f394ba57740 1 -- 172.21.15.45:0/1057575302 wait complete. 2024-03-12T05:24:44.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.102+0000 7f394ba57740 20 client.28375 trim_cache size 0 max 16384 2024-03-12T05:24:44.098 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.102+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 msgr2=0x7f392c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.102+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 0x7f392c0a4010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x557cf576cb10 tx=0x7f39340097b0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 msgr2=0x7f394419dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3938037040 tx=0x7f393802c740 comp rx=0 tx=0).stop 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 shutdown_connections 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f392c0a1b50 0x7f392c0a4010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f394410d730 0x7f39441a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39441080e0 0x7f394419e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 --2- 172.21.15.45:0/462984246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3944107720 0x7f394419dd40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 >> 172.21.15.45:0/462984246 conn(0x7f39440756c0 msgr2=0x7f3944075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 shutdown_connections 2024-03-12T05:24:44.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.103+0000 7f3948e8a640 1 -- 172.21.15.45:0/462984246 wait complete. 2024-03-12T05:24:44.115 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:44.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:24:44.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:43 smithi130 ceph-mon[35398]: pgmap v434: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 9.5 MiB/s wr, 993 op/s 2024-03-12T05:24:44.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:43 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/978032980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:44.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:43 smithi116 ceph-mon[35486]: pgmap v434: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 9.5 MiB/s wr, 993 op/s 2024-03-12T05:24:44.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:43 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/978032980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:24:44.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.344+0000 7efd80640640 1 -- 172.21.15.45:0/632452255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 msgr2=0x7efd7810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.344+0000 7efd80640640 1 --2- 172.21.15.45:0/632452255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd7810d490 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7efd74009900 tx=0x7efd7402bc40 comp rx=0 tx=0).stop 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 -- 172.21.15.45:0/632452255 shutdown_connections 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 --2- 172.21.15.45:0/632452255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7811a960 0x7efd7811cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 --2- 172.21.15.45:0/632452255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7810d9d0 0x7efd7810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 --2- 172.21.15.45:0/632452255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd7810d490 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 -- 172.21.15.45:0/632452255 >> 172.21.15.45:0/632452255 conn(0x7efd7806c260 msgr2=0x7efd7806c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 -- 172.21.15.45:0/632452255 shutdown_connections 2024-03-12T05:24:44.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.345+0000 7efd80640640 1 -- 172.21.15.45:0/632452255 wait complete. 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 Processor -- start 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 -- start start 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd78118400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7811a960 0x7efd781111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd78111950 con 0x7efd7810d9d0 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd78111ac0 con 0x7efd7811a960 2024-03-12T05:24:44.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd80640640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd78111c80 con 0x7efd7810f9a0 2024-03-12T05:24:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd7dbdb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd78118400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.346+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59990/0 (socket says 172.21.15.45:59990) 2024-03-12T05:24:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 -- 172.21.15.45:0/288243214 learned_addr learned my addr 172.21.15.45:0/288243214 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7811a960 0x7efd781111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 -- 172.21.15.45:0/288243214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 msgr2=0x7efd78118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd78118400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 -- 172.21.15.45:0/288243214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7811a960 msgr2=0x7efd781111c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7811a960 0x7efd781111c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.347+0000 7efd7e3dc640 1 -- 172.21.15.45:0/288243214 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74009560 con 0x7efd7810d9d0 2024-03-12T05:24:44.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.348+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efd74006010 tx=0x7efd74005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.348+0000 7efd6f7fe640 1 -- 172.21.15.45:0/288243214 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7402dea0 con 0x7efd7810d9d0 2024-03-12T05:24:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.349+0000 7efd6f7fe640 1 -- 172.21.15.45:0/288243214 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd740043d0 con 0x7efd7810d9d0 2024-03-12T05:24:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.349+0000 7efd6f7fe640 1 -- 172.21.15.45:0/288243214 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74005250 con 0x7efd7810d9d0 2024-03-12T05:24:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.349+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd781159e0 con 0x7efd7810d9d0 2024-03-12T05:24:44.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.349+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd78115ea0 con 0x7efd7810d9d0 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 Processor -- start 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 -- start start 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e009630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 0x55f63e012210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63dfc4960 con 0x55f63e009250 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63df78310 con 0x55f63e009b70 2024-03-12T05:24:44.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.353+0000 7efd83211740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63df75840 con 0x55f63e012750 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 0x55f63e012210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e014b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 0x55f63e012210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40282/0 (socket says 172.21.15.45:40282) 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3158206243 learned_addr learned my addr 172.21.15.45:0/3158206243 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 -- 172.21.15.45:0/3158206243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 msgr2=0x55f63e012210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 0x55f63e012210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 -- 172.21.15.45:0/3158206243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 msgr2=0x55f63e009630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e009630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.354+0000 7efd7dbdb640 1 -- 172.21.15.45:0/3158206243 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f63df33060 con 0x55f63e012750 2024-03-12T05:24:44.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.355+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e014b40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efd78112790 tx=0x7efd680113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66020bad089297e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.355+0000 7efd6cbfe640 1 -- 172.21.15.45:0/3158206243 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd68011e10 con 0x55f63e012750 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.355+0000 7efd6cbfe640 1 -- 172.21.15.45:0/3158206243 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd680044e0 con 0x55f63e012750 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.356+0000 7efd6cbfe640 1 -- 172.21.15.45:0/3158206243 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd680126b0 con 0x55f63e012750 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.356+0000 7efd83211740 1 -- 172.21.15.45:0/3158206243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 msgr2=0x55f63e014b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.356+0000 7efd83211740 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e014b40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efd78112790 tx=0x7efd680113e0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.356+0000 7efd6f7fe640 1 -- 172.21.15.45:0/288243214 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efd74003c30 con 0x7efd7810d9d0 2024-03-12T05:24:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd6f7fe640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 0x7efd640a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd6f7fe640 1 -- 172.21.15.45:0/288243214 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd740f4fb0 con 0x7efd7810d9d0 2024-03-12T05:24:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 0x7efd640a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd83211740 1 -- 172.21.15.45:0/3158206243 shutdown_connections 2024-03-12T05:24:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd83211740 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e014b40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd83211740 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009b70 0x55f63e012210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd83211740 1 --2- 172.21.15.45:0/3158206243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e009630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.357+0000 7efd83211740 1 -- 172.21.15.45:0/3158206243 >> 172.21.15.45:0/3158206243 conn(0x55f63e003980 msgr2=0x55f63e003d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.358+0000 7efd83211740 1 -- 172.21.15.45:0/3158206243 shutdown_connections 2024-03-12T05:24:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.358+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 0x7efd640a41b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd70005fd0 tx=0x7efd70004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.358+0000 7efd83211740 1 -- 172.21.15.45:0/3158206243 wait complete. 2024-03-12T05:24:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.358+0000 7efd83211740 1 -- start start 2024-03-12T05:24:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.358+0000 7efd83211740 1 Processor -- start 2024-03-12T05:24:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 0x55f63e00fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e010080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40286/0 (socket says 172.21.15.45:40286) 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3293304549 learned_addr learned my addr 172.21.15.45:0/3293304549 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63df71020 con 0x55f63e009b70 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63dc44430 con 0x55f63e009250 2024-03-12T05:24:44.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e010080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63dc00e00 con 0x55f63e012750 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd83211740 10 client.0 _ll_register_callbacks cb 0x55f63df18cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.359+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 0x55f63e00fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 msgr2=0x55f63e010080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e010080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3293304549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 msgr2=0x55f63e00fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 0x55f63e00fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd68009780 con 0x55f63e009250 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 0x55f63e00fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd74005ec0 tx=0x7efd740c18c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a42a8d4e3a9e7821 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:44.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd740c2120 con 0x55f63e009250 2024-03-12T05:24:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd740c5e10 con 0x55f63e009250 2024-03-12T05:24:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.360+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f63dc707e0 con 0x55f63e009250 2024-03-12T05:24:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd740ca460 con 0x55f63e009250 2024-03-12T05:24:44.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 10 client.28427 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f63df6f6a0 con 0x55f63e009250 2024-03-12T05:24:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd7408d2a0 con 0x55f63e009250 2024-03-12T05:24:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd7404f040 con 0x55f63e009250 2024-03-12T05:24:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 10 client.28427 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 10 client.28427 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 20 client.28427 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e0227c0 0x55f63e024c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 4 client.28427 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.361+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f63dc707e0 con 0x55f63e0227c0 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.362+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e0227c0 0x55f63e024c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.363+0000 7efd61ffb640 10 client.28427 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:44.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.363+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e0227c0 0x55f63e024c10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7dff7d1a28e082a6 server_cookie=cd96bf7b9192ef92 in_seq=0 out_seq=0 2024-03-12T05:24:44.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.364+0000 7efd61ffb640 1 -- 172.21.15.45:0/3293304549 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f63dc707e0 con 0x55f63e0227c0 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.364+0000 7efd61ffb640 10 client.28427 handle_command_reply: tid=0 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.365+0000 7efd83211740 1 client.28427 shutdown 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.365+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 msgr2=0x55f63e00f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.365+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd74005ec0 tx=0x7efd740c18c0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 shutdown_connections 2024-03-12T05:24:44.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009b70 0x55f63e00fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e0227c0 0x55f63e024c10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e012750 0x55f63e010080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 --2- 172.21.15.45:0/3293304549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e00f600 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.366+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 >> 172.21.15.45:0/3293304549 conn(0x55f63e003980 msgr2=0x55f63e0056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.367+0000 7efd61ffb640 0 client.28427 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:44.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.367+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 shutdown_connections 2024-03-12T05:24:44.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.367+0000 7efd83211740 1 -- 172.21.15.45:0/3293304549 wait complete. 2024-03-12T05:24:44.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.368+0000 7efd83211740 20 client.28427 trim_cache size 0 max 16384 2024-03-12T05:24:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.391+0000 7efd83211740 1 Processor -- start 2024-03-12T05:24:44.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.391+0000 7efd83211740 1 -- start start 2024-03-12T05:24:44.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd83211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd83211740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e029570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40298/0 (socket says 172.21.15.45:40298) 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd7e3dc640 1 -- 172.21.15.45:0/1216973890 learned_addr learned my addr 172.21.15.45:0/1216973890 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.392+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e029570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.393+0000 7efd83211740 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e012750 0x55f63e00c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.393+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e012750 0x55f63e00c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.393+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63def6130 con 0x55f63e012750 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.393+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63dc70810 con 0x55f63e009250 2024-03-12T05:24:44.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.393+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63e004980 con 0x55f63e009b70 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 -- 172.21.15.45:0/1216973890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 msgr2=0x55f63e029570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e029570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 -- 172.21.15.45:0/1216973890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e012750 msgr2=0x55f63e00c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e012750 0x55f63e00c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 -- 172.21.15.45:0/1216973890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f63df0bc20 con 0x55f63e009250 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.394+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd740f2cd0 tx=0x7efd74000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46fc713f32ed489f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.395+0000 7efd63fff640 1 -- 172.21.15.45:0/1216973890 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7408d4a0 con 0x55f63e009250 2024-03-12T05:24:44.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.395+0000 7efd63fff640 1 -- 172.21.15.45:0/1216973890 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd74005680 con 0x55f63e009250 2024-03-12T05:24:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.395+0000 7efd63fff640 1 -- 172.21.15.45:0/1216973890 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd740c2160 con 0x55f63e009250 2024-03-12T05:24:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.396+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 msgr2=0x55f63e011d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.396+0000 7efd83211740 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd740f2cd0 tx=0x7efd74000bc0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 shutdown_connections 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e012750 0x55f63e00c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e029570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 --2- 172.21.15.45:0/1216973890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e009250 0x55f63e011d30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 >> 172.21.15.45:0/1216973890 conn(0x55f63e003980 msgr2=0x55f63dffea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 shutdown_connections 2024-03-12T05:24:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.397+0000 7efd83211740 1 -- 172.21.15.45:0/1216973890 wait complete. 2024-03-12T05:24:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.398+0000 7efd83211740 1 -- start start 2024-03-12T05:24:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.398+0000 7efd83211740 1 Processor -- start 2024-03-12T05:24:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.398+0000 7efd83211740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e0179c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e017f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 0x55f63e071d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63e072270 con 0x55f63e009250 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63e0723e0 con 0x55f63e012750 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e0179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f63e0725a0 con 0x55f63e009b70 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7e3dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e0179c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60030/0 (socket says 172.21.15.45:60030) 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7e3dc640 1 -- 172.21.15.45:0/3699601065 learned_addr learned my addr 172.21.15.45:0/3699601065 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd83211740 10 client.0 _ll_register_callbacks cb 0x7efd68009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 0x55f63e071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7dbdb640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e017f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 -- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 msgr2=0x55f63e017f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e017f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 -- 172.21.15.45:0/3699601065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 msgr2=0x55f63e0179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e0179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.399+0000 7efd7ebdd640 1 -- 172.21.15.45:0/3699601065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd68009780 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.400+0000 7efd7ebdd640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 0x55f63e071d30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55f63dff5a80 tx=0x7efd70009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c75594b669e233a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.400+0000 7efd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.400+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70024070 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.400+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7efd70017070 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.401+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd680044e0 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.401+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd70004b20 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.401+0000 7efd83211740 10 client.28433 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.401+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f63df318c0 con 0x55f63e012750 2024-03-12T05:24:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.401+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7efd680044e0 con 0x55f63e012750 2024-03-12T05:24:44.397 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd7002f020 con 0x55f63e012750 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 10 client.28433 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 10 client.28433 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 20 client.28433 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e079d90 0x55f63e07c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 4 client.28433 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f63dcad720 con 0x55f63e079d90 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.402+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e079d90 0x55f63e07c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.403+0000 7efd61ffb640 10 client.28433 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.403+0000 7efd7e3dc640 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e079d90 0x55f63e07c1b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=40badab653efa39a server_cookie=fda63cab631d2264 in_seq=0 out_seq=0 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.403+0000 7efd61ffb640 1 -- 172.21.15.45:0/3699601065 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f63dcad720 con 0x55f63e079d90 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.403+0000 7efd61ffb640 10 client.28433 handle_command_reply: tid=0 2024-03-12T05:24:44.398 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.403+0000 7efd83211740 1 client.28433 shutdown 2024-03-12T05:24:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 msgr2=0x55f63e071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 0x55f63e071d30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55f63dff5a80 tx=0x7efd70009500 comp rx=0 tx=0).stop 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 shutdown_connections 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55f63e079d90 0x55f63e07c1b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f63e012750 0x55f63e071d30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd61ffb640 0 client.28433 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f63e009b70 0x55f63e017f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 --2- 172.21.15.45:0/3699601065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f63e009250 0x55f63e0179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 >> 172.21.15.45:0/3699601065 conn(0x55f63e003980 msgr2=0x55f63e029570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 shutdown_connections 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 1 -- 172.21.15.45:0/3699601065 wait complete. 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.404+0000 7efd83211740 20 client.28433 trim_cache size 0 max 16384 2024-03-12T05:24:44.400 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.405+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 msgr2=0x7efd640a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.405+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 0x7efd640a41b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd70005fd0 tx=0x7efd70004a40 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.405+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 msgr2=0x7efd78117ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.405+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7efd74006010 tx=0x7efd74005e10 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 shutdown_connections 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7efd640a1cf0 0x7efd640a41b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd7811a960 0x7efd781111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efd7810f9a0 0x7efd78118400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 --2- 172.21.15.45:0/288243214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efd7810d9d0 0x7efd78117ec0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 >> 172.21.15.45:0/288243214 conn(0x7efd7806c260 msgr2=0x7efd7811c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 shutdown_connections 2024-03-12T05:24:44.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.406+0000 7efd80640640 1 -- 172.21.15.45:0/288243214 wait complete. 2024-03-12T05:24:44.420 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:24:44.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:24:44.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.602+0000 7f4f7748d640 1 -- 172.21.15.45:0/2786623254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 msgr2=0x7f4f70073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.602+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2786623254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f70073900 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009900 tx=0x7f4f6002bcc0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 -- 172.21.15.45:0/2786623254 shutdown_connections 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2786623254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70112130 0x7f4f70114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2786623254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70073e40 0x7f4f70111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2786623254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f70073900 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 -- 172.21.15.45:0/2786623254 >> 172.21.15.45:0/2786623254 conn(0x7f4f700fcbb0 msgr2=0x7f4f700fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 -- 172.21.15.45:0/2786623254 shutdown_connections 2024-03-12T05:24:44.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.603+0000 7f4f7748d640 1 -- 172.21.15.45:0/2786623254 wait complete. 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.604+0000 7f4f7748d640 1 Processor -- start 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.604+0000 7f4f7748d640 1 -- start start 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.604+0000 7f4f7748d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f701a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.604+0000 7f4f7748d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70073e40 0x7f4f701a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.604+0000 7f4f7748d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 0x7f4f7019d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f7748d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7019df90 con 0x7f4f70073e40 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f7748d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7019e100 con 0x7f4f70112130 2024-03-12T05:24:44.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f7748d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7019e2c0 con 0x7f4f70073520 2024-03-12T05:24:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75229640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f701a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75229640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f701a3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41800/0 (socket says 172.21.15.45:41800) 2024-03-12T05:24:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75229640 1 -- 172.21.15.45:0/2554389239 learned_addr learned my addr 172.21.15.45:0/2554389239 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 0x7f4f7019d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 -- 172.21.15.45:0/2554389239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 msgr2=0x7f4f701a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f701a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 -- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70073e40 msgr2=0x7f4f701a3cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70073e40 0x7f4f701a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70073e40 0x7f4f701a3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.605+0000 7f4f75a2a640 1 -- 172.21.15.45:0/2554389239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f60009560 con 0x7f4f70112130 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 0x7f4f7019d940 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4f68005fd0 tx=0x7f4f6800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f667fc640 1 -- 172.21.15.45:0/2554389239 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800cdf0 con 0x7f4f70112130 2024-03-12T05:24:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f667fc640 1 -- 172.21.15.45:0/2554389239 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f68005020 con 0x7f4f70112130 2024-03-12T05:24:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7019e5a0 con 0x7f4f70112130 2024-03-12T05:24:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f701aa830 con 0x7f4f70112130 2024-03-12T05:24:44.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.606+0000 7f4f667fc640 1 -- 172.21.15.45:0/2554389239 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68010690 con 0x7f4f70112130 2024-03-12T05:24:44.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.609+0000 7f4f667fc640 1 -- 172.21.15.45:0/2554389239 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4f680107f0 con 0x7f4f70112130 2024-03-12T05:24:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 Processor -- start 2024-03-12T05:24:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 -- start start 2024-03-12T05:24:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c61630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c6cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993c1c960 con 0x555993c6a750 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993bd0310 con 0x555993c61b70 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.610+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993bcd840 con 0x555993c61250 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f667fc640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 0x7f4f5c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40330/0 (socket says 172.21.15.45:40330) 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 -- 172.21.15.45:0/3308337608 learned_addr learned my addr 172.21.15.45:0/3308337608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 0x7f4f5c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 -- 172.21.15.45:0/3308337608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 msgr2=0x555993c61630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c61630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.611+0000 7f4f75229640 1 -- 172.21.15.45:0/3308337608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 msgr2=0x555993c6cb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f74a28640 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c6cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f75229640 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c6cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f75229640 1 -- 172.21.15.45:0/3308337608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555993b8b060 con 0x555993c61b70 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f667fc640 1 -- 172.21.15.45:0/2554389239 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4f68014070 con 0x7f4f70112130 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f75229640 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4f60005d50 tx=0x7f4f60005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8e0f1a5ecc50a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f75a2a640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 0x7f4f5c0a40e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4f6809a720 tx=0x7f4f6809a8f0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f5b7fe640 1 -- 172.21.15.45:0/3308337608 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60004a90 con 0x555993c61b70 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.612+0000 7f4f5b7fe640 1 -- 172.21.15.45:0/3308337608 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f60004040 con 0x555993c61b70 2024-03-12T05:24:44.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.613+0000 7f4f5b7fe640 1 -- 172.21.15.45:0/3308337608 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60003b60 con 0x555993c61b70 2024-03-12T05:24:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.613+0000 7f4f7a05a740 1 -- 172.21.15.45:0/3308337608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 msgr2=0x555993c6a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.613+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4f60005d50 tx=0x7f4f60005d80 comp rx=0 tx=0).stop 2024-03-12T05:24:44.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.613+0000 7f4f7a05a740 1 -- 172.21.15.45:0/3308337608 shutdown_connections 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c6cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c6a210 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/3308337608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c61630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 -- 172.21.15.45:0/3308337608 >> 172.21.15.45:0/3308337608 conn(0x555993c5b980 msgr2=0x555993c5bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 -- 172.21.15.45:0/3308337608 shutdown_connections 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.614+0000 7f4f7a05a740 1 -- 172.21.15.45:0/3308337608 wait complete. 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.615+0000 7f4f7a05a740 1 -- start start 2024-03-12T05:24:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.615+0000 7f4f7a05a740 1 Processor -- start 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61250 0x555993c67630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c6a750 0x555993c680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993bc9020 con 0x555993c61250 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55599389c430 con 0x555993c6a750 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993858e00 con 0x555993c61b70 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f7a05a740 10 client.0 _ll_register_callbacks cb 0x555993b70cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f75229640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f75229640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41820/0 (socket says 172.21.15.45:41820) 2024-03-12T05:24:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f75229640 1 -- 172.21.15.45:0/2968257719 learned_addr learned my addr 172.21.15.45:0/2968257719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.616+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61250 0x555993c67630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 -- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c6a750 msgr2=0x555993c680b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c6a750 0x555993c680b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 -- 172.21.15.45:0/2968257719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61250 msgr2=0x555993c67630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61250 0x555993c67630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 -- 172.21.15.45:0/2968257719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f70114e50 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f75229640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4f60003800 tx=0x7f4f6002ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c458fffc4cbb4cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.617+0000 7f4f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60003cb0 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4f6002de40 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559938c87e0 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f60004a90 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f7a05a740 10 client.28402 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555993bc76a0 con 0x555993c61b70 2024-03-12T05:24:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.618+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f6005e080 con 0x555993c61b70 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f6003c050 con 0x555993c61b70 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 10 client.28402 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 10 client.28402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 20 client.28402 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993c7a7b0 0x555993c7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 4 client.28402 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555993905720 con 0x555993c7a7b0 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.619+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993c7a7b0 0x555993c7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.620+0000 7f4f58ff9640 10 client.28402 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.620+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993c7a7b0 0x555993c7cbd0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1badc8a8edead9e5 server_cookie=a9e24d5ae902a8c1 in_seq=0 out_seq=0 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.620+0000 7f4f58ff9640 1 -- 172.21.15.45:0/2968257719 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555993905720 con 0x555993c7a7b0 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.620+0000 7f4f58ff9640 10 client.28402 handle_command_reply: tid=0 2024-03-12T05:24:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.620+0000 7f4f7a05a740 1 client.28402 shutdown 2024-03-12T05:24:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.621+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 msgr2=0x555993c67b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.621+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4f60003800 tx=0x7f4f6002ca30 comp rx=0 tx=0).stop 2024-03-12T05:24:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.621+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 shutdown_connections 2024-03-12T05:24:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.621+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993c7a7b0 0x555993c7cbd0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.622+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c6a750 0x555993c680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.622+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61b70 0x555993c67b70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.622+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2968257719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61250 0x555993c67630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.622+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 >> 172.21.15.45:0/2968257719 conn(0x555993c5b980 msgr2=0x555993c5d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.622+0000 7f4f58ff9640 0 client.28402 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.623+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 shutdown_connections 2024-03-12T05:24:44.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.623+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2968257719 wait complete. 2024-03-12T05:24:44.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.623+0000 7f4f7a05a740 20 client.28402 trim_cache size 0 max 16384 2024-03-12T05:24:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.644+0000 7f4f7a05a740 1 Processor -- start 2024-03-12T05:24:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 -- start start 2024-03-12T05:24:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c66790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c5aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993b4e130 con 0x555993c6a750 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993c5c780 con 0x555993c61b70 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.645+0000 7f4f7a05a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993c5c980 con 0x555993c61250 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.646+0000 7f4f75229640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.646+0000 7f4f75229640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40358/0 (socket says 172.21.15.45:40358) 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.646+0000 7f4f75229640 1 -- 172.21.15.45:0/2266938586 learned_addr learned my addr 172.21.15.45:0/2266938586 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.647+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c66790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.647+0000 7f4f74a28640 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c5aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 -- 172.21.15.45:0/2266938586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 msgr2=0x555993c66790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c66790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 -- 172.21.15.45:0/2266938586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 msgr2=0x555993c5aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c5aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 -- 172.21.15.45:0/2266938586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555993b63c20 con 0x555993c61b70 2024-03-12T05:24:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f75229640 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f60036040 tx=0x7f4f60000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=368cd68d568128a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.648+0000 7f4f5affd640 1 -- 172.21.15.45:0/2266938586 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60051df0 con 0x555993c61b70 2024-03-12T05:24:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.649+0000 7f4f5affd640 1 -- 172.21.15.45:0/2266938586 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5559938c87e0 con 0x555993c61b70 2024-03-12T05:24:44.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.649+0000 7f4f5affd640 1 -- 172.21.15.45:0/2266938586 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6004ac10 con 0x555993c61b70 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2266938586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 msgr2=0x555993c67ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f60036040 tx=0x7f4f60000bc0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2266938586 shutdown_connections 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c6a750 0x555993c5aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61b70 0x555993c67ed0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/2266938586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c61250 0x555993c66790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2266938586 >> 172.21.15.45:0/2266938586 conn(0x555993c5b980 msgr2=0x555993c5a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2266938586 shutdown_connections 2024-03-12T05:24:44.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.651+0000 7f4f7a05a740 1 -- 172.21.15.45:0/2266938586 wait complete. 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.652+0000 7f4f7a05a740 1 -- start start 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.652+0000 7f4f7a05a740 1 Processor -- start 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.652+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.652+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 0x555993c70640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f7a05a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c6a750 0x555993c70b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f74a28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f74a28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40362/0 (socket says 172.21.15.45:40362) 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f75229640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 0x555993c70640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f75229640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 0x555993c70640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60102/0 (socket says 172.21.15.45:60102) 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f75229640 1 -- 172.21.15.45:0/723279385 learned_addr learned my addr 172.21.15.45:0/723279385 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993ce1cd0 con 0x555993c61b70 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993ce1e10 con 0x555993c61250 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555993ce1f50 con 0x555993c6a750 2024-03-12T05:24:44.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.653+0000 7f4f7a05a740 10 client.0 _ll_register_callbacks cb 0x7f4f6c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c6a750 0x555993c70b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 -- 172.21.15.45:0/723279385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c6a750 msgr2=0x555993c70b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c6a750 0x555993c70b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 -- 172.21.15.45:0/723279385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 msgr2=0x555993c70640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 0x555993c70640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555993bc76a0 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f74a28640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x555993c7d4b0 tx=0x7f4f6c014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f35d322109b17bc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.654+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c019dd0 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f60004040 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4f6c004020 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f6c0041e0 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 10 client.28448 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555993b898c0 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f60004040 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f6c003880 con 0x555993c61250 2024-03-12T05:24:44.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 10 client.28448 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 10 client.28448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 20 client.28448 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993ce37a0 0x555993ce5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 4 client.28448 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.655+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555993954890 con 0x555993ce37a0 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.656+0000 7f4f75229640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993ce37a0 0x555993ce5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:44.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.656+0000 7f4f58ff9640 10 client.28448 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f75229640 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993ce37a0 0x555993ce5bc0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f605b313084c4afc server_cookie=fc15ea7a15564ba0 in_seq=0 out_seq=0 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f58ff9640 1 -- 172.21.15.45:0/723279385 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555993954890 con 0x555993ce37a0 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f58ff9640 10 client.28448 handle_command_reply: tid=0 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 client.28448 shutdown 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 msgr2=0x555993c70100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x555993c7d4b0 tx=0x7f4f6c014040 comp rx=0 tx=0).stop 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 shutdown_connections 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555993ce37a0 0x555993ce5bc0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.657+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555993c61b70 0x555993c70640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555993c6a750 0x555993c70b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 1 --2- 172.21.15.45:0/723279385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555993c61250 0x555993c70100 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f58ff9640 0 client.28448 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 >> 172.21.15.45:0/723279385 conn(0x555993c5b980 msgr2=0x555993c67f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 shutdown_connections 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 1 -- 172.21.15.45:0/723279385 wait complete. 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.658+0000 7f4f7a05a740 20 client.28448 trim_cache size 0 max 16384 2024-03-12T05:24:44.654 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:24:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.659+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 msgr2=0x7f4f5c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.659+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 0x7f4f5c0a40e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4f6809a720 tx=0x7f4f6809a8f0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.659+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 msgr2=0x7f4f7019d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.659+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 0x7f4f7019d940 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4f68005fd0 tx=0x7f4f6800ef10 comp rx=0 tx=0).stop 2024-03-12T05:24:44.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 shutdown_connections 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f5c0a1c20 0x7f4f5c0a40e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f70073e40 0x7f4f701a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f70112130 0x7f4f7019d940 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 --2- 172.21.15.45:0/2554389239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f70073520 0x7f4f701a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 >> 172.21.15.45:0/2554389239 conn(0x7f4f700fcbb0 msgr2=0x7f4f700fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 shutdown_connections 2024-03-12T05:24:44.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:44.660+0000 7f4f7748d640 1 -- 172.21.15.45:0/2554389239 wait complete. 2024-03-12T05:24:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 760.0 s, 1320.6 tps, lat 76.134 ms stddev 28.160 2024-03-12T05:24:45.675 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:24:45.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:24:45.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 -- 172.21.15.45:0/2073409969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8107720 msgr2=0x7fddd8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 --2- 172.21.15.45:0/2073409969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8107720 0x7fddd8107ba0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009860 tx=0x7fddcc02bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 -- 172.21.15.45:0/2073409969 shutdown_connections 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 --2- 172.21.15.45:0/2073409969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 0x7fddd81145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 --2- 172.21.15.45:0/2073409969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd8107720 0x7fddd8107ba0 secure :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009860 tx=0x7fddcc02bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 --2- 172.21.15.45:0/2073409969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 0x7fddd810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.851+0000 7fddddd8a640 1 -- 172.21.15.45:0/2073409969 >> 172.21.15.45:0/2073409969 conn(0x7fddd80756c0 msgr2=0x7fddd8077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.852+0000 7fddddd8a640 1 -- 172.21.15.45:0/2073409969 shutdown_connections 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.852+0000 7fddddd8a640 1 -- 172.21.15.45:0/2073409969 wait complete. 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.852+0000 7fddddd8a640 1 Processor -- start 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 -- start start 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 0x7fddd819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 0x7fddd819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81a3140 con 0x7fddd810d730 2024-03-12T05:24:45.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81a32b0 con 0x7fddd81a2750 2024-03-12T05:24:45.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddddd8a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81a3420 con 0x7fddd81080e0 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40366/0 (socket says 172.21.15.45:40366) 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddd7fff640 1 -- 172.21.15.45:0/338074515 learned_addr learned my addr 172.21.15.45:0/338074515 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddd77fe640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 0x7fddd819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.853+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 0x7fddd819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 -- 172.21.15.45:0/338074515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 msgr2=0x7fddd819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 0x7fddd819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 -- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 msgr2=0x7fddd819e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 0x7fddd819e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 -- 172.21.15.45:0/338074515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8009560 con 0x7fddd81a2750 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd7fff640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fddd000eb10 tx=0x7fddd000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd4ff9640 1 -- 172.21.15.45:0/338074515 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0004b20 con 0x7fddd81a2750 2024-03-12T05:24:45.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc009490 con 0x7fddd81a2750 2024-03-12T05:24:45.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd81aacf0 con 0x7fddd81a2750 2024-03-12T05:24:45.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd4ff9640 1 -- 172.21.15.45:0/338074515 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddd0004c80 con 0x7fddd81a2750 2024-03-12T05:24:45.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.854+0000 7fddd4ff9640 1 -- 172.21.15.45:0/338074515 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0010690 con 0x7fddd81a2750 2024-03-12T05:24:45.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.858+0000 7fddd4ff9640 1 -- 172.21.15.45:0/338074515 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fddd00107f0 con 0x7fddd81a2750 2024-03-12T05:24:45.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.858+0000 7fdde0a37740 1 Processor -- start 2024-03-12T05:24:45.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.858+0000 7fdde0a37740 1 -- start start 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecbb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ec29f90 con 0x55b00ecbb380 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ec274c0 con 0x55b00ecc48b0 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fdde0a37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ec22ca0 con 0x55b00ecbbd30 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd4ff9640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 0x7fddc00a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd4ff9640 1 -- 172.21.15.45:0/338074515 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fddd0014070 con 0x7fddd81a2750 2024-03-12T05:24:45.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41856/0 (socket says 172.21.15.45:41856) 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 -- 172.21.15.45:0/3005559217 learned_addr learned my addr 172.21.15.45:0/3005559217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecbb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 0x7fddc00a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd7fff640 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 -- 172.21.15.45:0/3005559217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 msgr2=0x55b00ecc6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 -- 172.21.15.45:0/3005559217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 msgr2=0x55b00ecbb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecbb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.859+0000 7fddd77fe640 1 -- 172.21.15.45:0/3005559217 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b00ebe50b0 con 0x55b00ecbbd30 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.860+0000 7fddd77fe640 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009de0 tx=0x7fddc8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36f1bd2215a9cd7c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.860+0000 7fddbdffb640 1 -- 172.21.15.45:0/3005559217 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8005220 con 0x55b00ecbbd30 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.860+0000 7fddbdffb640 1 -- 172.21.15.45:0/3005559217 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddc8004c50 con 0x55b00ecbbd30 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.860+0000 7fddbdffb640 1 -- 172.21.15.45:0/3005559217 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc800e830 con 0x55b00ecbbd30 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.861+0000 7fdde0a37740 1 -- 172.21.15.45:0/3005559217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 msgr2=0x55b00ecc42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.861+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009de0 tx=0x7fddc8011810 comp rx=0 tx=0).stop 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.861+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 0x7fddc00a4230 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009830 tx=0x7fddcc005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 -- 172.21.15.45:0/3005559217 shutdown_connections 2024-03-12T05:24:45.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc42e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3005559217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecbb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 -- 172.21.15.45:0/3005559217 >> 172.21.15.45:0/3005559217 conn(0x55b00ecb5840 msgr2=0x55b00ecb5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 -- 172.21.15.45:0/3005559217 shutdown_connections 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 -- 172.21.15.45:0/3005559217 wait complete. 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 -- start start 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.862+0000 7fdde0a37740 1 Processor -- start 2024-03-12T05:24:45.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecc16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ebc7f40 con 0x55b00ecbb380 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00e8f64b0 con 0x55b00ecc48b0 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00eba8670 con 0x55b00ecbbd30 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fdde0a37740 10 client.0 _ll_register_callbacks cb 0x55b00ec15f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fddd77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecc16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40388/0 (socket says 172.21.15.45:40388) 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fddd7fff640 1 -- 172.21.15.45:0/581368550 learned_addr learned my addr 172.21.15.45:0/581368550 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.863+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 -- 172.21.15.45:0/581368550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 msgr2=0x55b00ecc1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 -- 172.21.15.45:0/581368550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 msgr2=0x55b00ecc16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecc16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 -- 172.21.15.45:0/581368550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc8009a10 con 0x55b00ecc48b0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddd7fff640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fddd009b660 tx=0x7fddd009b380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e525ddf69dd3df server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd009a690 con 0x55b00ecc48b0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b00ebfcf40 con 0x55b00ecc48b0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.864+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fddd009b960 con 0x55b00ecc48b0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.865+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddd009fab0 con 0x55b00ecc48b0 2024-03-12T05:24:45.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.865+0000 7fdde0a37740 10 client.28460 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:45.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.865+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b00ec21320 con 0x55b00ecc48b0 2024-03-12T05:24:45.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.865+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddd00aa080 con 0x55b00ecc48b0 2024-03-12T05:24:45.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.865+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fddd001f060 con 0x55b00ecc48b0 2024-03-12T05:24:45.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 10 client.28460 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:45.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 10 client.28460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:45.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 20 client.28460 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:45.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ecd4e10 0x55b00ecd7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 4 client.28460 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.866+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b00ec19310 con 0x55b00ecd4e10 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.867+0000 7fddd77fe640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ecd4e10 0x55b00ecd7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.867+0000 7fddd77fe640 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ecd4e10 0x55b00ecd7230 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56c563f39735de7d server_cookie=c81bf21924fa23fa in_seq=0 out_seq=0 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.867+0000 7fddb6ffd640 10 client.28460 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fddb6ffd640 1 -- 172.21.15.45:0/581368550 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b00ec19310 con 0x55b00ecd4e10 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fddb6ffd640 10 client.28460 handle_command_reply: tid=0 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 client.28460 shutdown 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 msgr2=0x55b00ecc2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fddd009b660 tx=0x7fddd009b380 comp rx=0 tx=0).stop 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 shutdown_connections 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ecd4e10 0x55b00ecd7230 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ecc2150 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 --2- 172.21.15.45:0/581368550 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00ecc16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.868+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 >> 172.21.15.45:0/581368550 conn(0x55b00ecb5840 msgr2=0x55b00ecb77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.869+0000 7fddb6ffd640 0 client.28460 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.869+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 shutdown_connections 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.869+0000 7fdde0a37740 1 -- 172.21.15.45:0/581368550 wait complete. 2024-03-12T05:24:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.869+0000 7fdde0a37740 20 client.28460 trim_cache size 0 max 16384 2024-03-12T05:24:45.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 Processor -- start 2024-03-12T05:24:45.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 -- start start 2024-03-12T05:24:45.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbb380 0x55b00ecb4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 0x55b00eccb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ecc6ca0 con 0x55b00ecc48b0 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ecc71a0 con 0x55b00ecbbd30 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fdde0a37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ecb12c0 con 0x55b00ecbb380 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 0x55b00eccb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fddd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fddd6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40396/0 (socket says 172.21.15.45:40396) 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 0x55b00eccb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60162/0 (socket says 172.21.15.45:60162) 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.903+0000 7fddd6ffd640 1 -- 172.21.15.45:0/2058008265 learned_addr learned my addr 172.21.15.45:0/2058008265 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 -- 172.21.15.45:0/2058008265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbb380 msgr2=0x55b00ecb4760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbb380 0x55b00ecb4760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 -- 172.21.15.45:0/2058008265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 msgr2=0x55b00eccb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 0x55b00eccb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 -- 172.21.15.45:0/2058008265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b00ecbb1f0 con 0x55b00ecbbd30 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.904+0000 7fddd6ffd640 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fddcc02c180 tx=0x7fddcc03b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f9989283392c7de server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.905+0000 7fddbd7fa640 1 -- 172.21.15.45:0/2058008265 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc02dea0 con 0x55b00ecbbd30 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.905+0000 7fddbd7fa640 1 -- 172.21.15.45:0/2058008265 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fddcc005040 con 0x55b00ecbbd30 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.905+0000 7fddbd7fa640 1 -- 172.21.15.45:0/2058008265 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc004920 con 0x55b00ecbbd30 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 -- 172.21.15.45:0/2058008265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 msgr2=0x55b00ecc1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fddcc02c180 tx=0x7fddcc03b000 comp rx=0 tx=0).stop 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 -- 172.21.15.45:0/2058008265 shutdown_connections 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecc48b0 0x55b00eccb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecbbd30 0x55b00ecc1f70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 --2- 172.21.15.45:0/2058008265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbb380 0x55b00ecb4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 -- 172.21.15.45:0/2058008265 >> 172.21.15.45:0/2058008265 conn(0x55b00ecb5840 msgr2=0x55b00ecb4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 -- 172.21.15.45:0/2058008265 shutdown_connections 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.906+0000 7fdde0a37740 1 -- 172.21.15.45:0/2058008265 wait complete. 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 -- start start 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 Processor -- start 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00eccafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00eccb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ed24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ed24970 con 0x55b00ecbb380 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ed24ae0 con 0x55b00ecc48b0 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b00ed24ca0 con 0x55b00ecbbd30 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fdde0a37740 10 client.0 _ll_register_callbacks cb 0x7fddd009b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fddd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00eccb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fddd77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00eccafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fddd6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00eccb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41890/0 (socket says 172.21.15.45:41890) 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fddd6ffd640 1 -- 172.21.15.45:0/3270014794 learned_addr learned my addr 172.21.15.45:0/3270014794 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.907+0000 7fddd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ed24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 -- 172.21.15.45:0/3270014794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 msgr2=0x55b00eccb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00eccb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 -- 172.21.15.45:0/3270014794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 msgr2=0x55b00eccafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00eccafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 -- 172.21.15.45:0/3270014794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd8114ef0 con 0x55b00ecc48b0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddd7fff640 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ed24430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fddd0099c10 tx=0x7fddd00cce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12216b5f579b3c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd009b960 con 0x55b00ecc48b0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fddd009fab0 con 0x55b00ecc48b0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.908+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b00ec48900 con 0x55b00ecc48b0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddd00a4100 con 0x55b00ecc48b0 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 10 client.28466 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:45.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b00ebbdc00 con 0x55b00ecc48b0 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b00ec48900 con 0x55b00ecc48b0 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fddd001f040 con 0x55b00ecc48b0 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 10 client.28466 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 10 client.28466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 20 client.28466 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ed2d340 0x55b00ed2f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 4 client.28466 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b00e922860 con 0x55b00ed2d340 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.909+0000 7fddd77fe640 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ed2d340 0x55b00ed2f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.910+0000 7fddd77fe640 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ed2d340 0x55b00ed2f760 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e615fa33416629e3 server_cookie=51027f3031cfad91 in_seq=0 out_seq=0 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.910+0000 7fddb6ffd640 10 client.28466 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.911+0000 7fddb6ffd640 1 -- 172.21.15.45:0/3270014794 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b00e922860 con 0x55b00ed2d340 2024-03-12T05:24:45.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.911+0000 7fddb6ffd640 10 client.28466 handle_command_reply: tid=0 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 client.28466 shutdown 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 msgr2=0x55b00ed24430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ed24430 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fddd0099c10 tx=0x7fddd00cce00 comp rx=0 tx=0).stop 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 shutdown_connections 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b00ed2d340 0x55b00ed2f760 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b00ecc48b0 0x55b00ed24430 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b00ecbbd30 0x55b00eccb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 --2- 172.21.15.45:0/3270014794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b00ecbb380 0x55b00eccafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 >> 172.21.15.45:0/3270014794 conn(0x55b00ecb5840 msgr2=0x55b00ecb6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.912+0000 7fddb6ffd640 0 client.28466 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.913+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 shutdown_connections 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.913+0000 7fdde0a37740 1 -- 172.21.15.45:0/3270014794 wait complete. 2024-03-12T05:24:45.910 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.913+0000 7fdde0a37740 20 client.28466 trim_cache size 0 max 16384 2024-03-12T05:24:45.911 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:45.911 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:24:45.911 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "3cf0be06-8088-4912-91d2-2285a801b9ed", 2024-03-12T05:24:45.911 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:24:45.911 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.916+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 msgr2=0x7fddc00a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.916+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 0x7fddc00a4230 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009830 tx=0x7fddcc005f90 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.916+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 msgr2=0x7fddd81a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.917+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fddd000eb10 tx=0x7fddd000c2d0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 shutdown_connections 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fddc00a1d70 0x7fddc00a4230 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fddd810d730 0x7fddd819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fddd81a2750 0x7fddd81a2c00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 --2- 172.21.15.45:0/338074515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddd81080e0 0x7fddd819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:45.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.918+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 >> 172.21.15.45:0/338074515 conn(0x7fddd80756c0 msgr2=0x7fddd80760b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:45.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.919+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 shutdown_connections 2024-03-12T05:24:45.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:45.919+0000 7fddddd8a640 1 -- 172.21.15.45:0/338074515 wait complete. 2024-03-12T05:24:45.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:24:45.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:24:45.975 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:45 smithi045 ceph-mon[32174]: pgmap v435: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 14 MiB/s wr, 1.51k op/s 2024-03-12T05:24:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.131+0000 7f5ab6dac640 1 -- 172.21.15.45:0/232300505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab010f8a0 msgr2=0x7f5ab010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.131+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/232300505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab010f8a0 0x7f5ab010d4c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009860 tx=0x7f5aa402bb70 comp rx=0 tx=0).stop 2024-03-12T05:24:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 -- 172.21.15.45:0/232300505 shutdown_connections 2024-03-12T05:24:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/232300505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ab011a960 0x7f5ab011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/232300505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab010fd10 0x7f5ab010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/232300505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab010f8a0 0x7f5ab010d4c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 -- 172.21.15.45:0/232300505 >> 172.21.15.45:0/232300505 conn(0x7f5ab006c270 msgr2=0x7f5ab006c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 -- 172.21.15.45:0/232300505 shutdown_connections 2024-03-12T05:24:46.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.132+0000 7f5ab6dac640 1 -- 172.21.15.45:0/232300505 wait complete. 2024-03-12T05:24:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 Processor -- start 2024-03-12T05:24:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 -- start start 2024-03-12T05:24:46.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ab010f8a0 0x7f5ab0119ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab010fd10 0x7f5ab0115f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0116a00 con 0x7f5ab011a960 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0116b70 con 0x7f5ab010fd10 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.133+0000 7f5ab6dac640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0116d30 con 0x7f5ab010f8a0 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60190/0 (socket says 172.21.15.45:60190) 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 -- 172.21.15.45:0/2943213767 learned_addr learned my addr 172.21.15.45:0/2943213767 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 -- 172.21.15.45:0/2943213767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ab010f8a0 msgr2=0x7f5ab0119ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ab010f8a0 0x7f5ab0119ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 -- 172.21.15.45:0/2943213767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab010fd10 msgr2=0x7f5ab0115f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab010fd10 0x7f5ab0115f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.134+0000 7f5ab5349640 1 -- 172.21.15.45:0/2943213767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4009490 con 0x7f5ab011a960 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.135+0000 7f5ab5349640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5aa801fd60 tx=0x7f5aa801d750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.135+0000 7f5aadffb640 1 -- 172.21.15.45:0/2943213767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa801de60 con 0x7f5ab011a960 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.135+0000 7f5aadffb640 1 -- 172.21.15.45:0/2943213767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5aa8003970 con 0x7f5ab011a960 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.135+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab01aa5c0 con 0x7f5ab011a960 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.136+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab01aa9f0 con 0x7f5ab011a960 2024-03-12T05:24:46.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.136+0000 7f5aadffb640 1 -- 172.21.15.45:0/2943213767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004590 con 0x7f5ab011a960 2024-03-12T05:24:46.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.139+0000 7f5aadffb640 1 -- 172.21.15.45:0/2943213767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5aa8003ae0 con 0x7f5ab011a960 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 Processor -- start 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 -- start start 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b13270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b15ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8a79280 con 0x5611a8b0abd0 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8a767b0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5ab997f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8a71f90 con 0x5611a8b137b0 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40434/0 (socket says 172.21.15.45:40434) 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5aaffff640 1 -- 172.21.15.45:0/4166968814 learned_addr learned my addr 172.21.15.45:0/4166968814 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:46.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.140+0000 7f5aaffff640 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b15ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b13270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aadffb640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 0x7f5a980a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 -- 172.21.15.45:0/4166968814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 msgr2=0x5611a8b15ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b15ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 -- 172.21.15.45:0/4166968814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 msgr2=0x5611a8b13270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b13270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 -- 172.21.15.45:0/4166968814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a8a30570 con 0x5611a8b0a2b0 2024-03-12T05:24:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5ab5349640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 0x7f5a980a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aaffff640 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0013700 tx=0x7f5aa0009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e6c642ee2caf21 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5aadffb640 1 -- 172.21.15.45:0/2943213767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5aa80de770 con 0x7f5ab011a960 2024-03-12T05:24:46.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.141+0000 7f5a96ffd640 1 -- 172.21.15.45:0/4166968814 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0005020 con 0x5611a8b0a2b0 2024-03-12T05:24:46.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.142+0000 7f5a96ffd640 1 -- 172.21.15.45:0/4166968814 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5aa0019070 con 0x5611a8b0a2b0 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.142+0000 7f5a96ffd640 1 -- 172.21.15.45:0/4166968814 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa000c6d0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab5349640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 0x7f5a980a4140 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80047a0 tx=0x7f5aa80cb040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- 172.21.15.45:0/4166968814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 msgr2=0x5611a8b0a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0013700 tx=0x7f5aa0009610 comp rx=0 tx=0).stop 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- 172.21.15.45:0/4166968814 shutdown_connections 2024-03-12T05:24:46.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b13270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b15ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 --2- 172.21.15.45:0/4166968814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b0a690 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- 172.21.15.45:0/4166968814 >> 172.21.15.45:0/4166968814 conn(0x5611a8b04990 msgr2=0x5611a8b04d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- 172.21.15.45:0/4166968814 shutdown_connections 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- 172.21.15.45:0/4166968814 wait complete. 2024-03-12T05:24:46.151 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.143+0000 7f5ab997f740 1 -- start start 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 Processor -- start 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 0x5611a8b10b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b137b0 0x5611a8b110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8745430 con 0x5611a8b137b0 2024-03-12T05:24:46.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8701e00 con 0x5611a8b0a2b0 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a89f6fd0 con 0x5611a8b0abd0 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5ab997f740 10 client.0 _ll_register_callbacks cb 0x5611a8a4cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.144+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40438/0 (socket says 172.21.15.45:40438) 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5aaffff640 1 -- 172.21.15.45:0/26434500 learned_addr learned my addr 172.21.15.45:0/26434500 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:46.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 0x5611a8b10b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5aaffff640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b137b0 0x5611a8b110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 -- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 msgr2=0x5611a8b10620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 -- 172.21.15.45:0/26434500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b137b0 msgr2=0x5611a8b110a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b137b0 0x5611a8b110a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 -- 172.21.15.45:0/26434500 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa0013400 con 0x5611a8b0abd0 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5aaffff640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 0x5611a8b10b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4005b60 tx=0x7f5aa4005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135bdb4b19c51a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.145+0000 7f5a8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4004410 con 0x5611a8b0abd0 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611a87717e0 con 0x5611a8b0abd0 2024-03-12T05:24:46.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5aa4034440 con 0x5611a8b0abd0 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5aa4034600 con 0x5611a8b0abd0 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5ab997f740 10 client.28429 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.146+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5611a8a70610 con 0x5611a8b0abd0 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.147+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5611a87717e0 con 0x5611a8b0abd0 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.147+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5aa402c740 con 0x5611a8b0abd0 2024-03-12T05:24:46.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.147+0000 7f5ab997f740 10 client.28429 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5ab997f740 10 client.28429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5ab997f740 20 client.28429 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b239f0 0x5611a8b25de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5ab997f740 4 client.28429 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611a87ae720 con 0x5611a8b239f0 2024-03-12T05:24:46.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.148+0000 7f5aaffff640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b239f0 0x5611a8b25de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.149+0000 7f5aaffff640 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b239f0 0x5611a8b25de0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7db62d6690a190 server_cookie=428787bc15a3bb2f in_seq=0 out_seq=0 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.149+0000 7f5a8ffff640 10 client.28429 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.150+0000 7f5a8ffff640 1 -- 172.21.15.45:0/26434500 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5611a87ae720 con 0x5611a8b239f0 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.150+0000 7f5a8ffff640 10 client.28429 handle_command_reply: tid=0 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.150+0000 7f5ab997f740 1 client.28429 shutdown 2024-03-12T05:24:46.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 msgr2=0x5611a8b10b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 0x5611a8b10b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4005b60 tx=0x7f5aa4005d90 comp rx=0 tx=0).stop 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 shutdown_connections 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b239f0 0x5611a8b25de0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b137b0 0x5611a8b110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0abd0 0x5611a8b10b60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 --2- 172.21.15.45:0/26434500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.159 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 >> 172.21.15.45:0/26434500 conn(0x5611a8b04990 msgr2=0x5611a8b06730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.152+0000 7f5a8ffff640 0 client.28429 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.153+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 shutdown_connections 2024-03-12T05:24:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.153+0000 7f5ab997f740 1 -- 172.21.15.45:0/26434500 wait complete. 2024-03-12T05:24:46.160 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.153+0000 7f5ab997f740 20 client.28429 trim_cache size 0 max 16384 2024-03-12T05:24:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.182+0000 7f5ab997f740 1 Processor -- start 2024-03-12T05:24:46.181 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.186+0000 7f5ab997f740 1 -- start start 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.187+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.187+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b0b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.187+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b137b0 0x5611a8b0be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.187+0000 7f5ab997f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b000d0 con 0x5611a8b0abd0 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.187+0000 7f5ab997f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b09f10 con 0x5611a8b137b0 2024-03-12T05:24:46.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5ab997f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b059e0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b137b0 0x5611a8b0be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:41932/0 (socket says 172.21.15.45:41932) 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5aaffff640 1 -- 172.21.15.45:0/168858066 learned_addr learned my addr 172.21.15.45:0/168858066 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.188+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b0b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 -- 172.21.15.45:0/168858066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b137b0 msgr2=0x5611a8b0be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b137b0 0x5611a8b0be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 -- 172.21.15.45:0/168858066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 msgr2=0x5611a8b0b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b0b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 -- 172.21.15.45:0/168858066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a8a0cac0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5aaffff640 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0013700 tx=0x7f5aa0009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2107cec88b1304a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.189+0000 7f5a967fc640 1 -- 172.21.15.45:0/168858066 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0004920 con 0x5611a8b0a2b0 2024-03-12T05:24:46.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.190+0000 7f5a967fc640 1 -- 172.21.15.45:0/168858066 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5aa0033020 con 0x5611a8b0a2b0 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.190+0000 7f5a967fc640 1 -- 172.21.15.45:0/168858066 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa002c7c0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.190+0000 7f5ab997f740 1 -- 172.21.15.45:0/168858066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 msgr2=0x5611a8b12d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0013700 tx=0x7f5aa0009610 comp rx=0 tx=0).stop 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 -- 172.21.15.45:0/168858066 shutdown_connections 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b0b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b137b0 0x5611a8b0be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 --2- 172.21.15.45:0/168858066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b0a2b0 0x5611a8b12d50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 -- 172.21.15.45:0/168858066 >> 172.21.15.45:0/168858066 conn(0x5611a8b04990 msgr2=0x5611a8b2a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 -- 172.21.15.45:0/168858066 shutdown_connections 2024-03-12T05:24:46.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 -- 172.21.15.45:0/168858066 wait complete. 2024-03-12T05:24:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.191+0000 7f5ab997f740 1 -- start start 2024-03-12T05:24:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.192+0000 7f5ab997f740 1 Processor -- start 2024-03-12T05:24:46.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b18d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b78eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b793f0 con 0x5611a8b0abd0 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b79530 con 0x5611a8b0a2b0 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a8b79670 con 0x5611a8b137b0 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5ab997f740 10 client.0 _ll_register_callbacks cb 0x7f5aa00098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:40454/0 (socket says 172.21.15.45:40454) 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 -- 172.21.15.45:0/2731700702 learned_addr learned my addr 172.21.15.45:0/2731700702 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:24:46.188 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b78eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 -- 172.21.15.45:0/2731700702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 msgr2=0x5611a8b78eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b78eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 -- 172.21.15.45:0/2731700702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 msgr2=0x5611a8b18d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:24:46.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b18d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.193+0000 7f5aaffff640 1 -- 172.21.15.45:0/2731700702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa0027040 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.194+0000 7f5aaffff640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5aa002c480 tx=0x7f5aa0009990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b69d2d8f396734 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.194+0000 7f5a8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.194+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa000c8b0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.194+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5aa0034030 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.194+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5aa0019070 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.195+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5aa000ca10 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.195+0000 7f5ab997f740 10 client.28481 fetch_fsmap learned FSMap version 33 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.195+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5611a8a326a0 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.195+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5aa0025070 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5aa0043020 con 0x5611a8b0a2b0 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 10 client.28481 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:24:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 10 client.28481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:24:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 20 client.28481 populate_metadata read hostname 'smithi045' 2024-03-12T05:24:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b7ab20 0x5611a8b7cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:24:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 4 client.28481 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:24:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f5aa0019070 con 0x5611a8b7ab20 2024-03-12T05:24:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.196+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b7ab20 0x5611a8b7cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:24:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.197+0000 7f5ab4b48640 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b7ab20 0x5611a8b7cf40 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4283bf5725001ef9 server_cookie=c1ab97d64a4fbd01 in_seq=0 out_seq=0 2024-03-12T05:24:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.197+0000 7f5a8ffff640 10 client.28481 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.197+0000 7f5a8ffff640 1 -- 172.21.15.45:0/2731700702 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f5aa0019070 con 0x5611a8b7ab20 2024-03-12T05:24:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.197+0000 7f5a8ffff640 10 client.28481 handle_command_reply: tid=0 2024-03-12T05:24:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.197+0000 7f5ab997f740 1 client.28481 shutdown 2024-03-12T05:24:46.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 msgr2=0x5611a8b187e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5aa002c480 tx=0x7f5aa0009990 comp rx=0 tx=0).stop 2024-03-12T05:24:46.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 shutdown_connections 2024-03-12T05:24:46.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5611a8b7ab20 0x5611a8b7cf40 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5611a8b0abd0 0x5611a8b18d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5611a8b137b0 0x5611a8b78eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.199+0000 7f5ab997f740 1 --2- 172.21.15.45:0/2731700702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611a8b0a2b0 0x5611a8b187e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.200+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 >> 172.21.15.45:0/2731700702 conn(0x5611a8b04990 msgr2=0x5611a8b0b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.201+0000 7f5a8ffff640 0 client.28481 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:24:46.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.202+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 shutdown_connections 2024-03-12T05:24:46.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.202+0000 7f5ab997f740 1 -- 172.21.15.45:0/2731700702 wait complete. 2024-03-12T05:24:46.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.202+0000 7f5ab997f740 20 client.28481 trim_cache size 0 max 16384 2024-03-12T05:24:46.197 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:24:46.197 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:24:46.197 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:24:46.197 INFO:teuthology.orchestra.run.smithi045.stdout: "3cf0be06-8088-4912-91d2-2285a801b9ed": { 2024-03-12T05:24:46.197 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:24:46.198 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "3cf0be06-8088-4912-91d2-2285a801b9ed", 2024-03-12T05:24:46.198 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:24:46.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:24:46.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:24:46.198 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 msgr2=0x7f5a980a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 0x7f5a980a4140 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80047a0 tx=0x7f5aa80cb040 comp rx=0 tx=0).stop 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 msgr2=0x7f5ab01164c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5aa801fd60 tx=0x7f5aa801d750 comp rx=0 tx=0).stop 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 shutdown_connections 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f5a980a1c80 0x7f5a980a4140 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ab011a960 0x7f5ab01164c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ab010fd10 0x7f5ab0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 --2- 172.21.15.45:0/2943213767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ab010f8a0 0x7f5ab0119ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:24:46.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.204+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 >> 172.21.15.45:0/2943213767 conn(0x7f5ab006c270 msgr2=0x7f5ab010c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:24:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.206+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 shutdown_connections 2024-03-12T05:24:46.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:24:46.206+0000 7f5ab6dac640 1 -- 172.21.15.45:0/2943213767 wait complete. 2024-03-12T05:24:46.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:45 smithi130 ceph-mon[35398]: pgmap v435: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 14 MiB/s wr, 1.51k op/s 2024-03-12T05:24:46.217 INFO:tasks.cephfs.filesystem:scrub status for tag:3cf0be06-8088-4912-91d2-2285a801b9ed - {'path': '/', 'tag': '3cf0be06-8088-4912-91d2-2285a801b9ed', 'options': 'recursive,force'} 2024-03-12T05:24:46.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:45 smithi116 ceph-mon[35486]: pgmap v435: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 14 MiB/s wr, 1.51k op/s 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:46 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:24:47.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:47.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:47.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:47.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:24:47.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:46 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:24:47.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:47.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:46 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:24:48.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:47 smithi045 ceph-mon[32174]: pgmap v436: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 9.4 MiB/s wr, 997 op/s 2024-03-12T05:24:48.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:47 smithi130 ceph-mon[35398]: pgmap v436: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 9.4 MiB/s wr, 997 op/s 2024-03-12T05:24:48.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:47 smithi116 ceph-mon[35486]: pgmap v436: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 9.4 MiB/s wr, 997 op/s 2024-03-12T05:24:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 765.0 s, 1320.8 tps, lat 75.671 ms stddev 25.261 2024-03-12T05:24:50.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:50 smithi045 ceph-mon[32174]: pgmap v437: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-03-12T05:24:50.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:50 smithi130 ceph-mon[35398]: pgmap v437: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-03-12T05:24:50.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:50 smithi116 ceph-mon[35486]: pgmap v437: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-03-12T05:24:52.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:52 smithi045 ceph-mon[32174]: pgmap v438: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:24:52.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:52 smithi130 ceph-mon[35398]: pgmap v438: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:24:52.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:52 smithi116 ceph-mon[35486]: pgmap v438: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:24:54.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:54 smithi045 ceph-mon[32174]: pgmap v439: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.7 MiB/s wr, 1.02k op/s 2024-03-12T05:24:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 770.0 s, 1328.2 tps, lat 75.130 ms stddev 42.577 2024-03-12T05:24:54.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:54 smithi130 ceph-mon[35398]: pgmap v439: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.7 MiB/s wr, 1.02k op/s 2024-03-12T05:24:54.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:54 smithi116 ceph-mon[35486]: pgmap v439: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.7 MiB/s wr, 1.02k op/s 2024-03-12T05:24:55.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:55 smithi045 ceph-mon[32174]: pgmap v440: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:24:55.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:55 smithi130 ceph-mon[35398]: pgmap v440: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:24:55.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:55 smithi116 ceph-mon[35486]: pgmap v440: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:24:58.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:57 smithi045 ceph-mon[32174]: pgmap v441: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 9.1 MiB/s wr, 1.06k op/s 2024-03-12T05:24:58.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:57 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:58.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:57 smithi130 ceph-mon[35398]: pgmap v441: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 9.1 MiB/s wr, 1.06k op/s 2024-03-12T05:24:58.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:57 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:58.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:57 smithi116 ceph-mon[35486]: pgmap v441: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 9.1 MiB/s wr, 1.06k op/s 2024-03-12T05:24:58.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:57 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:59.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:58 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:59.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:58 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:59.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:58 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:24:59.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:58 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:59.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:58 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:59.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:58 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:24:59.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:58 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:59.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:58 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:59.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:58 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:24:59.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 775.0 s, 1356.6 tps, lat 73.710 ms stddev 44.661 2024-03-12T05:25:00.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:24:59 smithi045 ceph-mon[32174]: pgmap v442: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 1.22k op/s 2024-03-12T05:25:00.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:24:59 smithi130 ceph-mon[35398]: pgmap v442: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 1.22k op/s 2024-03-12T05:25:00.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:24:59 smithi116 ceph-mon[35486]: pgmap v442: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 10 MiB/s wr, 1.22k op/s 2024-03-12T05:25:02.018 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:01 smithi130 ceph-mon[35398]: pgmap v443: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 13 MiB/s wr, 1.45k op/s 2024-03-12T05:25:02.042 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:01 smithi116 ceph-mon[35486]: pgmap v443: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 13 MiB/s wr, 1.45k op/s 2024-03-12T05:25:02.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:01 smithi045 ceph-mon[32174]: pgmap v443: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 13 MiB/s wr, 1.45k op/s 2024-03-12T05:25:04.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:03 smithi045 ceph-mon[32174]: pgmap v444: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:04.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:03 smithi130 ceph-mon[35398]: pgmap v444: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:04.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:03 smithi116 ceph-mon[35486]: pgmap v444: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 780.0 s, 1307.2 tps, lat 76.617 ms stddev 32.304 2024-03-12T05:25:06.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:05 smithi045 ceph-mon[32174]: pgmap v445: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:25:06.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:05 smithi130 ceph-mon[35398]: pgmap v445: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:25:06.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:05 smithi116 ceph-mon[35486]: pgmap v445: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 14 MiB/s wr, 1.62k op/s 2024-03-12T05:25:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:07 smithi045 ceph-mon[32174]: pgmap v446: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 9.2 MiB/s wr, 1.07k op/s 2024-03-12T05:25:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:07 smithi130 ceph-mon[35398]: pgmap v446: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 9.2 MiB/s wr, 1.07k op/s 2024-03-12T05:25:08.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:07 smithi116 ceph-mon[35486]: pgmap v446: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 9.2 MiB/s wr, 1.07k op/s 2024-03-12T05:25:09.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:08 smithi045 ceph-mon[32174]: osdmap e105: 12 total, 12 up, 12 in 2024-03-12T05:25:09.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:08 smithi130 ceph-mon[35398]: osdmap e105: 12 total, 12 up, 12 in 2024-03-12T05:25:09.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:08 smithi116 ceph-mon[35486]: osdmap e105: 12 total, 12 up, 12 in 2024-03-12T05:25:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 785.0 s, 1213.5 tps, lat 82.208 ms stddev 41.718 2024-03-12T05:25:10.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:09 smithi045 ceph-mon[32174]: pgmap v448: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:25:10.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:09 smithi130 ceph-mon[35398]: pgmap v448: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:25:10.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:09 smithi116 ceph-mon[35486]: pgmap v448: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-03-12T05:25:12.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:11 smithi045 ceph-mon[32174]: pgmap v449: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:12.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:11 smithi130 ceph-mon[35398]: pgmap v449: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:12.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:11 smithi116 ceph-mon[35486]: pgmap v449: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:14.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:13 smithi045 ceph-mon[32174]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:14.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:13 smithi130 ceph-mon[35398]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:14.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:13 smithi116 ceph-mon[35486]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-12T05:25:14.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 790.0 s, 1286.9 tps, lat 77.912 ms stddev 31.689 2024-03-12T05:25:15.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:15 smithi045 ceph-mon[32174]: osdmap e106: 12 total, 12 up, 12 in 2024-03-12T05:25:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:15 smithi130 ceph-mon[35398]: osdmap e106: 12 total, 12 up, 12 in 2024-03-12T05:25:15.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:15 smithi116 ceph-mon[35486]: osdmap e106: 12 total, 12 up, 12 in 2024-03-12T05:25:16.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:25:16.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:25:16.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 -- 172.21.15.45:0/2333766039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d41078a0 msgr2=0x7f30d4107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 --2- 172.21.15.45:0/2333766039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d41078a0 0x7f30d4107d00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009930 tx=0x7f30bc02bcb0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 -- 172.21.15.45:0/2333766039 shutdown_connections 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 --2- 172.21.15.45:0/2333766039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d4108240 0x7f30d4114610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 --2- 172.21.15.45:0/2333766039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d41078a0 0x7f30d4107d00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 --2- 172.21.15.45:0/2333766039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 -- 172.21.15.45:0/2333766039 >> 172.21.15.45:0/2333766039 conn(0x7f30d4076ff0 msgr2=0x7f30d4105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 -- 172.21.15.45:0/2333766039 shutdown_connections 2024-03-12T05:25:16.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.406+0000 7f30da89a640 1 -- 172.21.15.45:0/2333766039 wait complete. 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.407+0000 7f30da89a640 1 Processor -- start 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.407+0000 7f30da89a640 1 -- start start 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.407+0000 7f30da89a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.407+0000 7f30da89a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 0x7f30d419e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42176/0 (socket says 172.21.15.45:42176) 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30d3fff640 1 -- 172.21.15.45:0/628760778 learned_addr learned my addr 172.21.15.45:0/628760778 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d41a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d41a2c10 con 0x7f30d41078a0 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d41a2d80 con 0x7f30d410d7c0 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30d37fe640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 0x7f30d419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30d8e37640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d41a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.408+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d41a2f40 con 0x7f30d4108240 2024-03-12T05:25:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 -- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 msgr2=0x7f30d41a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d41a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 -- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 msgr2=0x7f30d419dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc009560 con 0x7f30d4108240 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d3fff640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d8e37640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d41a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.409+0000 7f30d37fe640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 0x7f30d419e290 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009230 tx=0x7f30bc0040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.410+0000 7f30d17fa640 1 -- 172.21.15.45:0/628760778 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0044e0 con 0x7f30d4108240 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.410+0000 7f30d17fa640 1 -- 172.21.15.45:0/628760778 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f30bc003aa0 con 0x7f30d4108240 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.410+0000 7f30d17fa640 1 -- 172.21.15.45:0/628760778 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0055b0 con 0x7f30d4108240 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.410+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d41a31c0 con 0x7f30d4108240 2024-03-12T05:25:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.410+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d41a3470 con 0x7f30d4108240 2024-03-12T05:25:16.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.413+0000 7f30d17fa640 1 -- 172.21.15.45:0/628760778 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f30bc003c10 con 0x7f30d4108240 2024-03-12T05:25:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d17fa640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 0x7f30c00a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 Processor -- start 2024-03-12T05:25:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 -- start start 2024-03-12T05:25:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d17fa640 1 -- 172.21.15.45:0/628760778 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f30bc0efe90 con 0x7f30d4108240 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9a6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d3fff640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 0x7f30c00a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca915280 con 0x5582ca9a6bd0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9127b0 con 0x5582ca9a62b0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30dd46c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca90df90 con 0x5582ca9af7b0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42182/0 (socket says 172.21.15.45:42182) 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d37fe640 1 -- 172.21.15.45:0/362114026 learned_addr learned my addr 172.21.15.45:0/362114026 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.414+0000 7f30d8e37640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d3fff640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9a6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 -- 172.21.15.45:0/362114026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 msgr2=0x5582ca9b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 -- 172.21.15.45:0/362114026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 msgr2=0x5582ca9a6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9a6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 -- 172.21.15.45:0/362114026 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582ca8cc570 con 0x5582ca9a6bd0 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d8e37640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d3fff640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 0x7f30c00a41b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005c40 tx=0x7f30c4007620 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30d37fe640 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0b9000 tx=0x7f30bc0bc5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad35e4792580b4d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.415+0000 7f30ca7fc640 1 -- 172.21.15.45:0/362114026 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0c0cf0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.416+0000 7f30ca7fc640 1 -- 172.21.15.45:0/362114026 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f30bc0c5340 con 0x5582ca9a6bd0 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.416+0000 7f30ca7fc640 1 -- 172.21.15.45:0/362114026 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0c4eb0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- 172.21.15.45:0/362114026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 msgr2=0x5582ca9af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0b9000 tx=0x7f30bc0bc5c0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- 172.21.15.45:0/362114026 shutdown_connections 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a6bd0 0x5582ca9af270 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 --2- 172.21.15.45:0/362114026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9a6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- 172.21.15.45:0/362114026 >> 172.21.15.45:0/362114026 conn(0x5582ca9a0990 msgr2=0x5582ca9a0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- 172.21.15.45:0/362114026 shutdown_connections 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- 172.21.15.45:0/362114026 wait complete. 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.417+0000 7f30dd46c740 1 -- start start 2024-03-12T05:25:16.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 Processor -- start 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9ac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca5e1430 con 0x5582ca9a62b0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca59de00 con 0x5582ca9a6bd0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca892fd0 con 0x5582ca9af7b0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30dd46c740 10 client.0 _ll_register_callbacks cb 0x5582ca8e8fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9ac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42828/0 (socket says 172.21.15.45:42828) 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.418+0000 7f30d37fe640 1 -- 172.21.15.45:0/1093163731 learned_addr learned my addr 172.21.15.45:0/1093163731 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d3fff640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 -- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 msgr2=0x5582ca9ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 -- 172.21.15.45:0/1093163731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 msgr2=0x5582ca9af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 -- 172.21.15.45:0/1093163731 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d41a3470 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.419+0000 7f30d8e37640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9accc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f30cc005fd0 tx=0x7f30cc010b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1ef58bf499c1781 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc00eda0 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5582ca60d7e0 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f30cc005020 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30cc0051e0 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30dd46c740 10 client.28447 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.420+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5582ca90c610 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f30cc024020 con 0x5582ca9af7b0 2024-03-12T05:25:16.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f30cc018550 con 0x5582ca9af7b0 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 10 client.28447 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 10 client.28447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 20 client.28447 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582ca9bf7c0 0x5582ca9c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 4 client.28447 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582ca60d7e0 con 0x5582ca9bf7c0 2024-03-12T05:25:16.419 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.421+0000 7f30d3fff640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582ca9bf7c0 0x5582ca9c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.422+0000 7f30d3fff640 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582ca9bf7c0 0x5582ca9c1c00 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87184cb87c4aecf3 server_cookie=f45e775afa65de1a in_seq=0 out_seq=0 2024-03-12T05:25:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.422+0000 7f30b37fe640 10 client.28447 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.423+0000 7f30b37fe640 1 -- 172.21.15.45:0/1093163731 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5582ca60d7e0 con 0x5582ca9bf7c0 2024-03-12T05:25:16.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.423+0000 7f30b37fe640 10 client.28447 handle_command_reply: tid=0 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.423+0000 7f30dd46c740 1 client.28447 shutdown 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.423+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 msgr2=0x5582ca9accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9accc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f30cc005fd0 tx=0x7f30cc010b40 comp rx=0 tx=0).stop 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 shutdown_connections 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582ca9bf7c0 0x5582ca9c1c00 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9accc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 --2- 172.21.15.45:0/1093163731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 >> 172.21.15.45:0/1093163731 conn(0x5582ca9a0990 msgr2=0x5582ca9a2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30b37fe640 0 client.28447 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 shutdown_connections 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 1 -- 172.21.15.45:0/1093163731 wait complete. 2024-03-12T05:25:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.424+0000 7f30dd46c740 20 client.28447 trim_cache size 0 max 16384 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.453+0000 7f30dd46c740 1 Processor -- start 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 -- start start 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9a7440 con 0x5582ca9a62b0 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9a81a0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30dd46c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9bf570 con 0x5582ca9af7b0 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42210/0 (socket says 172.21.15.45:42210) 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30d3fff640 1 -- 172.21.15.45:0/2878086584 learned_addr learned my addr 172.21.15.45:0/2878086584 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30d37fe640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.454+0000 7f30d8e37640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 -- 172.21.15.45:0/2878086584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 msgr2=0x5582ca9aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 -- 172.21.15.45:0/2878086584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 msgr2=0x5582ca9b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 -- 172.21.15.45:0/2878086584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582ca8a8ac0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d8e37640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d3fff640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30d37fe640 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b6960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0050a0 tx=0x7f30bc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=505e097d5a0b4fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.455+0000 7f30c9ffb640 1 -- 172.21.15.45:0/2878086584 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0c0cf0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.452 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.456+0000 7f30c9ffb640 1 -- 172.21.15.45:0/2878086584 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f30bc004dc0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.456+0000 7f30c9ffb640 1 -- 172.21.15.45:0/2878086584 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0c4eb0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 -- 172.21.15.45:0/2878086584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 msgr2=0x5582ca9b6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b6960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0050a0 tx=0x7f30bc009260 comp rx=0 tx=0).stop 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 -- 172.21.15.45:0/2878086584 shutdown_connections 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9af7b0 0x5582ca9aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b6960 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 --2- 172.21.15.45:0/2878086584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9a62b0 0x5582ca9b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 -- 172.21.15.45:0/2878086584 >> 172.21.15.45:0/2878086584 conn(0x5582ca9a0990 msgr2=0x5582ca9c6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 -- 172.21.15.45:0/2878086584 shutdown_connections 2024-03-12T05:25:16.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.457+0000 7f30dd46c740 1 -- 172.21.15.45:0/2878086584 wait complete. 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.458+0000 7f30dd46c740 1 -- start start 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.458+0000 7f30dd46c740 1 Processor -- start 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.459+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.459+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.459+0000 7f30dd46c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9af7b0 0x5582ca9b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9b65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42832/0 (socket says 172.21.15.45:42832) 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57006/0 (socket says 172.21.15.45:57006) 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30d3fff640 1 -- 172.21.15.45:0/733809333 learned_addr learned my addr 172.21.15.45:0/733809333 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9b5400 con 0x5582ca9af7b0 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9b5570 con 0x5582ca9a62b0 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582ca9b5730 con 0x5582ca9a6bd0 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.460+0000 7f30dd46c740 10 client.0 _ll_register_callbacks cb 0x7f30bc005710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.461+0000 7f30d37fe640 1 -- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 msgr2=0x5582ca9b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.461+0000 7f30d37fe640 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.461+0000 7f30d37fe640 1 -- 172.21.15.45:0/733809333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9af7b0 msgr2=0x5582ca9b4ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30d37fe640 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9af7b0 0x5582ca9b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30d37fe640 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c400d780 con 0x5582ca9a6bd0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30d37fe640 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x5582ca9c20c0 tx=0x7f30bc0bca80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a23de716934caf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0c0cf0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f30bc0c50c0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.462+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30bc0c5340 con 0x5582ca9a6bd0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30bc0035f0 con 0x5582ca9a6bd0 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 10 client.28453 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:16.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5582ca90c610 con 0x5582ca9a6bd0 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f30bc0c5340 con 0x5582ca9a6bd0 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f30bc03c070 con 0x5582ca9a6bd0 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 10 client.28453 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 10 client.28453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 20 client.28453 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582caa16cf0 0x5582caa190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 4 client.28453 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:25:16.459 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.463+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5582ca64a720 con 0x5582caa16cf0 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.464+0000 7f30d3fff640 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582caa16cf0 0x5582caa190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.464+0000 7f30b37fe640 10 client.28453 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30d3fff640 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582caa16cf0 0x5582caa190e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b65097aff0edb113 server_cookie=dae9095b8729f7fc in_seq=0 out_seq=0 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30b37fe640 1 -- 172.21.15.45:0/733809333 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5582ca64a720 con 0x5582caa16cf0 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30b37fe640 10 client.28453 handle_command_reply: tid=0 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 client.28453 shutdown 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 msgr2=0x5582ca9b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x5582ca9c20c0 tx=0x7f30bc0bca80 comp rx=0 tx=0).stop 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 shutdown_connections 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5582caa16cf0 0x5582caa190e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5582ca9af7b0 0x5582ca9b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.465+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5582ca9a6bd0 0x5582ca9b4980 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30dd46c740 1 --2- 172.21.15.45:0/733809333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5582ca9a62b0 0x5582ca9b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 >> 172.21.15.45:0/733809333 conn(0x5582ca9a0990 msgr2=0x5582ca9abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30b37fe640 0 client.28453 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 shutdown_connections 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30dd46c740 1 -- 172.21.15.45:0/733809333 wait complete. 2024-03-12T05:25:16.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.466+0000 7f30dd46c740 20 client.28453 trim_cache size 0 max 16384 2024-03-12T05:25:16.463 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:16.464 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:25:16.464 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:25:16.464 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 msgr2=0x7f30c00a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 0x7f30c00a41b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005c40 tx=0x7f30c4007620 comp rx=0 tx=0).stop 2024-03-12T05:25:16.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 msgr2=0x7f30d419e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 0x7f30d419e290 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009230 tx=0x7f30bc0040e0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 shutdown_connections 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f30c00a1cf0 0x7f30c00a41b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d410d7c0 0x7f30d41a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30d4108240 0x7f30d419e290 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 --2- 172.21.15.45:0/628760778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30d41078a0 0x7f30d419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.467+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 >> 172.21.15.45:0/628760778 conn(0x7f30d4076ff0 msgr2=0x7f30d4077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.468+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 shutdown_connections 2024-03-12T05:25:16.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.468+0000 7f30da89a640 1 -- 172.21.15.45:0/628760778 wait complete. 2024-03-12T05:25:16.482 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:25:16.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:25:16.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.658+0000 7f63a7c39640 1 -- 172.21.15.45:0/2281089749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0075020 msgr2=0x7f63a00733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.659+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2281089749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0075020 0x7f63a00733b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f639c009900 tx=0x7f639c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:25:16.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 -- 172.21.15.45:0/2281089749 shutdown_connections 2024-03-12T05:25:16.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2281089749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a0112050 0x7f63a0114440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2281089749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a00738f0 0x7f63a0073d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2281089749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0075020 0x7f63a00733b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 -- 172.21.15.45:0/2281089749 >> 172.21.15.45:0/2281089749 conn(0x7f63a0079170 msgr2=0x7f63a0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:16.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 -- 172.21.15.45:0/2281089749 shutdown_connections 2024-03-12T05:25:16.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.661+0000 7f63a7c39640 1 -- 172.21.15.45:0/2281089749 wait complete. 2024-03-12T05:25:16.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.662+0000 7f63a7c39640 1 Processor -- start 2024-03-12T05:25:16.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a7c39640 1 -- start start 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a7c39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 0x7f63a01a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a7c39640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a7c39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0112050 0x7f63a019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a51d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.663+0000 7f63a51d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57030/0 (socket says 172.21.15.45:57030) 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a51d4640 1 -- 172.21.15.45:0/2657987394 learned_addr learned my addr 172.21.15.45:0/2657987394 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a59d5640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 0x7f63a01a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a61d6640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0112050 0x7f63a019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a019e2b0 con 0x7f63a00738f0 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a019e420 con 0x7f63a0112050 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.664+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a019e5e0 con 0x7f63a0075020 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 -- 172.21.15.45:0/2657987394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0112050 msgr2=0x7f63a019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0112050 0x7f63a019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 -- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 msgr2=0x7f63a01a3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 0x7f63a01a3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009560 con 0x7f63a0075020 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a59d5640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 0x7f63a01a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a51d4640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f639000e9f0 tx=0x7f639000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390009720 con 0x7f63a0075020 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f639000cd80 con 0x7f63a0075020 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390010430 con 0x7f63a0075020 2024-03-12T05:25:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.665+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a019e8c0 con 0x7f63a0075020 2024-03-12T05:25:16.661 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.666+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a01aa9e0 con 0x7f63a0075020 2024-03-12T05:25:16.662 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.667+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a00748c0 con 0x7f63a0075020 2024-03-12T05:25:16.663 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.669+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6390004fa0 con 0x7f63a0075020 2024-03-12T05:25:16.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.670+0000 7f6396ffd640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 0x7f638c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:16.664 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.670+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6390014070 con 0x7f63a0075020 2024-03-12T05:25:16.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.671+0000 7f63a59d5640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 0x7f638c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:16.666 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.672+0000 7f63a59d5640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 0x7f638c0a4160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f639c005ab0 tx=0x7f639c000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:16.669 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:16.675+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f63900969c0 con 0x7f63a0075020 2024-03-12T05:25:16.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:16 smithi045 ceph-mon[32174]: pgmap v452: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:25:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:16 smithi130 ceph-mon[35398]: pgmap v452: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:25:16.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:16 smithi116 ceph-mon[35486]: pgmap v452: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-03-12T05:25:17.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.097+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a01177e0 con 0x7f63a0075020 2024-03-12T05:25:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.098+0000 7f6396ffd640 1 -- 172.21.15.45:0/2657987394 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f639009a080 con 0x7f63a0075020 2024-03-12T05:25:17.092 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:25:17.093 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:25:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:25:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.102+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 msgr2=0x7f638c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.102+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 0x7f638c0a4160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f639c005ab0 tx=0x7f639c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.102+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 msgr2=0x7f63a01a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.102+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f639000e9f0 tx=0x7f639000edf0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 shutdown_connections 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f638c0a1ca0 0x7f638c0a4160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63a0112050 0x7f63a019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63a0075020 0x7f63a01a3fa0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 --2- 172.21.15.45:0/2657987394 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63a00738f0 0x7f63a01a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 >> 172.21.15.45:0/2657987394 conn(0x7f63a0079170 msgr2=0x7f63a0110e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 shutdown_connections 2024-03-12T05:25:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.103+0000 7f63a7c39640 1 -- 172.21.15.45:0/2657987394 wait complete. 2024-03-12T05:25:17.115 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:17.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:25:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.292+0000 7fae61ffb640 1 -- 172.21.15.45:0/725713848 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c033730 con 0x7fae6410f6a0 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 -- 172.21.15.45:0/725713848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410f6a0 msgr2=0x7fae6410fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 --2- 172.21.15.45:0/725713848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410f6a0 0x7fae6410fa80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009be0 tx=0x7fae5c02bef0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 -- 172.21.15.45:0/725713848 shutdown_connections 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 --2- 172.21.15.45:0/725713848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 0x7fae6411c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 --2- 172.21.15.45:0/725713848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410d260 0x7fae6410d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 --2- 172.21.15.45:0/725713848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410f6a0 0x7fae6410fa80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 -- 172.21.15.45:0/725713848 >> 172.21.15.45:0/725713848 conn(0x7fae6406c030 msgr2=0x7fae6406c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 -- 172.21.15.45:0/725713848 shutdown_connections 2024-03-12T05:25:17.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.293+0000 7fae63fff640 1 -- 172.21.15.45:0/725713848 wait complete. 2024-03-12T05:25:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.294+0000 7fae63fff640 1 Processor -- start 2024-03-12T05:25:17.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.294+0000 7fae63fff640 1 -- start start 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 0x7fae641a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 0x7fae641a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 0x7fae641aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641ab190 con 0x7fae6410d260 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641ab300 con 0x7fae6410dce0 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae63fff640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641ab4c0 con 0x7fae6410f6a0 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 0x7fae641a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae637fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 0x7fae641aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 0x7fae641a62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42238/0 (socket says 172.21.15.45:42238) 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae62ffd640 1 -- 172.21.15.45:0/3218197152 learned_addr learned my addr 172.21.15.45:0/3218197152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.295+0000 7fae627fc640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 0x7fae641a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 -- 172.21.15.45:0/3218197152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 msgr2=0x7fae641a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 0x7fae641a6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 -- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 msgr2=0x7fae641a62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 0x7fae641a62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 -- 172.21.15.45:0/3218197152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c009840 con 0x7fae6410f6a0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae637fe640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 0x7fae641aab90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fae5400d750 tx=0x7fae5400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae4bfff640 1 -- 172.21.15.45:0/3218197152 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54009a30 con 0x7fae6410f6a0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae641ab7a0 con 0x7fae6410f6a0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae4bfff640 1 -- 172.21.15.45:0/3218197152 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae540044d0 con 0x7fae6410f6a0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.296+0000 7fae4bfff640 1 -- 172.21.15.45:0/3218197152 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54003d00 con 0x7fae6410f6a0 2024-03-12T05:25:17.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.297+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641b3020 con 0x7fae6410f6a0 2024-03-12T05:25:17.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.301+0000 7fae4bfff640 1 -- 172.21.15.45:0/3218197152 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fae54009b90 con 0x7fae6410f6a0 2024-03-12T05:25:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.302+0000 7fae6c009740 1 Processor -- start 2024-03-12T05:25:17.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.303+0000 7fae4bfff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 0x7fae4c0a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.303+0000 7fae4bfff640 1 -- 172.21.15.45:0/3218197152 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae540cd0d0 con 0x7fae6410f6a0 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.303+0000 7fae6c009740 1 -- start start 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae627fc640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 0x7fae4c0a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42888/0 (socket says 172.21.15.45:42888) 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae62ffd640 1 -- 172.21.15.45:0/3359689377 learned_addr learned my addr 172.21.15.45:0/3359689377 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557410080960 con 0x5574100c5d80 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557410034310 con 0x5574100c5460 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae627fc640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557410031840 con 0x5574100ce960 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae627fc640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 0x7fae4c0a41a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fae64072220 tx=0x7fae5800a250 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.304+0000 7fae637fe640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 -- 172.21.15.45:0/3359689377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 msgr2=0x5574100c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 -- 172.21.15.45:0/3359689377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 msgr2=0x5574100ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 -- 172.21.15.45:0/3359689377 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55740ffef060 con 0x5574100ce960 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae62ffd640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae637fe640 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d0d50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae540ca790 tx=0x7fae54099d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120b0390da05e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae497fa640 1 -- 172.21.15.45:0/3359689377 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54098020 con 0x5574100ce960 2024-03-12T05:25:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae497fa640 1 -- 172.21.15.45:0/3359689377 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae54096390 con 0x5574100ce960 2024-03-12T05:25:17.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.305+0000 7fae497fa640 1 -- 172.21.15.45:0/3359689377 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54095d60 con 0x5574100ce960 2024-03-12T05:25:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 msgr2=0x5574100d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d0d50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae540ca790 tx=0x7fae54099d20 comp rx=0 tx=0).stop 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 shutdown_connections 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d0d50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 --2- 172.21.15.45:0/3359689377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.307+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 >> 172.21.15.45:0/3359689377 conn(0x5574100bfb90 msgr2=0x5574100bff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.308+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 shutdown_connections 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.308+0000 7fae6c009740 1 -- 172.21.15.45:0/3359689377 wait complete. 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.308+0000 7fae6c009740 1 -- start start 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.309+0000 7fae6c009740 1 Processor -- start 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.309+0000 7fae6c009740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100cba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.309+0000 7fae6c009740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.309+0000 7fae6c009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae627fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae627fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42246/0 (socket says 172.21.15.45:42246) 2024-03-12T05:25:17.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741002d020 con 0x5574100c5d80 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae627fc640 1 -- 172.21.15.45:0/1347035399 learned_addr learned my addr 172.21.15.45:0/1347035399 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55740fd00430 con 0x5574100c5460 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55740fcbce00 con 0x5574100ce960 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae6c009740 10 client.0 _ll_register_callbacks cb 0x55740ffd4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 -- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 msgr2=0x5574100cc500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100cc500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 -- 172.21.15.45:0/1347035399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 msgr2=0x5574100cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.310+0000 7fae62ffd640 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641b3020 con 0x5574100c5460 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae62ffd640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100cba80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0096f0 tx=0x7fae5c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f188ffcaec98eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c003970 con 0x5574100c5460 2024-03-12T05:25:17.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55740fd2c7e0 con 0x5574100c5460 2024-03-12T05:25:17.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae5c003ad0 con 0x5574100c5460 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.311+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae5c003cd0 con 0x5574100c5460 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 10 client.28508 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55741002b6a0 con 0x5574100c5460 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55740fd2c7e0 con 0x5574100c5460 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae5c03c040 con 0x5574100c5460 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 10 client.28508 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 10 client.28508 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 20 client.28508 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5574100dee70 0x5574100e1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 4 client.28508 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:17.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.312+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55740fd69720 con 0x5574100dee70 2024-03-12T05:25:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.313+0000 7fae627fc640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5574100dee70 0x5574100e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.314+0000 7fae627fc640 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5574100dee70 0x5574100e1330 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=585f278f3cc13f3a server_cookie=cd93c1318895902b in_seq=0 out_seq=0 2024-03-12T05:25:17.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.314+0000 7fae46ff5640 10 client.28508 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.314+0000 7fae46ff5640 1 -- 172.21.15.45:0/1347035399 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55740fd69720 con 0x5574100dee70 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.314+0000 7fae46ff5640 10 client.28508 handle_command_reply: tid=0 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.314+0000 7fae6c009740 1 client.28508 shutdown 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 msgr2=0x5574100cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100cba80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0096f0 tx=0x7fae5c033c20 comp rx=0 tx=0).stop 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 shutdown_connections 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5574100dee70 0x5574100e1330 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 --2- 172.21.15.45:0/1347035399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100cba80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 >> 172.21.15.45:0/1347035399 conn(0x5574100bfb90 msgr2=0x5574100c18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae46ff5640 0 client.28508 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 shutdown_connections 2024-03-12T05:25:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 1 -- 172.21.15.45:0/1347035399 wait complete. 2024-03-12T05:25:17.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.315+0000 7fae6c009740 20 client.28508 trim_cache size 0 max 16384 2024-03-12T05:25:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.338+0000 7fae6c009740 1 Processor -- start 2024-03-12T05:25:17.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.338+0000 7fae6c009740 1 -- start start 2024-03-12T05:25:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae6c009740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae6c009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae6c009740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 0x5574100ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae627fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae627fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:57070/0 (socket says 172.21.15.45:57070) 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 0x5574100ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae637fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 0x5574100ca890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42262/0 (socket says 172.21.15.45:42262) 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.339+0000 7fae627fc640 1 -- 172.21.15.45:0/243646876 learned_addr learned my addr 172.21.15.45:0/243646876 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55740ffb2130 con 0x5574100ce960 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574100c4ea0 con 0x5574100c5460 2024-03-12T05:25:17.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574100c0dc0 con 0x5574100c5d80 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae627fc640 1 -- 172.21.15.45:0/243646876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 msgr2=0x5574100ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae627fc640 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae627fc640 1 -- 172.21.15.45:0/243646876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 msgr2=0x5574100ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae627fc640 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 0x5574100ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.340+0000 7fae627fc640 1 -- 172.21.15.45:0/243646876 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55740ffc7c20 con 0x5574100c5d80 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.341+0000 7fae627fc640 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fae5800a500 tx=0x7fae58012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2ef9566d30c8c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.341+0000 7fae48ff9640 1 -- 172.21.15.45:0/243646876 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58021040 con 0x5574100c5d80 2024-03-12T05:25:17.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.341+0000 7fae48ff9640 1 -- 172.21.15.45:0/243646876 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae58014070 con 0x5574100c5d80 2024-03-12T05:25:17.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.341+0000 7fae48ff9640 1 -- 172.21.15.45:0/243646876 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae580108d0 con 0x5574100c5d80 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.342+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 msgr2=0x5574100ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.342+0000 7fae6c009740 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fae5800a500 tx=0x7fae58012000 comp rx=0 tx=0).stop 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 shutdown_connections 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100ce960 0x5574100ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100c5d80 0x5574100ca350 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 --2- 172.21.15.45:0/243646876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 >> 172.21.15.45:0/243646876 conn(0x5574100bfb90 msgr2=0x5574100c6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 shutdown_connections 2024-03-12T05:25:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.343+0000 7fae6c009740 1 -- 172.21.15.45:0/243646876 wait complete. 2024-03-12T05:25:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.344+0000 7fae6c009740 1 -- start start 2024-03-12T05:25:17.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.344+0000 7fae6c009740 1 Processor -- start 2024-03-12T05:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.344+0000 7fae6c009740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.344+0000 7fae6c009740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100d4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.344+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:42926/0 (socket says 172.21.15.45:42926) 2024-03-12T05:25:17.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 -- 172.21.15.45:0/3716397975 learned_addr learned my addr 172.21.15.45:0/3716397975 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557410135420 con 0x5574100c5d80 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557410135560 con 0x5574100c5460 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574101356a0 con 0x5574100ce960 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae627fc640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100d4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae6c009740 10 client.0 _ll_register_callbacks cb 0x7fae54009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae637fe640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 -- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 msgr2=0x5574100d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 -- 172.21.15.45:0/3716397975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 msgr2=0x5574100d4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100d4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae62ffd640 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5800fdc0 con 0x5574100c5460 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.345+0000 7fae637fe640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae62ffd640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae5c002230 tx=0x7fae5c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=121cf7c18f27f6aa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae46ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c003d80 con 0x5574100c5460 2024-03-12T05:25:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fae5c0348e0 con 0x5574100c5460 2024-03-12T05:25:17.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae54096390 con 0x5574100c5460 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae5c034760 con 0x5574100c5460 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.346+0000 7fae6c009740 10 client.28511 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55740ffed8c0 con 0x5574100c5460 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fae5c033c00 con 0x5574100c5460 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae5c03c070 con 0x5574100c5460 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 10 client.28511 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 10 client.28511 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 20 client.28511 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557410136bb0 0x557410138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 4 client.28511 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.347+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fae54096390 con 0x557410136bb0 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae627fc640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557410136bb0 0x557410138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae627fc640 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557410136bb0 0x557410138fd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e54a88406b48217a server_cookie=315ab4f4aed22219 in_seq=0 out_seq=0 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae46ff5640 10 client.28511 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae46ff5640 1 -- 172.21.15.45:0/3716397975 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fae54096390 con 0x557410136bb0 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae46ff5640 10 client.28511 handle_command_reply: tid=0 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.348+0000 7fae6c009740 1 client.28511 shutdown 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 msgr2=0x5574100d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fae5c002230 tx=0x7fae5c005bd0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 shutdown_connections 2024-03-12T05:25:17.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x557410136bb0 0x557410138fd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae46ff5640 0 client.28511 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5574100c5d80 0x5574100d4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5574100ce960 0x5574100d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 --2- 172.21.15.45:0/3716397975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574100c5460 0x5574100d4220 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 >> 172.21.15.45:0/3716397975 conn(0x5574100bfb90 msgr2=0x5574100ca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 shutdown_connections 2024-03-12T05:25:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.349+0000 7fae6c009740 1 -- 172.21.15.45:0/3716397975 wait complete. 2024-03-12T05:25:17.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.352+0000 7fae6c009740 20 client.28511 trim_cache size 0 max 16384 2024-03-12T05:25:17.349 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 msgr2=0x7fae4c0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 0x7fae4c0a41a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fae64072220 tx=0x7fae5800a250 comp rx=0 tx=0).stop 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 msgr2=0x7fae641aab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 0x7fae641aab90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fae5400d750 tx=0x7fae5400dc10 comp rx=0 tx=0).stop 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 shutdown_connections 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fae4c0a1ce0 0x7fae4c0a41a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.355+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae6410f6a0 0x7fae641aab90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.356+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae6410dce0 0x7fae641a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.356+0000 7fae63fff640 1 --2- 172.21.15.45:0/3218197152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae6410d260 0x7fae641a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.356+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 >> 172.21.15.45:0/3218197152 conn(0x7fae6406c030 msgr2=0x7fae64070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.356+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 shutdown_connections 2024-03-12T05:25:17.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.356+0000 7fae63fff640 1 -- 172.21.15.45:0/3218197152 wait complete. 2024-03-12T05:25:17.371 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:17.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:25:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 -- 172.21.15.45:0/4182448564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 msgr2=0x7f8d540a8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 --2- 172.21.15.45:0/4182448564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 0x7f8d540a8f60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c066f90 tx=0x7f8d5c096db0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 -- 172.21.15.45:0/4182448564 shutdown_connections 2024-03-12T05:25:17.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 --2- 172.21.15.45:0/4182448564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009970 0x7f8d540a6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 --2- 172.21.15.45:0/4182448564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 0x7f8d540a8f60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 --2- 172.21.15.45:0/4182448564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009050 0x7f8d54009430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.554+0000 7f8d61725640 1 -- 172.21.15.45:0/4182448564 >> 172.21.15.45:0/4182448564 conn(0x7f8d540939e0 msgr2=0x7f8d54095e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.555+0000 7f8d61725640 1 -- 172.21.15.45:0/4182448564 shutdown_connections 2024-03-12T05:25:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.555+0000 7f8d61725640 1 -- 172.21.15.45:0/4182448564 wait complete. 2024-03-12T05:25:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.555+0000 7f8d61725640 1 Processor -- start 2024-03-12T05:25:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.555+0000 7f8d61725640 1 -- start start 2024-03-12T05:25:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 0x7f8d54138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 0x7f8d54132680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54132e10 con 0x7f8d54009050 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54132f80 con 0x7f8d54009970 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d61725640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54133140 con 0x7f8d540a6b70 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43138/0 (socket says 172.21.15.45:43138) 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.556+0000 7f8d5bfff640 1 -- 172.21.15.45:0/3694813211 learned_addr learned my addr 172.21.15.45:0/3694813211 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 0x7f8d54138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 -- 172.21.15.45:0/3694813211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 msgr2=0x7f8d54132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 0x7f8d54132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 -- 172.21.15.45:0/3694813211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 msgr2=0x7f8d54138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 0x7f8d54138ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 -- 172.21.15.45:0/3694813211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48009560 con 0x7f8d54009050 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 0x7f8d54138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.557+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000e9f0 tx=0x7f8d5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.558+0000 7f8d597fa640 1 -- 172.21.15.45:0/3694813211 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000cd80 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.558+0000 7f8d597fa640 1 -- 172.21.15.45:0/3694813211 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d500054d0 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.558+0000 7f8d597fa640 1 -- 172.21.15.45:0/3694813211 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50010430 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.558+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c04f750 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.558+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5413f4f0 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.562+0000 7f8d597fa640 1 -- 172.21.15.45:0/3694813211 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8d50005020 con 0x7f8d54009050 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.563+0000 7f8d597fa640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 0x7f8d4c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.563+0000 7f8d597fa640 1 -- 172.21.15.45:0/3694813211 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8d50014070 con 0x7f8d54009050 2024-03-12T05:25:17.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.563+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 0x7f8d4c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 0x7f8d4c0a4160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8d54133e80 tx=0x7f8d48009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 Processor -- start 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 -- start start 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6fd80 0x55734cf78420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf7ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cf2a960 con 0x55734cf6f460 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cede310 con 0x55734cf78960 2024-03-12T05:25:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.565+0000 7f8d65630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cedb840 con 0x55734cf6fd80 2024-03-12T05:25:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6fd80 0x55734cf78420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43140/0 (socket says 172.21.15.45:43140) 2024-03-12T05:25:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d5bfff640 1 -- 172.21.15.45:0/823134485 learned_addr learned my addr 172.21.15.45:0/823134485 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf7ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 -- 172.21.15.45:0/823134485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6fd80 msgr2=0x55734cf78420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6fd80 0x55734cf78420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 -- 172.21.15.45:0/823134485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 msgr2=0x55734cf6f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 -- 172.21.15.45:0/823134485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55734ce99060 con 0x55734cf78960 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.566+0000 7f8d60f24640 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf7ad50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0668b0 tx=0x7f8d5c0625f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c6e70111b8cf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.567+0000 7f8d467fc640 1 -- 172.21.15.45:0/823134485 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0762e0 con 0x55734cf78960 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.567+0000 7f8d467fc640 1 -- 172.21.15.45:0/823134485 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d5c06aba0 con 0x55734cf78960 2024-03-12T05:25:17.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.567+0000 7f8d467fc640 1 -- 172.21.15.45:0/823134485 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c067970 con 0x55734cf78960 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- 172.21.15.45:0/823134485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 msgr2=0x55734cf7ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf7ad50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0668b0 tx=0x7f8d5c0625f0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- 172.21.15.45:0/823134485 shutdown_connections 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf7ad50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6fd80 0x55734cf78420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 --2- 172.21.15.45:0/823134485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf6f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- 172.21.15.45:0/823134485 >> 172.21.15.45:0/823134485 conn(0x55734cf69b90 msgr2=0x55734cf69f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- 172.21.15.45:0/823134485 shutdown_connections 2024-03-12T05:25:17.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- 172.21.15.45:0/823134485 wait complete. 2024-03-12T05:25:17.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.568+0000 7f8d65630740 1 -- start start 2024-03-12T05:25:17.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 Processor -- start 2024-03-12T05:25:17.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf75820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6fd80 0x55734cf75d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734ced7020 con 0x55734cf6fd80 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cbaa430 con 0x55734cf78960 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cb66e00 con 0x55734cf6f460 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.569+0000 7f8d65630740 10 client.0 _ll_register_callbacks cb 0x55734ce7ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6fd80 0x55734cf75d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf75820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37700/0 (socket says 172.21.15.45:37700) 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 -- 172.21.15.45:0/2560261202 learned_addr learned my addr 172.21.15.45:0/2560261202 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 -- 172.21.15.45:0/2560261202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 msgr2=0x55734cf75820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf75820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 -- 172.21.15.45:0/2560261202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6fd80 msgr2=0x55734cf75d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6fd80 0x55734cf75d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.570+0000 7f8d60f24640 1 -- 172.21.15.45:0/2560261202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d540a99f0 con 0x55734cf78960 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.571+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0506c0 tx=0x7f8d5c069470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b94947d00f047ab server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.571+0000 7f8d3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.571+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c097960 con 0x55734cf78960 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.571+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8d5c067710 con 0x55734cf78960 2024-03-12T05:25:17.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.572+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55734cbd67e0 con 0x55734cf78960 2024-03-12T05:25:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.573+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d5c067910 con 0x55734cf78960 2024-03-12T05:25:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.573+0000 7f8d65630740 10 client.28520 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.573+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55734ced56a0 con 0x55734cf78960 2024-03-12T05:25:17.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.573+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55734cbd67e0 con 0x55734cf78960 2024-03-12T05:25:17.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d5c0b0050 con 0x55734cf78960 2024-03-12T05:25:17.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 10 client.28520 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 10 client.28520 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:25:17.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 20 client.28520 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734cfaaa70 0x55734cface90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 4 client.28520 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.574+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55734cc13720 con 0x55734cfaaa70 2024-03-12T05:25:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.575+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734cfaaa70 0x55734cface90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.575+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734cfaaa70 0x55734cface90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81650e181d6eab35 server_cookie=ed75174c8ec8a752 in_seq=0 out_seq=0 2024-03-12T05:25:17.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.575+0000 7f8d3b7fe640 10 client.28520 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.576+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/2560261202 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55734cc13720 con 0x55734cfaaa70 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.576+0000 7f8d3b7fe640 10 client.28520 handle_command_reply: tid=0 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.576+0000 7f8d65630740 1 client.28520 shutdown 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.576+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 msgr2=0x55734cf762a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.576+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0506c0 tx=0x7f8d5c069470 comp rx=0 tx=0).stop 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 shutdown_connections 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734cfaaa70 0x55734cface90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6fd80 0x55734cf75d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf78960 0x55734cf762a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 --2- 172.21.15.45:0/2560261202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf75820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 >> 172.21.15.45:0/2560261202 conn(0x55734cf69b90 msgr2=0x55734cf6b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d3b7fe640 0 client.28520 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 shutdown_connections 2024-03-12T05:25:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 1 -- 172.21.15.45:0/2560261202 wait complete. 2024-03-12T05:25:17.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.577+0000 7f8d65630740 20 client.28520 trim_cache size 0 max 16384 2024-03-12T05:25:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.599+0000 7f8d65630740 1 Processor -- start 2024-03-12T05:25:17.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.599+0000 7f8d65630740 1 -- start start 2024-03-12T05:25:17.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf77f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf72a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734ce5c130 con 0x55734cf6f460 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cf6e5b0 con 0x55734cf6fd80 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d65630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734cf6bea0 con 0x55734cf78960 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59326/0 (socket says 172.21.15.45:59326) 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 -- 172.21.15.45:0/2824858759 learned_addr learned my addr 172.21.15.45:0/2824858759 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf77f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf72a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 -- 172.21.15.45:0/2824858759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 msgr2=0x55734cf72a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf72a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 -- 172.21.15.45:0/2824858759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 msgr2=0x55734cf77f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf77f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d60f24640 1 -- 172.21.15.45:0/2824858759 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55734ce71c20 con 0x55734cf78960 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.600+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.601+0000 7f8d60f24640 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c066f90 tx=0x7f8d5c066d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=546a7fbce62800d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.601+0000 7f8d45ffb640 1 -- 172.21.15.45:0/2824858759 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0b3480 con 0x55734cf78960 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.601+0000 7f8d45ffb640 1 -- 172.21.15.45:0/2824858759 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d5c074530 con 0x55734cf78960 2024-03-12T05:25:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.601+0000 7f8d45ffb640 1 -- 172.21.15.45:0/2824858759 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c09f670 con 0x55734cf78960 2024-03-12T05:25:17.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 -- 172.21.15.45:0/2824858759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 msgr2=0x55734cf72fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c066f90 tx=0x7f8d5c066d80 comp rx=0 tx=0).stop 2024-03-12T05:25:17.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 -- 172.21.15.45:0/2824858759 shutdown_connections 2024-03-12T05:25:17.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf78960 0x55734cf72fc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 --2- 172.21.15.45:0/2824858759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf6f460 0x55734cf77f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.602+0000 7f8d65630740 1 -- 172.21.15.45:0/2824858759 >> 172.21.15.45:0/2824858759 conn(0x55734cf69b90 msgr2=0x55734cfb17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.603+0000 7f8d65630740 1 -- 172.21.15.45:0/2824858759 shutdown_connections 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.603+0000 7f8d65630740 1 -- 172.21.15.45:0/2824858759 wait complete. 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.603+0000 7f8d65630740 1 -- start start 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 Processor -- start 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf8e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf8eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf78960 0x55734cffff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734d000490 con 0x55734cf78960 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734d0005d0 con 0x55734cf6fd80 2024-03-12T05:25:17.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55734d000710 con 0x55734cf6f460 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d65630740 10 client.0 _ll_register_callbacks cb 0x7f8d48005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf8e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d5b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf8eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d5bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf8e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59332/0 (socket says 172.21.15.45:59332) 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.604+0000 7f8d5bfff640 1 -- 172.21.15.45:0/463909707 learned_addr learned my addr 172.21.15.45:0/463909707 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d60f24640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf78960 0x55734cffff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 -- 172.21.15.45:0/463909707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 msgr2=0x55734cf8e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf8e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 -- 172.21.15.45:0/463909707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf78960 msgr2=0x55734cffff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf78960 0x55734cffff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 -- 172.21.15.45:0/463909707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55734ced56a0 con 0x55734cf6fd80 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d5b7fe640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf8eef0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55734cf5e7b0 tx=0x7f8d480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbaca9d31ad425ee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48025070 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.605+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8d48011710 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d5c06aba0 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d480118d0 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d65630740 10 client.28526 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55734ce978c0 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8d48036020 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.606+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d4801ec10 con 0x55734cf6fd80 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 10 client.28526 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 10 client.28526 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 20 client.28526 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734d0020a0 0x55734d0044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 4 client.28526 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:17.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55734cc62890 con 0x55734d0020a0 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734d0020a0 0x55734d0044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.607+0000 7f8d3b7fe640 10 client.28526 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d5bfff640 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734d0020a0 0x55734d0044c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3fdd3b592b52b92e server_cookie=52f66c0d94878019 in_seq=0 out_seq=0 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d3b7fe640 1 -- 172.21.15.45:0/463909707 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55734cc62890 con 0x55734d0020a0 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d3b7fe640 10 client.28526 handle_command_reply: tid=0 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d65630740 1 client.28526 shutdown 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 msgr2=0x55734cf8eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.608+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf8eef0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55734cf5e7b0 tx=0x7f8d480046c0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 shutdown_connections 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x55734d0020a0 0x55734d0044c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55734cf78960 0x55734cffff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55734cf6fd80 0x55734cf8eef0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 --2- 172.21.15.45:0/463909707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55734cf6f460 0x55734cf8e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 >> 172.21.15.45:0/463909707 conn(0x55734cf69b90 msgr2=0x55734cf740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d3b7fe640 0 client.28526 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 shutdown_connections 2024-03-12T05:25:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 1 -- 172.21.15.45:0/463909707 wait complete. 2024-03-12T05:25:17.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.609+0000 7f8d65630740 20 client.28526 trim_cache size 0 max 16384 2024-03-12T05:25:17.606 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 msgr2=0x7f8d4c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 0x7f8d4c0a4160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8d54133e80 tx=0x7f8d48009260 comp rx=0 tx=0).stop 2024-03-12T05:25:17.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 msgr2=0x7f8d54138570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000e9f0 tx=0x7f8d5000edf0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 shutdown_connections 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8d4c0a1ca0 0x7f8d4c0a4160 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d540a6b70 0x7f8d54132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d54009970 0x7f8d54138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 --2- 172.21.15.45:0/3694813211 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8d54009050 0x7f8d54138570 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.610+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 >> 172.21.15.45:0/3694813211 conn(0x7f8d540939e0 msgr2=0x7f8d540a82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.611+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 shutdown_connections 2024-03-12T05:25:17.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.611+0000 7f8d61725640 1 -- 172.21.15.45:0/3694813211 wait complete. 2024-03-12T05:25:17.623 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:17 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2657987394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:17.624 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:17.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 -- 172.21.15.45:0/1694570265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e810a260 msgr2=0x7f93e810c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 --2- 172.21.15.45:0/1694570265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e810a260 0x7f93e810c650 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f93e000f0a0 tx=0x7f93e002ff20 comp rx=0 tx=0).stop 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 -- 172.21.15.45:0/1694570265 shutdown_connections 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 --2- 172.21.15.45:0/1694570265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e810a260 0x7f93e810c650 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 --2- 172.21.15.45:0/1694570265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e8104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 --2- 172.21.15.45:0/1694570265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e80ffbf0 0x7f93e8103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 -- 172.21.15.45:0/1694570265 >> 172.21.15.45:0/1694570265 conn(0x7f93e80761d0 msgr2=0x7f93e80fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 -- 172.21.15.45:0/1694570265 shutdown_connections 2024-03-12T05:25:17.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.803+0000 7f93eca64640 1 -- 172.21.15.45:0/1694570265 wait complete. 2024-03-12T05:25:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.804+0000 7f93eca64640 1 Processor -- start 2024-03-12T05:25:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.804+0000 7f93eca64640 1 -- start start 2024-03-12T05:25:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 0x7f93e8195480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e81959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e810a260 0x7f93e8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93e659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 0x7f93e8195480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e819a340 con 0x7f93e80ffbf0 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93e659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 0x7f93e8195480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43200/0 (socket says 172.21.15.45:43200) 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93e659c640 1 -- 172.21.15.45:0/767049253 learned_addr learned my addr 172.21.15.45:0/767049253 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e81959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e819a4b0 con 0x7f93e8104500 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e819a670 con 0x7f93e810a260 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.805+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e810a260 0x7f93e8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 -- 172.21.15.45:0/767049253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e810a260 msgr2=0x7f93e8199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e810a260 0x7f93e8199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 -- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 msgr2=0x7f93e8195480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 0x7f93e8195480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 -- 172.21.15.45:0/767049253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e000dd10 con 0x7f93e8104500 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e81959c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00d6d0 tx=0x7f93dc00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93d77fe640 1 -- 172.21.15.45:0/767049253 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc005220 con 0x7f93e8104500 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93d77fe640 1 -- 172.21.15.45:0/767049253 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93dc009cd0 con 0x7f93e8104500 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e819a950 con 0x7f93e8104500 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93d77fe640 1 -- 172.21.15.45:0/767049253 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc004d00 con 0x7f93e8104500 2024-03-12T05:25:17.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.806+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e81a21d0 con 0x7f93e8104500 2024-03-12T05:25:17.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.810+0000 7f93d77fe640 1 -- 172.21.15.45:0/767049253 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f93dc009820 con 0x7f93e8104500 2024-03-12T05:25:17.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 Processor -- start 2024-03-12T05:25:17.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 -- start start 2024-03-12T05:25:17.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x55717935a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935aa10 0x55717936ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557179315960 con 0x55717936bf60 2024-03-12T05:25:17.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571792c9310 con 0x55717935a060 2024-03-12T05:25:17.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.811+0000 7f93ef635740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571792c6840 con 0x55717935aa10 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43202/0 (socket says 172.21.15.45:43202) 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e5d9b640 1 -- 172.21.15.45:0/2087484252 learned_addr learned my addr 172.21.15.45:0/2087484252 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e659c640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935aa10 0x55717936ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x55717935a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93d77fe640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 0x7f93d00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 -- 172.21.15.45:0/2087484252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935aa10 msgr2=0x55717936ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935aa10 0x55717936ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 -- 172.21.15.45:0/2087484252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 msgr2=0x55717936e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 -- 172.21.15.45:0/2087484252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557179284060 con 0x55717935a060 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.812+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 0x7f93d00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.813+0000 7f93d77fe640 1 -- 172.21.15.45:0/767049253 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93dc0cd1f0 con 0x7f93e8104500 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.813+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:17.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.813+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x55717935a440 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f93e0030460 tx=0x7f93e000dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=763424cf59ea93f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.815+0000 7f93d4ff9640 1 -- 172.21.15.45:0/2087484252 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0038670 con 0x55717935a060 2024-03-12T05:25:17.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.815+0000 7f93d4ff9640 1 -- 172.21.15.45:0/2087484252 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93e0003e20 con 0x55717935a060 2024-03-12T05:25:17.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.815+0000 7f93d4ff9640 1 -- 172.21.15.45:0/2087484252 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e000d460 con 0x55717935a060 2024-03-12T05:25:17.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 -- 172.21.15.45:0/2087484252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 msgr2=0x55717935a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x55717935a440 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f93e0030460 tx=0x7f93e000dc10 comp rx=0 tx=0).stop 2024-03-12T05:25:17.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 0x7f93d00a40e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f93e004b000 tx=0x7f93e004c000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 -- 172.21.15.45:0/2087484252 shutdown_connections 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717936bf60 0x55717936e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935aa10 0x55717936ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 --2- 172.21.15.45:0/2087484252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x55717935a440 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.816+0000 7f93ef635740 1 -- 172.21.15.45:0/2087484252 >> 172.21.15.45:0/2087484252 conn(0x557179354980 msgr2=0x557179354d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.817+0000 7f93ef635740 1 -- 172.21.15.45:0/2087484252 shutdown_connections 2024-03-12T05:25:17.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.817+0000 7f93ef635740 1 -- 172.21.15.45:0/2087484252 wait complete. 2024-03-12T05:25:17.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.817+0000 7f93ef635740 1 -- start start 2024-03-12T05:25:17.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 Processor -- start 2024-03-12T05:25:17.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935a060 0x557179368fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 0x5571793694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x557179369a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571792c2020 con 0x55717935a060 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x557178f95430 con 0x55717935aa10 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 0x5571793694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93e6d9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x557179369a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 0x5571793694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37760/0 (socket says 172.21.15.45:37760) 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93e5d9b640 1 -- 172.21.15.45:0/3312927260 learned_addr learned my addr 172.21.15.45:0/3312927260 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.818+0000 7f93ef635740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557178f51e00 con 0x55717936bf60 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e659c640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935a060 0x557179368fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93ef635740 10 client.0 _ll_register_callbacks cb 0x557179269cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 -- 172.21.15.45:0/3312927260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 msgr2=0x5571793694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 0x5571793694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 -- 172.21.15.45:0/3312927260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935a060 msgr2=0x557179368fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935a060 0x557179368fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 -- 172.21.15.45:0/3312927260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e80ff6e0 con 0x55717936bf60 2024-03-12T05:25:17.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557178fc17e0 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x557179369a20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f93e003a040 tx=0x7f93e000d9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee55493e894ef5f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.819+0000 7f93cdffb640 10 client.28495 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93cdffb640 1 -- 172.21.15.45:0/3312927260 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0038670 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93cdffb640 1 -- 172.21.15.45:0/3312927260 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93e0030c40 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93cdffb640 1 -- 172.21.15.45:0/3312927260 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93e0030e00 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93ef635740 10 client.28495 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x5571792c06a0 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.820+0000 7f93cdffb640 1 -- 172.21.15.45:0/3312927260 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93e0003970 con 0x55717936bf60 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 10 client.28495 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 10 client.28495 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 20 client.28495 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55717937beb0 0x55717937e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 4 client.28495 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557178ffe720 con 0x55717937beb0 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93e659c640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55717937beb0 0x55717937e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93e659c640 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55717937beb0 0x55717937e2f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7c53a2d6bcbd987 server_cookie=7ba59fdcfeb013f5 in_seq=0 out_seq=0 2024-03-12T05:25:17.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.821+0000 7f93cdffb640 10 client.28495 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.822+0000 7f93cdffb640 1 -- 172.21.15.45:0/3312927260 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557178ffe720 con 0x55717937beb0 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.822+0000 7f93cdffb640 10 client.28495 handle_command_reply: tid=0 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.822+0000 7f93ef635740 1 client.28495 shutdown 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 msgr2=0x557179369a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x557179369a20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f93e003a040 tx=0x7f93e000d9d0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 shutdown_connections 2024-03-12T05:25:17.818 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55717937beb0 0x55717937e2f0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x557179369a20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935aa10 0x5571793694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 --2- 172.21.15.45:0/3312927260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935a060 0x557179368fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 >> 172.21.15.45:0/3312927260 conn(0x557179354980 msgr2=0x5571793566d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93cdffb640 0 client.28495 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 shutdown_connections 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 1 -- 172.21.15.45:0/3312927260 wait complete. 2024-03-12T05:25:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.823+0000 7f93ef635740 20 client.28495 trim_cache size 0 max 16384 2024-03-12T05:25:17.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.845+0000 7f93ef635740 1 Processor -- start 2024-03-12T05:25:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 -- start start 2024-03-12T05:25:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 0x55717936b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 0x557179366220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557179247130 con 0x55717935aa10 2024-03-12T05:25:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571793595b0 con 0x55717936bf60 2024-03-12T05:25:17.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.846+0000 7f93ef635740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571793505c0 con 0x55717935a060 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43218/0 (socket says 172.21.15.45:43218) 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 -- 172.21.15.45:0/3508440142 learned_addr learned my addr 172.21.15.45:0/3508440142 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 0x557179366220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e659c640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 0x55717936b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 -- 172.21.15.45:0/3508440142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 msgr2=0x55717936b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 0x55717936b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 -- 172.21.15.45:0/3508440142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 msgr2=0x557179366220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 0x557179366220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 -- 172.21.15.45:0/3508440142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55717925cc20 con 0x55717935aa10 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e659c640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 0x55717936b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 0x557179366220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.847+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009610 tx=0x7f93dc005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf46c85dc32b05c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.848+0000 7f93cffff640 1 -- 172.21.15.45:0/3508440142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc099980 con 0x55717935aa10 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.848+0000 7f93cffff640 1 -- 172.21.15.45:0/3508440142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93dc0997a0 con 0x55717935aa10 2024-03-12T05:25:17.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.848+0000 7f93cffff640 1 -- 172.21.15.45:0/3508440142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0a25a0 con 0x55717935aa10 2024-03-12T05:25:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 -- 172.21.15.45:0/3508440142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 msgr2=0x557179382c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009610 tx=0x7f93dc005480 comp rx=0 tx=0).stop 2024-03-12T05:25:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 -- 172.21.15.45:0/3508440142 shutdown_connections 2024-03-12T05:25:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179382c50 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717936bf60 0x557179366220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 --2- 172.21.15.45:0/3508440142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717935a060 0x55717936b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.849+0000 7f93ef635740 1 -- 172.21.15.45:0/3508440142 >> 172.21.15.45:0/3508440142 conn(0x557179354980 msgr2=0x557179364340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 -- 172.21.15.45:0/3508440142 shutdown_connections 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 -- 172.21.15.45:0/3508440142 wait complete. 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 -- start start 2024-03-12T05:25:17.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 Processor -- start 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x557179371c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x5571793d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x557179372a50 con 0x55717935aa10 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.850+0000 7f93ef635740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571793d2920 con 0x55717935a060 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93ef635740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571793d2a60 con 0x55717936bf60 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93ef635740 10 client.0 _ll_register_callbacks cb 0x7f93e000df30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43234/0 (socket says 172.21.15.45:43234) 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 -- 172.21.15.45:0/1258939928 learned_addr learned my addr 172.21.15.45:0/1258939928 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e659c640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x557179371c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e6d9d640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x5571793d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 -- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 msgr2=0x5571793d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x5571793d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 -- 172.21.15.45:0/1258939928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 msgr2=0x557179371c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x557179371c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.851+0000 7f93e5d9b640 1 -- 172.21.15.45:0/1258939928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571792c06a0 con 0x55717935aa10 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93e5d9b640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f93dc099dd0 tx=0x7f93dc0d0be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b5185327cde512 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93e659c640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x557179371c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc0d4070 con 0x55717935aa10 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93e0003e20 con 0x55717935aa10 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f93dc004e60 con 0x55717935aa10 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.852+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93dc0a2530 con 0x55717935aa10 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 10 client.19080 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:17.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f93e819a950 con 0x55717935aa10 2024-03-12T05:25:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f93dc0b6080 con 0x55717935aa10 2024-03-12T05:25:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f93dc0b8120 con 0x55717935aa10 2024-03-12T05:25:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 10 client.19080 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:17.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 10 client.19080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 20 client.19080 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5571793d3eb0 0x5571793d62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 4 client.19080 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.853+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55717904d890 con 0x5571793d3eb0 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.854+0000 7f93e659c640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5571793d3eb0 0x5571793d62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.855+0000 7f93e659c640 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5571793d3eb0 0x5571793d62a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a99673f302d59863 server_cookie=8d00d41a2ec8f9b1 in_seq=0 out_seq=0 2024-03-12T05:25:17.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.855+0000 7f93cdffb640 10 client.19080 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.855+0000 7f93cdffb640 1 -- 172.21.15.45:0/1258939928 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55717904d890 con 0x5571793d3eb0 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.855+0000 7f93cdffb640 10 client.19080 handle_command_reply: tid=0 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 client.19080 shutdown 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 msgr2=0x557179372180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f93dc099dd0 tx=0x7f93dc0d0be0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 shutdown_connections 2024-03-12T05:25:17.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5571793d3eb0 0x5571793d62a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55717935aa10 0x557179372180 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55717936bf60 0x5571793d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93cdffb640 0 client.19080 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 --2- 172.21.15.45:0/1258939928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55717935a060 0x557179371c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 >> 172.21.15.45:0/1258939928 conn(0x557179354980 msgr2=0x557179382c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.856+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 shutdown_connections 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.857+0000 7f93ef635740 1 -- 172.21.15.45:0/1258939928 wait complete. 2024-03-12T05:25:17.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.857+0000 7f93ef635740 20 client.19080 trim_cache size 0 max 16384 2024-03-12T05:25:17.853 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.858+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 msgr2=0x7f93d00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.858+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 0x7f93d00a40e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f93e004b000 tx=0x7f93e004c000 comp rx=0 tx=0).stop 2024-03-12T05:25:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 msgr2=0x7f93e81959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:17.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e81959c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00d6d0 tx=0x7f93dc00db90 comp rx=0 tx=0).stop 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 shutdown_connections 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f93d00a1c20 0x7f93d00a40e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f93e810a260 0x7f93e8199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93e8104500 0x7f93e81959c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.859+0000 7f93eca64640 1 --2- 172.21.15.45:0/767049253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93e80ffbf0 0x7f93e8195480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.860+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 >> 172.21.15.45:0/767049253 conn(0x7f93e80761d0 msgr2=0x7f93e810b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.860+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 shutdown_connections 2024-03-12T05:25:17.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:17.860+0000 7f93eca64640 1 -- 172.21.15.45:0/767049253 wait complete. 2024-03-12T05:25:17.876 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:17.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:25:17.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:17 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2657987394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:17.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:17 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2657987394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 -- 172.21.15.45:0/3194620930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c0ff5d0 msgr2=0x7faa0c0ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 --2- 172.21.15.45:0/3194620930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c0ff9b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ca0 tx=0x7faa0402c100 comp rx=0 tx=0).stop 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 -- 172.21.15.45:0/3194620930 shutdown_connections 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 --2- 172.21.15.45:0/3194620930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c1046a0 0x7faa0c10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 --2- 172.21.15.45:0/3194620930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c0fff80 0x7faa0c103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 --2- 172.21.15.45:0/3194620930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c0ff9b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 -- 172.21.15.45:0/3194620930 >> 172.21.15.45:0/3194620930 conn(0x7faa0c0761d0 msgr2=0x7faa0c0fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 -- 172.21.15.45:0/3194620930 shutdown_connections 2024-03-12T05:25:18.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.057+0000 7faa1084e640 1 -- 172.21.15.45:0/3194620930 wait complete. 2024-03-12T05:25:18.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.058+0000 7faa1084e640 1 Processor -- start 2024-03-12T05:25:18.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.058+0000 7faa1084e640 1 -- start start 2024-03-12T05:25:18.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 0x7faa0c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c197f90 con 0x7faa0c1046a0 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c198100 con 0x7faa0c1975a0 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa1084e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c1982c0 con 0x7faa0c0ff5d0 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 0x7faa0c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43254/0 (socket says 172.21.15.45:43254) 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 -- 172.21.15.45:0/3706179188 learned_addr learned my addr 172.21.15.45:0/3706179188 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa0a59c640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 -- 172.21.15.45:0/3706179188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c0ff5d0 msgr2=0x7faa0c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 -- 172.21.15.45:0/3706179188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 msgr2=0x7faa0c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 0x7faa0c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa09d9b640 1 -- 172.21.15.45:0/3706179188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04009900 con 0x7faa0c1046a0 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.059+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 0x7faa0c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.060+0000 7faa09d9b640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009c80 tx=0x7fa9fc00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.060+0000 7fa9fb7fe640 1 -- 172.21.15.45:0/3706179188 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc015070 con 0x7faa0c1046a0 2024-03-12T05:25:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.060+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c10e150 con 0x7faa0c1046a0 2024-03-12T05:25:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.060+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c10e6b0 con 0x7faa0c1046a0 2024-03-12T05:25:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.060+0000 7fa9fb7fe640 1 -- 172.21.15.45:0/3706179188 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fa9fc00be60 con 0x7faa0c1046a0 2024-03-12T05:25:18.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.061+0000 7fa9fb7fe640 1 -- 172.21.15.45:0/3706179188 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc004e80 con 0x7faa0c1046a0 2024-03-12T05:25:18.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 Processor -- start 2024-03-12T05:25:18.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 -- start start 2024-03-12T05:25:18.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 0x559693b23440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7fa9fb7fe640 1 -- 172.21.15.45:0/3706179188 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa9fc005000 con 0x7faa0c1046a0 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b37350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693ade960 con 0x559693b23a10 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693a92310 con 0x559693b34f60 2024-03-12T05:25:18.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.064+0000 7faa13424740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693a8f840 con 0x559693b23060 2024-03-12T05:25:18.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43270/0 (socket says 172.21.15.45:43270) 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0a59c640 1 -- 172.21.15.45:0/1443158826 learned_addr learned my addr 172.21.15.45:0/1443158826 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 0x559693b23440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa09d9b640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b37350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 -- 172.21.15.45:0/1443158826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 msgr2=0x559693b37350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b37350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 -- 172.21.15.45:0/1443158826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 msgr2=0x559693b34a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 -- 172.21.15.45:0/1443158826 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559693a4d060 con 0x559693b23060 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0a59c640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7fa9fb7fe640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 0x7fa9f40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.065+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 0x559693b23440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faa00009f60 tx=0x7faa00009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c541f423c517607 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.066+0000 7fa9fb7fe640 1 -- 172.21.15.45:0/3706179188 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fa9fc0ccd10 con 0x7faa0c1046a0 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.066+0000 7faa0a59c640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 0x7fa9f40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.066+0000 7fa9f8ff9640 1 -- 172.21.15.45:0/1443158826 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000042f0 con 0x559693b23060 2024-03-12T05:25:18.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.066+0000 7fa9f8ff9640 1 -- 172.21.15.45:0/1443158826 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faa00004450 con 0x559693b23060 2024-03-12T05:25:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.066+0000 7fa9f8ff9640 1 -- 172.21.15.45:0/1443158826 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000035f0 con 0x559693b23060 2024-03-12T05:25:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.067+0000 7faa13424740 1 -- 172.21.15.45:0/1443158826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 msgr2=0x559693b23440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.067+0000 7faa13424740 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 0x559693b23440 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faa00009f60 tx=0x7faa00009f90 comp rx=0 tx=0).stop 2024-03-12T05:25:18.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.067+0000 7faa0a59c640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 0x7fa9f40a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faa04000f80 tx=0x7faa04004a40 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 -- 172.21.15.45:0/1443158826 shutdown_connections 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b34a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b37350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 --2- 172.21.15.45:0/1443158826 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23060 0x559693b23440 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 -- 172.21.15.45:0/1443158826 >> 172.21.15.45:0/1443158826 conn(0x559693b1d980 msgr2=0x559693b1dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 -- 172.21.15.45:0/1443158826 shutdown_connections 2024-03-12T05:25:18.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.068+0000 7faa13424740 1 -- 172.21.15.45:0/1443158826 wait complete. 2024-03-12T05:25:18.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.069+0000 7faa13424740 1 -- start start 2024-03-12T05:25:18.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.069+0000 7faa13424740 1 Processor -- start 2024-03-12T05:25:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.069+0000 7faa13424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.069+0000 7faa13424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.069+0000 7faa13424740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b32a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37796/0 (socket says 172.21.15.45:37796) 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa09d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa09d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43278/0 (socket says 172.21.15.45:43278) 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa13424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693a8b020 con 0x559693b23a10 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa13424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55969375e430 con 0x559693b23060 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa13424740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55969371ae00 con 0x559693b34f60 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa13424740 10 client.0 _ll_register_callbacks cb 0x559693a32cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 -- 172.21.15.45:0/2501182719 learned_addr learned my addr 172.21.15.45:0/2501182719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b32a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 -- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 msgr2=0x559693b32a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b32a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 -- 172.21.15.45:0/2501182719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 msgr2=0x559693b32530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.070+0000 7faa0a59c640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa0a59c640 1 -- 172.21.15.45:0/2501182719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00009be0 con 0x559693b23060 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa09d9b640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa0a59c640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa0402c710 tx=0x7faa040095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=669d517a2a2cacc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7fa9f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04003b70 con 0x559693b23060 2024-03-12T05:25:18.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55969378a7e0 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faa04003cd0 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa04003e90 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa13424740 10 client.28556 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.071+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559693a896a0 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55969378a7e0 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faa04052410 con 0x559693b23060 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 10 client.28556 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 10 client.28556 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:25:18.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 20 client.28556 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b45440 0x559693b47860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 4 client.28556 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.072+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596937c7720 con 0x559693b45440 2024-03-12T05:25:18.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.073+0000 7faa09d9b640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b45440 0x559693b47860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.074+0000 7faa09d9b640 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b45440 0x559693b47860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d65944dda40eace2 server_cookie=b4e005b9ebb29c7d in_seq=0 out_seq=0 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.074+0000 7fa9f1ffb640 10 client.28556 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/2501182719 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5596937c7720 con 0x559693b45440 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7fa9f1ffb640 10 client.28556 handle_command_reply: tid=0 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 client.28556 shutdown 2024-03-12T05:25:18.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 msgr2=0x559693b31ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa0402c710 tx=0x7faa040095c0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 shutdown_connections 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b32530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b45440 0x559693b47860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b32a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.075+0000 7faa13424740 1 --2- 172.21.15.45:0/2501182719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b31ff0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.076+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 >> 172.21.15.45:0/2501182719 conn(0x559693b1d980 msgr2=0x559693b1f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.076+0000 7fa9f1ffb640 0 client.28556 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.076+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 shutdown_connections 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.076+0000 7faa13424740 1 -- 172.21.15.45:0/2501182719 wait complete. 2024-03-12T05:25:18.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.076+0000 7faa13424740 20 client.28556 trim_cache size 0 max 16384 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.099+0000 7faa13424740 1 Processor -- start 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 -- start start 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 0x559693b34720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23a10 0x559693b2d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b4c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693a10130 con 0x559693b23060 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693b41f40 con 0x559693b34f60 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa13424740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693b31de0 con 0x559693b23a10 2024-03-12T05:25:18.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 0x559693b34720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b4c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 0x559693b34720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43292/0 (socket says 172.21.15.45:43292) 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa0a59c640 1 -- 172.21.15.45:0/2081785683 learned_addr learned my addr 172.21.15.45:0/2081785683 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.100+0000 7faa09d9b640 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23a10 0x559693b2d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 -- 172.21.15.45:0/2081785683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23a10 msgr2=0x559693b2d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23a10 0x559693b2d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 -- 172.21.15.45:0/2081785683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 msgr2=0x559693b34720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 0x559693b34720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 -- 172.21.15.45:0/2081785683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559693a25c20 con 0x559693b34f60 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b4c1c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faa00011d60 tx=0x7faa00004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7f3139e8f53636 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7fa9f3fff640 1 -- 172.21.15.45:0/2081785683 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa00004940 con 0x559693b34f60 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7fa9f3fff640 1 -- 172.21.15.45:0/2081785683 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faa000050f0 con 0x559693b34f60 2024-03-12T05:25:18.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.101+0000 7fa9f3fff640 1 -- 172.21.15.45:0/2081785683 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0000f770 con 0x559693b34f60 2024-03-12T05:25:18.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 -- 172.21.15.45:0/2081785683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 msgr2=0x559693b4c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b4c1c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faa00011d60 tx=0x7faa00004710 comp rx=0 tx=0).stop 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 -- 172.21.15.45:0/2081785683 shutdown_connections 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b34f60 0x559693b4c1c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b23a10 0x559693b2d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 --2- 172.21.15.45:0/2081785683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23060 0x559693b34720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 -- 172.21.15.45:0/2081785683 >> 172.21.15.45:0/2081785683 conn(0x559693b1d980 msgr2=0x559693b18a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 -- 172.21.15.45:0/2081785683 shutdown_connections 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.102+0000 7faa13424740 1 -- 172.21.15.45:0/2081785683 wait complete. 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.103+0000 7faa13424740 1 -- start start 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.103+0000 7faa13424740 1 Processor -- start 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.103+0000 7faa13424740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.103+0000 7faa13424740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b3c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa13424740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b3a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa13424740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693b3aa70 con 0x559693b23a10 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa13424740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693b3abe0 con 0x559693b23060 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa13424740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559693b3ada0 con 0x559693b34f60 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa13424740 10 client.0 _ll_register_callbacks cb 0x7faa00009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37812/0 (socket says 172.21.15.45:37812) 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 -- 172.21.15.45:0/1025222320 learned_addr learned my addr 172.21.15.45:0/1025222320 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa09d9b640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b3c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0ad9d640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b3a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 -- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 msgr2=0x559693b3a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b3a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 -- 172.21.15.45:0/1025222320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 msgr2=0x559693b3c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b3c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 -- 172.21.15.45:0/1025222320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0001d040 con 0x559693b23060 2024-03-12T05:25:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7faa0a59c640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ca0 tx=0x7faa04032b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f04565e4f750cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.104+0000 7fa9f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04003b70 con 0x559693b23060 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7faa04003cd0 con 0x559693b23060 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa00004450 con 0x559693b23060 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa04003e90 con 0x559693b23060 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7faa13424740 10 client.28562 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:18.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559693a4b8c0 con 0x559693b23060 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.105+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7faa040044e0 con 0x559693b23060 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7faa040637f0 con 0x559693b23060 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 10 client.28562 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 10 client.28562 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 20 client.28562 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b9c9d0 0x559693b9edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 4 client.28562 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7faa00004450 con 0x559693b9c9d0 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.106+0000 7faa09d9b640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b9c9d0 0x559693b9edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa09d9b640 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b9c9d0 0x559693b9edf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e1265b89b607992 server_cookie=793e16e1d1aa07f0 in_seq=0 out_seq=0 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7fa9f1ffb640 10 client.28562 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7fa9f1ffb640 1 -- 172.21.15.45:0/1025222320 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7faa00004450 con 0x559693b9c9d0 2024-03-12T05:25:18.104 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7fa9f1ffb640 10 client.28562 handle_command_reply: tid=0 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 client.28562 shutdown 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 msgr2=0x559693b3be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ca0 tx=0x7faa04032b80 comp rx=0 tx=0).stop 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 shutdown_connections 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559693b23a10 0x559693b3c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x559693b9c9d0 0x559693b9edf0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.107+0000 7fa9f1ffb640 0 client.28562 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.108+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559693b34f60 0x559693b3a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.108+0000 7faa13424740 1 --2- 172.21.15.45:0/1025222320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559693b23060 0x559693b3be00 secure :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ca0 tx=0x7faa04032b80 comp rx=0 tx=0).stop 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.108+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 >> 172.21.15.45:0/1025222320 conn(0x559693b1d980 msgr2=0x559693b2d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.108+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 shutdown_connections 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.108+0000 7faa13424740 1 -- 172.21.15.45:0/1025222320 wait complete. 2024-03-12T05:25:18.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.109+0000 7faa13424740 20 client.28562 trim_cache size 0 max 16384 2024-03-12T05:25:18.106 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 msgr2=0x7fa9f40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 0x7fa9f40a3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faa04000f80 tx=0x7faa04004a40 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 msgr2=0x7faa0c19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009c80 tx=0x7fa9fc00b690 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 shutdown_connections 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fa9f40a1ac0 0x7fa9f40a3f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa0c1046a0 0x7faa0c19d930 secure :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009c80 tx=0x7fa9fc00b690 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa0c1975a0 0x7faa0c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 --2- 172.21.15.45:0/3706179188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa0c0ff5d0 0x7faa0c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.110+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 >> 172.21.15.45:0/3706179188 conn(0x7faa0c0761d0 msgr2=0x7faa0c102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.111+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 shutdown_connections 2024-03-12T05:25:18.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.111+0000 7faa1084e640 1 -- 172.21.15.45:0/3706179188 wait complete. 2024-03-12T05:25:18.126 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:18.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:25:18.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.307+0000 7f671c95a640 1 -- 172.21.15.45:0/3031180759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718100a50 msgr2=0x7f6718104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.307+0000 7f671c95a640 1 --2- 172.21.15.45:0/3031180759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718100a50 0x7f6718104a90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f670c009900 tx=0x7f670c02bc40 comp rx=0 tx=0).stop 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 -- 172.21.15.45:0/3031180759 shutdown_connections 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 --2- 172.21.15.45:0/3031180759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6718105170 0x7f671810cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 --2- 172.21.15.45:0/3031180759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718100a50 0x7f6718104a90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 --2- 172.21.15.45:0/3031180759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 -- 172.21.15.45:0/3031180759 >> 172.21.15.45:0/3031180759 conn(0x7f6718076c30 msgr2=0x7f67180fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 -- 172.21.15.45:0/3031180759 shutdown_connections 2024-03-12T05:25:18.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.308+0000 7f671c95a640 1 -- 172.21.15.45:0/3031180759 wait complete. 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.309+0000 7f671c95a640 1 Processor -- start 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 -- start start 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 0x7f6718197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671819be30 0x7f671819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819c820 con 0x7f67181000a0 2024-03-12T05:25:18.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819c990 con 0x7f671819be30 2024-03-12T05:25:18.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f671c95a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819cb00 con 0x7f6718105170 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f67177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f6717fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671819be30 0x7f671819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f67177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43318/0 (socket says 172.21.15.45:43318) 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f67177fe640 1 -- 172.21.15.45:0/1554610270 learned_addr learned my addr 172.21.15.45:0/1554610270 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.310+0000 7f6716ffd640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 0x7f6718197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 -- 172.21.15.45:0/1554610270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 msgr2=0x7f6718197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 0x7f6718197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 -- 172.21.15.45:0/1554610270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671819be30 msgr2=0x7f671819c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671819be30 0x7f671819c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 -- 172.21.15.45:0/1554610270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c009560 con 0x7f67181000a0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f6716ffd640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 0x7f6718197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f67177fe640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f670800e9f0 tx=0x7f670800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.311+0000 7f6714ff9640 1 -- 172.21.15.45:0/1554610270 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708009700 con 0x7f67181000a0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.312+0000 7f6714ff9640 1 -- 172.21.15.45:0/1554610270 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f670800cd80 con 0x7f67181000a0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.312+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671810d280 con 0x7f67181000a0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.312+0000 7f6714ff9640 1 -- 172.21.15.45:0/1554610270 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708018840 con 0x7f67181000a0 2024-03-12T05:25:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.312+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671810d700 con 0x7f67181000a0 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 Processor -- start 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 -- start start 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6714ff9640 1 -- 172.21.15.45:0/1554610270 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f67080189a0 con 0x7f67181000a0 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241568050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x564241570c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424151a960 con 0x564241568590 2024-03-12T05:25:18.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642414ce310 con 0x564241571170 2024-03-12T05:25:18.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.315+0000 7f6720794740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642414cb840 con 0x564241567c70 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f67177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x564241570c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37830/0 (socket says 172.21.15.45:37830) 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 -- 172.21.15.45:0/499828608 learned_addr learned my addr 172.21.15.45:0/499828608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6717fff640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241568050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 -- 172.21.15.45:0/499828608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 msgr2=0x564241568050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241568050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 -- 172.21.15.45:0/499828608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 msgr2=0x564241570c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x564241570c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6716ffd640 1 -- 172.21.15.45:0/499828608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564241489060 con 0x564241571170 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.316+0000 7f6714ff9640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 0x7f67000a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f6717fff640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241568050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f67177fe640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x564241570c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f6714ff9640 1 -- 172.21.15.45:0/1554610270 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f6708014070 con 0x7f67181000a0 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f6716ffd640 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d88a76d6f2d9cdc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f66fdffb640 1 -- 172.21.15.45:0/499828608 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c02c940 con 0x564241571170 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f66fdffb640 1 -- 172.21.15.45:0/499828608 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f670c0044f0 con 0x564241571170 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f66fdffb640 1 -- 172.21.15.45:0/499828608 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c003d80 con 0x564241571170 2024-03-12T05:25:18.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.317+0000 7f6717fff640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 0x7f67000a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- 172.21.15.45:0/499828608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 msgr2=0x564241573560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6717fff640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 0x7f67000a4160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f671000a510 tx=0x7f6710009650 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c005e80 comp rx=0 tx=0).stop 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- 172.21.15.45:0/499828608 shutdown_connections 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x564241570c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241573560 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 --2- 172.21.15.45:0/499828608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241568050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- 172.21.15.45:0/499828608 >> 172.21.15.45:0/499828608 conn(0x5642415623a0 msgr2=0x564241562780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- 172.21.15.45:0/499828608 shutdown_connections 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- 172.21.15.45:0/499828608 wait complete. 2024-03-12T05:25:18.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.318+0000 7f6720794740 1 -- start start 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.319+0000 7f6720794740 1 Processor -- start 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.319+0000 7f6720794740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 0x56424156e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.319+0000 7f6720794740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6720794740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6716ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6716ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43328/0 (socket says 172.21.15.45:43328) 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6716ffd640 1 -- 172.21.15.45:0/3890511206 learned_addr learned my addr 172.21.15.45:0/3890511206 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642414c7020 con 0x564241568590 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424119a430 con 0x564241567c70 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564241156e00 con 0x564241571170 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.320+0000 7f6720794740 10 client.0 _ll_register_callbacks cb 0x56424146ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f67177fe640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 0x56424156e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f6716ffd640 1 -- 172.21.15.45:0/3890511206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 msgr2=0x56424156ec30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f6716ffd640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156ec30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f6716ffd640 1 -- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 msgr2=0x56424156e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f6716ffd640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 0x56424156e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f6716ffd640 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6718068660 con 0x564241568590 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.321+0000 7f67177fe640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 0x56424156e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f6716ffd640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f670c005ab0 tx=0x7f670c003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df6167916a4129a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f66f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c003f90 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f670c0040f0 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642411c67e0 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f670c0042b0 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f6720794740 10 client.19110 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.322+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5642414c56a0 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f670c05e080 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f670c03c050 con 0x564241568590 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 10 client.19110 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:18.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 10 client.19110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 20 client.19110 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564241581050 0x564241583480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 4 client.19110 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642411c67e0 con 0x564241581050 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.323+0000 7f67177fe640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564241581050 0x564241583480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.324+0000 7f66f6ffd640 10 client.19110 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.324+0000 7f67177fe640 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564241581050 0x564241583480 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daeec7f9795debae server_cookie=ab3a3098c5b2d268 in_seq=0 out_seq=0 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.325+0000 7f66f6ffd640 1 -- 172.21.15.45:0/3890511206 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5642411c67e0 con 0x564241581050 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.325+0000 7f66f6ffd640 10 client.19110 handle_command_reply: tid=0 2024-03-12T05:25:18.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.325+0000 7f6720794740 1 client.19110 shutdown 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 msgr2=0x56424156e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f670c005ab0 tx=0x7f670c003960 comp rx=0 tx=0).stop 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 shutdown_connections 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564241581050 0x564241583480 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x56424156e6f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 --2- 172.21.15.45:0/3890511206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241567c70 0x56424156e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f66f6ffd640 0 client.19110 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:18.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.326+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 >> 172.21.15.45:0/3890511206 conn(0x5642415623a0 msgr2=0x5642415640f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.327+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 shutdown_connections 2024-03-12T05:25:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.328+0000 7f6720794740 1 -- 172.21.15.45:0/3890511206 wait complete. 2024-03-12T05:25:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.329+0000 7f6720794740 20 client.19110 trim_cache size 0 max 16384 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 Processor -- start 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 -- start start 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 0x56424156ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424144c130 con 0x564241567c70 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564241567090 con 0x564241568590 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.350+0000 7f6720794740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56424156f690 con 0x564241571170 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43332/0 (socket says 172.21.15.45:43332) 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 -- 172.21.15.45:0/2675235111 learned_addr learned my addr 172.21.15.45:0/2675235111 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 0x56424156ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f6717fff640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 -- 172.21.15.45:0/2675235111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 msgr2=0x56424156b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 -- 172.21.15.45:0/2675235111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 msgr2=0x56424156ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 0x56424156ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 -- 172.21.15.45:0/2675235111 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564241583830 con 0x564241567c70 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 0x56424156ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f67177fe640 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f670809a430 tx=0x7f67080cfcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583b0fa93a17a499 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.351+0000 7f66fd7fa640 1 -- 172.21.15.45:0/2675235111 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670809e7d0 con 0x564241567c70 2024-03-12T05:25:18.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.352+0000 7f66fd7fa640 1 -- 172.21.15.45:0/2675235111 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f67080a2e20 con 0x564241567c70 2024-03-12T05:25:18.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.352+0000 7f66fd7fa640 1 -- 172.21.15.45:0/2675235111 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080ba0b0 con 0x564241567c70 2024-03-12T05:25:18.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 -- 172.21.15.45:0/2675235111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 msgr2=0x56424156e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f670809a430 tx=0x7f67080cfcb0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 -- 172.21.15.45:0/2675235111 shutdown_connections 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241571170 0x56424156b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241568590 0x56424156ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 --2- 172.21.15.45:0/2675235111 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241567c70 0x56424156e700 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 -- 172.21.15.45:0/2675235111 >> 172.21.15.45:0/2675235111 conn(0x5642415623a0 msgr2=0x56424156ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.353+0000 7f6720794740 1 -- 172.21.15.45:0/2675235111 shutdown_connections 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.354+0000 7f6720794740 1 -- 172.21.15.45:0/2675235111 wait complete. 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.354+0000 7f6720794740 1 -- start start 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.354+0000 7f6720794740 1 Processor -- start 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x5642415766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f67177fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f67177fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59482/0 (socket says 172.21.15.45:59482) 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241576c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f67177fe640 1 -- 172.21.15.45:0/2550857321 learned_addr learned my addr 172.21.15.45:0/2550857321 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564241577310 con 0x564241568590 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564241577480 con 0x564241571170 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642415d75b0 con 0x564241567c70 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6720794740 10 client.0 _ll_register_callbacks cb 0x7f670c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x5642415766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.355+0000 7f6717fff640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241576c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 -- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 msgr2=0x564241576c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241576c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 -- 172.21.15.45:0/2550857321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 msgr2=0x5642415766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x5642415766d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671810d700 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x5642415766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f67177fe640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6708096a80 tx=0x7f6708005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cd2edfb64c12e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.356+0000 7f66f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670809e7d0 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f67080d0e40 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f670c0044f0 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6708010b50 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 10 client.28531 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564241461c20 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6708096da0 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67080cfd20 con 0x564241567c70 2024-03-12T05:25:18.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 10 client.28531 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 10 client.28531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 20 client.28531 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5642415d89b0 0x5642415dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 4 client.28531 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.357+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f670c0044f0 con 0x5642415d89b0 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5642415d89b0 0x5642415dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f6716ffd640 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5642415d89b0 0x5642415dadd0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4de98837066cb5a server_cookie=ae3d3383af0fee6d in_seq=0 out_seq=0 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f66f6ffd640 10 client.28531 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f66f6ffd640 1 -- 172.21.15.45:0/2550857321 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f670c0044f0 con 0x5642415d89b0 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f66f6ffd640 10 client.28531 handle_command_reply: tid=0 2024-03-12T05:25:18.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.358+0000 7f6720794740 1 client.28531 shutdown 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 msgr2=0x564241578330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6708096a80 tx=0x7f6708005760 comp rx=0 tx=0).stop 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 shutdown_connections 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5642415d89b0 0x5642415dadd0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f66f6ffd640 0 client.28531 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564241568590 0x5642415766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564241571170 0x564241576c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 --2- 172.21.15.45:0/2550857321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564241567c70 0x564241578330 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 >> 172.21.15.45:0/2550857321 conn(0x5642415623a0 msgr2=0x564241558440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 shutdown_connections 2024-03-12T05:25:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 1 -- 172.21.15.45:0/2550857321 wait complete. 2024-03-12T05:25:18.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.359+0000 7f6720794740 20 client.28531 trim_cache size 0 max 16384 2024-03-12T05:25:18.359 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 msgr2=0x7f67000a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 0x7f67000a4160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f671000a510 tx=0x7f6710009650 comp rx=0 tx=0).stop 2024-03-12T05:25:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 msgr2=0x7f6718197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f670800e9f0 tx=0x7f670800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 shutdown_connections 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f67000a1ca0 0x7f67000a4160 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f671819be30 0x7f671819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718105170 0x7f6718197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 --2- 172.21.15.45:0/1554610270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67181000a0 0x7f6718197570 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 >> 172.21.15.45:0/1554610270 conn(0x7f6718076c30 msgr2=0x7f67180fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 shutdown_connections 2024-03-12T05:25:18.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:18.360+0000 7f671c95a640 1 -- 172.21.15.45:0/1554610270 wait complete. 2024-03-12T05:25:18.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:18 smithi045 ceph-mon[32174]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:25:18.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:18 smithi130 ceph-mon[35398]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:25:18.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:18 smithi116 ceph-mon[35486]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-03-12T05:25:19.373 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:25:19.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:25:19.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:25:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.555+0000 7ffb5a82c640 1 -- 172.21.15.45:0/1282467980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54112110 msgr2=0x7ffb54114500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.555+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/1282467980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54112110 0x7ffb54114500 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009be0 tx=0x7ffb4c02bf80 comp rx=0 tx=0).stop 2024-03-12T05:25:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 -- 172.21.15.45:0/1282467980 shutdown_connections 2024-03-12T05:25:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/1282467980 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54073e40 0x7ffb54111bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/1282467980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54112110 0x7ffb54114500 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/1282467980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb54073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 -- 172.21.15.45:0/1282467980 >> 172.21.15.45:0/1282467980 conn(0x7ffb540fcc10 msgr2=0x7ffb540ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 -- 172.21.15.45:0/1282467980 shutdown_connections 2024-03-12T05:25:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.556+0000 7ffb5a82c640 1 -- 172.21.15.45:0/1282467980 wait complete. 2024-03-12T05:25:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 Processor -- start 2024-03-12T05:25:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 -- start start 2024-03-12T05:25:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54073e40 0x7ffb541a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54112110 0x7ffb5419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.557+0000 7ffb5a82c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5419e0e0 con 0x7ffb54112110 2024-03-12T05:25:19.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb5a82c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5419e250 con 0x7ffb54073520 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb5a82c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5419e410 con 0x7ffb54073e40 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37870/0 (socket says 172.21.15.45:37870) 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 -- 172.21.15.45:0/4077284607 learned_addr learned my addr 172.21.15.45:0/4077284607 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb537fe640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54073e40 0x7ffb541a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54112110 0x7ffb5419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 -- 172.21.15.45:0/4077284607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54073e40 msgr2=0x7ffb541a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54073e40 0x7ffb541a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 -- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54112110 msgr2=0x7ffb5419da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54112110 0x7ffb5419da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.558+0000 7ffb53fff640 1 -- 172.21.15.45:0/4077284607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c009840 con 0x7ffb54073520 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb53fff640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400e9c0 tx=0x7ffb4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb517fa640 1 -- 172.21.15.45:0/4077284607 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4400ccc0 con 0x7ffb54073520 2024-03-12T05:25:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5419e6f0 con 0x7ffb54073520 2024-03-12T05:25:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb541aa980 con 0x7ffb54073520 2024-03-12T05:25:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb517fa640 1 -- 172.21.15.45:0/4077284607 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb4400ce20 con 0x7ffb54073520 2024-03-12T05:25:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.559+0000 7ffb517fa640 1 -- 172.21.15.45:0/4077284607 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44010430 con 0x7ffb54073520 2024-03-12T05:25:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.562+0000 7ffb5d3fd740 1 Processor -- start 2024-03-12T05:25:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb517fa640 1 -- 172.21.15.45:0/4077284607 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ffb44010590 con 0x7ffb54073520 2024-03-12T05:25:19.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 -- start start 2024-03-12T05:25:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f6e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f772e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7edcf90 con 0x558df7f6e380 2024-03-12T05:25:19.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7eda4c0 con 0x558df7f6ed30 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.563+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7ed5ca0 con 0x558df7f778b0 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59500/0 (socket says 172.21.15.45:59500) 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 -- 172.21.15.45:0/3293572507 learned_addr learned my addr 172.21.15.45:0/3293572507 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f6e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb53fff640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f772e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 -- 172.21.15.45:0/3293572507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 msgr2=0x558df7f772e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f772e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 -- 172.21.15.45:0/3293572507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 msgr2=0x558df7f6e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f6e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 -- 172.21.15.45:0/3293572507 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df7e980b0 con 0x558df7f778b0 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f6e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:19.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb537fe640 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009c20 tx=0x7ffb48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96f6ec447e49bfb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.564+0000 7ffb517fa640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 0x7ffb3c0a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.565+0000 7ffb3a7fc640 1 -- 172.21.15.45:0/3293572507 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48004050 con 0x558df7f778b0 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.565+0000 7ffb517fa640 1 -- 172.21.15.45:0/4077284607 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffb44014070 con 0x7ffb54073520 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.565+0000 7ffb3a7fc640 1 -- 172.21.15.45:0/3293572507 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb48005020 con 0x558df7f778b0 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.565+0000 7ffb3a7fc640 1 -- 172.21.15.45:0/3293572507 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48004c60 con 0x558df7f778b0 2024-03-12T05:25:19.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.565+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 0x7ffb3c0a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/3293572507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 msgr2=0x558df7f79d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 0x7ffb3c0a4120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c005ec0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009c20 tx=0x7ffb48011810 comp rx=0 tx=0).stop 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/3293572507 shutdown_connections 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f778b0 0x558df7f79d00 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f772e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/3293572507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f6e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/3293572507 >> 172.21.15.45:0/3293572507 conn(0x558df7f68840 msgr2=0x558df7f68c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/3293572507 shutdown_connections 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.566+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/3293572507 wait complete. 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.567+0000 7ffb5d3fd740 1 -- start start 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.567+0000 7ffb5d3fd740 1 Processor -- start 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.567+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.567+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.567+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 0x558df7f75230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7e7af40 con 0x558df7f778b0 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7ba94b0 con 0x558df7f6ed30 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7e5b670 con 0x558df7f6e380 2024-03-12T05:25:19.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37886/0 (socket says 172.21.15.45:37886) 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 -- 172.21.15.45:0/670704350 learned_addr learned my addr 172.21.15.45:0/670704350 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb53fff640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 0x558df7f75230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb5d3fd740 10 client.0 _ll_register_callbacks cb 0x558df7ec8f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 -- 172.21.15.45:0/670704350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 msgr2=0x558df7f747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 -- 172.21.15.45:0/670704350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 msgr2=0x558df7f75230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 0x558df7f75230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 -- 172.21.15.45:0/670704350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb48009850 con 0x558df7f6ed30 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 0x558df7f75230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb537fe640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb48004940 tx=0x7ffb48003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40dc01895462d5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.568+0000 7ffb337fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:19.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48003c20 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558df7eaff40 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffb48003d80 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb480305d0 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb5d3fd740 10 client.28586 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558df7ed4320 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.569+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558df7eaff40 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffb48021040 con 0x558df7f6ed30 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 10 client.28586 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 10 client.28586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 20 client.28586 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f87ec0 0x558df7f8a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 4 client.28586 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558df7ecc310 con 0x558df7f87ec0 2024-03-12T05:25:19.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb53fff640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f87ec0 0x558df7f8a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.570+0000 7ffb53fff640 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f87ec0 0x558df7f8a2e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754a979413224d56 server_cookie=3a042549e4486433 in_seq=0 out_seq=0 2024-03-12T05:25:19.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb337fe640 10 client.28586 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb337fe640 1 -- 172.21.15.45:0/670704350 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558df7ecc310 con 0x558df7f87ec0 2024-03-12T05:25:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb337fe640 10 client.28586 handle_command_reply: tid=0 2024-03-12T05:25:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb5d3fd740 1 client.28586 shutdown 2024-03-12T05:25:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 msgr2=0x558df7f74cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.571+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb48004940 tx=0x7ffb48003820 comp rx=0 tx=0).stop 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 shutdown_connections 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f87ec0 0x558df7f8a2e0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f778b0 0x558df7f75230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f6ed30 0x558df7f74cf0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/670704350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 >> 172.21.15.45:0/670704350 conn(0x558df7f68840 msgr2=0x558df7f6a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb337fe640 0 client.28586 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 shutdown_connections 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/670704350 wait complete. 2024-03-12T05:25:19.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.572+0000 7ffb5d3fd740 20 client.28586 trim_cache size 0 max 16384 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 Processor -- start 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 -- start start 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f71a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 0x558df7f71f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7f75b00 con 0x558df7f6ed30 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7f69410 con 0x558df7f68840 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7f7a3c0 con 0x558df7f6e380 2024-03-12T05:25:19.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb58dc9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 0x558df7f71f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb537fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f71a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37892/0 (socket says 172.21.15.45:37892) 2024-03-12T05:25:19.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.600+0000 7ffb53fff640 1 -- 172.21.15.45:0/2030936277 learned_addr learned my addr 172.21.15.45:0/2030936277 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 -- 172.21.15.45:0/2030936277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 msgr2=0x558df7f71a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f71a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 -- 172.21.15.45:0/2030936277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 msgr2=0x558df7f76ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 -- 172.21.15.45:0/2030936277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df7e70c00 con 0x558df7f6ed30 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb53fff640 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 0x558df7f71f50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c036040 tx=0x7ffb4c040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5a74e5aff8f447 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb39ffb640 1 -- 172.21.15.45:0/2030936277 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c033e90 con 0x558df7f6ed30 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb39ffb640 1 -- 172.21.15.45:0/2030936277 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ffb4c042070 con 0x558df7f6ed30 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.601+0000 7ffb39ffb640 1 -- 172.21.15.45:0/2030936277 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c034690 con 0x558df7f6ed30 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/2030936277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 msgr2=0x558df7f71f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 0x558df7f71f50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c036040 tx=0x7ffb4c040000 comp rx=0 tx=0).stop 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/2030936277 shutdown_connections 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6ed30 0x558df7f71f50 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6e380 0x558df7f71a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/2030936277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f76ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.602+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/2030936277 >> 172.21.15.45:0/2030936277 conn(0x558df7f6f480 msgr2=0x558df7f73080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.603+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/2030936277 shutdown_connections 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.603+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/2030936277 wait complete. 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.603+0000 7ffb5d3fd740 1 -- start start 2024-03-12T05:25:19.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.603+0000 7ffb5d3fd740 1 Processor -- start 2024-03-12T05:25:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f7e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6ed30 0x558df7fd8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7fd8950 con 0x558df7f6e380 2024-03-12T05:25:19.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7fd8ac0 con 0x558df7f68840 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df7fd8c80 con 0x558df7f6ed30 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb5d3fd740 10 client.0 _ll_register_callbacks cb 0x7ffb480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37898/0 (socket says 172.21.15.45:37898) 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 -- 172.21.15.45:0/921175154 learned_addr learned my addr 172.21.15.45:0/921175154 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb58dc9640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6ed30 0x558df7fd8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 -- 172.21.15.45:0/921175154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6ed30 msgr2=0x558df7fd8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6ed30 0x558df7fd8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb53fff640 1 -- 172.21.15.45:0/921175154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 msgr2=0x558df7f7e0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.604+0000 7ffb537fe640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f7e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb53fff640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f7e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb53fff640 1 -- 172.21.15.45:0/921175154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb541aa980 con 0x558df7f68840 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb537fe640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f7e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:25:19.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb53fff640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c6a0 tx=0x7ffb44005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b5a00285655144 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb337fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb440a2e30 con 0x558df7f68840 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ffb44096d00 con 0x558df7f68840 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.605+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558df7efb900 con 0x558df7f68840 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.606+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb44096ec0 con 0x558df7f68840 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.606+0000 7ffb5d3fd740 10 client.28592 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:19.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.606+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7ffb54114e30 con 0x558df7f68840 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558df7efb900 con 0x558df7f68840 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ffb4401f060 con 0x558df7f68840 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 10 client.28592 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 10 client.28592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 20 client.28592 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f778b0 0x558df7fe2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 4 client.28592 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.607+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558df7bd5860 con 0x558df7f778b0 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.608+0000 7ffb537fe640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f778b0 0x558df7fe2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.609+0000 7ffb337fe640 10 client.28592 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.609+0000 7ffb537fe640 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f778b0 0x558df7fe2540 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50a72c17dbe46044 server_cookie=eb055cc9ae0bb1f4 in_seq=0 out_seq=0 2024-03-12T05:25:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.610+0000 7ffb337fe640 1 -- 172.21.15.45:0/921175154 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558df7bd5860 con 0x558df7f778b0 2024-03-12T05:25:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.610+0000 7ffb337fe640 10 client.28592 handle_command_reply: tid=0 2024-03-12T05:25:19.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.610+0000 7ffb5d3fd740 1 client.28592 shutdown 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 msgr2=0x558df7f7dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c6a0 tx=0x7ffb44005880 comp rx=0 tx=0).stop 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 shutdown_connections 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558df7f778b0 0x558df7fe2540 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558df7f6e380 0x558df7f7e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558df7f6ed30 0x558df7fd8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 --2- 172.21.15.45:0/921175154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558df7f68840 0x558df7f7dbb0 secure :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c6a0 tx=0x7ffb44005880 comp rx=0 tx=0).stop 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 >> 172.21.15.45:0/921175154 conn(0x558df7f6f480 msgr2=0x558df7f67050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb337fe640 0 client.28592 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.611+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 shutdown_connections 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.612+0000 7ffb5d3fd740 1 -- 172.21.15.45:0/921175154 wait complete. 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.612+0000 7ffb5d3fd740 20 client.28592 trim_cache size 0 max 16384 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "eeaddff0-2729-44f5-814a-2a15b8dfb810", 2024-03-12T05:25:19.608 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:25:19.609 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:19.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.613+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 msgr2=0x7ffb3c0a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.613+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 0x7ffb3c0a4120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c006010 tx=0x7ffb4c005ec0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.613+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 msgr2=0x7ffb541a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.613+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400e9c0 tx=0x7ffb4400ee80 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 shutdown_connections 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ffb3c0a1c60 0x7ffb3c0a4120 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb54112110 0x7ffb5419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb54073e40 0x7ffb541a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 --2- 172.21.15.45:0/4077284607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb54073520 0x7ffb541a38e0 secure :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400e9c0 tx=0x7ffb4400ee80 comp rx=0 tx=0).stop 2024-03-12T05:25:19.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.614+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 >> 172.21.15.45:0/4077284607 conn(0x7ffb540fcc10 msgr2=0x7ffb540fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.618+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 shutdown_connections 2024-03-12T05:25:19.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.618+0000 7ffb5a82c640 1 -- 172.21.15.45:0/4077284607 wait complete. 2024-03-12T05:25:19.634 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:25:19.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:25:19.678 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:19 smithi045 ceph-mon[32174]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.9 MiB/s wr, 1.18k op/s 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 -- 172.21.15.45:0/317833227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c0756b0 msgr2=0x7f2c8c075a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 --2- 172.21.15.45:0/317833227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c0756b0 0x7f2c8c075a90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009ca0 tx=0x7f2c8402c120 comp rx=0 tx=0).stop 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 -- 172.21.15.45:0/317833227 shutdown_connections 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 --2- 172.21.15.45:0/317833227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c111e00 0x7f2c8c1141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 --2- 172.21.15.45:0/317833227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c1118c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 --2- 172.21.15.45:0/317833227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c0756b0 0x7f2c8c075a90 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 -- 172.21.15.45:0/317833227 >> 172.21.15.45:0/317833227 conn(0x7f2c8c0febf0 msgr2=0x7f2c8c101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 -- 172.21.15.45:0/317833227 shutdown_connections 2024-03-12T05:25:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.832+0000 7f2c90975640 1 -- 172.21.15.45:0/317833227 wait complete. 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.833+0000 7f2c90975640 1 Processor -- start 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.833+0000 7f2c90975640 1 -- start start 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c111e00 0x7f2c8c19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 0x7f2c8c1a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1a3050 con 0x7f2c8c075fd0 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1a31c0 con 0x7f2c8c111e00 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c90975640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1a3330 con 0x7f2c8c1a2660 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 0x7f2c8c1a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43434/0 (socket says 172.21.15.45:43434) 2024-03-12T05:25:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/2391428186 learned_addr learned my addr 172.21.15.45:0/2391428186 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.834+0000 7f2c8affd640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c111e00 0x7f2c8c19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 -- 172.21.15.45:0/2391428186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c111e00 msgr2=0x7f2c8c19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c111e00 0x7f2c8c19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 -- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 msgr2=0x7f2c8c19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 -- 172.21.15.45:0/2391428186 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84009900 con 0x7f2c8c1a2660 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.835+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 0x7f2c8c1a2b10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c06a2e0 tx=0x7f2c8000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.836+0000 7f2c88ff9640 1 -- 172.21.15.45:0/2391428186 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c80011830 con 0x7f2c8c1a2660 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.836+0000 7f2c88ff9640 1 -- 172.21.15.45:0/2391428186 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c80011e40 con 0x7f2c8c1a2660 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.836+0000 7f2c88ff9640 1 -- 172.21.15.45:0/2391428186 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c80010ea0 con 0x7f2c8c1a2660 2024-03-12T05:25:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.836+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c1aa910 con 0x7f2c8c1a2660 2024-03-12T05:25:19.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.836+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c1aad70 con 0x7f2c8c1a2660 2024-03-12T05:25:19.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.839+0000 7f2c88ff9640 1 -- 172.21.15.45:0/2391428186 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2c8001a440 con 0x7f2c8c1a2660 2024-03-12T05:25:19.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.840+0000 7f2c88ff9640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 0x7f2c740a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.840+0000 7f2c88ff9640 1 -- 172.21.15.45:0/2391428186 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2c800ce0b0 con 0x7f2c8c1a2660 2024-03-12T05:25:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.840+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 0x7f2c740a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.841+0000 7f2c947a9740 1 Processor -- start 2024-03-12T05:25:19.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.841+0000 7f2c947a9740 1 -- start start 2024-03-12T05:25:19.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.841+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 0x7f2c740a4160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006010 tx=0x7f2c84005d80 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.841+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c947a9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21cd80280 con 0x55f21ce11df0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c947a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21cd7d7b0 con 0x55f21ce114d0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c947a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21cd78f90 con 0x55f21ce1a9d0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:37922/0 (socket says 172.21.15.45:37922) 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59570/0 (socket says 172.21.15.45:59570) 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/3558863107 learned_addr learned my addr 172.21.15.45:0/3558863107 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 -- 172.21.15.45:0/3558863107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 msgr2=0x55f21ce118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce118b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 -- 172.21.15.45:0/3558863107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 msgr2=0x55f21ce1a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.842+0000 7f2c8bfff640 1 -- 172.21.15.45:0/3558863107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f21cd37570 con 0x55f21ce1a9d0 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.843+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2c800cb880 tx=0x7f2c8009a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dad32d6c514d17d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.843+0000 7f2c71ffb640 1 -- 172.21.15.45:0/3558863107 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8009e030 con 0x55f21ce1a9d0 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.843+0000 7f2c71ffb640 1 -- 172.21.15.45:0/3558863107 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c8009ad60 con 0x55f21ce1a9d0 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.843+0000 7f2c71ffb640 1 -- 172.21.15.45:0/3558863107 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c800cb9f0 con 0x55f21ce1a9d0 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 -- 172.21.15.45:0/3558863107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 msgr2=0x55f21ce1cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2c800cb880 tx=0x7f2c8009a6c0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 -- 172.21.15.45:0/3558863107 shutdown_connections 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce1cdc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 --2- 172.21.15.45:0/3558863107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce118b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.844+0000 7f2c947a9740 1 -- 172.21.15.45:0/3558863107 >> 172.21.15.45:0/3558863107 conn(0x55f21ce0bc00 msgr2=0x55f21ce0bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.845+0000 7f2c947a9740 1 -- 172.21.15.45:0/3558863107 shutdown_connections 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.845+0000 7f2c947a9740 1 -- 172.21.15.45:0/3558863107 wait complete. 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.845+0000 7f2c947a9740 1 -- start start 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.845+0000 7f2c947a9740 1 Processor -- start 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce11df0 0x55f21ce177f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 0x55f21ce17ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ca4c430 con 0x55f21ce1a9d0 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ca08e00 con 0x55f21ce11df0 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ccfdfd0 con 0x55f21ce114d0 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c947a9740 10 client.0 _ll_register_callbacks cb 0x55f21cd53fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:19.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce11df0 0x55f21ce177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59578/0 (socket says 172.21.15.45:59578) 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/598443955 learned_addr learned my addr 172.21.15.45:0/598443955 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 0x55f21ce17ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce11df0 msgr2=0x55f21ce177f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce11df0 0x55f21ce177f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/598443955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 msgr2=0x55f21ce17ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.846+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 0x55f21ce17ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c8b7fe640 1 -- 172.21.15.45:0/598443955 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c1aad70 con 0x55f21ce114d0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 0x55f21ce17ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2c840325a0 tx=0x7f2c8402c120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dec74884dfba7a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c84003d80 con 0x55f21ce114d0 2024-03-12T05:25:19.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f21ca787e0 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2c84046030 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c84009780 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c947a9740 10 client.28558 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.847+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f21cd77610 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55f21ca787e0 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2c8403f510 con 0x55f21ce114d0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 10 client.28558 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 10 client.28558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 20 client.28558 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce3bcc0 0x55f21ce3e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 4 client.28558 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f21cab5720 con 0x55f21ce3bcc0 2024-03-12T05:25:19.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.848+0000 7f2c8affd640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce3bcc0 0x55f21ce3e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.849+0000 7f2c8affd640 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce3bcc0 0x55f21ce3e0e0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9caefedad71358 server_cookie=bb75041690ed7f7a in_seq=0 out_seq=0 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.849+0000 7f2c6affd640 10 client.28558 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c6affd640 1 -- 172.21.15.45:0/598443955 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f21cab5720 con 0x55f21ce3bcc0 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c6affd640 10 client.28558 handle_command_reply: tid=0 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 client.28558 shutdown 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 msgr2=0x55f21ce1a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2c840325a0 tx=0x7f2c8402c120 comp rx=0 tx=0).stop 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 shutdown_connections 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce3bcc0 0x55f21ce3e0e0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce1a9d0 0x55f21ce17ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce11df0 0x55f21ce177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 --2- 172.21.15.45:0/598443955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce1a460 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 >> 172.21.15.45:0/598443955 conn(0x55f21ce0bc00 msgr2=0x55f21ce0d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c6affd640 0 client.28558 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.850+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 shutdown_connections 2024-03-12T05:25:19.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.851+0000 7f2c947a9740 1 -- 172.21.15.45:0/598443955 wait complete. 2024-03-12T05:25:19.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.851+0000 7f2c947a9740 20 client.28558 trim_cache size 0 max 16384 2024-03-12T05:25:19.874 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.879+0000 7f2c947a9740 1 Processor -- start 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.879+0000 7f2c947a9740 1 -- start start 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.879+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce19f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce32890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c947a9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ce1d1c0 con 0x55f21ce11df0 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43454/0 (socket says 172.21.15.45:43454) 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c8affd640 1 -- 172.21.15.45:0/1884184425 learned_addr learned my addr 172.21.15.45:0/1884184425 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ce07300 con 0x55f21ce114d0 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce32890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.875 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.880+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21cab5720 con 0x55f21ce1a9d0 2024-03-12T05:25:19.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 -- 172.21.15.45:0/1884184425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 msgr2=0x55f21ce32890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce32890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.876 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 -- 172.21.15.45:0/1884184425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 msgr2=0x55f21ce19f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:25:19.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce19f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 -- 172.21.15.45:0/1884184425 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f21cd13ac0 con 0x55f21ce11df0 2024-03-12T05:25:19.877 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.881+0000 7f2c8affd640 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c004650 tx=0x7f2c7c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258a2be28e6640c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.882+0000 7f2c717fa640 1 -- 172.21.15.45:0/1884184425 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c016030 con 0x55f21ce11df0 2024-03-12T05:25:19.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.882+0000 7f2c717fa640 1 -- 172.21.15.45:0/1884184425 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c7c017040 con 0x55f21ce11df0 2024-03-12T05:25:19.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.882+0000 7f2c717fa640 1 -- 172.21.15.45:0/1884184425 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0116a0 con 0x55f21ce11df0 2024-03-12T05:25:19.878 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.883+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 msgr2=0x55f21ce1a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.883+0000 7f2c947a9740 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c004650 tx=0x7f2c7c00b390 comp rx=0 tx=0).stop 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 shutdown_connections 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce1a460 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce1a9d0 0x55f21ce32890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 --2- 172.21.15.45:0/1884184425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce114d0 0x55f21ce19f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.879 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 >> 172.21.15.45:0/1884184425 conn(0x55f21ce0bc00 msgr2=0x55f21ce06bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 shutdown_connections 2024-03-12T05:25:19.880 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 -- 172.21.15.45:0/1884184425 wait complete. 2024-03-12T05:25:19.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.884+0000 7f2c947a9740 1 -- start start 2024-03-12T05:25:19.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.885+0000 7f2c947a9740 1 Processor -- start 2024-03-12T05:25:19.881 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.885+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce31b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.885+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.885+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce31b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.885+0000 7f2c8b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce31b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:59586/0 (socket says 172.21.15.45:59586) 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c947a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce1a9d0 0x55f21ce8c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c947a9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ce8c6e0 con 0x55f21ce11df0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c947a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ce8c850 con 0x55f21ce1a9d0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c947a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f21ce8ca10 con 0x55f21ce114d0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43464/0 (socket says 172.21.15.45:43464) 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 -- 172.21.15.45:0/2706199744 learned_addr learned my addr 172.21.15.45:0/2706199744 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c947a9740 10 client.0 _ll_register_callbacks cb 0x7f2c800108a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8bfff640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce1a9d0 0x55f21ce8c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.882 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 -- 172.21.15.45:0/2706199744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 msgr2=0x55f21ce31b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce31b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 -- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce1a9d0 msgr2=0x55f21ce8c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce1a9d0 0x55f21ce8c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 -- 172.21.15.45:0/2706199744 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c0043f0 con 0x55f21ce11df0 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c8affd640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00b890 tx=0x7f2c7c01f320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a00cd5b2d59e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.886+0000 7f2c6affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.887+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c016040 con 0x55f21ce11df0 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.887+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2c7c01f6a0 con 0x55f21ce11df0 2024-03-12T05:25:19.883 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.887+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c8009ad60 con 0x55f21ce11df0 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.888+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c7c01f860 con 0x55f21ce11df0 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.888+0000 7f2c947a9740 10 client.19149 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.888+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f21cd396a0 con 0x55f21ce11df0 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.888+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2c8009ad60 con 0x55f21ce11df0 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2c7c02e020 con 0x55f21ce11df0 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 10 client.19149 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 10 client.19149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 20 client.19149 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:19.884 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce940c0 0x55f21ce964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:19.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 4 client.19149 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:25:19.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.889+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f21cb04890 con 0x55f21ce940c0 2024-03-12T05:25:19.885 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.890+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce940c0 0x55f21ce964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:19.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.892+0000 7f2c8b7fe640 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce940c0 0x55f21ce964e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b79b5c9f936b14a server_cookie=f18889f330130321 in_seq=0 out_seq=0 2024-03-12T05:25:19.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.892+0000 7f2c6affd640 10 client.19149 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.892+0000 7f2c6affd640 1 -- 172.21.15.45:0/2706199744 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f21cb04890 con 0x55f21ce940c0 2024-03-12T05:25:19.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.892+0000 7f2c6affd640 10 client.19149 handle_command_reply: tid=0 2024-03-12T05:25:19.886 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.892+0000 7f2c947a9740 1 client.19149 shutdown 2024-03-12T05:25:19.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 msgr2=0x55f21ce32050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00b890 tx=0x7f2c7c01f320 comp rx=0 tx=0).stop 2024-03-12T05:25:19.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 shutdown_connections 2024-03-12T05:25:19.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55f21ce940c0 0x55f21ce964e0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f21ce11df0 0x55f21ce32050 secure :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00b890 tx=0x7f2c7c01f320 comp rx=0 tx=0).stop 2024-03-12T05:25:19.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f21ce1a9d0 0x55f21ce8c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 --2- 172.21.15.45:0/2706199744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f21ce114d0 0x55f21ce31b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.894+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 >> 172.21.15.45:0/2706199744 conn(0x55f21ce0bc00 msgr2=0x55f21ce12ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.895+0000 7f2c6affd640 0 client.19149 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.895+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 shutdown_connections 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.895+0000 7f2c947a9740 1 -- 172.21.15.45:0/2706199744 wait complete. 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.895+0000 7f2c947a9740 20 client.19149 trim_cache size 0 max 16384 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:25:19.890 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: "eeaddff0-2729-44f5-814a-2a15b8dfb810": { 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "eeaddff0-2729-44f5-814a-2a15b8dfb810", 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:25:19.891 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 msgr2=0x7f2c740a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 0x7f2c740a4160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006010 tx=0x7f2c84005d80 comp rx=0 tx=0).stop 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 msgr2=0x7f2c8c1a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 0x7f2c8c1a2b10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c06a2e0 tx=0x7f2c8000b870 comp rx=0 tx=0).stop 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 shutdown_connections 2024-03-12T05:25:19.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f2c740a1ca0 0x7f2c740a4160 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c8c1a2660 0x7f2c8c1a2b10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c8c111e00 0x7f2c8c19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 --2- 172.21.15.45:0/2391428186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c8c075fd0 0x7f2c8c19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.896+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 >> 172.21.15.45:0/2391428186 conn(0x7f2c8c0febf0 msgr2=0x7f2c8c100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.897+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 shutdown_connections 2024-03-12T05:25:19.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:19.897+0000 7f2c90975640 1 -- 172.21.15.45:0/2391428186 wait complete. 2024-03-12T05:25:19.910 INFO:tasks.cephfs.filesystem:scrub status for tag:eeaddff0-2729-44f5-814a-2a15b8dfb810 - {'path': '/', 'tag': 'eeaddff0-2729-44f5-814a-2a15b8dfb810', 'options': 'recursive,force'} 2024-03-12T05:25:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 795.0 s, 1241.8 tps, lat 80.376 ms stddev 32.746 2024-03-12T05:25:19.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:19 smithi130 ceph-mon[35398]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.9 MiB/s wr, 1.18k op/s 2024-03-12T05:25:19.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:19 smithi116 ceph-mon[35486]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.9 MiB/s wr, 1.18k op/s 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:20 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.967 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:20 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:20.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:20 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:21.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:21 smithi045 ceph-mon[32174]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:21.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:21 smithi130 ceph-mon[35398]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:21.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:21 smithi116 ceph-mon[35486]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:24.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:23 smithi045 ceph-mon[32174]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:23 smithi130 ceph-mon[35398]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:24.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:23 smithi116 ceph-mon[35486]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-03-12T05:25:24.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 800.0 s, 1351.2 tps, lat 73.865 ms stddev 46.325 2024-03-12T05:25:26.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:25 smithi045 ceph-mon[32174]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:25:26.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:25 smithi130 ceph-mon[35398]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:25:26.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:25 smithi116 ceph-mon[35486]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 11 MiB/s wr, 1.24k op/s 2024-03-12T05:25:28.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:27 smithi045 ceph-mon[32174]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 8.9 MiB/s wr, 1.05k op/s 2024-03-12T05:25:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:27 smithi130 ceph-mon[35398]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 8.9 MiB/s wr, 1.05k op/s 2024-03-12T05:25:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:27 smithi116 ceph-mon[35486]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 8.9 MiB/s wr, 1.05k op/s 2024-03-12T05:25:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 805.0 s, 1224.6 tps, lat 82.011 ms stddev 29.726 2024-03-12T05:25:30.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:29 smithi045 ceph-mon[32174]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 10 MiB/s wr, 1.19k op/s 2024-03-12T05:25:30.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:29 smithi130 ceph-mon[35398]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 10 MiB/s wr, 1.19k op/s 2024-03-12T05:25:30.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:29 smithi116 ceph-mon[35486]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 10 MiB/s wr, 1.19k op/s 2024-03-12T05:25:32.017 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:31 smithi130 ceph-mon[35398]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:25:32.017 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:31 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:25:32.042 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:31 smithi116 ceph-mon[35486]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:25:32.042 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:31 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:25:32.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:31 smithi045 ceph-mon[32174]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-03-12T05:25:32.126 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:31 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:25:33.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:32 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:25:33.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:32 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:25:33.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:32 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:25:33.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:32 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:25:33.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:32 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:25:33.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:32 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:25:33.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:32 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:25:33.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:32 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:25:33.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:32 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:25:33.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:32 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:25:33.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:32 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:25:33.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:32 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:25:33.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:32 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:25:33.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:32 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:25:33.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:32 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:25:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:33 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:25:34.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:33 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:25:34.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:33 smithi045 ceph-mon[32174]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:25:34.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:33 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:25:34.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:33 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:25:34.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:33 smithi130 ceph-mon[35398]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:25:34.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:33 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:25:34.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:33 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:25:34.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:33 smithi116 ceph-mon[35486]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:25:34.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 810.0 s, 1297.0 tps, lat 77.038 ms stddev 34.886 2024-03-12T05:25:36.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:35 smithi045 ceph-mon[32174]: pgmap v462: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 15 MiB/s wr, 1.57k op/s 2024-03-12T05:25:36.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:35 smithi130 ceph-mon[35398]: pgmap v462: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 15 MiB/s wr, 1.57k op/s 2024-03-12T05:25:36.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:35 smithi116 ceph-mon[35486]: pgmap v462: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 15 MiB/s wr, 1.57k op/s 2024-03-12T05:25:38.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:37 smithi045 ceph-mon[32174]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 9.7 MiB/s wr, 1.00k op/s 2024-03-12T05:25:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:37 smithi130 ceph-mon[35398]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 9.7 MiB/s wr, 1.00k op/s 2024-03-12T05:25:38.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:37 smithi116 ceph-mon[35486]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 9.7 MiB/s wr, 1.00k op/s 2024-03-12T05:25:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 815.0 s, 1365.6 tps, lat 72.893 ms stddev 28.297 2024-03-12T05:25:40.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:39 smithi045 ceph-mon[32174]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:25:40.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:39 smithi130 ceph-mon[35398]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:25:40.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:39 smithi116 ceph-mon[35486]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-03-12T05:25:42.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:41 smithi045 ceph-mon[32174]: pgmap v465: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:25:42.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:41 smithi130 ceph-mon[35398]: pgmap v465: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:25:42.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:41 smithi116 ceph-mon[35486]: pgmap v465: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:25:44.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:43 smithi045 ceph-mon[32174]: pgmap v466: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-03-12T05:25:44.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:43 smithi130 ceph-mon[35398]: pgmap v466: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-03-12T05:25:44.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:43 smithi116 ceph-mon[35486]: pgmap v466: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-03-12T05:25:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 820.0 s, 1293.0 tps, lat 77.559 ms stddev 41.247 2024-03-12T05:25:46.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:45 smithi045 ceph-mon[32174]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-03-12T05:25:46.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:45 smithi130 ceph-mon[35398]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-03-12T05:25:46.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:45 smithi116 ceph-mon[35486]: pgmap v467: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 15 MiB/s wr, 1.61k op/s 2024-03-12T05:25:48.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:47 smithi045 ceph-mon[32174]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:48.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:47 smithi130 ceph-mon[35398]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:48.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:47 smithi116 ceph-mon[35486]: pgmap v468: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 9.3 MiB/s wr, 1.08k op/s 2024-03-12T05:25:49.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:25:49.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:25:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 825.0 s, 1391.6 tps, lat 71.881 ms stddev 49.100 2024-03-12T05:25:50.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.089+0000 7fc3d9f81640 1 -- 172.21.15.45:0/3746755182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d4108660 msgr2=0x7fc3d4110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.089+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/3746755182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d4108660 0x7fc3d4110230 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009930 tx=0x7fc3c402bc30 comp rx=0 tx=0).stop 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 -- 172.21.15.45:0/3746755182 shutdown_connections 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/3746755182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d4108660 0x7fc3d4110230 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/3746755182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103f40 0x7fc3d4107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/3746755182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4103590 0x7fc3d4103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 -- 172.21.15.45:0/3746755182 >> 172.21.15.45:0/3746755182 conn(0x7fc3d40fee30 msgr2=0x7fc3d4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 -- 172.21.15.45:0/3746755182 shutdown_connections 2024-03-12T05:25:50.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.090+0000 7fc3d9f81640 1 -- 172.21.15.45:0/3746755182 wait complete. 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 Processor -- start 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 -- start start 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 0x7fc3d4072820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 0x7fc3d406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d406c530 con 0x7fc3d406b8f0 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d406c6a0 con 0x7fc3d4108660 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d9f81640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4070340 con 0x7fc3d4103590 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.091+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33936/0 (socket says 172.21.15.45:33936) 2024-03-12T05:25:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/4181984805 learned_addr learned my addr 172.21.15.45:0/4181984805 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 0x7fc3d4072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 0x7fc3d406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/4181984805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 msgr2=0x7fc3d4072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 0x7fc3d4072820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 msgr2=0x7fc3d406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 0x7fc3d406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/4181984805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4009560 con 0x7fc3d4108660 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 0x7fc3d4072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 0x7fc3d406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.092+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00ebc0 tx=0x7fc3bc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.093+0000 7fc3d0ff9640 1 -- 172.21.15.45:0/4181984805 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc004d20 con 0x7fc3d4108660 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.093+0000 7fc3d0ff9640 1 -- 172.21.15.45:0/4181984805 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3bc00eec0 con 0x7fc3d4108660 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.093+0000 7fc3d0ff9640 1 -- 172.21.15.45:0/4181984805 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc010690 con 0x7fc3d4108660 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.093+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d4070620 con 0x7fc3d4108660 2024-03-12T05:25:50.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.093+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d4070b30 con 0x7fc3d4108660 2024-03-12T05:25:50.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.096+0000 7fc3d0ff9640 1 -- 172.21.15.45:0/4181984805 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc3bc005020 con 0x7fc3d4108660 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.096+0000 7fc3dcc32740 1 Processor -- start 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 -- start start 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d0ff9640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 0x7fc3c00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d0ff9640 1 -- 172.21.15.45:0/4181984805 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc3bc014070 con 0x7fc3d4108660 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600c5fc0 0x55e3600c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e360023280 con 0x55e3600c5fc0 2024-03-12T05:25:50.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600207b0 con 0x55e3600b4900 2024-03-12T05:25:50.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e36001bf90 con 0x55e3600b3fe0 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33952/0 (socket says 172.21.15.45:33952) 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1936104075 learned_addr learned my addr 172.21.15.45:0/1936104075 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600c5fc0 0x55e3600c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 0x7fc3c00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1936104075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 msgr2=0x55e3600b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1936104075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600c5fc0 msgr2=0x55e3600c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600c5fc0 0x55e3600c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.097+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1936104075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e35ffda570 con 0x55e3600b4900 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc096fc0 tx=0x7fc3bc09a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ff88bfdcd918fa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 0x7fc3c00a40e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4036040 tx=0x7fc3c4005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3c9ffb640 1 -- 172.21.15.45:0/1936104075 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0b2070 con 0x55e3600b4900 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3c9ffb640 1 -- 172.21.15.45:0/1936104075 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3bc09be90 con 0x55e3600b4900 2024-03-12T05:25:50.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.098+0000 7fc3c9ffb640 1 -- 172.21.15.45:0/1936104075 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0cb900 con 0x55e3600b4900 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1936104075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 msgr2=0x55e3600c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc096fc0 tx=0x7fc3bc09a580 comp rx=0 tx=0).stop 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1936104075 shutdown_connections 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600c5fc0 0x55e3600c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600c5a80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1936104075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1936104075 >> 172.21.15.45:0/1936104075 conn(0x55e3600ae990 msgr2=0x55e3600aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.099+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1936104075 shutdown_connections 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.100+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1936104075 wait complete. 2024-03-12T05:25:50.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.100+0000 7fc3dcc32740 1 -- start start 2024-03-12T05:25:50.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.100+0000 7fc3dcc32740 1 Processor -- start 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 0x55e3600c3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e35fcef430 con 0x55e3600b4900 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e35fcabe00 con 0x55e3600c5fc0 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e35ffa0fd0 con 0x55e3600b3fe0 2024-03-12T05:25:50.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3dcc32740 10 client.0 _ll_register_callbacks cb 0x55e35fff6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.101+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57804/0 (socket says 172.21.15.45:57804) 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/834694085 learned_addr learned my addr 172.21.15.45:0/834694085 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 0x55e3600c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 -- 172.21.15.45:0/834694085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 msgr2=0x55e3600c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 -- 172.21.15.45:0/834694085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 msgr2=0x55e3600c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 -- 172.21.15.45:0/834694085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d4070b30 con 0x55e3600c5fc0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 0x55e3600c3540 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc3c402c170 tx=0x7fc3c402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f952370d24a6a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.102+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4005520 con 0x55e3600c5fc0 2024-03-12T05:25:50.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc3c402de40 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e35fd1b7e0 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3c4005680 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 10 client.28619 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e36001a610 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55e35fd1b7e0 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc3c404f5a0 con 0x55e3600c5fc0 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 10 client.28619 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 10 client.28619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.103+0000 7fc3dcc32740 20 client.28619 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.104+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e3600d62d0 0x55e3600d86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.104+0000 7fc3dcc32740 4 client.28619 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:50.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.104+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e35fd58720 con 0x55e3600d62d0 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.104+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e3600d62d0 0x55e3600d86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.105+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e3600d62d0 0x55e3600d86c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc47bc40a2147c1c server_cookie=4094687c1c5ef0af in_seq=0 out_seq=0 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.105+0000 7fc3b2ffd640 10 client.28619 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.106+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/834694085 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e35fd58720 con 0x55e3600d62d0 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.106+0000 7fc3b2ffd640 10 client.28619 handle_command_reply: tid=0 2024-03-12T05:25:50.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.106+0000 7fc3dcc32740 1 client.28619 shutdown 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.106+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 msgr2=0x55e3600c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.106+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 0x55e3600c3540 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc3c402c170 tx=0x7fc3c402bc30 comp rx=0 tx=0).stop 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 shutdown_connections 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e3600d62d0 0x55e3600d86c0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b4900 0x55e3600c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600c5fc0 0x55e3600c3540 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/834694085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600b3fe0 0x55e3600c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 >> 172.21.15.45:0/834694085 conn(0x55e3600ae990 msgr2=0x55e3600b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3b2ffd640 0 client.28619 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 shutdown_connections 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.107+0000 7fc3dcc32740 1 -- 172.21.15.45:0/834694085 wait complete. 2024-03-12T05:25:50.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.108+0000 7fc3dcc32740 20 client.28619 trim_cache size 0 max 16384 2024-03-12T05:25:50.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:49 smithi045 ceph-mon[32174]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:25:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.138+0000 7fc3dcc32740 1 Processor -- start 2024-03-12T05:25:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.138+0000 7fc3dcc32740 1 -- start start 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600ae240 con 0x55e3600b3fe0 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600aa320 con 0x55e3600b4900 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600c42e0 con 0x55e3600c5fc0 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57806/0 (socket says 172.21.15.45:57806) 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d37fe640 1 -- 172.21.15.45:0/1115623797 learned_addr learned my addr 172.21.15.45:0/1115623797 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1115623797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 msgr2=0x55e3600cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1115623797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 msgr2=0x55e3600c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.139+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/1115623797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e35ffb6ac0 con 0x55e3600b4900 2024-03-12T05:25:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.140+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cce80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0cbc80 tx=0x7fc3bc096f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b8c8b4b82167d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.140+0000 7fc3c97fa640 1 -- 172.21.15.45:0/1115623797 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0b2070 con 0x55e3600b4900 2024-03-12T05:25:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.140+0000 7fc3c97fa640 1 -- 172.21.15.45:0/1115623797 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3bc09ecb0 con 0x55e3600b4900 2024-03-12T05:25:50.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.140+0000 7fc3c97fa640 1 -- 172.21.15.45:0/1115623797 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0971d0 con 0x55e3600b4900 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.141+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1115623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 msgr2=0x55e3600cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.141+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cce80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0cbc80 tx=0x7fc3bc096f30 comp rx=0 tx=0).stop 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.141+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1115623797 shutdown_connections 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cce80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/1115623797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1115623797 >> 172.21.15.45:0/1115623797 conn(0x55e3600ae990 msgr2=0x55e3600c1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1115623797 shutdown_connections 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 -- 172.21.15.45:0/1115623797 wait complete. 2024-03-12T05:25:50.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.142+0000 7fc3dcc32740 1 -- start start 2024-03-12T05:25:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 Processor -- start 2024-03-12T05:25:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600cc300 con 0x55e3600b3fe0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3600cc470 con 0x55e3600b4900 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e36012c7b0 con 0x55e3600c5fc0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3dcc32740 10 client.0 _ll_register_callbacks cb 0x7fc3bc00ce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33974/0 (socket says 172.21.15.45:33974) 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/703206671 learned_addr learned my addr 172.21.15.45:0/703206671 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3d3fff640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.143+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/703206671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 msgr2=0x55e3600cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/703206671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 msgr2=0x55e3600cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 -- 172.21.15.45:0/703206671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e35ffda570 con 0x55e3600b4900 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:50.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3d2ffd640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0ce080 tx=0x7fc3bc099070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c258935f816a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc0b2030 con 0x55e3600b4900 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.144+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc3bc0b3070 con 0x55e3600b4900 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.145+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3bc09be90 con 0x55e3600b4900 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.145+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3bc0a2e70 con 0x55e3600b4900 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.145+0000 7fc3dcc32740 10 client.28625 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.145+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e35ffdc6a0 con 0x55e3600b4900 2024-03-12T05:25:50.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.145+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc3bc09be90 con 0x55e3600b4900 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc3bc01f070 con 0x55e3600b4900 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 10 client.28625 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 10 client.28625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 20 client.28625 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e36012dab0 0x55e36012fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 4 client.28625 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e35fda7890 con 0x55e36012dab0 2024-03-12T05:25:50.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.146+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e36012dab0 0x55e36012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.147+0000 7fc3d37fe640 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e36012dab0 0x55e36012fea0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5bbd93be879eb12 server_cookie=d6df3488546a75da in_seq=0 out_seq=0 2024-03-12T05:25:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.147+0000 7fc3b2ffd640 10 client.28625 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.147+0000 7fc3b2ffd640 1 -- 172.21.15.45:0/703206671 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e35fda7890 con 0x55e36012dab0 2024-03-12T05:25:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.147+0000 7fc3b2ffd640 10 client.28625 handle_command_reply: tid=0 2024-03-12T05:25:50.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.147+0000 7fc3dcc32740 1 client.28625 shutdown 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 msgr2=0x55e3600cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0ce080 tx=0x7fc3bc099070 comp rx=0 tx=0).stop 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 shutdown_connections 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55e36012dab0 0x55e36012fea0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3600c5fc0 0x55e3600cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3600b4900 0x55e3600cb630 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 --2- 172.21.15.45:0/703206671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3600b3fe0 0x55e3600cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 >> 172.21.15.45:0/703206671 conn(0x55e3600ae990 msgr2=0x55e3600c2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.151+0000 7fc3b2ffd640 0 client.28625 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.152+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 shutdown_connections 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.152+0000 7fc3dcc32740 1 -- 172.21.15.45:0/703206671 wait complete. 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.152+0000 7fc3dcc32740 20 client.28625 trim_cache size 0 max 16384 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:25:50.147 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:50.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.154+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 msgr2=0x7fc3c00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.154+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 0x7fc3c00a40e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4036040 tx=0x7fc3c4005a20 comp rx=0 tx=0).stop 2024-03-12T05:25:50.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.154+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 msgr2=0x7fc3d4072d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.148 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.154+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00ebc0 tx=0x7fc3bc00c6a0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 shutdown_connections 2024-03-12T05:25:50.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3c00a1c20 0x7fc3c00a40e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3d406b8f0 0x7fc3d406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d4108660 0x7fc3d4072d60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.149 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 --2- 172.21.15.45:0/4181984805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3d4103590 0x7fc3d4072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 >> 172.21.15.45:0/4181984805 conn(0x7fc3d40fee30 msgr2=0x7fc3d41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 shutdown_connections 2024-03-12T05:25:50.150 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.155+0000 7fc3d9f81640 1 -- 172.21.15.45:0/4181984805 wait complete. 2024-03-12T05:25:50.167 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:25:50.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:25:50.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:49 smithi130 ceph-mon[35398]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:25:50.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:49 smithi116 ceph-mon[35486]: pgmap v469: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 11 MiB/s wr, 1.25k op/s 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 -- 172.21.15.45:0/323932641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc10b7c0 msgr2=0x7fd9cc10dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 --2- 172.21.15.45:0/323932641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc10dc60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009be0 tx=0x7fd9c402bef0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 -- 172.21.15.45:0/323932641 shutdown_connections 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 --2- 172.21.15.45:0/323932641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc102d10 0x7fd9cc10b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 --2- 172.21.15.45:0/323932641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc10dc60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 --2- 172.21.15.45:0/323932641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 -- 172.21.15.45:0/323932641 >> 172.21.15.45:0/323932641 conn(0x7fd9cc077180 msgr2=0x7fd9cc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.376 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 -- 172.21.15.45:0/323932641 shutdown_connections 2024-03-12T05:25:50.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.381+0000 7fd9d260f640 1 -- 172.21.15.45:0/323932641 wait complete. 2024-03-12T05:25:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.382+0000 7fd9d260f640 1 Processor -- start 2024-03-12T05:25:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.382+0000 7fd9d260f640 1 -- start start 2024-03-12T05:25:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9d260f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.378 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9d260f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 0x7fd9cc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9d260f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43720/0 (socket says 172.21.15.45:43720) 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9d0bac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9d0bac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57828/0 (socket says 172.21.15.45:57828) 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.383+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 0x7fd9cc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.384+0000 7fd9cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 0x7fd9cc197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:33992/0 (socket says 172.21.15.45:33992) 2024-03-12T05:25:50.379 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.384+0000 7fd9d260f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19c380 con 0x7fd9cc10b7c0 2024-03-12T05:25:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.384+0000 7fd9cbfff640 1 -- 172.21.15.45:0/2803490653 learned_addr learned my addr 172.21.15.45:0/2803490653 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.384+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19c4f0 con 0x7fd9cc102d10 2024-03-12T05:25:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.384+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc19c6b0 con 0x7fd9cc1023f0 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 -- 172.21.15.45:0/2803490653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 msgr2=0x7fd9cc197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 0x7fd9cc197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 -- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 msgr2=0x7fd9cc19bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0009560 con 0x7fd9cc1023f0 2024-03-12T05:25:50.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9d0bac640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.385+0000 7fd9cbfff640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009fd0 tx=0x7fd9bc00edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.386+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc009700 con 0x7fd9cc1023f0 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.386+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c4009840 con 0x7fd9cc1023f0 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.387+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc0fff70 con 0x7fd9cc1023f0 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.388+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd9bc00cd80 con 0x7fd9cc1023f0 2024-03-12T05:25:50.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.388+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc01bb50 con 0x7fd9cc1023f0 2024-03-12T05:25:50.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.389+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b8005290 con 0x7fd9cc1023f0 2024-03-12T05:25:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.391+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd9bc0137d0 con 0x7fd9cc1023f0 2024-03-12T05:25:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.391+0000 7fd9c97fa640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 0x7fd9b40a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:50.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.391+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd9bc0d1a40 con 0x7fd9cc1023f0 2024-03-12T05:25:50.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.392+0000 7fd9cb7fe640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 0x7fd9b40a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:50.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.392+0000 7fd9cb7fe640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 0x7fd9b40a4160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0ff830 tx=0x7fd9c0009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:50.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.399+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd9bc09aa40 con 0x7fd9cc1023f0 2024-03-12T05:25:50.804 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.809+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9b8005810 con 0x7fd9cc1023f0 2024-03-12T05:25:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.812+0000 7fd9c97fa640 1 -- 172.21.15.45:0/2803490653 <== mon.1 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7fd9bc0bce60 con 0x7fd9cc1023f0 2024-03-12T05:25:50.806 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:25:50.807 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:25:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:25:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 msgr2=0x7fd9b40a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 0x7fd9b40a4160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0ff830 tx=0x7fd9c0009260 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 msgr2=0x7fd9cc1974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009fd0 tx=0x7fd9bc00edd0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 shutdown_connections 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd9b40a1ca0 0x7fd9b40a4160 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9cc10b7c0 0x7fd9cc19bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9cc102d10 0x7fd9cc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 --2- 172.21.15.45:0/2803490653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc1023f0 0x7fd9cc1974c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 >> 172.21.15.45:0/2803490653 conn(0x7fd9cc077180 msgr2=0x7fd9cc0fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 shutdown_connections 2024-03-12T05:25:50.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:50.816+0000 7fd9d260f640 1 -- 172.21.15.45:0/2803490653 wait complete. 2024-03-12T05:25:50.828 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:50.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/1575083274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880ff5d0 msgr2=0x7f1a880ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/1575083274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880ff5d0 0x7f1a880ffa50 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009900 tx=0x7f1a7802bc30 comp rx=0 tx=0).stop 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/1575083274 shutdown_connections 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/1575083274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880ff5d0 0x7f1a880ffa50 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/1575083274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a880fff90 0x7f1a8810c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/1575083274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a88105690 0x7f1a88105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/1575083274 >> 172.21.15.45:0/1575083274 conn(0x7f1a88076380 msgr2=0x7f1a880fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/1575083274 shutdown_connections 2024-03-12T05:25:51.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.010+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/1575083274 wait complete. 2024-03-12T05:25:51.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 Processor -- start 2024-03-12T05:25:51.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 -- start start 2024-03-12T05:25:51.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 0x7f1a881979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 0x7f1a8819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8819c310 con 0x7f1a880fff90 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8819c480 con 0x7f1a88105690 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.011+0000 7f1a8e0e6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8819c640 con 0x7f1a880ff5d0 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 0x7f1a8819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8c8e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 0x7f1a881979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43730/0 (socket says 172.21.15.45:43730) 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 -- 172.21.15.45:0/2579441121 learned_addr learned my addr 172.21.15.45:0/2579441121 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 -- 172.21.15.45:0/2579441121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 msgr2=0x7f1a8819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 0x7f1a8819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 -- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 msgr2=0x7f1a881979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 0x7f1a881979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 -- 172.21.15.45:0/2579441121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80009840 con 0x7f1a880ff5d0 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 0x7f1a8819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 0x7f1a881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.012+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400ea20 tx=0x7f1a8400eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.013+0000 7f1a7e7fc640 1 -- 172.21.15.45:0/2579441121 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400cd40 con 0x7f1a880ff5d0 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.013+0000 7f1a7e7fc640 1 -- 172.21.15.45:0/2579441121 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a840168c0 con 0x7f1a880ff5d0 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.013+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a78009560 con 0x7f1a880ff5d0 2024-03-12T05:25:51.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.013+0000 7f1a7e7fc640 1 -- 172.21.15.45:0/2579441121 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400b860 con 0x7f1a880ff5d0 2024-03-12T05:25:51.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.013+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a88194f80 con 0x7f1a880ff5d0 2024-03-12T05:25:51.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.016+0000 7f1a91ff8740 1 Processor -- start 2024-03-12T05:25:51.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.016+0000 7f1a91ff8740 1 -- start start 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a7e7fc640 1 -- 172.21.15.45:0/2579441121 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1a84016a30 con 0x7f1a880ff5d0 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb5559960 con 0x55cdb559ed80 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb550d310 con 0x55cdb559e460 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb550a840 con 0x55cdb55a7960 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8c8e3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34022/0 (socket says 172.21.15.45:34022) 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/2618874471 learned_addr learned my addr 172.21.15.45:0/2618874471 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/2618874471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 msgr2=0x55cdb55a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/2618874471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 msgr2=0x55cdb55a7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/2618874471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdb54c8060 con 0x55cdb559e460 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.017+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:51.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009fd0 tx=0x7f1a80010a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc823d01a3181ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a7e7fc640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 0x7f1a740a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a737fe640 1 -- 172.21.15.45:0/2618874471 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000e7f0 con 0x55cdb559e460 2024-03-12T05:25:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a7e7fc640 1 -- 172.21.15.45:0/2579441121 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a840cd670 con 0x7f1a880ff5d0 2024-03-12T05:25:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 0x7f1a740a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a737fe640 1 -- 172.21.15.45:0/2618874471 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a80015070 con 0x55cdb559e460 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.018+0000 7f1a737fe640 1 -- 172.21.15.45:0/2618874471 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80011d90 con 0x55cdb559e460 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 -- 172.21.15.45:0/2618874471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 msgr2=0x55cdb559e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009fd0 tx=0x7f1a80010a70 comp rx=0 tx=0).stop 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a8d0e4640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 0x7f1a740a41a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55cdb559df30 tx=0x7f1a8400a830 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 -- 172.21.15.45:0/2618874471 shutdown_connections 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/2618874471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb559e840 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 -- 172.21.15.45:0/2618874471 >> 172.21.15.45:0/2618874471 conn(0x55cdb5598b90 msgr2=0x55cdb5598f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 -- 172.21.15.45:0/2618874471 shutdown_connections 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.019+0000 7f1a91ff8740 1 -- 172.21.15.45:0/2618874471 wait complete. 2024-03-12T05:25:51.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.020+0000 7f1a91ff8740 1 -- start start 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.020+0000 7f1a91ff8740 1 Processor -- start 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.020+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.020+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb5506020 con 0x55cdb559ed80 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb51d9430 con 0x55cdb559e460 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb5195e00 con 0x55cdb55a7960 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a91ff8740 10 client.0 _ll_register_callbacks cb 0x55cdb54adcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8c8e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34028/0 (socket says 172.21.15.45:34028) 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/293440029 learned_addr learned my addr 172.21.15.45:0/293440029 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 msgr2=0x55cdb55a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/293440029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 msgr2=0x55cdb55a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/293440029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a80009e40 con 0x55cdb559e460 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1a80010f70 tx=0x7f1a80009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70bc74ee86519453 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.021+0000 7f1a70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80003b70 con 0x55cdb559e460 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a80022040 con 0x55cdb559e460 2024-03-12T05:25:51.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cdb52057e0 con 0x55cdb559e460 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a80004e90 con 0x55cdb559e460 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a91ff8740 10 client.28643 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cdb55046a0 con 0x55cdb559e460 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.022+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55cdb52057e0 con 0x55cdb559e460 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1a80019030 con 0x55cdb559e460 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 10 client.28643 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 10 client.28643 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 20 client.28643 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb55b7e30 0x55cdb55ba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 4 client.28643 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cdb5242720 con 0x55cdb55b7e30 2024-03-12T05:25:51.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.023+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb55b7e30 0x55cdb55ba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.024+0000 7f1a70ff9640 10 client.28643 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:51.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.024+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb55b7e30 0x55cdb55ba220 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff2616a5be9d55d7 server_cookie=d841b01f202075ec in_seq=0 out_seq=0 2024-03-12T05:25:51.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.024+0000 7f1a70ff9640 1 -- 172.21.15.45:0/293440029 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cdb5242720 con 0x55cdb55b7e30 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.024+0000 7f1a70ff9640 10 client.28643 handle_command_reply: tid=0 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.024+0000 7f1a91ff8740 1 client.28643 shutdown 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 msgr2=0x55cdb55a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1a80010f70 tx=0x7f1a80009280 comp rx=0 tx=0).stop 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 shutdown_connections 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb55b7e30 0x55cdb55ba220 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb559ed80 0x55cdb55a4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb55a7960 0x55cdb55a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/293440029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55a49e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 >> 172.21.15.45:0/293440029 conn(0x55cdb5598b90 msgr2=0x55cdb559a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a70ff9640 0 client.28643 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 shutdown_connections 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 1 -- 172.21.15.45:0/293440029 wait complete. 2024-03-12T05:25:51.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.025+0000 7f1a91ff8740 20 client.28643 trim_cache size 0 max 16384 2024-03-12T05:25:51.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.050+0000 7f1a91ff8740 1 Processor -- start 2024-03-12T05:25:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 -- start start 2024-03-12T05:25:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 0x55cdb55a7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559ed80 0x55cdb55a5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb548b130 con 0x55cdb55a7960 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb55aa630 con 0x55cdb559ed80 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb5594540 con 0x55cdb559e460 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 0x55cdb55a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 0x55cdb55a7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43766/0 (socket says 172.21.15.45:43766) 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/3868845733 learned_addr learned my addr 172.21.15.45:0/3868845733 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.053+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/3868845733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 msgr2=0x55cdb55a7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 0x55cdb55a7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/3868845733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559ed80 msgr2=0x55cdb55a5280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559ed80 0x55cdb55a5280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/3868845733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdb54a0c20 con 0x55cdb55a7960 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.054+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55a1c40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1a80006010 tx=0x7f1a80000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a83efe28de69899 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.055+0000 7f1a72ffd640 1 -- 172.21.15.45:0/3868845733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a800119b0 con 0x55cdb55a7960 2024-03-12T05:25:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.055+0000 7f1a72ffd640 1 -- 172.21.15.45:0/3868845733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a80004a00 con 0x55cdb55a7960 2024-03-12T05:25:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.055+0000 7f1a72ffd640 1 -- 172.21.15.45:0/3868845733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80009ce0 con 0x55cdb55a7960 2024-03-12T05:25:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 -- 172.21.15.45:0/3868845733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 msgr2=0x55cdb55a1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55a1c40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1a80006010 tx=0x7f1a80000f40 comp rx=0 tx=0).stop 2024-03-12T05:25:51.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 -- 172.21.15.45:0/3868845733 shutdown_connections 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55a1c40 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559ed80 0x55cdb55a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/3868845733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559e460 0x55cdb55a7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 -- 172.21.15.45:0/3868845733 >> 172.21.15.45:0/3868845733 conn(0x55cdb5598b90 msgr2=0x55cdb55a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 -- 172.21.15.45:0/3868845733 shutdown_connections 2024-03-12T05:25:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.056+0000 7f1a91ff8740 1 -- 172.21.15.45:0/3868845733 wait complete. 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 -- start start 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 Processor -- start 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559ed80 0x55cdb55accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb55ad770 con 0x55cdb55a7960 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb55ad8e0 con 0x55cdb559e460 2024-03-12T05:25:51.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdb55adaa0 con 0x55cdb559ed80 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.057+0000 7f1a91ff8740 10 client.0 _ll_register_callbacks cb 0x7f1a80009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57876/0 (socket says 172.21.15.45:57876) 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/204298402 learned_addr learned my addr 172.21.15.45:0/204298402 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559ed80 0x55cdb55accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559ed80 msgr2=0x55cdb55accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559ed80 0x55cdb55accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/204298402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 msgr2=0x55cdb55ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.058+0000 7f1a8d8e5640 1 -- 172.21.15.45:0/204298402 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdb55046a0 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.059+0000 7f1a8d8e5640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1a80004b90 tx=0x7f1a80009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1d9c37f1833313 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.059+0000 7f1a70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.059+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80009ce0 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.059+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a80015070 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.059+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1a80028e50 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a800289f0 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a91ff8740 10 client.19191 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cdb54c68c0 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1a80015070 con 0x55cdb55a7960 2024-03-12T05:25:51.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1a80038030 con 0x55cdb55a7960 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a91ff8740 10 client.19191 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.060+0000 7f1a91ff8740 10 client.19191 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.061+0000 7f1a91ff8740 20 client.19191 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.061+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb5617ae0 0x55cdb5619f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.061+0000 7f1a91ff8740 4 client.19191 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.061+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cdb5291890 con 0x55cdb5617ae0 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.061+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb5617ae0 0x55cdb5619f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.062+0000 7f1a8c8e3640 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb5617ae0 0x55cdb5619f00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d0362aee67a3da81 server_cookie=af85eaf843f5d691 in_seq=0 out_seq=0 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.062+0000 7f1a70ff9640 10 client.19191 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.062+0000 7f1a70ff9640 1 -- 172.21.15.45:0/204298402 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cdb5291890 con 0x55cdb5617ae0 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.062+0000 7f1a70ff9640 10 client.19191 handle_command_reply: tid=0 2024-03-12T05:25:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.062+0000 7f1a91ff8740 1 client.19191 shutdown 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 msgr2=0x55cdb55ad230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1a80004b90 tx=0x7f1a80009280 comp rx=0 tx=0).stop 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 shutdown_connections 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x55cdb5617ae0 0x55cdb5619f00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55cdb55a7960 0x55cdb55ad230 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdb559ed80 0x55cdb55accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 --2- 172.21.15.45:0/204298402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cdb559e460 0x55cdb55ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.063+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 >> 172.21.15.45:0/204298402 conn(0x55cdb5598b90 msgr2=0x55cdb5597440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.068+0000 7f1a70ff9640 0 client.19191 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:25:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.070+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 shutdown_connections 2024-03-12T05:25:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.070+0000 7f1a91ff8740 1 -- 172.21.15.45:0/204298402 wait complete. 2024-03-12T05:25:51.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.070+0000 7f1a91ff8740 20 client.19191 trim_cache size 0 max 16384 2024-03-12T05:25:51.065 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.076+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 msgr2=0x7f1a740a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 0x7f1a740a41a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55cdb559df30 tx=0x7f1a8400a830 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 msgr2=0x7f1a881974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400ea20 tx=0x7f1a8400eee0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 shutdown_connections 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1a740a1ce0 0x7f1a740a41a0 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x55cdb559df30 tx=0x7f1a8400a830 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1a880fff90 0x7f1a881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a88105690 0x7f1a8819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 --2- 172.21.15.45:0/2579441121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a880ff5d0 0x7f1a881974a0 secure :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400ea20 tx=0x7f1a8400eee0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.071 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.077+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 >> 172.21.15.45:0/2579441121 conn(0x7f1a88076380 msgr2=0x7f1a880fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.078+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 shutdown_connections 2024-03-12T05:25:51.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.078+0000 7f1a8e0e6640 1 -- 172.21.15.45:0/2579441121 wait complete. 2024-03-12T05:25:51.088 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:51.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 -- 172.21.15.45:0/3200702804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010cd30 msgr2=0x7fbb1010d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 --2- 172.21.15.45:0/3200702804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010cd30 0x7fbb1010d110 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000fc30 tx=0x7fbb0002bab0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 -- 172.21.15.45:0/3200702804 shutdown_connections 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 --2- 172.21.15.45:0/3200702804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010daf0 0x7fbb1010df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 --2- 172.21.15.45:0/3200702804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010e490 0x7fbb10114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 --2- 172.21.15.45:0/3200702804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010cd30 0x7fbb1010d110 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 -- 172.21.15.45:0/3200702804 >> 172.21.15.45:0/3200702804 conn(0x7fbb1006c030 msgr2=0x7fbb1006c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.265+0000 7fbb151ec640 1 -- 172.21.15.45:0/3200702804 shutdown_connections 2024-03-12T05:25:51.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.266+0000 7fbb151ec640 1 -- 172.21.15.45:0/3200702804 wait complete. 2024-03-12T05:25:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.266+0000 7fbb151ec640 1 Processor -- start 2024-03-12T05:25:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.266+0000 7fbb151ec640 1 -- start start 2024-03-12T05:25:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 0x7fbb101a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010daf0 0x7fbb101a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 0x7fbb1019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019def0 con 0x7fbb1010e490 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019e060 con 0x7fbb1010cd30 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb151ec640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019e220 con 0x7fbb1010daf0 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 0x7fbb101a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb149eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 0x7fbb1019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 0x7fbb101a36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34060/0 (socket says 172.21.15.45:34060) 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb0ffff640 1 -- 172.21.15.45:0/1445160231 learned_addr learned my addr 172.21.15.45:0/1445160231 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010daf0 0x7fbb101a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb149eb640 1 -- 172.21.15.45:0/1445160231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010daf0 msgr2=0x7fbb101a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.267+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010daf0 0x7fbb101a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb149eb640 1 -- 172.21.15.45:0/1445160231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 msgr2=0x7fbb101a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 0x7fbb101a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb149eb640 1 -- 172.21.15.45:0/1445160231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0400b4b0 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 0x7fbb1019d8a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005fd0 tx=0x7fbb08016850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb0d7fa640 1 -- 172.21.15.45:0/1445160231 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800fde0 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb0d7fa640 1 -- 172.21.15.45:0/1445160231 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb08017040 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb0d7fa640 1 -- 172.21.15.45:0/1445160231 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08011440 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0000f870 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.268+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb101aa5b0 con 0x7fbb1010e490 2024-03-12T05:25:51.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.271+0000 7fbb0d7fa640 1 -- 172.21.15.45:0/1445160231 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbb08011600 con 0x7fbb1010e490 2024-03-12T05:25:51.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.272+0000 7fbb19023740 1 Processor -- start 2024-03-12T05:25:51.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.272+0000 7fbb19023740 1 -- start start 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 0x559ccf28e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28ed80 0x559ccf297420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf299d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf249960 con 0x559ccf28e460 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf1fd310 con 0x559ccf28ed80 2024-03-12T05:25:51.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb19023740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf1fa840 con 0x559ccf297960 2024-03-12T05:25:51.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf299d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 0x559ccf28e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 0x559ccf28e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57900/0 (socket says 172.21.15.45:57900) 2024-03-12T05:25:51.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.273+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/2530760678 learned_addr learned my addr 172.21.15.45:0/2530760678 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/2530760678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28ed80 msgr2=0x559ccf297420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28ed80 0x559ccf297420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/2530760678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 msgr2=0x559ccf28e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 0x559ccf28e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/2530760678 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ccf1b8060 con 0x559ccf297960 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0d7fa640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 0x7fbaf80a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf299d50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400a9e0 tx=0x7fbb04005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef659be8323d492 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0d7fa640 1 -- 172.21.15.45:0/1445160231 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbb080d5a00 con 0x7fbb1010e490 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 0x7fbaf80a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.274+0000 7fbaf67fc640 1 -- 172.21.15.45:0/2530760678 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04020070 con 0x559ccf297960 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.275+0000 7fbaf67fc640 1 -- 172.21.15.45:0/2530760678 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb04005020 con 0x559ccf297960 2024-03-12T05:25:51.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.275+0000 7fbaf67fc640 1 -- 172.21.15.45:0/2530760678 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04010040 con 0x559ccf297960 2024-03-12T05:25:51.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 -- 172.21.15.45:0/2530760678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 msgr2=0x559ccf299d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf299d50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400a9e0 tx=0x7fbb04005f90 comp rx=0 tx=0).stop 2024-03-12T05:25:51.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 0x7fbaf80a4160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x559ccf28df30 tx=0x7fbb0003b040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 -- 172.21.15.45:0/2530760678 shutdown_connections 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf299d50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28ed80 0x559ccf297420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 --2- 172.21.15.45:0/2530760678 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28e460 0x559ccf28e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 -- 172.21.15.45:0/2530760678 >> 172.21.15.45:0/2530760678 conn(0x559ccf288b90 msgr2=0x559ccf288f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.276+0000 7fbb19023740 1 -- 172.21.15.45:0/2530760678 shutdown_connections 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.277+0000 7fbb19023740 1 -- 172.21.15.45:0/2530760678 wait complete. 2024-03-12T05:25:51.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.277+0000 7fbb19023740 1 -- start start 2024-03-12T05:25:51.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.278+0000 7fbb19023740 1 Processor -- start 2024-03-12T05:25:51.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf294a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf2954c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf1f6020 con 0x559ccf28ed80 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57904/0 (socket says 172.21.15.45:57904) 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/4123675909 learned_addr learned my addr 172.21.15.45:0/4123675909 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cceec9430 con 0x559ccf28e460 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccee85e00 con 0x559ccf297960 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf294a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb149eb640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf2954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.279+0000 7fbb19023740 10 client.0 _ll_register_callbacks cb 0x559ccf19dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/4123675909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 msgr2=0x559ccf2954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf2954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 msgr2=0x559ccf294a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf294a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0400a6c0 con 0x559ccf28ed80 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb149eb640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf2954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559cceef57e0 con 0x559ccf28ed80 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.280+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbb04001eb0 tx=0x7fbb0401bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763204dec2f546f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.281+0000 7fbaef7fe640 10 client.19203 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.281+0000 7fbaef7fe640 1 -- 172.21.15.45:0/4123675909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04020040 con 0x559ccf28ed80 2024-03-12T05:25:51.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.281+0000 7fbaef7fe640 1 -- 172.21.15.45:0/4123675909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbb04004ad0 con 0x559ccf28ed80 2024-03-12T05:25:51.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.281+0000 7fbaef7fe640 1 -- 172.21.15.45:0/4123675909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb04004cd0 con 0x559ccf28ed80 2024-03-12T05:25:51.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.284+0000 7fbb19023740 10 client.19203 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.284+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x559ccf1f46a0 con 0x559ccf28ed80 2024-03-12T05:25:51.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.287+0000 7fbaef7fe640 1 -- 172.21.15.45:0/4123675909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbb04016070 con 0x559ccf28ed80 2024-03-12T05:25:51.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.287+0000 7fbb19023740 10 client.19203 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb19023740 10 client.19203 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:25:51.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb19023740 20 client.19203 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2a7be0 0x559ccf2aa000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb19023740 4 client.19203 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:51.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ccef32720 con 0x559ccf2a7be0 2024-03-12T05:25:51.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.288+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2a7be0 0x559ccf2aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.289+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2a7be0 0x559ccf2aa000 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7674f43396c6b323 server_cookie=31406084ae00699e in_seq=0 out_seq=0 2024-03-12T05:25:51.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.289+0000 7fbaef7fe640 10 client.19203 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbaef7fe640 1 -- 172.21.15.45:0/4123675909 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559ccef32720 con 0x559ccf2a7be0 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbaef7fe640 10 client.19203 handle_command_reply: tid=0 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 client.19203 shutdown 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 msgr2=0x559ccf294f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbb04001eb0 tx=0x7fbb0401bd50 comp rx=0 tx=0).stop 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 shutdown_connections 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2a7be0 0x559ccf2aa000 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf294f80 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf2954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 --2- 172.21.15.45:0/4123675909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf294a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 >> 172.21.15.45:0/4123675909 conn(0x559ccf288b90 msgr2=0x559ccf28a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbaef7fe640 0 client.19203 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 shutdown_connections 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 1 -- 172.21.15.45:0/4123675909 wait complete. 2024-03-12T05:25:51.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.290+0000 7fbb19023740 20 client.19203 trim_cache size 0 max 16384 2024-03-12T05:25:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb19023740 1 Processor -- start 2024-03-12T05:25:51.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb19023740 1 -- start start 2024-03-12T05:25:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb19023740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf297170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb19023740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf2952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf297170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.312+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf297170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34088/0 (socket says 172.21.15.45:34088) 2024-03-12T05:25:51.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb0ffff640 1 -- 172.21.15.45:0/1676277746 learned_addr learned my addr 172.21.15.45:0/1676277746 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb19023740 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf291cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf284340 con 0x559ccf28ed80 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf17b130 con 0x559ccf28e460 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf2aa3c0 con 0x559ccf297960 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf291cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 -- 172.21.15.45:0/1676277746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 msgr2=0x559ccf297170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf297170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 -- 172.21.15.45:0/1676277746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 msgr2=0x559ccf2952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf2952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.313+0000 7fbb149eb640 1 -- 172.21.15.45:0/1676277746 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ccf190c20 con 0x559ccf297960 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.314+0000 7fbb149eb640 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf291cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbb080d30c0 tx=0x7fbb0809e1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d80617098fb18b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.314+0000 7fbaf5ffb640 1 -- 172.21.15.45:0/1676277746 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb080a2630 con 0x559ccf297960 2024-03-12T05:25:51.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.314+0000 7fbaf5ffb640 1 -- 172.21.15.45:0/1676277746 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb080a6780 con 0x559ccf297960 2024-03-12T05:25:51.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.314+0000 7fbaf5ffb640 1 -- 172.21.15.45:0/1676277746 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0809ee80 con 0x559ccf297960 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 msgr2=0x559ccf291cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf291cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbb080d30c0 tx=0x7fbb0809e1b0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 shutdown_connections 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf28ed80 0x559ccf2952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf297960 0x559ccf291cc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 --2- 172.21.15.45:0/1676277746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf297170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.315+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 >> 172.21.15.45:0/1676277746 conn(0x559ccf288b90 msgr2=0x559ccf293310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.316+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 shutdown_connections 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.316+0000 7fbb19023740 1 -- 172.21.15.45:0/1676277746 wait complete. 2024-03-12T05:25:51.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.317+0000 7fbb19023740 1 -- start start 2024-03-12T05:25:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.317+0000 7fbb19023740 1 Processor -- start 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf28ed80 0x559ccf29cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf297960 0x559ccf29d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf29d8d0 con 0x559ccf297960 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34098/0 (socket says 172.21.15.45:34098) 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb0ffff640 1 -- 172.21.15.45:0/2336200441 learned_addr learned my addr 172.21.15.45:0/2336200441 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf29da40 con 0x559ccf28e460 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ccf29dc00 con 0x559ccf28ed80 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb19023740 10 client.0 _ll_register_callbacks cb 0x7fbb0400a380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.318+0000 7fbb149eb640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf297960 0x559ccf29d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.319+0000 7fbb0ffff640 1 -- 172.21.15.45:0/2336200441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf28ed80 msgr2=0x559ccf29cdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.319+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf28ed80 0x559ccf29cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.319+0000 7fbb0ffff640 1 -- 172.21.15.45:0/2336200441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf297960 msgr2=0x559ccf29d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.319+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf297960 0x559ccf29d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.319+0000 7fbb0ffff640 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb101aa5b0 con 0x559ccf28e460 2024-03-12T05:25:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.321+0000 7fbb0ffff640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbb00004a60 tx=0x7fbb0000fa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2b75f7ebefca53 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.321+0000 7fbaef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.321+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00055110 con 0x559ccf28e460 2024-03-12T05:25:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.322+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbb00055270 con 0x559ccf28e460 2024-03-12T05:25:51.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.322+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb04005020 con 0x559ccf28e460 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.322+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb00055430 con 0x559ccf28e460 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.322+0000 7fbb19023740 10 client.28661 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.322+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559ccf1b68c0 con 0x559ccf28e460 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbb04005020 con 0x559ccf28e460 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbb00050040 con 0x559ccf28e460 2024-03-12T05:25:51.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 10 client.28661 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 10 client.28661 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 20 client.28661 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2ff790 0x559ccf301bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 4 client.28661 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559ccef81890 con 0x559ccf2ff790 2024-03-12T05:25:51.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.323+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2ff790 0x559ccf301bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.324+0000 7fbaef7fe640 10 client.28661 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.324+0000 7fbb0f7fe640 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2ff790 0x559ccf301bb0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=722f43eab16c5ea7 server_cookie=3510f81885dcfa50 in_seq=0 out_seq=0 2024-03-12T05:25:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.324+0000 7fbaef7fe640 1 -- 172.21.15.45:0/2336200441 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559ccef81890 con 0x559ccf2ff790 2024-03-12T05:25:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.324+0000 7fbaef7fe640 10 client.28661 handle_command_reply: tid=0 2024-03-12T05:25:51.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.324+0000 7fbb19023740 1 client.28661 shutdown 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 msgr2=0x559ccf29ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbb00004a60 tx=0x7fbb0000fa90 comp rx=0 tx=0).stop 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 shutdown_connections 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x559ccf2ff790 0x559ccf301bb0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x559ccf297960 0x559ccf29d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ccf28ed80 0x559ccf29cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 --2- 172.21.15.45:0/2336200441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ccf28e460 0x559ccf29ea10 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 >> 172.21.15.45:0/2336200441 conn(0x559ccf288b90 msgr2=0x559ccf287cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbaef7fe640 0 client.28661 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 shutdown_connections 2024-03-12T05:25:51.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 1 -- 172.21.15.45:0/2336200441 wait complete. 2024-03-12T05:25:51.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.325+0000 7fbb19023740 20 client.28661 trim_cache size 0 max 16384 2024-03-12T05:25:51.321 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 msgr2=0x7fbaf80a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 0x7fbaf80a4160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x559ccf28df30 tx=0x7fbb0003b040 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 msgr2=0x7fbb1019d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 0x7fbb1019d8a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005fd0 tx=0x7fbb08016850 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 shutdown_connections 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbaf80a1ca0 0x7fbaf80a4160 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb1010e490 0x7fbb1019d8a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb1010daf0 0x7fbb101a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 --2- 172.21.15.45:0/1445160231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb1010cd30 0x7fbb101a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 >> 172.21.15.45:0/1445160231 conn(0x7fbb1006c030 msgr2=0x7fbb10071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 shutdown_connections 2024-03-12T05:25:51.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.326+0000 7fbb151ec640 1 -- 172.21.15.45:0/1445160231 wait complete. 2024-03-12T05:25:51.338 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:51.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:25:51.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.531+0000 7fb5f5223640 1 -- 172.21.15.45:0/1847572497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0107720 msgr2=0x7fb5f0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.531+0000 7fb5f5223640 1 --2- 172.21.15.45:0/1847572497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0107720 0x7fb5f0107ba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009860 tx=0x7fb5e402bb90 comp rx=0 tx=0).stop 2024-03-12T05:25:51.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 -- 172.21.15.45:0/1847572497 shutdown_connections 2024-03-12T05:25:51.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 --2- 172.21.15.45:0/1847572497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f01080e0 0x7fb5f01145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 --2- 172.21.15.45:0/1847572497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0107720 0x7fb5f0107ba0 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009860 tx=0x7fb5e402bb90 comp rx=0 tx=0).stop 2024-03-12T05:25:51.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 --2- 172.21.15.45:0/1847572497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f010d730 0x7fb5f010db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 -- 172.21.15.45:0/1847572497 >> 172.21.15.45:0/1847572497 conn(0x7fb5f00756c0 msgr2=0x7fb5f0077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 -- 172.21.15.45:0/1847572497 shutdown_connections 2024-03-12T05:25:51.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.532+0000 7fb5f5223640 1 -- 172.21.15.45:0/1847572497 wait complete. 2024-03-12T05:25:51.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.533+0000 7fb5f5223640 1 Processor -- start 2024-03-12T05:25:51.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 -- start start 2024-03-12T05:25:51.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f01080e0 0x7fb5f019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 0x7fb5f019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 0x7fb5f01a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a3260 con 0x7fb5f010d730 2024-03-12T05:25:51.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5f5223640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a33d0 con 0x7fb5f01a2870 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.534+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 0x7fb5f019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.535+0000 7fb5f5223640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f01a3540 con 0x7fb5f01080e0 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.535+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 0x7fb5f019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57944/0 (socket says 172.21.15.45:57944) 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.535+0000 7fb5ee59c640 1 -- 172.21.15.45:0/635424413 learned_addr learned my addr 172.21.15.45:0/635424413 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.535+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f01080e0 0x7fb5f019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.535+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 0x7fb5f01a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 -- 172.21.15.45:0/635424413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f01080e0 msgr2=0x7fb5f019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f01080e0 0x7fb5f019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 -- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 msgr2=0x7fb5f019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 0x7fb5f019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 -- 172.21.15.45:0/635424413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009490 con 0x7fb5f01a2870 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.536+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 0x7fb5f01a2d20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800ed70 tx=0x7fb5e800c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.537+0000 7fb5d7fff640 1 -- 172.21.15.45:0/635424413 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8003b70 con 0x7fb5f01a2870 2024-03-12T05:25:51.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.537+0000 7fb5d7fff640 1 -- 172.21.15.45:0/635424413 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5e800ee60 con 0x7fb5f01a2870 2024-03-12T05:25:51.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.537+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f01aab20 con 0x7fb5f01a2870 2024-03-12T05:25:51.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.537+0000 7fb5d7fff640 1 -- 172.21.15.45:0/635424413 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8017790 con 0x7fb5f01a2870 2024-03-12T05:25:51.532 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.537+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f01aafa0 con 0x7fb5f01a2870 2024-03-12T05:25:51.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.540+0000 7fb5d7fff640 1 -- 172.21.15.45:0/635424413 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb5e800a040 con 0x7fb5f01a2870 2024-03-12T05:25:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.541+0000 7fb5d7fff640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 0x7fb5d80a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.541+0000 7fb5f7dfa740 1 Processor -- start 2024-03-12T05:25:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5d7fff640 1 -- 172.21.15.45:0/635424413 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb5e80cd560 con 0x7fb5f01a2870 2024-03-12T05:25:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 0x7fb5d80a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.536 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 -- start start 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf05630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf10b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bec0960 con 0x562a1bf05250 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1be74310 con 0x562a1bf0e750 2024-03-12T05:25:51.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1be71840 con 0x562a1bf05b70 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43840/0 (socket says 172.21.15.45:43840) 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5ee59c640 1 -- 172.21.15.45:0/3809945255 learned_addr learned my addr 172.21.15.45:0/3809945255 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.542+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf10b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf05630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 -- 172.21.15.45:0/3809945255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 msgr2=0x562a1bf0e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 -- 172.21.15.45:0/3809945255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 msgr2=0x562a1bf05630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf05630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 -- 172.21.15.45:0/3809945255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a1be2f060 con 0x562a1bf0e750 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 0x7fb5d80a4120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0006fd0 tx=0x7fb5e00090e0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf10b40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80cad90 tx=0x7fb5e8095f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecadc38ed550793f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5d57fa640 1 -- 172.21.15.45:0/3809945255 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8098620 con 0x562a1bf0e750 2024-03-12T05:25:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5d57fa640 1 -- 172.21.15.45:0/3809945255 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5e8003cd0 con 0x562a1bf0e750 2024-03-12T05:25:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.543+0000 7fb5d57fa640 1 -- 172.21.15.45:0/3809945255 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80a27e0 con 0x562a1bf0e750 2024-03-12T05:25:51.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3809945255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 msgr2=0x562a1bf10b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf10b40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80cad90 tx=0x7fb5e8095f20 comp rx=0 tx=0).stop 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3809945255 shutdown_connections 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf10b40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3809945255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf05630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3809945255 >> 172.21.15.45:0/3809945255 conn(0x562a1beff980 msgr2=0x562a1beffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3809945255 shutdown_connections 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.544+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3809945255 wait complete. 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 -- start start 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 Processor -- start 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf0b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf0c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1be6d020 con 0x562a1bf05250 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bb40430 con 0x562a1bf0e750 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bafce00 con 0x562a1bf05b70 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.545+0000 7fb5f7dfa740 10 client.0 _ll_register_callbacks cb 0x562a1be14cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5eed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf0b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43848/0 (socket says 172.21.15.45:43848) 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 -- 172.21.15.45:0/3870166470 learned_addr learned my addr 172.21.15.45:0/3870166470 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5eed9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf0b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57960/0 (socket says 172.21.15.45:57960) 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf0c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 -- 172.21.15.45:0/3870166470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 msgr2=0x562a1bf0c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf0c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 -- 172.21.15.45:0/3870166470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 msgr2=0x562a1bf0b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf0b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 -- 172.21.15.45:0/3870166470 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f01aafa0 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009830 tx=0x7fb5e4034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e88a0a81dcf22f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.546+0000 7fb5d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e402c530 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb5e402c690 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a1bb6c7e0 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5e402c850 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5f7dfa740 10 client.28630 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.547+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562a1be6b6a0 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x562a1bb6c7e0 con 0x562a1bf05b70 2024-03-12T05:25:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5e4053420 con 0x562a1bf05b70 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 10 client.28630 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 10 client.28630 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 20 client.28630 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf1eb80 0x562a1bf20fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 4 client.28630 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a1bba9720 con 0x562a1bf1eb80 2024-03-12T05:25:51.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.548+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf1eb80 0x562a1bf20fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.550+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf1eb80 0x562a1bf20fa0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e14fee1c87f722fd server_cookie=efd913a869d2b05e in_seq=0 out_seq=0 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.550+0000 7fb5d2ff5640 10 client.28630 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/3870166470 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562a1bba9720 con 0x562a1bf1eb80 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5d2ff5640 10 client.28630 handle_command_reply: tid=0 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 client.28630 shutdown 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 msgr2=0x562a1bf0bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009830 tx=0x7fb5e4034440 comp rx=0 tx=0).stop 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 shutdown_connections 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf1eb80 0x562a1bf20fa0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf0e750 0x562a1bf0c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.546 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf05b70 0x562a1bf0bd30 secure :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009830 tx=0x7fb5e4034440 comp rx=0 tx=0).stop 2024-03-12T05:25:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/3870166470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf05250 0x562a1bf0b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 >> 172.21.15.45:0/3870166470 conn(0x562a1beff980 msgr2=0x562a1bf016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.551+0000 7fb5d2ff5640 0 client.28630 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:51.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.558+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 shutdown_connections 2024-03-12T05:25:51.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.560+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/3870166470 wait complete. 2024-03-12T05:25:51.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.560+0000 7fb5f7dfa740 20 client.28630 trim_cache size 0 max 16384 2024-03-12T05:25:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.585+0000 7fb5f7dfa740 1 Processor -- start 2024-03-12T05:25:51.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.585+0000 7fb5f7dfa740 1 -- start start 2024-03-12T05:25:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.585+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf0df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf08a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf09410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bdf2130 con 0x562a1bf0e750 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bf07420 con 0x562a1bf05250 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf0df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bf00760 con 0x562a1bf08f90 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf0df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34154/0 (socket says 172.21.15.45:34154) 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5eed9d640 1 -- 172.21.15.45:0/1053003810 learned_addr learned my addr 172.21.15.45:0/1053003810 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf08a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 -- 172.21.15.45:0/1053003810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 msgr2=0x562a1bf09410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf09410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 -- 172.21.15.45:0/1053003810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 msgr2=0x562a1bf0df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf0df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.581 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.586+0000 7fb5ee59c640 1 -- 172.21.15.45:0/1053003810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a1be07c20 con 0x562a1bf0e750 2024-03-12T05:25:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.587+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf08a50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x562a1bf0c880 tx=0x7fb5e402c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c0fae3282badc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.587+0000 7fb5d4ff9640 1 -- 172.21.15.45:0/1053003810 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4004500 con 0x562a1bf0e750 2024-03-12T05:25:51.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.587+0000 7fb5d4ff9640 1 -- 172.21.15.45:0/1053003810 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5e4034af0 con 0x562a1bf0e750 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.587+0000 7fb5d4ff9640 1 -- 172.21.15.45:0/1053003810 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4053420 con 0x562a1bf0e750 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/1053003810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 msgr2=0x562a1bf08a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf08a50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x562a1bf0c880 tx=0x7fb5e402c370 comp rx=0 tx=0).stop 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/1053003810 shutdown_connections 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf08a50 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf09410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/1053003810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf0df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/1053003810 >> 172.21.15.45:0/1053003810 conn(0x562a1beff980 msgr2=0x562a1bf25940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.588+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/1053003810 shutdown_connections 2024-03-12T05:25:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.589+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/1053003810 wait complete. 2024-03-12T05:25:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.589+0000 7fb5f7dfa740 1 -- start start 2024-03-12T05:25:51.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.590+0000 7fb5f7dfa740 1 Processor -- start 2024-03-12T05:25:51.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf148e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf6f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bf6f650 con 0x562a1bf0e750 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bf6f7c0 con 0x562a1bf05250 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5f7dfa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a1bf6f980 con 0x562a1bf08f90 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5ee59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43860/0 (socket says 172.21.15.45:43860) 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5ee59c640 1 -- 172.21.15.45:0/750805254 learned_addr learned my addr 172.21.15.45:0/750805254 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.591+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf6f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5f7dfa740 10 client.0 _ll_register_callbacks cb 0x7fb5e80966b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf148e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 -- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 msgr2=0x562a1bf14e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 -- 172.21.15.45:0/750805254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 msgr2=0x562a1bf148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf148e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 -- 172.21.15.45:0/750805254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562a1be6b6a0 con 0x562a1bf0e750 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ee59c640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5ef59e640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf6f110 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800cbe0 tx=0x7fb5e80965c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21e5c0525e0800d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.592+0000 7fb5d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80a2580 con 0x562a1bf0e750 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb5e80a26e0 con 0x562a1bf0e750 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5e8003cd0 con 0x562a1bf0e750 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5e8099770 con 0x562a1bf0e750 2024-03-12T05:25:51.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5f7dfa740 10 client.19224 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.593+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562a1be2d8c0 con 0x562a1bf0e750 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb5e8003cd0 con 0x562a1bf0e750 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5e8096210 con 0x562a1bf0e750 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 10 client.19224 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 10 client.19224 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 20 client.19224 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf76e90 0x562a1bf79280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 4 client.19224 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.594+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562a1bbf8890 con 0x562a1bf76e90 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.595+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf76e90 0x562a1bf79280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.595+0000 7fb5eed9d640 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf76e90 0x562a1bf79280 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=955c8b7e882472d1 server_cookie=eed79176d9c3b887 in_seq=0 out_seq=0 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5d2ff5640 10 client.19224 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5d2ff5640 1 -- 172.21.15.45:0/750805254 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562a1bbf8890 con 0x562a1bf76e90 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5d2ff5640 10 client.19224 handle_command_reply: tid=0 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 client.19224 shutdown 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 msgr2=0x562a1bf6f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf6f110 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800cbe0 tx=0x7fb5e80965c0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 shutdown_connections 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x562a1bf76e90 0x562a1bf79280 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x562a1bf0e750 0x562a1bf6f110 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a1bf08f90 0x562a1bf14e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 --2- 172.21.15.45:0/750805254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a1bf05250 0x562a1bf148e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 >> 172.21.15.45:0/750805254 conn(0x562a1beff980 msgr2=0x562a1befe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5d2ff5640 0 client.19224 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:25:51.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 shutdown_connections 2024-03-12T05:25:51.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.596+0000 7fb5f7dfa740 1 -- 172.21.15.45:0/750805254 wait complete. 2024-03-12T05:25:51.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.597+0000 7fb5f7dfa740 20 client.19224 trim_cache size 0 max 16384 2024-03-12T05:25:51.593 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 msgr2=0x7fb5d80a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 0x7fb5d80a4120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0006fd0 tx=0x7fb5e00090e0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 msgr2=0x7fb5f01a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 0x7fb5f01a2d20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800ed70 tx=0x7fb5e800c6e0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 shutdown_connections 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb5d80a1c60 0x7fb5d80a4120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5f010d730 0x7fb5f019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f01a2870 0x7fb5f01a2d20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 --2- 172.21.15.45:0/635424413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5f01080e0 0x7fb5f019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.598+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 >> 172.21.15.45:0/635424413 conn(0x7fb5f00756c0 msgr2=0x7fb5f00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.599+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 shutdown_connections 2024-03-12T05:25:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.599+0000 7fb5f5223640 1 -- 172.21.15.45:0/635424413 wait complete. 2024-03-12T05:25:51.612 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:51.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:25:51.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.792+0000 7f3d82ffd640 1 -- 172.21.15.45:0/4123598392 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c034440 con 0x7f3d84068380 2024-03-12T05:25:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/4123598392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84068380 msgr2=0x7f3d84068760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/4123598392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84068380 0x7f3d84068760 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009ca0 tx=0x7f3d7c02c190 comp rx=0 tx=0).stop 2024-03-12T05:25:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/4123598392 shutdown_connections 2024-03-12T05:25:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/4123598392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8410db20 0x7f3d8410ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/4123598392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d84068ca0 0x7f3d8410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/4123598392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84068380 0x7f3d84068760 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.793+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/4123598392 >> 172.21.15.45:0/4123598392 conn(0x7f3d84077180 msgr2=0x7f3d84077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.794+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/4123598392 shutdown_connections 2024-03-12T05:25:51.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.794+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/4123598392 wait complete. 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.794+0000 7f3d8a3e2640 1 Processor -- start 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 -- start start 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84068ca0 0x7f3d8419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84199950 0x7f3d84199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8419a340 con 0x7f3d84068ca0 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8419a4b0 con 0x7f3d8410db20 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8a3e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8419a670 con 0x7f3d84199950 2024-03-12T05:25:51.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d83fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84068ca0 0x7f3d8419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34178/0 (socket says 172.21.15.45:34178) 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d837fe640 1 -- 172.21.15.45:0/3246124855 learned_addr learned my addr 172.21.15.45:0/3246124855 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.795+0000 7f3d8897f640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84199950 0x7f3d84199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 -- 172.21.15.45:0/3246124855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84199950 msgr2=0x7f3d84199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84199950 0x7f3d84199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 -- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84068ca0 msgr2=0x7f3d8419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84068ca0 0x7f3d8419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 -- 172.21.15.45:0/3246124855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c009900 con 0x7f3d8410db20 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d837fe640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400b450 tx=0x7f3d7400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.796+0000 7f3d817fa640 1 -- 172.21.15.45:0/3246124855 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74011890 con 0x7f3d8410db20 2024-03-12T05:25:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.797+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d841a6660 con 0x7f3d8410db20 2024-03-12T05:25:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.797+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d841a6b90 con 0x7f3d8410db20 2024-03-12T05:25:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.799+0000 7f3d817fa640 1 -- 172.21.15.45:0/3246124855 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d740103e0 con 0x7f3d8410db20 2024-03-12T05:25:51.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.799+0000 7f3d817fa640 1 -- 172.21.15.45:0/3246124855 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74011ea0 con 0x7f3d8410db20 2024-03-12T05:25:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.800+0000 7f3d817fa640 1 -- 172.21.15.45:0/3246124855 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3d7400f620 con 0x7f3d8410db20 2024-03-12T05:25:51.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.801+0000 7f3d817fa640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 0x7f3d6c0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 0x7f3d6c0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 Processor -- start 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d817fa640 1 -- 172.21.15.45:0/3246124855 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d740ce380 con 0x7f3d8410db20 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 -- start start 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b097630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b052960 con 0x55fd1b0a0750 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b006310 con 0x55fd1b097b70 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.802+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b003840 con 0x55fd1b097250 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d8897f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d8897f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58008/0 (socket says 172.21.15.45:58008) 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34194/0 (socket says 172.21.15.45:34194) 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d8897f640 1 -- 172.21.15.45:0/1602723889 learned_addr learned my addr 172.21.15.45:0/1602723889 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d83fff640 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b097630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 0x7f3d6c0a4160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f80 tx=0x7f3d7c005f50 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 -- 172.21.15.45:0/1602723889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 msgr2=0x55fd1b097630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b097630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 -- 172.21.15.45:0/1602723889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 msgr2=0x55fd1b0a2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.803+0000 7f3d837fe640 1 -- 172.21.15.45:0/1602723889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd1afc1060 con 0x55fd1b097b70 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.804+0000 7f3d837fe640 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3d740cbb50 tx=0x7f3d7409b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a3aecd11c6d4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.804+0000 7f3d6a7fc640 1 -- 172.21.15.45:0/1602723889 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d740b0020 con 0x55fd1b097b70 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.804+0000 7f3d6a7fc640 1 -- 172.21.15.45:0/1602723889 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d740a3880 con 0x55fd1b097b70 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.804+0000 7f3d6a7fc640 1 -- 172.21.15.45:0/1602723889 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7409a790 con 0x55fd1b097b70 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/1602723889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 msgr2=0x55fd1b0a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3d740cbb50 tx=0x7f3d7409b8a0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/1602723889 shutdown_connections 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a0210 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/1602723889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b097630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/1602723889 >> 172.21.15.45:0/1602723889 conn(0x55fd1b091980 msgr2=0x55fd1b091d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/1602723889 shutdown_connections 2024-03-12T05:25:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.805+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/1602723889 wait complete. 2024-03-12T05:25:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.806+0000 7f3d8cfbc740 1 -- start start 2024-03-12T05:25:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.806+0000 7f3d8cfbc740 1 Processor -- start 2024-03-12T05:25:51.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.806+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b09dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b09e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1afff020 con 0x55fd1b0a0750 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1acd2430 con 0x55fd1b097b70 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1ac8ee00 con 0x55fd1b097250 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43894/0 (socket says 172.21.15.45:43894) 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 -- 172.21.15.45:0/2107687457 learned_addr learned my addr 172.21.15.45:0/2107687457 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d8cfbc740 10 client.0 _ll_register_callbacks cb 0x55fd1afa6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d837fe640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b09dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 -- 172.21.15.45:0/2107687457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 msgr2=0x55fd1b09dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b09dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 -- 172.21.15.45:0/2107687457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 msgr2=0x55fd1b09e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b09e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.807+0000 7f3d83fff640 1 -- 172.21.15.45:0/2107687457 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d841a6b90 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d83fff640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009ca0 tx=0x7f3d7c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63abbf23b276193e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d637fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c032a90 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d7c032bf0 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fd1acfe7e0 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d7c032db0 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d8cfbc740 10 client.28645 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.808+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fd1affd6a0 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55fd1acfe7e0 con 0x55fd1b097250 2024-03-12T05:25:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3d7c056670 con 0x55fd1b097250 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 10 client.28645 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 10 client.28645 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 20 client.28645 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b0b0990 0x55fd1b0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 4 client.28645 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.809+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd1ad3b720 con 0x55fd1b0b0990 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.810+0000 7f3d837fe640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b0b0990 0x55fd1b0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.811+0000 7f3d837fe640 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b0b0990 0x55fd1b0b2db0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9f86edd428dde6b server_cookie=e5f6f9535afd4903 in_seq=0 out_seq=0 2024-03-12T05:25:51.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.811+0000 7f3d637fe640 10 client.28645 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.812+0000 7f3d637fe640 1 -- 172.21.15.45:0/2107687457 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f3d7407fa90 con 0x55fd1b0b0990 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.812+0000 7f3d637fe640 10 client.28645 handle_command_reply: tid=0 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.812+0000 7f3d8cfbc740 1 client.28645 shutdown 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.812+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 msgr2=0x55fd1b09d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.812+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009ca0 tx=0x7f3d7c0095c0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 shutdown_connections 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b09e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b0b0990 0x55fd1b0b2db0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b09dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/2107687457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09d6d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 >> 172.21.15.45:0/2107687457 conn(0x55fd1b091980 msgr2=0x55fd1b0936d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d637fe640 0 client.28645 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 shutdown_connections 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/2107687457 wait complete. 2024-03-12T05:25:51.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.814+0000 7f3d8cfbc740 20 client.28645 trim_cache size 0 max 16384 2024-03-12T05:25:51.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.837+0000 7f3d8cfbc740 1 Processor -- start 2024-03-12T05:25:51.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.837+0000 7f3d8cfbc740 1 -- start start 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b090a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d83fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d83fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43908/0 (socket says 172.21.15.45:43908) 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b090a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d83fff640 1 -- 172.21.15.45:0/975904046 learned_addr learned my addr 172.21.15.45:0/975904046 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1af84130 con 0x55fd1b0a0750 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b08ca10 con 0x55fd1b097b70 2024-03-12T05:25:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b096eb0 con 0x55fd1b097250 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.838+0000 7f3d8897f640 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 -- 172.21.15.45:0/975904046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 msgr2=0x55fd1b09df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 -- 172.21.15.45:0/975904046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 msgr2=0x55fd1b090a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b090a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 -- 172.21.15.45:0/975904046 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd1af99c20 con 0x55fd1b0a0750 2024-03-12T05:25:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.839+0000 7f3d8897f640 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a7660 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800b680 tx=0x7f3d7800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2297851f08ebfebe server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.840+0000 7f3d69ffb640 1 -- 172.21.15.45:0/975904046 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78003a70 con 0x55fd1b0a0750 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.840+0000 7f3d69ffb640 1 -- 172.21.15.45:0/975904046 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d780054d0 con 0x55fd1b0a0750 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.840+0000 7f3d69ffb640 1 -- 172.21.15.45:0/975904046 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78004c30 con 0x55fd1b0a0750 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.841+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 msgr2=0x55fd1b0a7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.841+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a7660 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800b680 tx=0x7f3d7800bb40 comp rx=0 tx=0).stop 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.841+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 shutdown_connections 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.841+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b0a7660 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.841+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b090a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.842+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/975904046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b09df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.842+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 >> 172.21.15.45:0/975904046 conn(0x55fd1b091980 msgr2=0x55fd1b0902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.842+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 shutdown_connections 2024-03-12T05:25:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.842+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/975904046 wait complete. 2024-03-12T05:25:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.843+0000 7f3d8cfbc740 1 -- start start 2024-03-12T05:25:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.843+0000 7f3d8cfbc740 1 Processor -- start 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b0a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8cfbc740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8897f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8897f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58050/0 (socket says 172.21.15.45:58050) 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.844+0000 7f3d8897f640 1 -- 172.21.15.45:0/3335500189 learned_addr learned my addr 172.21.15.45:0/3335500189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b107420 con 0x55fd1b0a0750 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b0a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b107560 con 0x55fd1b097b70 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd1b1076a0 con 0x55fd1b097250 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d8cfbc740 10 client.0 _ll_register_callbacks cb 0x7f3d74097380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 -- 172.21.15.45:0/3335500189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 msgr2=0x55fd1b0a6d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:25:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a6d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 -- 172.21.15.45:0/3335500189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 msgr2=0x55fd1b106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.845+0000 7f3d83fff640 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d780042d0 con 0x55fd1b097250 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.846+0000 7f3d83fff640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b0a67f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c006010 tx=0x7f3d7c032b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80f5660fda2bbecd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.846+0000 7f3d637fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.846+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c003d40 con 0x55fd1b097250 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.846+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3d7c003f80 con 0x55fd1b097250 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.846+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d740a3880 con 0x55fd1b097250 2024-03-12T05:25:51.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d7c0347e0 con 0x55fd1b097250 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 10 client.28651 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fd1afbf8c0 con 0x55fd1b097250 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3d7c073080 con 0x55fd1b097250 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3d7c067560 con 0x55fd1b097250 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 10 client.28651 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 10 client.28651 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 20 client.28651 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b108b50 0x55fd1b10af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 4 client.28651 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.847+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f3d740a3880 con 0x55fd1b108b50 2024-03-12T05:25:51.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.848+0000 7f3d837fe640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b108b50 0x55fd1b10af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.848+0000 7f3d637fe640 10 client.28651 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.848+0000 7f3d837fe640 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b108b50 0x55fd1b10af70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d90bfa725ea26bb9 server_cookie=39e78aa8b8e34e8c in_seq=0 out_seq=0 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.848+0000 7f3d637fe640 1 -- 172.21.15.45:0/3335500189 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f3d740a3880 con 0x55fd1b108b50 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.848+0000 7f3d637fe640 10 client.28651 handle_command_reply: tid=0 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 client.28651 shutdown 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 msgr2=0x55fd1b0a67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b0a67f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c006010 tx=0x7f3d7c032b40 comp rx=0 tx=0).stop 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 shutdown_connections 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fd1b0a0750 0x55fd1b106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55fd1b108b50 0x55fd1b10af70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd1b097b70 0x55fd1b0a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d637fe640 0 client.28651 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.849+0000 7f3d8cfbc740 1 --2- 172.21.15.45:0/3335500189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fd1b097250 0x55fd1b0a67f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.850+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 >> 172.21.15.45:0/3335500189 conn(0x55fd1b091980 msgr2=0x55fd1b090b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.850+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 shutdown_connections 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.850+0000 7f3d8cfbc740 1 -- 172.21.15.45:0/3335500189 wait complete. 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.850+0000 7f3d8cfbc740 20 client.28651 trim_cache size 0 max 16384 2024-03-12T05:25:51.847 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:51.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.851+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 msgr2=0x7f3d6c0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.851+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 0x7f3d6c0a4160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f80 tx=0x7f3d7c005f50 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.851+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 msgr2=0x7f3d8419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.851+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400b450 tx=0x7f3d7400b910 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 shutdown_connections 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3d6c0a1ca0 0x7f3d6c0a4160 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d84199950 0x7f3d84199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d8410db20 0x7f3d8419fce0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 --2- 172.21.15.45:0/3246124855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d84068ca0 0x7f3d8419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 >> 172.21.15.45:0/3246124855 conn(0x7f3d84077180 msgr2=0x7f3d8410f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 shutdown_connections 2024-03-12T05:25:51.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:51.852+0000 7f3d8a3e2640 1 -- 172.21.15.45:0/3246124855 wait complete. 2024-03-12T05:25:51.866 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:25:51.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:25:51.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:51 smithi045 ceph-mon[32174]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 12 MiB/s wr, 1.40k op/s 2024-03-12T05:25:51.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:51 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2803490653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/1771744051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a4de0 msgr2=0x7f1dc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/1771744051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a4de0 0x7f1dc40a51c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc066e50 tx=0x7f1dcc08ebb0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/1771744051 shutdown_connections 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/1771744051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc40b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/1771744051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a5700 0x7f1dc40b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/1771744051 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a4de0 0x7f1dc40a51c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.049+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/1771744051 >> 172.21.15.45:0/1771744051 conn(0x7f1dc401a640 msgr2=0x7f1dc401aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.050+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/1771744051 shutdown_connections 2024-03-12T05:25:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.050+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/1771744051 wait complete. 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 Processor -- start 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 -- start start 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 0x7f1dc4149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc4143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4143e00 con 0x7f1dc40a5700 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4143f70 con 0x7f1dc40a4de0 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd2d6e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4144130 con 0x7f1dc40b7a70 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34258/0 (socket says 172.21.15.45:34258) 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd256d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc4143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/2033527781 learned_addr learned my addr 172.21.15.45:0/2033527781 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.051+0000 7f1dd156b640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 0x7f1dc4149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/2033527781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 msgr2=0x7f1dc4143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc4143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 msgr2=0x7f1dc4149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 0x7f1dc4149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/2033527781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc04f750 con 0x7f1dc40a4de0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd156b640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 0x7f1dc4149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd256d640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc4143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc000b60 tx=0x7f1dcc069320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dc2ffd640 1 -- 172.21.15.45:0/2033527781 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc09d070 con 0x7f1dc40a4de0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dc2ffd640 1 -- 172.21.15.45:0/2033527781 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1dcc08fdd0 con 0x7f1dc40a4de0 2024-03-12T05:25:52.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dc2ffd640 1 -- 172.21.15.45:0/2033527781 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc098e50 con 0x7f1dc40a4de0 2024-03-12T05:25:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc41443b0 con 0x7f1dc40a4de0 2024-03-12T05:25:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.052+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4150530 con 0x7f1dc40a4de0 2024-03-12T05:25:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.056+0000 7f1dc2ffd640 1 -- 172.21.15.45:0/2033527781 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1dcc095070 con 0x7f1dc40a4de0 2024-03-12T05:25:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.056+0000 7f1dd6b9c740 1 Processor -- start 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.056+0000 7f1dd6b9c740 1 -- start start 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8410840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a841bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a83cb960 con 0x55b9a8419960 2024-03-12T05:25:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a837f310 con 0x55b9a8410d80 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a837c840 con 0x55b9a8410460 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34260/0 (socket says 172.21.15.45:34260) 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3475510492 learned_addr learned my addr 172.21.15.45:0/3475510492 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dd156b640 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a841bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.057+0000 7f1dc2ffd640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 0x7f1db80a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3475510492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 msgr2=0x55b9a8410840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8410840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3475510492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 msgr2=0x55b9a841bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a841bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3475510492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9a833a060 con 0x55b9a8410d80 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd256d640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 0x7f1db80a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd156b640 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a841bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc14ec20 tx=0x7f1dcc0e9470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d06085009ff919e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1dc2ffd640 1 -- 172.21.15.45:0/2033527781 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1dcc151510 con 0x7f1dc40a4de0 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1db7fff640 1 -- 172.21.15.45:0/3475510492 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc11dd00 con 0x55b9a8410d80 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.058+0000 7f1db7fff640 1 -- 172.21.15.45:0/3475510492 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1dcc11db20 con 0x55b9a8410d80 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.059+0000 7f1db7fff640 1 -- 172.21.15.45:0/3475510492 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0f2bd0 con 0x55b9a8410d80 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.059+0000 7f1dd256d640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 0x7f1db80a41a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8005fd0 tx=0x7f1dc8005d60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.059+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3475510492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 msgr2=0x55b9a8419420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.059+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc14ec20 tx=0x7f1dcc0e9470 comp rx=0 tx=0).stop 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3475510492 shutdown_connections 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a841bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8419420 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3475510492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8410840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3475510492 >> 172.21.15.45:0/3475510492 conn(0x55b9a840ab90 msgr2=0x55b9a840af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3475510492 shutdown_connections 2024-03-12T05:25:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3475510492 wait complete. 2024-03-12T05:25:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.060+0000 7f1dd6b9c740 1 -- start start 2024-03-12T05:25:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 Processor -- start 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8416f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8417460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a8378020 con 0x55b9a8419960 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a804b430 con 0x55b9a8410d80 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a8007e00 con 0x55b9a8410460 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd6b9c740 10 client.0 _ll_register_callbacks cb 0x55b9a831fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8416f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8416f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34272/0 (socket says 172.21.15.45:34272) 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43960/0 (socket says 172.21.15.45:43960) 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd156b640 1 -- 172.21.15.45:0/3363488793 learned_addr learned my addr 172.21.15.45:0/3363488793 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.061+0000 7f1dd256d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8417460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 msgr2=0x55b9a8416f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8416f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3363488793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 msgr2=0x55b9a8417460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8417460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/3363488793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4150530 con 0x55b9a8410460 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc066cc0 tx=0x7f1dcc08ff40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e26fe10eb4853418 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.062+0000 7f1db57fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc11dd00 con 0x55b9a8410460 2024-03-12T05:25:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1dcc098470 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9a80777e0 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dcc14f060 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1dd6b9c740 10 client.28657 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b9a83766a0 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.063+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55b9a80777e0 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1dcc098650 con 0x55b9a8410460 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 10 client.28657 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 10 client.28657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 20 client.28657 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a84294f0 0x55b9a842b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 4 client.28657 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:52.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9a80b4720 con 0x55b9a84294f0 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.064+0000 7f1dd156b640 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a84294f0 0x55b9a842b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.065+0000 7f1dd156b640 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a84294f0 0x55b9a842b910 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f3e3669fdd5b2fe server_cookie=43795fc41d184677 in_seq=0 out_seq=0 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.065+0000 7f1db57fa640 10 client.28657 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.065+0000 7f1db57fa640 1 -- 172.21.15.45:0/3363488793 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b9a80b4720 con 0x55b9a84294f0 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.065+0000 7f1db57fa640 10 client.28657 handle_command_reply: tid=0 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 client.28657 shutdown 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 msgr2=0x55b9a84169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc066cc0 tx=0x7f1dcc08ff40 comp rx=0 tx=0).stop 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 shutdown_connections 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a84294f0 0x55b9a842b910 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1db57fa640 0 client.28657 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:52.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8417460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8416f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/3363488793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a84169e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 >> 172.21.15.45:0/3363488793 conn(0x55b9a840ab90 msgr2=0x55b9a840c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.066+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 shutdown_connections 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.067+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/3363488793 wait complete. 2024-03-12T05:25:52.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.067+0000 7f1dd6b9c740 20 client.28657 trim_cache size 0 max 16384 2024-03-12T05:25:52.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.087+0000 7f1dd6b9c740 1 Processor -- start 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 -- start start 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8419110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8430230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a8406340 con 0x55b9a8419960 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a82fd130 con 0x55b9a8410d80 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a841c120 con 0x55b9a8410460 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd156b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34276/0 (socket says 172.21.15.45:34276) 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd156b640 1 -- 172.21.15.45:0/191782838 learned_addr learned my addr 172.21.15.45:0/191782838 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd256d640 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8430230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.088+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8419110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 -- 172.21.15.45:0/191782838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 msgr2=0x55b9a8419110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8419110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 -- 172.21.15.45:0/191782838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 msgr2=0x55b9a8430230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8430230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 -- 172.21.15.45:0/191782838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9a8312c20 con 0x55b9a8410d80 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1dd156b640 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x55b9a842c1b0 tx=0x7f1dbc00a2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfd97ab37eaac87d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1db77fe640 1 -- 172.21.15.45:0/191782838 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc015040 con 0x55b9a8410d80 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1db77fe640 1 -- 172.21.15.45:0/191782838 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1dbc016040 con 0x55b9a8410d80 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.089+0000 7f1db77fe640 1 -- 172.21.15.45:0/191782838 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc005280 con 0x55b9a8410d80 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.090+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/191782838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 msgr2=0x55b9a8417280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.090+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x55b9a842c1b0 tx=0x7f1dbc00a2d0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/191782838 shutdown_connections 2024-03-12T05:25:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8419960 0x55b9a8430230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8410d80 0x55b9a8417280 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/191782838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410460 0x55b9a8419110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/191782838 >> 172.21.15.45:0/191782838 conn(0x55b9a840ab90 msgr2=0x55b9a84112a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/191782838 shutdown_connections 2024-03-12T05:25:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.091+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/191782838 wait complete. 2024-03-12T05:25:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.092+0000 7f1dd6b9c740 1 -- start start 2024-03-12T05:25:52.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.092+0000 7f1dd6b9c740 1 Processor -- start 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 0x55b9a8420840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 0x55b9a841ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8419960 0x55b9a841f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 0x55b9a8420840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd1d6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 0x55b9a8420840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58098/0 (socket says 172.21.15.45:58098) 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a841f700 con 0x55b9a8410460 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a841f870 con 0x55b9a8419960 2024-03-12T05:25:52.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a841fa30 con 0x55b9a8410d80 2024-03-12T05:25:52.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd1d6c640 1 -- 172.21.15.45:0/1261125280 learned_addr learned my addr 172.21.15.45:0/1261125280 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:52.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd156b640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 0x55b9a841ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.093+0000 7f1dd6b9c740 10 client.0 _ll_register_callbacks cb 0x7f1dcc154dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:52.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd256d640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8419960 0x55b9a841f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd156b640 1 -- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8419960 msgr2=0x55b9a841f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd156b640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8419960 0x55b9a841f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd156b640 1 -- 172.21.15.45:0/1261125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 msgr2=0x55b9a8420840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd156b640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 0x55b9a8420840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.094+0000 7f1dd156b640 1 -- 172.21.15.45:0/1261125280 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dbc0106d0 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1dd156b640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 0x55b9a841ec30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55b9a8417ea0 tx=0x7f1dbc004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19196a9242e2213f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1db57fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00fce0 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f1dbc00fe40 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1dcc11db20 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dbc01f0f0 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1dd6b9c740 10 client.28663 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b9a83388c0 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.095+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f1dcc11db20 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1dbc003b20 con 0x55b9a8410d80 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 10 client.28663 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:52.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 10 client.28663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:52.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 20 client.28663 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:52.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a8480ec0 0x55b9a84832e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:52.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 4 client.28663 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:25:52.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b9a8103890 con 0x55b9a8480ec0 2024-03-12T05:25:52.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.096+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a8480ec0 0x55b9a84832e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:52.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.097+0000 7f1dd1d6c640 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a8480ec0 0x55b9a84832e0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=567f37741a28c98 server_cookie=79341970b218412 in_seq=0 out_seq=0 2024-03-12T05:25:52.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.097+0000 7f1db57fa640 10 client.28663 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.097+0000 7f1db57fa640 1 -- 172.21.15.45:0/1261125280 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b9a8103890 con 0x55b9a8480ec0 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.097+0000 7f1db57fa640 10 client.28663 handle_command_reply: tid=0 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.098+0000 7f1dd6b9c740 1 client.28663 shutdown 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 msgr2=0x55b9a841ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 0x55b9a841ec30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55b9a8417ea0 tx=0x7f1dbc004510 comp rx=0 tx=0).stop 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 shutdown_connections 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55b9a8480ec0 0x55b9a84832e0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b9a8419960 0x55b9a841f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a8410d80 0x55b9a841ec30 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 --2- 172.21.15.45:0/1261125280 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55b9a8410460 0x55b9a8420840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 >> 172.21.15.45:0/1261125280 conn(0x55b9a840ab90 msgr2=0x55b9a8430230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1db57fa640 0 client.28663 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 shutdown_connections 2024-03-12T05:25:52.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 1 -- 172.21.15.45:0/1261125280 wait complete. 2024-03-12T05:25:52.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.099+0000 7f1dd6b9c740 20 client.28663 trim_cache size 0 max 16384 2024-03-12T05:25:52.098 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:25:52.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 msgr2=0x7f1db80a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 0x7f1db80a41a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8005fd0 tx=0x7f1dc8005d60 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 msgr2=0x7f1dc41495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc000b60 tx=0x7f1dcc069320 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 shutdown_connections 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f1db80a1ce0 0x7f1db80a41a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40a5700 0x7f1dc4149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1dc40b7a70 0x7f1dc4143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 --2- 172.21.15.45:0/2033527781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dc40a4de0 0x7f1dc41495b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 >> 172.21.15.45:0/2033527781 conn(0x7f1dc401a640 msgr2=0x7f1dc40b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 shutdown_connections 2024-03-12T05:25:52.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:52.102+0000 7f1dd2d6e640 1 -- 172.21.15.45:0/2033527781 wait complete. 2024-03-12T05:25:52.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:51 smithi130 ceph-mon[35398]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 12 MiB/s wr, 1.40k op/s 2024-03-12T05:25:52.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:51 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2803490653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:52.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:51 smithi116 ceph-mon[35486]: pgmap v470: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 12 MiB/s wr, 1.40k op/s 2024-03-12T05:25:52.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:51 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2803490653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:25:53.118 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:25:53.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:25:53.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:25:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/2939840177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 msgr2=0x7f4cbc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/2939840177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc105800 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009be0 tx=0x7f4cb402bf80 comp rx=0 tx=0).stop 2024-03-12T05:25:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/2939840177 shutdown_connections 2024-03-12T05:25:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/2939840177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc105800 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/2939840177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc100870 0x7f4cbc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/2939840177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0fff50 0x7f4cbc100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/2939840177 >> 172.21.15.45:0/2939840177 conn(0x7f4cbc076c30 msgr2=0x7f4cbc0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.306+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/2939840177 shutdown_connections 2024-03-12T05:25:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.307+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/2939840177 wait complete. 2024-03-12T05:25:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.307+0000 7f4cc1b3a640 1 Processor -- start 2024-03-12T05:25:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 -- start start 2024-03-12T05:25:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc0fff50 0x7f4cbc1a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc100870 0x7f4cbc1a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc19df40 con 0x7f4cbc100870 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc19e080 con 0x7f4cbc0fff50 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cc1b3a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc19e240 con 0x7f4cbc105350 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc0fff50 0x7f4cbc1a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cbbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cbbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:43996/0 (socket says 172.21.15.45:43996) 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.308+0000 7f4cbbfff640 1 -- 172.21.15.45:0/3458214431 learned_addr learned my addr 172.21.15.45:0/3458214431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 -- 172.21.15.45:0/3458214431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc0fff50 msgr2=0x7f4cbc1a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc0fff50 0x7f4cbc1a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 -- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc100870 msgr2=0x7f4cbc1a3c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc100870 0x7f4cbc1a3c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 -- 172.21.15.45:0/3458214431 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb0009490 con 0x7f4cbc105350 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.309+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4036040 tx=0x7f4cb4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.310+0000 7f4cb8ff9640 1 -- 172.21.15.45:0/3458214431 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4004570 con 0x7f4cbc105350 2024-03-12T05:25:53.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.310+0000 7f4cb8ff9640 1 -- 172.21.15.45:0/3458214431 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cb40046d0 con 0x7f4cbc105350 2024-03-12T05:25:53.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.310+0000 7f4cb8ff9640 1 -- 172.21.15.45:0/3458214431 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb403e470 con 0x7f4cbc105350 2024-03-12T05:25:53.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.310+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4009840 con 0x7f4cbc105350 2024-03-12T05:25:53.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.310+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc106300 con 0x7f4cbc105350 2024-03-12T05:25:53.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.314+0000 7f4cb8ff9640 1 -- 172.21.15.45:0/3458214431 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4cb4034430 con 0x7f4cbc105350 2024-03-12T05:25:53.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cb8ff9640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 0x7f4ca40a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 Processor -- start 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 -- start start 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 0x7f4ca40a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2ccf00 0x55c62c2d54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c23af90 con 0x55c62c2cc550 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c2384c0 con 0x55c62c2d5a80 2024-03-12T05:25:53.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.316+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c233ca0 con 0x55c62c2ccf00 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34324/0 (socket says 172.21.15.45:34324) 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 -- 172.21.15.45:0/4053380719 learned_addr learned my addr 172.21.15.45:0/4053380719 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cb8ff9640 1 -- 172.21.15.45:0/3458214431 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4cb40f02f0 con 0x7f4cbc105350 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 -- 172.21.15.45:0/4053380719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2ccf00 msgr2=0x55c62c2d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2ccf00 0x55c62c2d54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 -- 172.21.15.45:0/4053380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 msgr2=0x55c62c2cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 -- 172.21.15.45:0/4053380719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c62c1f60b0 con 0x55c62c2d5a80 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40bcca0 tx=0x7f4cb40bc900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995fb95051822ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.317+0000 7f4ca1ffb640 1 -- 172.21.15.45:0/4053380719 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb403a240 con 0x55c62c2d5a80 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.318+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 0x7f4ca40a4270 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc19ea30 tx=0x7f4cac011810 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.318+0000 7f4ca1ffb640 1 -- 172.21.15.45:0/4053380719 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cb40d5070 con 0x55c62c2d5a80 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.318+0000 7f4ca1ffb640 1 -- 172.21.15.45:0/4053380719 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb404edf0 con 0x55c62c2d5a80 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4053380719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 msgr2=0x55c62c2d7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40bcca0 tx=0x7f4cb40bc900 comp rx=0 tx=0).stop 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4053380719 shutdown_connections 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2d5a80 0x55c62c2d7ed0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2ccf00 0x55c62c2d54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4053380719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4053380719 >> 172.21.15.45:0/4053380719 conn(0x55c62c2c6a10 msgr2=0x55c62c2c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4053380719 shutdown_connections 2024-03-12T05:25:53.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.319+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4053380719 wait complete. 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.320+0000 7f4cc47e9740 1 -- start start 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.320+0000 7f4cc47e9740 1 Processor -- start 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.320+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.320+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44022/0 (socket says 172.21.15.45:44022) 2024-03-12T05:25:53.316 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34340/0 (socket says 172.21.15.45:34340) 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/4063714403 learned_addr learned my addr 172.21.15.45:0/4063714403 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c1d8f40 con 0x55c62c2d5a80 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62bf074b0 con 0x55c62c2ccf00 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c1b9670 con 0x55c62c2cc550 2024-03-12T05:25:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.321+0000 7f4cc47e9740 10 client.0 _ll_register_callbacks cb 0x55c62c226f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.322+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 msgr2=0x55c62c2d2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.322+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.322+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/4063714403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2d5a80 msgr2=0x55c62c2d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.322+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.322+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc106300 con 0x55c62c2cc550 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.323+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00fb40 tx=0x7f4cac018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a1f70a6c3bd78ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.323+0000 7f4c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.323+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00ee00 con 0x55c62c2cc550 2024-03-12T05:25:53.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.325+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4cac028410 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.325+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c62c20df40 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.325+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cac0285d0 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.325+0000 7f4cc47e9740 10 client.28672 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.325+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c62c232320 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cac01e8b0 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4cac02f600 con 0x55c62c2cc550 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 10 client.28672 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 10 client.28672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 20 client.28672 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c2f6ba0 0x55c62c2f9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 4 client.28672 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c62c20df40 con 0x55c62c2f6ba0 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.326+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c2f6ba0 0x55c62c2f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.327+0000 7f4c9affd640 10 client.28672 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.328+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c2f6ba0 0x55c62c2f9000 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2decdf3f7cecd52 server_cookie=14ea335ee3fccd91 in_seq=0 out_seq=0 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.328+0000 7f4c9affd640 1 -- 172.21.15.45:0/4063714403 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c62c20df40 con 0x55c62c2f6ba0 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.328+0000 7f4c9affd640 10 client.28672 handle_command_reply: tid=0 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.328+0000 7f4cc47e9740 1 client.28672 shutdown 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 msgr2=0x55c62c2d2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00fb40 tx=0x7f4cac018000 comp rx=0 tx=0).stop 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 shutdown_connections 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c2f6ba0 0x55c62c2f9000 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/4063714403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2cc550 0x55c62c2d2a30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 >> 172.21.15.45:0/4063714403 conn(0x55c62c2c6a10 msgr2=0x55c62c2c5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.329+0000 7f4c9affd640 0 client.28672 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.330+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 shutdown_connections 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.330+0000 7f4cc47e9740 1 -- 172.21.15.45:0/4063714403 wait complete. 2024-03-12T05:25:53.326 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.330+0000 7f4cc47e9740 20 client.28672 trim_cache size 0 max 16384 2024-03-12T05:25:53.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 Processor -- start 2024-03-12T05:25:53.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 -- start start 2024-03-12T05:25:53.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c2c6540 con 0x55c62c2cc550 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c20df40 con 0x55c62c2ccf00 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.364+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c2c7810 con 0x55c62c2d5a80 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2d5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58136/0 (socket says 172.21.15.45:58136) 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/3121007286 learned_addr learned my addr 172.21.15.45:0/3121007286 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 -- 172.21.15.45:0/3121007286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 msgr2=0x55c62c2d34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 -- 172.21.15.45:0/3121007286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 msgr2=0x55c62c2d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.365+0000 7f4cbaffd640 1 -- 172.21.15.45:0/3121007286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c62c1cec00 con 0x55c62c2ccf00 2024-03-12T05:25:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.366+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x55c62c2c7c90 tx=0x7f4cb0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b778fc6ff22911c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.366+0000 7f4ca17fa640 1 -- 172.21.15.45:0/3121007286 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb00122d0 con 0x55c62c2ccf00 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.366+0000 7f4ca17fa640 1 -- 172.21.15.45:0/3121007286 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cb00044e0 con 0x55c62c2ccf00 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.366+0000 7f4ca17fa640 1 -- 172.21.15.45:0/3121007286 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0019740 con 0x55c62c2ccf00 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- 172.21.15.45:0/3121007286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 msgr2=0x55c62c2d2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x55c62c2c7c90 tx=0x7f4cb0004730 comp rx=0 tx=0).stop 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- 172.21.15.45:0/3121007286 shutdown_connections 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c2d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2d2f80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/3121007286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2d5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- 172.21.15.45:0/3121007286 >> 172.21.15.45:0/3121007286 conn(0x55c62c2c6a10 msgr2=0x55c62c2d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- 172.21.15.45:0/3121007286 shutdown_connections 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- 172.21.15.45:0/3121007286 wait complete. 2024-03-12T05:25:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.367+0000 7f4cc47e9740 1 -- start start 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.368+0000 7f4cc47e9740 1 Processor -- start 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.368+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.368+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2ece50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cc47e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c34d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c2ed530 con 0x55c62c2cc550 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c34d630 con 0x55c62c2ccf00 2024-03-12T05:25:53.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cc47e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62c34d770 con 0x55c62c2d5a80 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cc47e9740 10 client.0 _ll_register_callbacks cb 0x7f4cb4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58152/0 (socket says 172.21.15.45:58152) 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/952052568 learned_addr learned my addr 172.21.15.45:0/952052568 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2ece50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbbfff640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c34d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/952052568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 msgr2=0x55c62c34d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c34d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 msgr2=0x55c62c2ece50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2ece50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.369+0000 7f4cbb7fe640 1 -- 172.21.15.45:0/952052568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb0009d40 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4cbb7fe640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02feb0 tx=0x7f4cac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc80162f4866fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4c9affd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac02f7a0 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4cac005040 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c62c259900 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4cac00f760 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4cc47e9740 10 client.19269 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.370+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c62c1f4910 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4cac046020 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4cac004c80 con 0x55c62c2cc550 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 10 client.19269 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 10 client.19269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 20 client.19269 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:53.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c34efd0 0x55c62c3513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 4 client.19269 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c62c259900 con 0x55c62c34efd0 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.371+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c34efd0 0x55c62c3513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.372+0000 7f4cbaffd640 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c34efd0 0x55c62c3513f0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0644d173074c5f4 server_cookie=ee1bdf879774ba0b in_seq=0 out_seq=0 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.372+0000 7f4c9affd640 10 client.19269 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.373+0000 7f4c9affd640 1 -- 172.21.15.45:0/952052568 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c62c259900 con 0x55c62c34efd0 2024-03-12T05:25:53.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.373+0000 7f4c9affd640 10 client.19269 handle_command_reply: tid=0 2024-03-12T05:25:53.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.373+0000 7f4cc47e9740 1 client.19269 shutdown 2024-03-12T05:25:53.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 msgr2=0x55c62c2ec910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02feb0 tx=0x7f4cac0057f0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 shutdown_connections 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c62c34efd0 0x55c62c3513f0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c62c2d5a80 0x55c62c34d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c62c2ccf00 0x55c62c2ece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 --2- 172.21.15.45:0/952052568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c62c2cc550 0x55c62c2ec910 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 >> 172.21.15.45:0/952052568 conn(0x55c62c2c6a10 msgr2=0x55c62c2c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4c9affd640 0 client.19269 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 shutdown_connections 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.374+0000 7f4cc47e9740 1 -- 172.21.15.45:0/952052568 wait complete. 2024-03-12T05:25:53.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.375+0000 7f4cc47e9740 20 client.19269 trim_cache size 0 max 16384 2024-03-12T05:25:53.370 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:53.370 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:25:53.370 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "7278cce9-3a6a-4c90-b5e3-3354c2f40860", 2024-03-12T05:25:53.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:25:53.370 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:53.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 msgr2=0x7f4ca40a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 0x7f4ca40a4270 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc19ea30 tx=0x7f4cac011810 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 msgr2=0x7f4cbc19d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4036040 tx=0x7f4cb4004270 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 shutdown_connections 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4ca40a1db0 0x7f4ca40a4270 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc100870 0x7f4cbc1a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cbc105350 0x7f4cbc19d8f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 --2- 172.21.15.45:0/3458214431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc0fff50 0x7f4cbc1a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.376+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 >> 172.21.15.45:0/3458214431 conn(0x7f4cbc076c30 msgr2=0x7f4cbc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.377+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 shutdown_connections 2024-03-12T05:25:53.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.377+0000 7f4cc1b3a640 1 -- 172.21.15.45:0/3458214431 wait complete. 2024-03-12T05:25:53.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:25:53.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 -- 172.21.15.45:0/147504877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f269410c640 msgr2=0x7f269410ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 --2- 172.21.15.45:0/147504877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f269410c640 0x7f269410ca20 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2688009890 tx=0x7f268802bc00 comp rx=0 tx=0).stop 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 -- 172.21.15.45:0/147504877 shutdown_connections 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 --2- 172.21.15.45:0/147504877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694108100 0x7f26941145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 --2- 172.21.15.45:0/147504877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2694107760 0x7f2694107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 --2- 172.21.15.45:0/147504877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f269410c640 0x7f269410ca20 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 -- 172.21.15.45:0/147504877 >> 172.21.15.45:0/147504877 conn(0x7f2694076f50 msgr2=0x7f2694105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 -- 172.21.15.45:0/147504877 shutdown_connections 2024-03-12T05:25:53.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.582+0000 7f269a281640 1 -- 172.21.15.45:0/147504877 wait complete. 2024-03-12T05:25:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.584+0000 7f269a281640 1 Processor -- start 2024-03-12T05:25:53.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.584+0000 7f269a281640 1 -- start start 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.584+0000 7f269a281640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f269a281640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f269a281640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 0x7f269419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f2693fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f2693fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34386/0 (socket says 172.21.15.45:34386) 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f26937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f26937fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58170/0 (socket says 172.21.15.45:58170) 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f2693fff640 1 -- 172.21.15.45:0/3435357741 learned_addr learned my addr 172.21.15.45:0/3435357741 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f269881e640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 0x7f269419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269419e150 con 0x7f2694108100 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.585+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269419e2c0 con 0x7f2694107760 2024-03-12T05:25:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269419e480 con 0x7f269410c640 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269881e640 1 -- 172.21.15.45:0/3435357741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 msgr2=0x7f26941a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269881e640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269881e640 1 -- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 msgr2=0x7f26941a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269881e640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.586+0000 7f269881e640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684009560 con 0x7f269410c640 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f26937fe640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f2693fff640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f269881e640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 0x7f269419db00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f268c00ec50 tx=0x7f268c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f26917fa640 1 -- 172.21.15.45:0/3435357741 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c003b70 con 0x7f269410c640 2024-03-12T05:25:53.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f26917fa640 1 -- 172.21.15.45:0/3435357741 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268c00ce70 con 0x7f269410c640 2024-03-12T05:25:53.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c009840 con 0x7f269410c640 2024-03-12T05:25:53.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.587+0000 7f26917fa640 1 -- 172.21.15.45:0/3435357741 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c010670 con 0x7f269410c640 2024-03-12T05:25:53.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.588+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2688009490 con 0x7f269410c640 2024-03-12T05:25:53.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.591+0000 7f26917fa640 1 -- 172.21.15.45:0/3435357741 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f268c0108d0 con 0x7f269410c640 2024-03-12T05:25:53.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.591+0000 7f269ce4d740 1 Processor -- start 2024-03-12T05:25:53.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f26917fa640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 0x7f267c0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f26917fa640 1 -- 172.21.15.45:0/3435357741 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f268c014070 con 0x7f269410c640 2024-03-12T05:25:53.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f269ce4d740 1 -- start start 2024-03-12T05:25:53.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f26937fe640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 0x7f267c0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.592+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa784270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269ce4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa6ea280 con 0x5570aa77b2b0 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269ce4d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa6e77b0 con 0x5570aa7847b0 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269ce4d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa6e2f90 con 0x5570aa77bbd0 2024-03-12T05:25:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34394/0 (socket says 172.21.15.45:34394) 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 -- 172.21.15.45:0/2476816877 learned_addr learned my addr 172.21.15.45:0/2476816877 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f2693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f26937fe640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 0x7f267c0a4230 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2694107290 tx=0x7f2684009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f26937fe640 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa784270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 -- 172.21.15.45:0/2476816877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 msgr2=0x5570aa784270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa784270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 -- 172.21.15.45:0/2476816877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 msgr2=0x5570aa77b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.593+0000 7f269881e640 1 -- 172.21.15.45:0/2476816877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570aa6a1570 con 0x5570aa7847b0 2024-03-12T05:25:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.594+0000 7f269881e640 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f268c003fd0 tx=0x7f268c09a970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bd0cfef64c6151a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.595+0000 7f267a7fc640 1 -- 172.21.15.45:0/2476816877 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c097880 con 0x5570aa7847b0 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.595+0000 7f267a7fc640 1 -- 172.21.15.45:0/2476816877 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268c0cba80 con 0x5570aa7847b0 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 -- 172.21.15.45:0/2476816877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 msgr2=0x5570aa786ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f268c003fd0 tx=0x7f268c09a970 comp rx=0 tx=0).stop 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 -- 172.21.15.45:0/2476816877 shutdown_connections 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa7847b0 0x5570aa786ba0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa784270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 --2- 172.21.15.45:0/2476816877 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 -- 172.21.15.45:0/2476816877 >> 172.21.15.45:0/2476816877 conn(0x5570aa775990 msgr2=0x5570aa775d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 -- 172.21.15.45:0/2476816877 shutdown_connections 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.596+0000 7f269ce4d740 1 -- 172.21.15.45:0/2476816877 wait complete. 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.597+0000 7f269ce4d740 1 -- start start 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.597+0000 7f269ce4d740 1 Processor -- start 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.598+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa7867d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.598+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.598+0000 7f2693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa7867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.598+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa784480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.598+0000 7f2693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa7867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58192/0 (socket says 172.21.15.45:58192) 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44086/0 (socket says 172.21.15.45:44086) 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f2693fff640 1 -- 172.21.15.45:0/4162946674 learned_addr learned my addr 172.21.15.45:0/4162946674 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa3b6430 con 0x5570aa77b2b0 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa372e00 con 0x5570aa784000 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f269881e640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa784480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa667fd0 con 0x5570aa77bbd0 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.599+0000 7f269ce4d740 10 client.0 _ll_register_callbacks cb 0x5570aa6bdfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 -- 172.21.15.45:0/4162946674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 msgr2=0x5570aa783ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 -- 172.21.15.45:0/4162946674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 msgr2=0x5570aa7867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa7867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2688009490 con 0x5570aa784000 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f26937fe640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269881e640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa784480 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f268c0a3440 tx=0x7f268c003d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6207696f2119f3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f26737fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c0972c0 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f268c09b1e0 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.600+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570aa41f720 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f268c09ec30 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 10 client.28736 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570aa6e1610 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f268c09aae0 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f268c01f060 con 0x5570aa784000 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 10 client.28736 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 10 client.28736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 20 client.28736 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7945d0 0x5570aa7969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 4 client.28736 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.601+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570aa41f720 con 0x5570aa7945d0 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.602+0000 7f2693fff640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7945d0 0x5570aa7969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.602+0000 7f26737fe640 10 client.28736 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.602+0000 7f2693fff640 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7945d0 0x5570aa7969e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f53918cf1a3ed9a server_cookie=3188cf9153c42aab in_seq=0 out_seq=0 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.603+0000 7f26737fe640 1 -- 172.21.15.45:0/4162946674 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5570aa41f720 con 0x5570aa7945d0 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.603+0000 7f26737fe640 10 client.28736 handle_command_reply: tid=0 2024-03-12T05:25:53.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.603+0000 7f269ce4d740 1 client.28736 shutdown 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 msgr2=0x5570aa784480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa784480 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f268c0a3440 tx=0x7f268c003d70 comp rx=0 tx=0).stop 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 shutdown_connections 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7945d0 0x5570aa7969e0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa784480 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa783ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4162946674 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa7867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 >> 172.21.15.45:0/4162946674 conn(0x5570aa775990 msgr2=0x5570aa777730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.604+0000 7f26737fe640 0 client.28736 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.605+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 shutdown_connections 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.605+0000 7f269ce4d740 1 -- 172.21.15.45:0/4162946674 wait complete. 2024-03-12T05:25:53.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.605+0000 7f269ce4d740 20 client.28736 trim_cache size 0 max 16384 2024-03-12T05:25:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.638+0000 7f269ce4d740 1 Processor -- start 2024-03-12T05:25:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.638+0000 7f269ce4d740 1 -- start start 2024-03-12T05:25:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.638+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa78b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269ce4d740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa794400 con 0x5570aa77b2b0 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f2693fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44102/0 (socket says 172.21.15.45:44102) 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f26937fe640 1 -- 172.21.15.45:0/4015986138 learned_addr learned my addr 172.21.15.45:0/4015986138 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269881e640 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa78b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa787480 con 0x5570aa784000 2024-03-12T05:25:53.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.639+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa776790 con 0x5570aa77bbd0 2024-03-12T05:25:53.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.640+0000 7f26937fe640 1 -- 172.21.15.45:0/4015986138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 msgr2=0x5570aa78b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.640+0000 7f26937fe640 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa78b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.640+0000 7f26937fe640 1 -- 172.21.15.45:0/4015986138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 msgr2=0x5570aa77e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.640+0000 7f26937fe640 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.640+0000 7f26937fe640 1 -- 172.21.15.45:0/4015986138 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570aa67dac0 con 0x5570aa77bbd0 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.641+0000 7f26937fe640 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2684005fd0 tx=0x7f2684014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5d55614bad0769 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.641+0000 7f2679ffb640 1 -- 172.21.15.45:0/4015986138 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268402b070 con 0x5570aa77bbd0 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.641+0000 7f2679ffb640 1 -- 172.21.15.45:0/4015986138 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f2684022040 con 0x5570aa77bbd0 2024-03-12T05:25:53.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.641+0000 7f2679ffb640 1 -- 172.21.15.45:0/4015986138 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2684021070 con 0x5570aa77bbd0 2024-03-12T05:25:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.642+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 msgr2=0x5570aa786300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.642+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2684005fd0 tx=0x7f2684014000 comp rx=0 tx=0).stop 2024-03-12T05:25:53.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 shutdown_connections 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa784000 0x5570aa78b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa786300 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 --2- 172.21.15.45:0/4015986138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa77b2b0 0x5570aa77e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 >> 172.21.15.45:0/4015986138 conn(0x5570aa775990 msgr2=0x5570aa775d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 shutdown_connections 2024-03-12T05:25:53.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.643+0000 7f269ce4d740 1 -- 172.21.15.45:0/4015986138 wait complete. 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.644+0000 7f269ce4d740 1 -- start start 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.644+0000 7f269ce4d740 1 Processor -- start 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.644+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa78a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:34416/0 (socket says 172.21.15.45:34416) 2024-03-12T05:25:53.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa784000 0x5570aa78ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa78a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f26937fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa78a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:44104/0 (socket says 172.21.15.45:44104) 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 -- 172.21.15.45:0/1003865298 learned_addr learned my addr 172.21.15.45:0/1003865298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa7eab10 con 0x5570aa784000 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa7eac80 con 0x5570aa77b2b0 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa7eae40 con 0x5570aa77bbd0 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f269ce4d740 10 client.0 _ll_register_callbacks cb 0x7f268c003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 -- 172.21.15.45:0/1003865298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 msgr2=0x5570aa78a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa78a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 -- 172.21.15.45:0/1003865298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa784000 msgr2=0x5570aa78ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:25:53.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa784000 0x5570aa78ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.645+0000 7f2693fff640 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2684017050 con 0x5570aa77b2b0 2024-03-12T05:25:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.646+0000 7f2693fff640 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f26880040e0 tx=0x7f2688004210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8883a0e0d19c31d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:25:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.646+0000 7f26737fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:25:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.646+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688034d40 con 0x5570aa77b2b0 2024-03-12T05:25:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570aa3e27e0 con 0x5570aa77b2b0 2024-03-12T05:25:53.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f2688045b00 con 0x5570aa77b2b0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2688034ea0 con 0x5570aa77b2b0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f269ce4d740 10 client.28742 fetch_fsmap learned FSMap version 33 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570aa6a36a0 con 0x5570aa77b2b0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5570aa3e27e0 con 0x5570aa77b2b0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.647+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f26880053a0 con 0x5570aa77b2b0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 10 client.28742 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 10 client.28742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 20 client.28742 populate_metadata read hostname 'smithi045' 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7ec720 0x5570aa7eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 4 client.28742 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5570aa46e890 con 0x5570aa7ec720 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.648+0000 7f26937fe640 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7ec720 0x5570aa7eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.649+0000 7f26937fe640 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7ec720 0x5570aa7eeb40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67f8aefe4f96fb4 server_cookie=5f9591562a9caf26 in_seq=0 out_seq=0 2024-03-12T05:25:53.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.649+0000 7f26737fe640 10 client.28742 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.649+0000 7f26737fe640 1 -- 172.21.15.45:0/1003865298 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5570aa46e890 con 0x5570aa7ec720 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f26737fe640 10 client.28742 handle_command_reply: tid=0 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 client.28742 shutdown 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 msgr2=0x5570aa78a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f26880040e0 tx=0x7f2688004210 comp rx=0 tx=0).stop 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 shutdown_connections 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5570aa7ec720 0x5570aa7eeb40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570aa784000 0x5570aa78ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570aa77bbd0 0x5570aa78a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 --2- 172.21.15.45:0/1003865298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570aa77b2b0 0x5570aa78a2a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 >> 172.21.15.45:0/1003865298 conn(0x5570aa775990 msgr2=0x5570aa77e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f26737fe640 0 client.28742 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:25:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 shutdown_connections 2024-03-12T05:25:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.650+0000 7f269ce4d740 1 -- 172.21.15.45:0/1003865298 wait complete. 2024-03-12T05:25:53.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.651+0000 7f269ce4d740 20 client.28742 trim_cache size 0 max 16384 2024-03-12T05:25:53.648 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:25:53.648 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:25:53.648 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:25:53.648 INFO:teuthology.orchestra.run.smithi045.stdout: "7278cce9-3a6a-4c90-b5e3-3354c2f40860": { 2024-03-12T05:25:53.648 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:25:53.649 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "7278cce9-3a6a-4c90-b5e3-3354c2f40860", 2024-03-12T05:25:53.649 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:25:53.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:25:53.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:25:53.649 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:25:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.651+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 msgr2=0x7f267c0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.651+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 0x7f267c0a4230 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2694107290 tx=0x7f2684009260 comp rx=0 tx=0).stop 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.651+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 msgr2=0x7f269419db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.651+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 0x7f269419db00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f268c00ec50 tx=0x7f268c00c6a0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 shutdown_connections 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f267c0a1d70 0x7f267c0a4230 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694108100 0x7f26941a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f269410c640 0x7f269419db00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 --2- 172.21.15.45:0/3435357741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2694107760 0x7f26941a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:25:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 >> 172.21.15.45:0/3435357741 conn(0x7f2694076f50 msgr2=0x7f2694077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:25:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 shutdown_connections 2024-03-12T05:25:53.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:25:53.652+0000 7f269a281640 1 -- 172.21.15.45:0/3435357741 wait complete. 2024-03-12T05:25:53.667 INFO:tasks.cephfs.filesystem:scrub status for tag:7278cce9-3a6a-4c90-b5e3-3354c2f40860 - {'path': '/', 'tag': '7278cce9-3a6a-4c90-b5e3-3354c2f40860', 'options': 'recursive,force'} 2024-03-12T05:25:54.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: pgmap v471: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 9.1 MiB/s wr, 1.07k op/s 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:53 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: pgmap v471: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 9.1 MiB/s wr, 1.07k op/s 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:53 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: pgmap v471: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 9.1 MiB/s wr, 1.07k op/s 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:53 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 830.0 s, 1253.0 tps, lat 79.828 ms stddev 28.600 2024-03-12T05:25:55.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:54 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:25:55.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:54 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:55.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:54 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:55.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:54 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:25:55.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:54 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:55.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:54 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:55.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:54 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:25:55.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:54 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:55.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:54 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:25:56.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:55 smithi045 ceph-mon[32174]: pgmap v472: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:25:56.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:55 smithi130 ceph-mon[35398]: pgmap v472: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:25:56.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:55 smithi116 ceph-mon[35486]: pgmap v472: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-03-12T05:25:58.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:57 smithi045 ceph-mon[32174]: pgmap v473: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.1 MiB/s wr, 1.04k op/s 2024-03-12T05:25:58.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:57 smithi130 ceph-mon[35398]: pgmap v473: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.1 MiB/s wr, 1.04k op/s 2024-03-12T05:25:58.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:57 smithi116 ceph-mon[35486]: pgmap v473: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.1 MiB/s wr, 1.04k op/s 2024-03-12T05:25:59.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 835.0 s, 1256.6 tps, lat 79.259 ms stddev 49.509 2024-03-12T05:26:00.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:25:59 smithi045 ceph-mon[32174]: pgmap v474: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:26:00.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:25:59 smithi130 ceph-mon[35398]: pgmap v474: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:26:00.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:25:59 smithi116 ceph-mon[35486]: pgmap v474: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 MiB/s wr, 1.22k op/s 2024-03-12T05:26:02.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:01 smithi045 ceph-mon[32174]: pgmap v475: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:26:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:01 smithi130 ceph-mon[35398]: pgmap v475: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:26:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:01 smithi116 ceph-mon[35486]: pgmap v475: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 13 MiB/s wr, 1.43k op/s 2024-03-12T05:26:04.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:03 smithi045 ceph-mon[32174]: pgmap v476: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:26:04.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:03 smithi130 ceph-mon[35398]: pgmap v476: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:26:04.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:03 smithi116 ceph-mon[35486]: pgmap v476: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.2 MiB/s wr, 1.04k op/s 2024-03-12T05:26:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 840.0 s, 1243.0 tps, lat 80.622 ms stddev 66.829 2024-03-12T05:26:06.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:05 smithi045 ceph-mon[32174]: pgmap v477: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:26:06.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:05 smithi130 ceph-mon[35398]: pgmap v477: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:26:06.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:05 smithi116 ceph-mon[35486]: pgmap v477: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-03-12T05:26:08.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:07 smithi045 ceph-mon[32174]: pgmap v478: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-03-12T05:26:08.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:07 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:07 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:07 smithi130 ceph-mon[35398]: pgmap v478: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-03-12T05:26:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:07 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:07 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:08.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:07 smithi116 ceph-mon[35486]: pgmap v478: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 9.0 MiB/s wr, 1.03k op/s 2024-03-12T05:26:08.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:07 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:08.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:07 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:08 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:09.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:08 smithi045 ceph-mon[32174]: osdmap e107: 12 total, 12 up, 12 in 2024-03-12T05:26:09.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:09.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:08 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:09.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:08 smithi130 ceph-mon[35398]: osdmap e107: 12 total, 12 up, 12 in 2024-03-12T05:26:09.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:09.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:08 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:09.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:08 smithi116 ceph-mon[35486]: osdmap e107: 12 total, 12 up, 12 in 2024-03-12T05:26:09.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 845.0 s, 1190.2 tps, lat 83.901 ms stddev 80.117 2024-03-12T05:26:10.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:09 smithi045 ceph-mon[32174]: pgmap v480: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:26:10.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:09 smithi130 ceph-mon[35398]: pgmap v480: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:26:10.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:09 smithi116 ceph-mon[35486]: pgmap v480: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.21k op/s 2024-03-12T05:26:12.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:11 smithi045 ceph-mon[32174]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:12.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:11 smithi130 ceph-mon[35398]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:12.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:11 smithi116 ceph-mon[35486]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:14.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:13 smithi045 ceph-mon[32174]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:14.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:13 smithi130 ceph-mon[35398]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:14.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:13 smithi116 ceph-mon[35486]: pgmap v482: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.7 MiB/s wr, 1.15k op/s 2024-03-12T05:26:14.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 850.0 s, 1327.6 tps, lat 75.535 ms stddev 50.686 2024-03-12T05:26:15.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:15 smithi045 ceph-mon[32174]: osdmap e108: 12 total, 12 up, 12 in 2024-03-12T05:26:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:15 smithi130 ceph-mon[35398]: osdmap e108: 12 total, 12 up, 12 in 2024-03-12T05:26:15.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:15 smithi116 ceph-mon[35486]: osdmap e108: 12 total, 12 up, 12 in 2024-03-12T05:26:16.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:16 smithi045 ceph-mon[32174]: pgmap v484: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.47k op/s 2024-03-12T05:26:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:16 smithi130 ceph-mon[35398]: pgmap v484: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.47k op/s 2024-03-12T05:26:16.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:16 smithi116 ceph-mon[35486]: pgmap v484: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.47k op/s 2024-03-12T05:26:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:18 smithi045 ceph-mon[32174]: pgmap v485: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:26:18.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:18 smithi130 ceph-mon[35398]: pgmap v485: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:26:18.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:18 smithi116 ceph-mon[35486]: pgmap v485: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-03-12T05:26:19.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:19 smithi045 ceph-mon[32174]: pgmap v486: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-03-12T05:26:19.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 855.0 s, 1297.6 tps, lat 76.663 ms stddev 29.832 2024-03-12T05:26:19.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:19 smithi130 ceph-mon[35398]: pgmap v486: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-03-12T05:26:19.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:19 smithi116 ceph-mon[35486]: pgmap v486: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-03-12T05:26:22.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:21 smithi045 ceph-mon[32174]: pgmap v487: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:22.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:21 smithi130 ceph-mon[35398]: pgmap v487: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:22.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:21 smithi116 ceph-mon[35486]: pgmap v487: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:23.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:26:23.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 -- 172.21.15.45:0/3106410456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 msgr2=0x7fbeec100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 --2- 172.21.15.45:0/3106410456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec100650 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009900 tx=0x7fbed402bc30 comp rx=0 tx=0).stop 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 -- 172.21.15.45:0/3106410456 shutdown_connections 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 --2- 172.21.15.45:0/3106410456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec105340 0x7fbeec0682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 --2- 172.21.15.45:0/3106410456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec100c20 0x7fbeec104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 --2- 172.21.15.45:0/3106410456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec100650 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 -- 172.21.15.45:0/3106410456 >> 172.21.15.45:0/3106410456 conn(0x7fbeec077230 msgr2=0x7fbeec077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 -- 172.21.15.45:0/3106410456 shutdown_connections 2024-03-12T05:26:23.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.848+0000 7fbef106f640 1 -- 172.21.15.45:0/3106410456 wait complete. 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 Processor -- start 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 -- start start 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec10c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 0x7fbeec1a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec10d040 con 0x7fbeec100270 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1aa3d0 con 0x7fbeec1a7aa0 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.849+0000 7fbef106f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1aa560 con 0x7fbeec105340 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec10c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36344/0 (socket says 172.21.15.45:36344) 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 -- 172.21.15.45:0/2548339039 learned_addr learned my addr 172.21.15.45:0/2548339039 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:23.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 0x7fbeec1a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 -- 172.21.15.45:0/2548339039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 msgr2=0x7fbeec1a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 0x7fbeec1a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 -- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 msgr2=0x7fbeec10c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec10c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeea59c640 1 -- 172.21.15.45:0/2548339039 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee4009490 con 0x7fbeec105340 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 0x7fbeec1a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.850+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec10c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.851+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00ebc0 tx=0x7fbedc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.851+0000 7fbee3fff640 1 -- 172.21.15.45:0/2548339039 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc010630 con 0x7fbeec105340 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.851+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed4009560 con 0x7fbeec105340 2024-03-12T05:26:23.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.851+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec1aac10 con 0x7fbeec105340 2024-03-12T05:26:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.852+0000 7fbee3fff640 1 -- 172.21.15.45:0/2548339039 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbedc00ced0 con 0x7fbeec105340 2024-03-12T05:26:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.852+0000 7fbee3fff640 1 -- 172.21.15.45:0/2548339039 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc018a90 con 0x7fbeec105340 2024-03-12T05:26:23.848 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.854+0000 7fbee3fff640 1 -- 172.21.15.45:0/2548339039 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbedc018bf0 con 0x7fbeec105340 2024-03-12T05:26:23.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.855+0000 7fbee3fff640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 0x7fbed80a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbee3fff640 1 -- 172.21.15.45:0/2548339039 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbedc014070 con 0x7fbeec105340 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 Processor -- start 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 -- start start 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 0x7fbed80a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900e8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e8bd0 0x55ea900f1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea90057280 con 0x55ea900e8bd0 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea900547b0 con 0x55ea900e82b0 2024-03-12T05:26:23.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbef3c42740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea9004ff90 con 0x55ea900f17b0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbeeb59e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbeeb59e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36354/0 (socket says 172.21.15.45:36354) 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbeeb59e640 1 -- 172.21.15.45:0/3550243195 learned_addr learned my addr 172.21.15.45:0/3550243195 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.856+0000 7fbeead9d640 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 -- 172.21.15.45:0/3550243195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 msgr2=0x55ea900e8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900e8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 -- 172.21.15.45:0/3550243195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e8bd0 msgr2=0x55ea900f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e8bd0 0x55ea900f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 -- 172.21.15.45:0/3550243195 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea9000e570 con 0x55ea900f17b0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 0x7fbed80a4160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005940 tx=0x7fbed40047a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.857+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009d10 tx=0x7fbee40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a147ee0a7048ce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.858+0000 7fbee17fa640 1 -- 172.21.15.45:0/3550243195 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4012450 con 0x55ea900f17b0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.858+0000 7fbee17fa640 1 -- 172.21.15.45:0/3550243195 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbee4012a60 con 0x55ea900f17b0 2024-03-12T05:26:23.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.858+0000 7fbee17fa640 1 -- 172.21.15.45:0/3550243195 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee400fb60 con 0x55ea900f17b0 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- 172.21.15.45:0/3550243195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 msgr2=0x55ea900f3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009d10 tx=0x7fbee40117b0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- 172.21.15.45:0/3550243195 shutdown_connections 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e8bd0 0x55ea900f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f3ba0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3550243195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900e8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- 172.21.15.45:0/3550243195 >> 172.21.15.45:0/3550243195 conn(0x55ea900e2990 msgr2=0x55ea900e2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- 172.21.15.45:0/3550243195 shutdown_connections 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- 172.21.15.45:0/3550243195 wait complete. 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 -- start start 2024-03-12T05:26:23.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.859+0000 7fbef3c42740 1 Processor -- start 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900eed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8fd23430 con 0x55ea900e82b0 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8fcdfe00 con 0x55ea900e8bd0 2024-03-12T05:26:23.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea8ffd4fd0 con 0x55ea900f17b0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbef3c42740 10 client.0 _ll_register_callbacks cb 0x55ea9002afd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54480/0 (socket says 172.21.15.45:54480) 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeead9d640 1 -- 172.21.15.45:0/3640129191 learned_addr learned my addr 172.21.15.45:0/3640129191 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 -- 172.21.15.45:0/3640129191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 msgr2=0x55ea900eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 -- 172.21.15.45:0/3640129191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 msgr2=0x55ea900f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.860+0000 7fbeea59c640 1 -- 172.21.15.45:0/3640129191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee4009970 con 0x55ea900e8bd0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbeead9d640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbeea59c640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900ee5c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cb410 tx=0x7fbedc09a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2172af4bdadaf3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc018ea0 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea8fd4f7e0 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbedc09ea40 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbedc0cb8a0 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbef3c42740 10 client.28754 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.861+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ea9004e610 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbedc0b7080 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbedc099dd0 con 0x55ea900e8bd0 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 10 client.28754 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 10 client.28754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 20 client.28754 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90101b40 0x55ea90103f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 4 client.28754 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea8fd8c720 con 0x55ea90101b40 2024-03-12T05:26:23.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.862+0000 7fbeead9d640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90101b40 0x55ea90103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.863+0000 7fbed27fc640 10 client.28754 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.863+0000 7fbeead9d640 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90101b40 0x55ea90103f60 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c955d41b1e8273d server_cookie=fbb727eb7c62e0e8 in_seq=0 out_seq=0 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbed27fc640 1 -- 172.21.15.45:0/3640129191 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ea8fd8c720 con 0x55ea90101b40 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbed27fc640 10 client.28754 handle_command_reply: tid=0 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 client.28754 shutdown 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 msgr2=0x55ea900ee5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900ee5c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cb410 tx=0x7fbedc09a250 comp rx=0 tx=0).stop 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 shutdown_connections 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90101b40 0x55ea90103f60 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900ee5c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 --2- 172.21.15.45:0/3640129191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 >> 172.21.15.45:0/3640129191 conn(0x55ea900e2990 msgr2=0x55ea900e4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbed27fc640 0 client.28754 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 shutdown_connections 2024-03-12T05:26:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 1 -- 172.21.15.45:0/3640129191 wait complete. 2024-03-12T05:26:23.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.864+0000 7fbef3c42740 20 client.28754 trim_cache size 0 max 16384 2024-03-12T05:26:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 Processor -- start 2024-03-12T05:26:23.887 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 -- start start 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea900f3fa0 con 0x55ea900e82b0 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea900e3c10 con 0x55ea900e8bd0 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbef3c42740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea900ecb10 con 0x55ea900f17b0 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbeeb59e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbeeb59e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36370/0 (socket says 172.21.15.45:36370) 2024-03-12T05:26:23.888 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbeeb59e640 1 -- 172.21.15.45:0/4145254790 learned_addr learned my addr 172.21.15.45:0/4145254790 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.892+0000 7fbeead9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54492/0 (socket says 172.21.15.45:54492) 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeea59c640 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 -- 172.21.15.45:0/4145254790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 msgr2=0x55ea900f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900f0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 -- 172.21.15.45:0/4145254790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 msgr2=0x55ea900ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 -- 172.21.15.45:0/4145254790 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea8ffeaac0 con 0x55ea900f17b0 2024-03-12T05:26:23.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeead9d640 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.893+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009940 tx=0x7fbee400f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5502ca750091408 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.894+0000 7fbee0ff9640 1 -- 172.21.15.45:0/4145254790 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4009e80 con 0x55ea900f17b0 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.894+0000 7fbee0ff9640 1 -- 172.21.15.45:0/4145254790 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbee400f440 con 0x55ea900f17b0 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.894+0000 7fbee0ff9640 1 -- 172.21.15.45:0/4145254790 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4005040 con 0x55ea900f17b0 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 -- 172.21.15.45:0/4145254790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 msgr2=0x55ea900f1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009940 tx=0x7fbee400f6c0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 -- 172.21.15.45:0/4145254790 shutdown_connections 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900f17b0 0x55ea900f1230 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e8bd0 0x55ea900f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 --2- 172.21.15.45:0/4145254790 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900e82b0 0x55ea900ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 -- 172.21.15.45:0/4145254790 >> 172.21.15.45:0/4145254790 conn(0x55ea900e2990 msgr2=0x55ea900ede50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 -- 172.21.15.45:0/4145254790 shutdown_connections 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.895+0000 7fbef3c42740 1 -- 172.21.15.45:0/4145254790 wait complete. 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.896+0000 7fbef3c42740 1 -- start start 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.896+0000 7fbef3c42740 1 Processor -- start 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.896+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.896+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbef3c42740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 0x55ea90158010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbef3c42740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea900f8410 con 0x55ea900f17b0 2024-03-12T05:26:23.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbef3c42740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea90158550 con 0x55ea900e82b0 2024-03-12T05:26:23.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbef3c42740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea90158690 con 0x55ea900e8bd0 2024-03-12T05:26:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbef3c42740 10 client.0 _ll_register_callbacks cb 0x7fbee4009760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36382/0 (socket says 172.21.15.45:36382) 2024-03-12T05:26:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 -- 172.21.15.45:0/2434413718 learned_addr learned my addr 172.21.15.45:0/2434413718 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:23.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 0x55ea90158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 -- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 msgr2=0x55ea900f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900f75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 -- 172.21.15.45:0/2434413718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 msgr2=0x55ea90158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 0x55ea90158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeea59c640 1 -- 172.21.15.45:0/2434413718 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed4049050 con 0x55ea900e8bd0 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.897+0000 7fbeeb59e640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 0x55ea90158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.898+0000 7fbeea59c640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09a5f0 tx=0x7fbedc000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbfdf9a1ee2b7090 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.898+0000 7fbed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:23.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.898+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc018ea0 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.898+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fbedc09ea40 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.899+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbee4012a60 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.899+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbedc09b070 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.899+0000 7fbef3c42740 10 client.28714 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.899+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fbee401e040 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.899+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55ea8fd4f7e0 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbedc0cbb60 con 0x55ea900e8bd0 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 10 client.28714 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 10 client.28714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 20 client.28714 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90159be0 0x55ea9015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 4 client.28714 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:26:23.899 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fbee4012a60 con 0x55ea90159be0 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.900+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90159be0 0x55ea9015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbed27fc640 10 client.28714 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbeead9d640 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90159be0 0x55ea9015c000 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=738d83b851513b9b server_cookie=1890590277465d12 in_seq=0 out_seq=0 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbed27fc640 1 -- 172.21.15.45:0/2434413718 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fbee4012a60 con 0x55ea90159be0 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbed27fc640 10 client.28714 handle_command_reply: tid=0 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbef3c42740 1 client.28714 shutdown 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 msgr2=0x55ea900f7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.901+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbedc09a5f0 tx=0x7fbedc000f40 comp rx=0 tx=0).stop 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 shutdown_connections 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55ea90159be0 0x55ea9015c000 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55ea900f17b0 0x55ea90158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea900e8bd0 0x55ea900f7b10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 --2- 172.21.15.45:0/2434413718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea900e82b0 0x55ea900f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 >> 172.21.15.45:0/2434413718 conn(0x55ea900e2990 msgr2=0x55ea900ece90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbed27fc640 0 client.28714 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 shutdown_connections 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 1 -- 172.21.15.45:0/2434413718 wait complete. 2024-03-12T05:26:23.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.902+0000 7fbef3c42740 20 client.28714 trim_cache size 0 max 16384 2024-03-12T05:26:23.902 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:26:23.902 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:26:23.902 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:26:23.902 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 msgr2=0x7fbed80a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 0x7fbed80a4160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005940 tx=0x7fbed40047a0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 msgr2=0x7fbeec10c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00ebc0 tx=0x7fbedc00c6a0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 shutdown_connections 2024-03-12T05:26:23.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbed80a1ca0 0x7fbed80a4160 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbeec1a7aa0 0x7fbeec1a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105340 0x7fbeec10c900 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 --2- 172.21.15.45:0/2548339039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeec100270 0x7fbeec10c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 >> 172.21.15.45:0/2548339039 conn(0x7fbeec077230 msgr2=0x7fbeec107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 shutdown_connections 2024-03-12T05:26:23.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:23.903+0000 7fbef106f640 1 -- 172.21.15.45:0/2548339039 wait complete. 2024-03-12T05:26:23.915 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:26:23.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:26:23.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:23 smithi045 ceph-mon[32174]: pgmap v488: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:24.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 -- 172.21.15.45:0/1940324638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0204100ca0 msgr2=0x7f0204104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 --2- 172.21.15.45:0/1940324638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0204100ca0 0x7f0204104ce0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009860 tx=0x7f01ec02bb70 comp rx=0 tx=0).stop 2024-03-12T05:26:24.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 -- 172.21.15.45:0/1940324638 shutdown_connections 2024-03-12T05:26:24.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 --2- 172.21.15.45:0/1940324638 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 0x7f020410cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 --2- 172.21.15.45:0/1940324638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0204100ca0 0x7f0204104ce0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 --2- 172.21.15.45:0/1940324638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02041002f0 0x7f02041006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.099+0000 7f020a181640 1 -- 172.21.15.45:0/1940324638 >> 172.21.15.45:0/1940324638 conn(0x7f0204077180 msgr2=0x7f0204077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.100+0000 7f020a181640 1 -- 172.21.15.45:0/1940324638 shutdown_connections 2024-03-12T05:26:24.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.101+0000 7f020a181640 1 -- 172.21.15.45:0/1940324638 wait complete. 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 Processor -- start 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 -- start start 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02041002f0 0x7f0204195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 0x7f0204195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419a800 con 0x7f02041053c0 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419a970 con 0x7f0204199e10 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.102+0000 7f020a181640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419aae0 con 0x7f02041002f0 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43604/0 (socket says 172.21.15.45:43604) 2024-03-12T05:26:24.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 -- 172.21.15.45:0/2082444874 learned_addr learned my addr 172.21.15.45:0/2082444874 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f01faffd640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 0x7f0204195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 -- 172.21.15.45:0/2082444874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02041002f0 msgr2=0x7f0204195550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02041002f0 0x7f0204195550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 -- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 msgr2=0x7f0204195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 0x7f0204195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.103+0000 7f0203fff640 1 -- 172.21.15.45:0/2082444874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4009490 con 0x7f0204199e10 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f0203fff640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009fd0 tx=0x7f01fc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc004b20 con 0x7f0204199e10 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f01fc003e20 con 0x7f0204199e10 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0047b0 con 0x7f0204199e10 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec009490 con 0x7f0204199e10 2024-03-12T05:26:24.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.104+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02041a2360 con 0x7f0204199e10 2024-03-12T05:26:24.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.108+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f01fc00a040 con 0x7f0204199e10 2024-03-12T05:26:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.108+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02041017c0 con 0x7f0204199e10 2024-03-12T05:26:24.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.109+0000 7f02017fa640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 0x7f01f00a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.109+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f01fc0cd7a0 con 0x7f0204199e10 2024-03-12T05:26:24.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.110+0000 7f02037fe640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 0x7f01f00a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.111+0000 7f02037fe640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 0x7f01f00a8640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009c10 tx=0x7f01f4005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.114+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f01fc0965e0 con 0x7f0204199e10 2024-03-12T05:26:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:23 smithi130 ceph-mon[35398]: pgmap v488: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:24.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:23 smithi116 ceph-mon[35486]: pgmap v488: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-03-12T05:26:24.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.512+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02041968a0 con 0x7f0204199e10 2024-03-12T05:26:24.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.513+0000 7f02017fa640 1 -- 172.21.15.45:0/2082444874 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f01fc01a090 con 0x7f0204199e10 2024-03-12T05:26:24.509 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:26:24.509 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:26:24.509 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.519+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 msgr2=0x7f01f00a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 0x7f01f00a8640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009c10 tx=0x7f01f4005f90 comp rx=0 tx=0).stop 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 msgr2=0x7f020419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009fd0 tx=0x7f01fc00c800 comp rx=0 tx=0).stop 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 shutdown_connections 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01f00a6180 0x7f01f00a8640 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02041053c0 0x7f0204195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0204199e10 0x7f020419a2c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 --2- 172.21.15.45:0/2082444874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02041002f0 0x7f0204195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 >> 172.21.15.45:0/2082444874 conn(0x7f0204077180 msgr2=0x7f02040fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 shutdown_connections 2024-03-12T05:26:24.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.520+0000 7f020a181640 1 -- 172.21.15.45:0/2082444874 wait complete. 2024-03-12T05:26:24.533 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:24.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:26:24.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 -- 172.21.15.45:0/3805407103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068ca0 msgr2=0x7f4b1810d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/3805407103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068ca0 0x7f4b1810d5e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009ca0 tx=0x7f4b1002c190 comp rx=0 tx=0).stop 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 -- 172.21.15.45:0/3805407103 shutdown_connections 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/3805407103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1810db20 0x7f4b1810ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/3805407103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068ca0 0x7f4b1810d5e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/3805407103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b18068380 0x7f4b18068760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 -- 172.21.15.45:0/3805407103 >> 172.21.15.45:0/3805407103 conn(0x7f4b18077180 msgr2=0x7f4b18077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 -- 172.21.15.45:0/3805407103 shutdown_connections 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 -- 172.21.15.45:0/3805407103 wait complete. 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.758+0000 7f4b1fe39640 1 Processor -- start 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 -- start start 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 0x7f4b1810c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b1810db20 0x7f4b181a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181a3e20 con 0x7f4b18068ca0 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181a3f90 con 0x7f4b18068380 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1fe39640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181a4150 con 0x7f4b1810db20 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1d3d4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 0x7f4b1810c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1d3d4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54522/0 (socket says 172.21.15.45:54522) 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/2315672704 learned_addr learned my addr 172.21.15.45:0/2315672704 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b1810db20 0x7f4b181a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/2315672704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b1810db20 msgr2=0x7f4b181a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b1810db20 0x7f4b181a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 msgr2=0x7f4b1810c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/2315672704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10009900 con 0x7f4b18068380 2024-03-12T05:26:24.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.759+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 0x7f4b1810c040 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4b08005fd0 tx=0x7f4b0800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b0effd640 1 -- 172.21.15.45:0/2315672704 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08009c30 con 0x7f4b18068380 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b181a4430 con 0x7f4b18068380 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b0effd640 1 -- 172.21.15.45:0/2315672704 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4b080044e0 con 0x7f4b18068380 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b181aaab0 con 0x7f4b18068380 2024-03-12T05:26:24.758 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.760+0000 7f4b0effd640 1 -- 172.21.15.45:0/2315672704 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08003da0 con 0x7f4b18068380 2024-03-12T05:26:24.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.762+0000 7f4b0effd640 1 -- 172.21.15.45:0/2315672704 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4b08010460 con 0x7f4b18068380 2024-03-12T05:26:24.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.763+0000 7f4b0effd640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 0x7f4b040a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.764+0000 7f4b0effd640 1 -- 172.21.15.45:0/2315672704 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4b080cced0 con 0x7f4b18068380 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.764+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 0x7f4b040a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.764+0000 7f4b22a0a740 1 Processor -- start 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.764+0000 7f4b22a0a740 1 -- start start 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 0x558d36704210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d36706b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d366b6960 con 0x558d366fb250 2024-03-12T05:26:24.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3666a310 con 0x558d366fbb70 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d36667840 con 0x558d36704750 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1dbd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 0x7f4b040a41a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4b1810daf0 tx=0x7f4b1002e040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1e3d6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d36706b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1dbd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54528/0 (socket says 172.21.15.45:54528) 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/3531246949 learned_addr learned my addr 172.21.15.45:0/3531246949 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.765+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 0x558d36704210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/3531246949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 msgr2=0x558d36706b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d36706b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/3531246949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 msgr2=0x558d366fb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/3531246949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d36625060 con 0x558d366fbb70 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:24.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d36706b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:24.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 0x558d36704210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006010 tx=0x7f4b1003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5bea96b99e8d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b03fff640 1 -- 172.21.15.45:0/3531246949 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10032880 con 0x558d366fbb70 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.766+0000 7f4b03fff640 1 -- 172.21.15.45:0/3531246949 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4b10032e90 con 0x558d366fbb70 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.767+0000 7f4b03fff640 1 -- 172.21.15.45:0/3531246949 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1004c3f0 con 0x558d366fbb70 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.767+0000 7f4b22a0a740 1 -- 172.21.15.45:0/3531246949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 msgr2=0x558d36704210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.767+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 0x558d36704210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006010 tx=0x7f4b1003f000 comp rx=0 tx=0).stop 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 -- 172.21.15.45:0/3531246949 shutdown_connections 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d36706b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fbb70 0x558d36704210 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/3531246949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d366fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 -- 172.21.15.45:0/3531246949 >> 172.21.15.45:0/3531246949 conn(0x558d366f5980 msgr2=0x558d366f5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 -- 172.21.15.45:0/3531246949 shutdown_connections 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 -- 172.21.15.45:0/3531246949 wait complete. 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.768+0000 7f4b22a0a740 1 -- start start 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 Processor -- start 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fb250 0x558d36701760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 0x558d367021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d36663020 con 0x558d36704750 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d36336430 con 0x558d366fb250 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d362f2e00 con 0x558d366fbb70 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.769+0000 7f4b22a0a740 10 client.0 _ll_register_callbacks cb 0x558d3660acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1e3d6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 0x558d367021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36460/0 (socket says 172.21.15.45:36460) 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/2920869427 learned_addr learned my addr 172.21.15.45:0/2920869427 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fb250 0x558d36701760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/2920869427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fb250 msgr2=0x558d36701760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fb250 0x558d36701760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/2920869427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 msgr2=0x558d367021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 0x558d367021e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1d3d4640 1 -- 172.21.15.45:0/2920869427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b18110a70 con 0x558d366fbb70 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.770+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 0x558d367021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4b1003fa70 tx=0x7f4b1003faa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=198da37e5d9b59ee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b017fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10032a90 con 0x558d366fbb70 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558d363627e0 con 0x558d366fbb70 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4b10047040 con 0x558d366fbb70 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.771+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b1003fe80 con 0x558d366fbb70 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 10 client.28729 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:24.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558d366616a0 con 0x558d366fbb70 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4b1006b020 con 0x558d366fbb70 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4b1005f3f0 con 0x558d366fbb70 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 10 client.28729 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 10 client.28729 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 20 client.28729 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d36714740 0x558d36716b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 4 client.28729 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.772+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d3639f720 con 0x558d36714740 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.773+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d36714740 0x558d36716b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.773+0000 7f4b017fa640 10 client.28729 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:24.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.773+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d36714740 0x558d36716b60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dc17c72049c8a325 server_cookie=b185a4bd33f118ea in_seq=0 out_seq=0 2024-03-12T05:26:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.774+0000 7f4b017fa640 1 -- 172.21.15.45:0/2920869427 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558d3639f720 con 0x558d36714740 2024-03-12T05:26:24.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.774+0000 7f4b017fa640 10 client.28729 handle_command_reply: tid=0 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.774+0000 7f4b22a0a740 1 client.28729 shutdown 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.774+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 msgr2=0x558d36701ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.774+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4b1003fa70 tx=0x7f4b1003faa0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 shutdown_connections 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d36714740 0x558d36716b60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d36704750 0x558d367021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36701ca0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2920869427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d366fb250 0x558d36701760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 >> 172.21.15.45:0/2920869427 conn(0x558d366f5980 msgr2=0x558d366f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b017fa640 0 client.28729 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 shutdown_connections 2024-03-12T05:26:24.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2920869427 wait complete. 2024-03-12T05:26:24.773 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.775+0000 7f4b22a0a740 20 client.28729 trim_cache size 0 max 16384 2024-03-12T05:26:24.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:24 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2082444874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:24.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.795+0000 7f4b22a0a740 1 Processor -- start 2024-03-12T05:26:24.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.795+0000 7f4b22a0a740 1 -- start start 2024-03-12T05:26:24.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.795+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36702000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36704750 0x558d366fe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d365e8130 con 0x558d366fb250 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d366f12b0 con 0x558d36704750 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d366fabf0 con 0x558d366fbb70 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54550/0 (socket says 172.21.15.45:54550) 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/1015646205 learned_addr learned my addr 172.21.15.45:0/1015646205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36704750 0x558d366fe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36702000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/1015646205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 msgr2=0x558d36702000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36702000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/1015646205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36704750 msgr2=0x558d366fe9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36704750 0x558d366fe9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 -- 172.21.15.45:0/1015646205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d365fdc20 con 0x558d366fb250 2024-03-12T05:26:24.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1d3d4640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36702000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:24.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.796+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009610 tx=0x7f4b08003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673b81274a4bfc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.797+0000 7f4b037fe640 1 -- 172.21.15.45:0/1015646205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08098ce0 con 0x558d366fb250 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.797+0000 7f4b037fe640 1 -- 172.21.15.45:0/1015646205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4b08009d90 con 0x558d366fb250 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.797+0000 7f4b037fe640 1 -- 172.21.15.45:0/1015646205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080ca810 con 0x558d366fb250 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 -- 172.21.15.45:0/1015646205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 msgr2=0x558d36703e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009610 tx=0x7f4b08003f00 comp rx=0 tx=0).stop 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 -- 172.21.15.45:0/1015646205 shutdown_connections 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36704750 0x558d366fe9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d366fbb70 0x558d36702000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/1015646205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fb250 0x558d36703e90 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.798+0000 7f4b22a0a740 1 -- 172.21.15.45:0/1015646205 >> 172.21.15.45:0/1015646205 conn(0x558d366f5980 msgr2=0x558d366fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.799+0000 7f4b22a0a740 1 -- 172.21.15.45:0/1015646205 shutdown_connections 2024-03-12T05:26:24.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.799+0000 7f4b22a0a740 1 -- 172.21.15.45:0/1015646205 wait complete. 2024-03-12T05:26:24.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.800+0000 7f4b22a0a740 1 -- start start 2024-03-12T05:26:24.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.800+0000 7f4b22a0a740 1 Processor -- start 2024-03-12T05:26:24.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.800+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 0x558d3670a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d3670ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3670b2f0 con 0x558d366fbb70 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3676b560 con 0x558d36768c00 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d3676b720 con 0x558d36704750 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1d3d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d3670ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43666/0 (socket says 172.21.15.45:43666) 2024-03-12T05:26:24.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 -- 172.21.15.45:0/2977944052 learned_addr learned my addr 172.21.15.45:0/2977944052 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 0x558d3670a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 10 client.0 _ll_register_callbacks cb 0x7f4b14005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 -- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 msgr2=0x558d3670ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d3670ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 -- 172.21.15.45:0/2977944052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 msgr2=0x558d3670a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 0x558d3670a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1e3d6640 1 -- 172.21.15.45:0/2977944052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b181a4430 con 0x558d36768c00 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 0x558d3670a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.801+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b10032e90 con 0x558d36768c00 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b1e3d6640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4b14010c90 tx=0x7f4b1400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2b8a827c7702a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b017fa640 10 client.28781 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b017fa640 1 -- 172.21.15.45:0/2977944052 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14021070 con 0x558d36768c00 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b017fa640 1 -- 172.21.15.45:0/2977944052 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f4b14009cd0 con 0x558d36768c00 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b017fa640 1 -- 172.21.15.45:0/2977944052 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b14017040 con 0x558d36768c00 2024-03-12T05:26:24.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b22a0a740 10 client.28781 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:24.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x558d366238c0 con 0x558d36768c00 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b017fa640 1 -- 172.21.15.45:0/2977944052 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4b14024030 con 0x558d36768c00 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b22a0a740 10 client.28781 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b22a0a740 10 client.28781 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.802+0000 7f4b22a0a740 20 client.28781 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d3676cd30 0x558d3676f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:24.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b22a0a740 4 client.28781 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558d363ee890 con 0x558d3676cd30 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d3676cd30 0x558d3676f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b017fa640 10 client.28781 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.803+0000 7f4b1dbd5640 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d3676cd30 0x558d3676f150 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2ee3a1858974b47c server_cookie=31144bf392bc4487 in_seq=0 out_seq=0 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b017fa640 1 -- 172.21.15.45:0/2977944052 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558d363ee890 con 0x558d3676cd30 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b017fa640 10 client.28781 handle_command_reply: tid=0 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 client.28781 shutdown 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 msgr2=0x558d3676aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4b14010c90 tx=0x7f4b1400f790 comp rx=0 tx=0).stop 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 shutdown_connections 2024-03-12T05:26:24.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x558d3676cd30 0x558d3676f150 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558d36768c00 0x558d3676aff0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d36704750 0x558d3670ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 --2- 172.21.15.45:0/2977944052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558d366fbb70 0x558d3670a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b017fa640 0 client.28781 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.804+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 >> 172.21.15.45:0/2977944052 conn(0x558d366f5980 msgr2=0x558d366f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.805+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 shutdown_connections 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.805+0000 7f4b22a0a740 1 -- 172.21.15.45:0/2977944052 wait complete. 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.805+0000 7f4b22a0a740 20 client.28781 trim_cache size 0 max 16384 2024-03-12T05:26:24.801 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.806+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 msgr2=0x7f4b040a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.806+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 0x7f4b040a41a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4b1810daf0 tx=0x7f4b1002e040 comp rx=0 tx=0).stop 2024-03-12T05:26:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.806+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 msgr2=0x7f4b1810c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.806+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 0x7f4b1810c040 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4b08005fd0 tx=0x7f4b0800dde0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 shutdown_connections 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4b040a1ce0 0x7f4b040a41a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b18068ca0 0x7f4b1810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b1810db20 0x7f4b181a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 --2- 172.21.15.45:0/2315672704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b18068380 0x7f4b1810c040 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 >> 172.21.15.45:0/2315672704 conn(0x7f4b18077180 msgr2=0x7f4b1810f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 shutdown_connections 2024-03-12T05:26:24.803 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:24.807+0000 7f4b1fe39640 1 -- 172.21.15.45:0/2315672704 wait complete. 2024-03-12T05:26:24.819 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:24.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:26:24.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 860.0 s, 1334.6 tps, lat 74.807 ms stddev 41.414 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 -- 172.21.15.45:0/538140806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05140968f0 msgr2=0x7f051409a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 --2- 172.21.15.45:0/538140806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05140968f0 0x7f051409a930 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f050c009890 tx=0x7f050c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 -- 172.21.15.45:0/538140806 shutdown_connections 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 --2- 172.21.15.45:0/538140806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f05140968f0 0x7f051409a930 secure :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f050c009890 tx=0x7f050c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 --2- 172.21.15.45:0/538140806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140a2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 --2- 172.21.15.45:0/538140806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514095f40 0x7f0514096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.003+0000 7f052231f640 1 -- 172.21.15.45:0/538140806 >> 172.21.15.45:0/538140806 conn(0x7f051400ce90 msgr2=0x7f051400d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:24.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.004+0000 7f052231f640 1 -- 172.21.15.45:0/538140806 shutdown_connections 2024-03-12T05:26:24.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.004+0000 7f052231f640 1 -- 172.21.15.45:0/538140806 wait complete. 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.004+0000 7f052231f640 1 Processor -- start 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 -- start start 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 0x7f0514006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051413f030 con 0x7f0514095f40 2024-03-12T05:26:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051413f1a0 con 0x7f0514008b40 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052231f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051413f360 con 0x7f051409b010 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f052131d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 0x7f0514006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f0520b1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43676/0 (socket says 172.21.15.45:43676) 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.005+0000 7f0521b1e640 1 -- 172.21.15.45:0/1128396068 learned_addr learned my addr 172.21.15.45:0/1128396068 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 -- 172.21.15.45:0/1128396068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 msgr2=0x7f0514007920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 -- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 msgr2=0x7f0514006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 0x7f0514006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 -- 172.21.15.45:0/1128396068 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c009490 con 0x7f051409b010 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f052131d640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 0x7f0514006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.006+0000 7f0520b1c640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140073e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0514096e40 tx=0x7f050c0344c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.007+0000 7f05127fc640 1 -- 172.21.15.45:0/1128396068 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c02c710 con 0x7f051409b010 2024-03-12T05:26:25.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.007+0000 7f05127fc640 1 -- 172.21.15.45:0/1128396068 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f050c02ce90 con 0x7f051409b010 2024-03-12T05:26:25.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.008+0000 7f05127fc640 1 -- 172.21.15.45:0/1128396068 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c03d3f0 con 0x7f051409b010 2024-03-12T05:26:25.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.008+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051413f5e0 con 0x7f051409b010 2024-03-12T05:26:25.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.008+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051413f9f0 con 0x7f051409b010 2024-03-12T05:26:25.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.011+0000 7f05127fc640 1 -- 172.21.15.45:0/1128396068 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f050c03b040 con 0x7f051409b010 2024-03-12T05:26:25.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.012+0000 7f0526232740 1 Processor -- start 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.012+0000 7f0526232740 1 -- start start 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f05127fc640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 0x7f05080a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fbffe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc000d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbfb0960 con 0x5572fbff5460 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbf64310 con 0x5572fbff5d80 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0526232740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbf61840 con 0x5572fbffe960 2024-03-12T05:26:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f05127fc640 1 -- 172.21.15.45:0/1128396068 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f050c0ee650 con 0x7f051409b010 2024-03-12T05:26:25.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 0x7f05080a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f052131d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f052131d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54588/0 (socket says 172.21.15.45:54588) 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc000d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.013+0000 7f052131d640 1 -- 172.21.15.45:0/1405715182 learned_addr learned my addr 172.21.15.45:0/1405715182 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f0520b1c640 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fbffe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 -- 172.21.15.45:0/1405715182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 msgr2=0x5572fc000d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc000d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 -- 172.21.15.45:0/1405715182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 msgr2=0x5572fbffe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fbffe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 -- 172.21.15.45:0/1405715182 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572fbf1f060 con 0x5572fbff5460 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 0x7f05080a4160 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f051c066920 tx=0x7f051c068e60 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f052131d640 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0518011490 tx=0x7f0518011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bb8d28cb73844b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.014+0000 7f05077fe640 1 -- 172.21.15.45:0/1405715182 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518004020 con 0x5572fbff5460 2024-03-12T05:26:25.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f05077fe640 1 -- 172.21.15.45:0/1405715182 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f05180035f0 con 0x5572fbff5460 2024-03-12T05:26:25.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f05077fe640 1 -- 172.21.15.45:0/1405715182 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051800f720 con 0x5572fbff5460 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 -- 172.21.15.45:0/1405715182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 msgr2=0x5572fbff5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0518011490 tx=0x7f0518011950 comp rx=0 tx=0).stop 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 -- 172.21.15.45:0/1405715182 shutdown_connections 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc000d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fbffe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 --2- 172.21.15.45:0/1405715182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5460 0x5572fbff5840 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.015+0000 7f0526232740 1 -- 172.21.15.45:0/1405715182 >> 172.21.15.45:0/1405715182 conn(0x5572fbfefb90 msgr2=0x5572fbfeff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.016+0000 7f0526232740 1 -- 172.21.15.45:0/1405715182 shutdown_connections 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.016+0000 7f0526232740 1 -- 172.21.15.45:0/1405715182 wait complete. 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 -- start start 2024-03-12T05:26:25.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 Processor -- start 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffe960 0x5572fbffbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffc380 0x5572fc00a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbf5d020 con 0x5572fbffe960 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbc30430 con 0x5572fbffc380 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.017+0000 7f0526232740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbbece00 con 0x5572fbff5d80 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36504/0 (socket says 172.21.15.45:36504) 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 -- 172.21.15.45:0/3169530906 learned_addr learned my addr 172.21.15.45:0/3169530906 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f0520b1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffe960 0x5572fbffbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f0521b1e640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffc380 0x5572fc00a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f0526232740 10 client.0 _ll_register_callbacks cb 0x5572fbf04cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 -- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffc380 msgr2=0x5572fc00a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffc380 0x5572fc00a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 -- 172.21.15.45:0/3169530906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffe960 msgr2=0x5572fbffbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffe960 0x5572fbffbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 -- 172.21.15.45:0/3169530906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518009d00 con 0x5572fbff5d80 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f052131d640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f051801b040 tx=0x7f05180042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebb40201bf131cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f0504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.018+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518003d00 con 0x5572fbff5d80 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572fbc5c7e0 con 0x5572fbff5d80 2024-03-12T05:26:25.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f05180044a0 con 0x5572fbff5d80 2024-03-12T05:26:25.016 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0518005250 con 0x5572fbff5d80 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0526232740 10 client.28744 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5572fbf5b6a0 con 0x5572fbff5d80 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.019+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5572fbc5c7e0 con 0x5572fbff5d80 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0518021070 con 0x5572fbff5d80 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 10 client.28744 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 10 client.28744 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 20 client.28744 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc00f1e0 0x5572fc011640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 4 client.28744 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572fbc99720 con 0x5572fc00f1e0 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.020+0000 7f0520b1c640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc00f1e0 0x5572fc011640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.021+0000 7f0520b1c640 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc00f1e0 0x5572fc011640 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ba046efb0a15518d server_cookie=7466cf98c1f50cdf in_seq=0 out_seq=0 2024-03-12T05:26:25.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.021+0000 7f0504ff9640 10 client.28744 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0504ff9640 1 -- 172.21.15.45:0/3169530906 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5572fbc99720 con 0x5572fc00f1e0 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0504ff9640 10 client.28744 handle_command_reply: tid=0 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 client.28744 shutdown 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 msgr2=0x5572fbffb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f051801b040 tx=0x7f05180042b0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 shutdown_connections 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc00f1e0 0x5572fc011640 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffe960 0x5572fbffbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffc380 0x5572fc00a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 --2- 172.21.15.45:0/3169530906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbff5d80 0x5572fbffb900 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0504ff9640 0 client.28744 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 >> 172.21.15.45:0/3169530906 conn(0x5572fbfefb90 msgr2=0x5572fbff18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 shutdown_connections 2024-03-12T05:26:25.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 1 -- 172.21.15.45:0/3169530906 wait complete. 2024-03-12T05:26:25.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.022+0000 7f0526232740 20 client.28744 trim_cache size 0 max 16384 2024-03-12T05:26:25.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.046+0000 7f0526232740 1 Processor -- start 2024-03-12T05:26:25.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.046+0000 7f0526232740 1 -- start start 2024-03-12T05:26:25.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5d80 0x5572fc005840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 0x5572fc005d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 0x5572fbffb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbee2130 con 0x5572fbff5d80 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbff7530 con 0x5572fbffe960 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0520b1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 0x5572fc005d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0520b1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 0x5572fc005d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36516/0 (socket says 172.21.15.45:36516) 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 0x5572fbffb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0520b1c640 1 -- 172.21.15.45:0/2461034136 learned_addr learned my addr 172.21.15.45:0/2461034136 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.047+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fbfef9f0 con 0x5572fbffc380 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 -- 172.21.15.45:0/2461034136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 msgr2=0x5572fc005d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 0x5572fc005d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 -- 172.21.15.45:0/2461034136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5d80 msgr2=0x5572fc005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5d80 0x5572fc005840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 -- 172.21.15.45:0/2461034136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572fbef7c20 con 0x5572fbffe960 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0521b1e640 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 0x5572fbffb8c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f051c04fc90 tx=0x7f051c08a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52282f624fde74d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.048+0000 7f0506ffd640 1 -- 172.21.15.45:0/2461034136 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c07b070 con 0x5572fbffe960 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.049+0000 7f0506ffd640 1 -- 172.21.15.45:0/2461034136 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f051c07c040 con 0x5572fbffe960 2024-03-12T05:26:25.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.049+0000 7f0506ffd640 1 -- 172.21.15.45:0/2461034136 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c087690 con 0x5572fbffe960 2024-03-12T05:26:25.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 msgr2=0x5572fbffb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 0x5572fbffb8c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f051c04fc90 tx=0x7f051c08a000 comp rx=0 tx=0).stop 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 shutdown_connections 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbffe960 0x5572fbffb8c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffc380 0x5572fc005d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 --2- 172.21.15.45:0/2461034136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbff5d80 0x5572fc005840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 >> 172.21.15.45:0/2461034136 conn(0x5572fbfefb90 msgr2=0x5572fc015fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.050+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 shutdown_connections 2024-03-12T05:26:25.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.051+0000 7f0526232740 1 -- 172.21.15.45:0/2461034136 wait complete. 2024-03-12T05:26:25.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.051+0000 7f0526232740 1 -- start start 2024-03-12T05:26:25.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.052+0000 7f0526232740 1 Processor -- start 2024-03-12T05:26:25.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.052+0000 7f0526232740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fc0059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0526232740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 0x5572fc003d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.047 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0526232740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0520b1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 0x5572fc003d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0521b1e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36518/0 (socket says 172.21.15.45:36518) 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0521b1e640 1 -- 172.21.15.45:0/1474850353 learned_addr learned my addr 172.21.15.45:0/1474850353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fc0047c0 con 0x5572fbffc380 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f052131d640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fc0059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fc004930 con 0x5572fbff5d80 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.053+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fc004af0 con 0x5572fbffe960 2024-03-12T05:26:25.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0526232740 10 client.0 _ll_register_callbacks cb 0x7f0518009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0521b1e640 1 -- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 msgr2=0x5572fc0059a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fc0059a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0521b1e640 1 -- 172.21.15.45:0/1474850353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 msgr2=0x5572fc003d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 0x5572fc003d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0521b1e640 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c075680 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.054+0000 7f0520b1c640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 0x5572fc003d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0521b1e640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f051c0641a0 tx=0x7f051c092980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9418cc2c7fa82dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c07b040 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f051c08e040 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05180035f0 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f051c068a60 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0526232740 10 client.28750 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.055+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5572fbf1d8c0 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f051c09d8f0 con 0x5572fbffe960 2024-03-12T05:26:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f051c09b450 con 0x5572fbffe960 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 10 client.28750 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 10 client.28750 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 20 client.28750 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc066bb0 0x5572fc068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 4 client.28750 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f05180035f0 con 0x5572fc066bb0 2024-03-12T05:26:25.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.056+0000 7f052131d640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc066bb0 0x5572fc068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0504ff9640 10 client.28750 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f052131d640 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc066bb0 0x5572fc068fd0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b2218b33a7f05b4b server_cookie=cebf18aa8cdfc558 in_seq=0 out_seq=0 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0504ff9640 1 -- 172.21.15.45:0/1474850353 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f05180035f0 con 0x5572fc066bb0 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0504ff9640 10 client.28750 handle_command_reply: tid=0 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 client.28750 shutdown 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 msgr2=0x5572fc004280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f051c0641a0 tx=0x7f051c092980 comp rx=0 tx=0).stop 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 shutdown_connections 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5572fc066bb0 0x5572fc068fd0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5572fbffc380 0x5572fc003d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0504ff9640 0 client.28750 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5572fbffe960 0x5572fc004280 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 --2- 172.21.15.45:0/1474850353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572fbff5d80 0x5572fc0059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.057+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 >> 172.21.15.45:0/1474850353 conn(0x5572fbfefb90 msgr2=0x5572fbffb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.058+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 shutdown_connections 2024-03-12T05:26:25.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.058+0000 7f0526232740 1 -- 172.21.15.45:0/1474850353 wait complete. 2024-03-12T05:26:25.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.058+0000 7f0526232740 20 client.28750 trim_cache size 0 max 16384 2024-03-12T05:26:25.055 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.058+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 msgr2=0x7f05080a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 0x7f05080a4160 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f051c066920 tx=0x7f051c068e60 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 msgr2=0x7f05140073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140073e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0514096e40 tx=0x7f050c0344c0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 shutdown_connections 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f05080a1ca0 0x7f05080a4160 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0514008b40 0x7f0514007920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f051409b010 0x7f05140073e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 --2- 172.21.15.45:0/1128396068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0514095f40 0x7f0514006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 >> 172.21.15.45:0/1128396068 conn(0x7f051400ce90 msgr2=0x7f0514098820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 shutdown_connections 2024-03-12T05:26:25.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.059+0000 7f052231f640 1 -- 172.21.15.45:0/1128396068 wait complete. 2024-03-12T05:26:25.074 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:25.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:26:25.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:24 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2082444874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:25.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:24 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2082444874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.261+0000 7f034e18c640 1 -- 172.21.15.45:0/2971677975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101d10 msgr2=0x7f0348105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.261+0000 7f034e18c640 1 --2- 172.21.15.45:0/2971677975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101d10 0x7f0348105d50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f033c009900 tx=0x7f033c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 -- 172.21.15.45:0/2971677975 shutdown_connections 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 --2- 172.21.15.45:0/2971677975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0348106430 0x7f034810e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 --2- 172.21.15.45:0/2971677975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101d10 0x7f0348105d50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 --2- 172.21.15.45:0/2971677975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348101360 0x7f0348101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 -- 172.21.15.45:0/2971677975 >> 172.21.15.45:0/2971677975 conn(0x7f03480fcbe0 msgr2=0x7f03480ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 -- 172.21.15.45:0/2971677975 shutdown_connections 2024-03-12T05:26:25.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.262+0000 7f034e18c640 1 -- 172.21.15.45:0/2971677975 wait complete. 2024-03-12T05:26:25.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 Processor -- start 2024-03-12T05:26:25.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 -- start start 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 0x7f0348199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 0x7f0348199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 0x7f034819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034819e9e0 con 0x7f0348106430 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034819eb50 con 0x7f034819dff0 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f034e18c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034819ecc0 con 0x7f0348101360 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f03477fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 0x7f0348199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f03477fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 0x7f0348199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36546/0 (socket says 172.21.15.45:36546) 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.263+0000 7f03477fe640 1 -- 172.21.15.45:0/3780855302 learned_addr learned my addr 172.21.15.45:0/3780855302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 0x7f034819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0346ffd640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 0x7f0348199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 -- 172.21.15.45:0/3780855302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 msgr2=0x7f0348199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 0x7f0348199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 -- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 msgr2=0x7f0348199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 0x7f0348199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 -- 172.21.15.45:0/3780855302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c009560 con 0x7f034819dff0 2024-03-12T05:26:25.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0346ffd640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 0x7f0348199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.264+0000 7f0347fff640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 0x7f034819e4a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f034000ca30 tx=0x7f034000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.265+0000 7f0344ff9640 1 -- 172.21.15.45:0/3780855302 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340007d90 con 0x7f034819dff0 2024-03-12T05:26:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.265+0000 7f0344ff9640 1 -- 172.21.15.45:0/3780855302 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0340005020 con 0x7f034819dff0 2024-03-12T05:26:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.265+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03481a62a0 con 0x7f034819dff0 2024-03-12T05:26:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.265+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03481a66c0 con 0x7f034819dff0 2024-03-12T05:26:25.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.265+0000 7f0344ff9640 1 -- 172.21.15.45:0/3780855302 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034000f6b0 con 0x7f034819dff0 2024-03-12T05:26:25.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.268+0000 7f0344ff9640 1 -- 172.21.15.45:0/3780855302 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f034000f810 con 0x7f034819dff0 2024-03-12T05:26:25.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.268+0000 7f0350d66740 1 Processor -- start 2024-03-12T05:26:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 -- start start 2024-03-12T05:26:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0344ff9640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 0x7f03300a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe57a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe57f60 0x5570cbe5a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe01960 con 0x5570cbe57f60 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbdb5310 con 0x5570cbe4f380 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0350d66740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbdb2840 con 0x5570cbe4ea60 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0346ffd640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 0x7f03300a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f03477fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f03477fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36552/0 (socket says 172.21.15.45:36552) 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f03477fe640 1 -- 172.21.15.45:0/2815694571 learned_addr learned my addr 172.21.15.45:0/2815694571 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.269+0000 7f0347fff640 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe57f60 0x5570cbe5a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 -- 172.21.15.45:0/2815694571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 msgr2=0x5570cbe57a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe57a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 -- 172.21.15.45:0/2815694571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe57f60 msgr2=0x5570cbe5a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe57f60 0x5570cbe5a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 -- 172.21.15.45:0/2815694571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570cbd70060 con 0x5570cbe4ea60 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f0344ff9640 1 -- 172.21.15.45:0/3780855302 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f03400cd9c0 con 0x7f034819dff0 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f0346ffd640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 0x7f03300a4160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f033c02c170 tx=0x7f033c000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f03477fe640 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f033800b7e0 tx=0x7f033800bca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32782a46145ec14d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f032dffb640 1 -- 172.21.15.45:0/2815694571 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338003dd0 con 0x5570cbe4ea60 2024-03-12T05:26:25.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f032dffb640 1 -- 172.21.15.45:0/2815694571 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f03380118f0 con 0x5570cbe4ea60 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.270+0000 7f032dffb640 1 -- 172.21.15.45:0/2815694571 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338010870 con 0x5570cbe4ea60 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 -- 172.21.15.45:0/2815694571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 msgr2=0x5570cbe4ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f033800b7e0 tx=0x7f033800bca0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 -- 172.21.15.45:0/2815694571 shutdown_connections 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe57f60 0x5570cbe5a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe57a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 --2- 172.21.15.45:0/2815694571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4ea60 0x5570cbe4ee40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 -- 172.21.15.45:0/2815694571 >> 172.21.15.45:0/2815694571 conn(0x5570cbe49190 msgr2=0x5570cbe49570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 -- 172.21.15.45:0/2815694571 shutdown_connections 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.271+0000 7f0350d66740 1 -- 172.21.15.45:0/2815694571 wait complete. 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 -- start start 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 Processor -- start 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe55500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe55a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbdae020 con 0x5570cbe4ea60 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cba81430 con 0x5570cbe4f380 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cba3de00 con 0x5570cbe57f60 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f0350d66740 10 client.0 _ll_register_callbacks cb 0x5570cbd55cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f03477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.272+0000 7f03477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54656/0 (socket says 172.21.15.45:54656) 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 -- 172.21.15.45:0/2601896173 learned_addr learned my addr 172.21.15.45:0/2601896173 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f0347fff640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe55a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f0346ffd640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe55500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 -- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 msgr2=0x5570cbe55a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe55a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 -- 172.21.15.45:0/2601896173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 msgr2=0x5570cbe55500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe55500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 -- 172.21.15.45:0/2601896173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0338004290 con 0x5570cbe4ea60 2024-03-12T05:26:25.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f03477fe640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f03380105b0 tx=0x7f0338003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e987a9abed1d760 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.273+0000 7f0326ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:26:25.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338021070 con 0x5570cbe4ea60 2024-03-12T05:26:25.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0338011650 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570cbaad7e0 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0338011a60 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0350d66740 10 client.19347 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570cbdac6a0 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.274+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5570cbaad7e0 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0338030470 con 0x5570cbe4ea60 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 10 client.19347 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 10 client.19347 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 20 client.19347 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbe68290 0x5570cbe6a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 4 client.19347 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570cbaea720 con 0x5570cbe68290 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.275+0000 7f0346ffd640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbe68290 0x5570cbe6a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.276+0000 7f0346ffd640 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbe68290 0x5570cbe6a680 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9a353af9d0e095 server_cookie=c8baaf418da950d in_seq=0 out_seq=0 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.276+0000 7f0326ffd640 10 client.19347 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.277+0000 7f0326ffd640 1 -- 172.21.15.45:0/2601896173 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5570cbaea720 con 0x5570cbe68290 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.277+0000 7f0326ffd640 10 client.19347 handle_command_reply: tid=0 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.277+0000 7f0350d66740 1 client.19347 shutdown 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.279+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 msgr2=0x5570cbe54fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.279+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f03380105b0 tx=0x7f0338003e00 comp rx=0 tx=0).stop 2024-03-12T05:26:25.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 shutdown_connections 2024-03-12T05:26:25.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbe68290 0x5570cbe6a680 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe55a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe55500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 --2- 172.21.15.45:0/2601896173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe54fc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 >> 172.21.15.45:0/2601896173 conn(0x5570cbe49190 msgr2=0x5570cbe4a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0326ffd640 0 client.19347 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 shutdown_connections 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 1 -- 172.21.15.45:0/2601896173 wait complete. 2024-03-12T05:26:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.280+0000 7f0350d66740 20 client.19347 trim_cache size 0 max 16384 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.306+0000 7f0350d66740 1 Processor -- start 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.306+0000 7f0350d66740 1 -- start start 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe5ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbd33130 con 0x5570cbe4ea60 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe64ea0 con 0x5570cbe4f380 2024-03-12T05:26:25.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0350d66740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe56340 con 0x5570cbe57f60 2024-03-12T05:26:25.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0346ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe5ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0347fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0347fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36564/0 (socket says 172.21.15.45:36564) 2024-03-12T05:26:25.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f0347fff640 1 -- 172.21.15.45:0/2710265912 learned_addr learned my addr 172.21.15.45:0/2710265912 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.307+0000 7f03477fe640 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 -- 172.21.15.45:0/2710265912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 msgr2=0x5570cbe5ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe5ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 -- 172.21.15.45:0/2710265912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 msgr2=0x5570cbe576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 -- 172.21.15.45:0/2710265912 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570cbd48c20 con 0x5570cbe57f60 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.308+0000 7f0347fff640 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f034000ca00 tx=0x7f03400ba040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57a4108f49e7987a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.309+0000 7f032d7fa640 1 -- 172.21.15.45:0/2710265912 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340099f70 con 0x5570cbe57f60 2024-03-12T05:26:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.309+0000 7f032d7fa640 1 -- 172.21.15.45:0/2710265912 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f034009a610 con 0x5570cbe57f60 2024-03-12T05:26:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f032d7fa640 1 -- 172.21.15.45:0/2710265912 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340005620 con 0x5570cbe57f60 2024-03-12T05:26:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 -- 172.21.15.45:0/2710265912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 msgr2=0x5570cbe5f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f034000ca00 tx=0x7f03400ba040 comp rx=0 tx=0).stop 2024-03-12T05:26:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 -- 172.21.15.45:0/2710265912 shutdown_connections 2024-03-12T05:26:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe57f60 0x5570cbe5f350 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe4f380 0x5570cbe5ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 --2- 172.21.15.45:0/2710265912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 -- 172.21.15.45:0/2710265912 >> 172.21.15.45:0/2710265912 conn(0x5570cbe49190 msgr2=0x5570cbe6efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 -- 172.21.15.45:0/2710265912 shutdown_connections 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.310+0000 7f0350d66740 1 -- 172.21.15.45:0/2710265912 wait complete. 2024-03-12T05:26:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 -- start start 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 Processor -- start 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4f380 0x5570cbe5d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe57f60 0x5570cbe5d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe5dfa0 con 0x5570cbe4ea60 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe5e110 con 0x5570cbe57f60 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570cbe5e2d0 con 0x5570cbe4f380 2024-03-12T05:26:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.311+0000 7f0350d66740 10 client.0 _ll_register_callbacks cb 0x7f0338004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f03477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f03477fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54676/0 (socket says 172.21.15.45:54676) 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f03477fe640 1 -- 172.21.15.45:0/723478104 learned_addr learned my addr 172.21.15.45:0/723478104 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f0346ffd640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4f380 0x5570cbe5d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f0347fff640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe57f60 0x5570cbe5d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.312+0000 7f03477fe640 1 -- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4f380 msgr2=0x5570cbe5d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f03477fe640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4f380 0x5570cbe5d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f03477fe640 1 -- 172.21.15.45:0/723478104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe57f60 msgr2=0x5570cbe5d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f03477fe640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe57f60 0x5570cbe5d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f03477fe640 1 -- 172.21.15.45:0/723478104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03481a66c0 con 0x5570cbe4ea60 2024-03-12T05:26:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f03477fe640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0338003a40 tx=0x7f03380041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5c1751488f38d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.308 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f0326ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0338021070 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03380118f0 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.313+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0338011a60 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0338011c40 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 10 client.19353 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5570cbd6e8c0 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f03380118f0 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0338010690 con 0x5570cbe4ea60 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 10 client.19353 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 10 client.19353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:26:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 20 client.19353 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.314+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbebfad0 0x5570cbec1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.315+0000 7f0350d66740 4 client.19353 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.315+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5570cbb39890 con 0x5570cbebfad0 2024-03-12T05:26:25.310 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.315+0000 7f0346ffd640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbebfad0 0x5570cbec1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.315+0000 7f0346ffd640 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbebfad0 0x5570cbec1ef0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d858e75e083fd3b server_cookie=a78270122b105437 in_seq=0 out_seq=0 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0326ffd640 10 client.19353 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0326ffd640 1 -- 172.21.15.45:0/723478104 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5570cbb39890 con 0x5570cbebfad0 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0326ffd640 10 client.19353 handle_command_reply: tid=0 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0350d66740 1 client.19353 shutdown 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 msgr2=0x5570cbe5f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.316+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0338003a40 tx=0x7f03380041c0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 shutdown_connections 2024-03-12T05:26:25.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x5570cbebfad0 0x5570cbec1ef0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570cbe57f60 0x5570cbe5d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5570cbe4f380 0x5570cbe5d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 --2- 172.21.15.45:0/723478104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5570cbe4ea60 0x5570cbe5f0b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 >> 172.21.15.45:0/723478104 conn(0x5570cbe49190 msgr2=0x5570cbe502a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0326ffd640 0 client.19353 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 shutdown_connections 2024-03-12T05:26:25.312 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.317+0000 7f0350d66740 1 -- 172.21.15.45:0/723478104 wait complete. 2024-03-12T05:26:25.313 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.318+0000 7f0350d66740 20 client.19353 trim_cache size 0 max 16384 2024-03-12T05:26:25.313 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.318+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 msgr2=0x7f03300a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.318+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 0x7f03300a4160 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f033c02c170 tx=0x7f033c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 msgr2=0x7f034819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 0x7f034819e4a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f034000ca30 tx=0x7f034000cef0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 shutdown_connections 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f03300a1ca0 0x7f03300a4160 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0348106430 0x7f0348199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f034819dff0 0x7f034819e4a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 --2- 172.21.15.45:0/3780855302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0348101360 0x7f0348199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 >> 172.21.15.45:0/3780855302 conn(0x7f03480fcbe0 msgr2=0x7f03480fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 shutdown_connections 2024-03-12T05:26:25.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.319+0000 7f034e18c640 1 -- 172.21.15.45:0/3780855302 wait complete. 2024-03-12T05:26:25.334 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:25.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:26:25.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.512+0000 7f96cb7fe640 1 -- 172.21.15.45:0/1219408735 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c002cb60 con 0x7f96cc101260 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.513+0000 7f96d18dc640 1 -- 172.21.15.45:0/1219408735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc101260 msgr2=0x7f96cc101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.513+0000 7f96d18dc640 1 --2- 172.21.15.45:0/1219408735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc101260 0x7f96cc101640 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009900 tx=0x7f96c002bc30 comp rx=0 tx=0).stop 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 -- 172.21.15.45:0/1219408735 shutdown_connections 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 --2- 172.21.15.45:0/1219408735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101c10 0x7f96cc105c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 --2- 172.21.15.45:0/1219408735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc106330 0x7f96cc10df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 --2- 172.21.15.45:0/1219408735 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc101260 0x7f96cc101640 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 -- 172.21.15.45:0/1219408735 >> 172.21.15.45:0/1219408735 conn(0x7f96cc0fcae0 msgr2=0x7f96cc0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 -- 172.21.15.45:0/1219408735 shutdown_connections 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 -- 172.21.15.45:0/1219408735 wait complete. 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.514+0000 7f96d18dc640 1 Processor -- start 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 -- start start 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 0x7f96cc10de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc106330 0x7f96cc10af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 0x7f96cc10b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d08da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 0x7f96cc10de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d08da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 0x7f96cc10de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54694/0 (socket says 172.21.15.45:54694) 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc071d00 con 0x7f96cc101260 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc071e70 con 0x7f96cc10b490 2024-03-12T05:26:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96d18dc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc072030 con 0x7f96cc106330 2024-03-12T05:26:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.515+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc106330 0x7f96cc10af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 0x7f96cc10b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d08da640 1 -- 172.21.15.45:0/3357540479 learned_addr learned my addr 172.21.15.45:0/3357540479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 -- 172.21.15.45:0/3357540479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc106330 msgr2=0x7f96cc10af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc106330 0x7f96cc10af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 -- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 msgr2=0x7f96cc10de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 0x7f96cc10de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 -- 172.21.15.45:0/3357540479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c0009560 con 0x7f96cc10b490 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.516+0000 7f96d10db640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 0x7f96cc10b8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f96c400e940 tx=0x7f96c400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.512 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.517+0000 7f96c9ffb640 1 -- 172.21.15.45:0/3357540479 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4004d20 con 0x7f96cc10b490 2024-03-12T05:26:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.517+0000 7f96c9ffb640 1 -- 172.21.15.45:0/3357540479 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96c4004e80 con 0x7f96cc10b490 2024-03-12T05:26:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.517+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc072310 con 0x7f96cc10b490 2024-03-12T05:26:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.518+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc072870 con 0x7f96cc10b490 2024-03-12T05:26:25.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.518+0000 7f96c9ffb640 1 -- 172.21.15.45:0/3357540479 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4011470 con 0x7f96cc10b490 2024-03-12T05:26:25.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.521+0000 7f96c9ffb640 1 -- 172.21.15.45:0/3357540479 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f96c400cb20 con 0x7f96cc10b490 2024-03-12T05:26:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.522+0000 7f96d57ed740 1 Processor -- start 2024-03-12T05:26:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.522+0000 7f96d57ed740 1 -- start start 2024-03-12T05:26:25.518 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.522+0000 7f96c9ffb640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 0x7f96bc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a5301a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a5301cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52fcc960 con 0x558a53011d60 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52f80310 con 0x558a5301a940 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d57ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52f7d840 con 0x558a53011440 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96c9ffb640 1 -- 172.21.15.45:0/3357540479 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f96c40cd7f0 con 0x7f96cc10b490 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36616/0 (socket says 172.21.15.45:36616) 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 -- 172.21.15.45:0/3758817397 learned_addr learned my addr 172.21.15.45:0/3758817397 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d10db640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a5301cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a5301a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 -- 172.21.15.45:0/3758817397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 msgr2=0x558a5301cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a5301cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 -- 172.21.15.45:0/3758817397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 msgr2=0x558a5301a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a5301a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d08da640 1 -- 172.21.15.45:0/3758817397 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a52f3b060 con 0x558a53011440 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96d10db640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a5301cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.523+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 0x7f96bc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.524+0000 7f96d08da640 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f96c0036040 tx=0x7f96c0005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772d7f76bb65d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.524+0000 7f96b6ffd640 1 -- 172.21.15.45:0/3758817397 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c002cb60 con 0x558a53011440 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.524+0000 7f96b6ffd640 1 -- 172.21.15.45:0/3758817397 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96c00044e0 con 0x558a53011440 2024-03-12T05:26:25.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.524+0000 7f96b6ffd640 1 -- 172.21.15.45:0/3758817397 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0003ae0 con 0x558a53011440 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.525+0000 7f96d57ed740 1 -- 172.21.15.45:0/3758817397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 msgr2=0x558a53011820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.525+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 0x7f96bc0a40e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b6c0 tx=0x7f96b800b850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.525+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f96c0036040 tx=0x7f96c0005d50 comp rx=0 tx=0).stop 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 -- 172.21.15.45:0/3758817397 shutdown_connections 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a5301a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a5301cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3758817397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53011820 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 -- 172.21.15.45:0/3758817397 >> 172.21.15.45:0/3758817397 conn(0x558a5300bb70 msgr2=0x558a5300bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 -- 172.21.15.45:0/3758817397 shutdown_connections 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.526+0000 7f96d57ed740 1 -- 172.21.15.45:0/3758817397 wait complete. 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.527+0000 7f96d57ed740 1 -- start start 2024-03-12T05:26:25.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.527+0000 7f96d57ed740 1 Processor -- start 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53017910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a53017e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52f79020 con 0x558a53011d60 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52c4c430 con 0x558a5301a940 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.528+0000 7f96d57ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52c08e00 con 0x558a53011440 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d57ed740 10 client.0 _ll_register_callbacks cb 0x558a52f20cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43792/0 (socket says 172.21.15.45:43792) 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a53017e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 -- 172.21.15.45:0/2144050442 learned_addr learned my addr 172.21.15.45:0/2144050442 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d08da640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53017910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 -- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 msgr2=0x558a53017910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53017910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 -- 172.21.15.45:0/2144050442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 msgr2=0x558a53017e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a53017e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96d10db640 1 -- 172.21.15.45:0/2144050442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc10e710 con 0x558a5301a940 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.529+0000 7f96cbfff640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a53017e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96d10db640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f96c409a3d0 tx=0x7f96c40130a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d969e00406403ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.525 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c40054c0 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f96c4005000 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a52c787e0 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c409a440 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96d57ed740 10 client.28823 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.530+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558a52f776a0 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x558a52c787e0 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f96c4018030 con 0x558a5301a940 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 10 client.28823 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 10 client.28823 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 20 client.28823 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a5302abe0 0x558a5302d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 4 client.28823 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a52cb5720 con 0x558a5302abe0 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.531+0000 7f96d08da640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a5302abe0 0x558a5302d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.532+0000 7f96d08da640 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a5302abe0 0x558a5302d020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f136f16cd793ff4 server_cookie=31b58151bfa50dfe in_seq=0 out_seq=0 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.532+0000 7f96abfff640 10 client.28823 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96abfff640 1 -- 172.21.15.45:0/2144050442 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558a52cb5720 con 0x558a5302abe0 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96abfff640 10 client.28823 handle_command_reply: tid=0 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 client.28823 shutdown 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 msgr2=0x558a53018390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f96c409a3d0 tx=0x7f96c40130a0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 shutdown_connections 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011d60 0x558a53017e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a5302abe0 0x558a5302d020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a5301a940 0x558a53018390 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 --2- 172.21.15.45:0/2144050442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a53017910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96abfff640 0 client.28823 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 >> 172.21.15.45:0/2144050442 conn(0x558a5300bb70 msgr2=0x558a5300d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 shutdown_connections 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.533+0000 7f96d57ed740 1 -- 172.21.15.45:0/2144050442 wait complete. 2024-03-12T05:26:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.534+0000 7f96d57ed740 20 client.28823 trim_cache size 0 max 16384 2024-03-12T05:26:25.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.562+0000 7f96d57ed740 1 Processor -- start 2024-03-12T05:26:25.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d57ed740 1 -- start start 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43806/0 (socket says 172.21.15.45:43806) 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d08da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.563+0000 7f96d08da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36636/0 (socket says 172.21.15.45:36636) 2024-03-12T05:26:25.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96d10db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96d10db640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54714/0 (socket says 172.21.15.45:54714) 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96cbfff640 1 -- 172.21.15.45:0/1113690941 learned_addr learned my addr 172.21.15.45:0/1113690941 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52efe130 con 0x558a5301a940 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a52c787e0 con 0x558a53011d60 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.564+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a53007570 con 0x558a53011440 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96cbfff640 1 -- 172.21.15.45:0/1113690941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 msgr2=0x558a5301a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96cbfff640 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96cbfff640 1 -- 172.21.15.45:0/1113690941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 msgr2=0x558a53013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96cbfff640 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96cbfff640 1 -- 172.21.15.45:0/1113690941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a52f13c20 con 0x558a53011d60 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96d08da640 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.565+0000 7f96d10db640 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:26:25.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.566+0000 7f96cbfff640 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f96b8005fd0 tx=0x7f96b801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed6ef0e9675ce5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.566+0000 7f96b67fc640 1 -- 172.21.15.45:0/1113690941 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8025070 con 0x558a53011d60 2024-03-12T05:26:25.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.566+0000 7f96b67fc640 1 -- 172.21.15.45:0/1113690941 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96b80044d0 con 0x558a53011d60 2024-03-12T05:26:25.562 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.566+0000 7f96b67fc640 1 -- 172.21.15.45:0/1113690941 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8004b10 con 0x558a53011d60 2024-03-12T05:26:25.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 msgr2=0x558a53012c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f96b8005fd0 tx=0x7f96b801d000 comp rx=0 tx=0).stop 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 shutdown_connections 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a5301a940 0x558a53013160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53012c20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 --2- 172.21.15.45:0/1113690941 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a53011440 0x558a5301a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 >> 172.21.15.45:0/1113690941 conn(0x558a5300bb70 msgr2=0x558a530161e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 shutdown_connections 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.567+0000 7f96d57ed740 1 -- 172.21.15.45:0/1113690941 wait complete. 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.568+0000 7f96d57ed740 1 -- start start 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.568+0000 7f96d57ed740 1 Processor -- start 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.568+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53020f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 0x558a53081150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a53081690 con 0x558a53011440 2024-03-12T05:26:25.564 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a530817d0 con 0x558a53011d60 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a53081910 con 0x558a5301a940 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d57ed740 10 client.0 _ll_register_callbacks cb 0x7f96c0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d08da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d08da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54720/0 (socket says 172.21.15.45:54720) 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d08da640 1 -- 172.21.15.45:0/3841384606 learned_addr learned my addr 172.21.15.45:0/3841384606 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53020f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.569+0000 7f96d10db640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 0x558a53081150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96cbfff640 1 -- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 msgr2=0x558a53081150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 0x558a53081150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96cbfff640 1 -- 172.21.15.45:0/3841384606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 msgr2=0x558a530209c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96cbfff640 1 -- 172.21.15.45:0/3841384606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a52f3b060 con 0x558a53011d60 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96d10db640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 0x558a53081150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.570+0000 7f96d08da640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.571+0000 7f96cbfff640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53020f00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x558a5301d630 tx=0x7f96b8021f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62b95fc765c588a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.571+0000 7f96abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:25.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.573+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8025040 con 0x558a53011d60 2024-03-12T05:26:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.574+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a53017740 con 0x558a53011d60 2024-03-12T05:26:25.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.574+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f96b8029820 con 0x558a53011d60 2024-03-12T05:26:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.576+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96b80299e0 con 0x558a53011d60 2024-03-12T05:26:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.576+0000 7f96d57ed740 10 client.28829 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.576+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f96b8010530 con 0x558a53011d60 2024-03-12T05:26:25.570 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.576+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f96b803e020 con 0x558a53011d60 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.576+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f96b80307a0 con 0x558a53011d60 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 10 client.28829 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 10 client.28829 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 20 client.28829 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a53083040 0x558a53085460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 4 client.28829 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:25.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f96c00044e0 con 0x558a53083040 2024-03-12T05:26:25.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.577+0000 7f96d08da640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a53083040 0x558a53085460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96abfff640 10 client.28829 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d08da640 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a53083040 0x558a53085460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9327662832ac620c server_cookie=19d6d2a765e4f713 in_seq=0 out_seq=0 2024-03-12T05:26:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96abfff640 1 -- 172.21.15.45:0/3841384606 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f96c00044e0 con 0x558a53083040 2024-03-12T05:26:25.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96abfff640 10 client.28829 handle_command_reply: tid=0 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 client.28829 shutdown 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 msgr2=0x558a53020f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53020f00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x558a5301d630 tx=0x7f96b8021f20 comp rx=0 tx=0).stop 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 shutdown_connections 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x558a53083040 0x558a53085460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5301a940 0x558a53081150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558a53011d60 0x558a53020f00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 --2- 172.21.15.45:0/3841384606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558a53011440 0x558a530209c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 >> 172.21.15.45:0/3841384606 conn(0x558a5300bb70 msgr2=0x558a53012c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.578+0000 7f96abfff640 0 client.28829 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.579+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 shutdown_connections 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.579+0000 7f96d57ed740 1 -- 172.21.15.45:0/3841384606 wait complete. 2024-03-12T05:26:25.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.579+0000 7f96d57ed740 20 client.28829 trim_cache size 0 max 16384 2024-03-12T05:26:25.575 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:25.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 msgr2=0x7f96bc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 0x7f96bc0a40e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b6c0 tx=0x7f96b800b850 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 msgr2=0x7f96cc10b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 0x7f96cc10b8d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f96c400e940 tx=0x7f96c400ee00 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 shutdown_connections 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f96bc0a1c20 0x7f96bc0a40e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96cc10b490 0x7f96cc10b8d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f96cc106330 0x7f96cc10af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 --2- 172.21.15.45:0/3357540479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96cc101260 0x7f96cc10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 >> 172.21.15.45:0/3357540479 conn(0x7f96cc0fcae0 msgr2=0x7f96cc0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 shutdown_connections 2024-03-12T05:26:25.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.580+0000 7f96d18dc640 1 -- 172.21.15.45:0/3357540479 wait complete. 2024-03-12T05:26:25.603 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:25.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:26:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.802+0000 7f8ed3199640 1 -- 172.21.15.45:0/3145240756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc100c10 msgr2=0x7f8ecc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.802+0000 7f8ed3199640 1 --2- 172.21.15.45:0/3145240756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc100c10 0x7f8ecc104c50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000d910 tx=0x7f8ec002b9c0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 -- 172.21.15.45:0/3145240756 shutdown_connections 2024-03-12T05:26:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 --2- 172.21.15.45:0/3145240756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc105330 0x7f8ecc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 --2- 172.21.15.45:0/3145240756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc100c10 0x7f8ecc104c50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 --2- 172.21.15.45:0/3145240756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 0x7f8ecc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 -- 172.21.15.45:0/3145240756 >> 172.21.15.45:0/3145240756 conn(0x7f8ecc077180 msgr2=0x7f8ecc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.803+0000 7f8ed3199640 1 -- 172.21.15.45:0/3145240756 shutdown_connections 2024-03-12T05:26:25.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.804+0000 7f8ed3199640 1 -- 172.21.15.45:0/3145240756 wait complete. 2024-03-12T05:26:25.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 Processor -- start 2024-03-12T05:26:25.799 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 -- start start 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 0x7f8ecc1954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 0x7f8ecc195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc19a770 con 0x7f8ecc100260 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc19a8e0 con 0x7f8ecc199d80 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.805+0000 7f8ed3199640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc19aa50 con 0x7f8ecc105330 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 0x7f8ecc1954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43832/0 (socket says 172.21.15.45:43832) 2024-03-12T05:26:25.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 -- 172.21.15.45:0/1524376227 learned_addr learned my addr 172.21.15.45:0/1524376227 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 0x7f8ecc195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 -- 172.21.15.45:0/1524376227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 msgr2=0x7f8ecc195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 0x7f8ecc195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 -- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 msgr2=0x7f8ecc1954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 0x7f8ecc1954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 -- 172.21.15.45:0/1524376227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec000d570 con 0x7f8ecc199d80 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 0x7f8ecc195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.806+0000 7f8ed1736640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400edb0 tx=0x7f8ec4012490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.807+0000 7f8ec9ffb640 1 -- 172.21.15.45:0/1524376227 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4014040 con 0x7f8ecc199d80 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.807+0000 7f8ec9ffb640 1 -- 172.21.15.45:0/1524376227 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ec4012db0 con 0x7f8ecc199d80 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.807+0000 7f8ec9ffb640 1 -- 172.21.15.45:0/1524376227 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4018400 con 0x7f8ecc199d80 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.807+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc1a2030 con 0x7f8ecc199d80 2024-03-12T05:26:25.802 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.807+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc1a2460 con 0x7f8ecc199d80 2024-03-12T05:26:25.805 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.811+0000 7f8ed5d68740 1 Processor -- start 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ec9ffb640 1 -- 172.21.15.45:0/1524376227 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8ec4005240 con 0x7f8ecc199d80 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 -- start start 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687d95010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687d50960 con 0x555687da6f60 2024-03-12T05:26:25.806 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687d04310 con 0x555687d955e0 2024-03-12T05:26:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.812+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687d01840 con 0x555687d94c30 2024-03-12T05:26:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.813+0000 7f8ec9ffb640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 0x7f8eb40a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.813+0000 7f8ec9ffb640 1 -- 172.21.15.45:0/1524376227 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8ec40d17f0 con 0x7f8ecc199d80 2024-03-12T05:26:25.807 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.813+0000 7f8ed1736640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 0x7f8eb40a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43838/0 (socket says 172.21.15.45:43838) 2024-03-12T05:26:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 -- 172.21.15.45:0/1552441842 learned_addr learned my addr 172.21.15.45:0/1552441842 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687d95010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 -- 172.21.15.45:0/1552441842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 msgr2=0x555687d95010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687d95010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 -- 172.21.15.45:0/1552441842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 msgr2=0x555687da9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed0f35640 1 -- 172.21.15.45:0/1552441842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555687cbf060 con 0x555687d955e0 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.814+0000 7f8ed1736640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 0x7f8eb40a41a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40d1a90 tx=0x7f8ec40d1a00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.815+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00ac20 tx=0x7f8ebc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9896e74f53fef38b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.815+0000 7f8eb2ffd640 1 -- 172.21.15.45:0/1552441842 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00a720 con 0x555687d955e0 2024-03-12T05:26:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.815+0000 7f8eb2ffd640 1 -- 172.21.15.45:0/1552441842 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ebc018af0 con 0x555687d955e0 2024-03-12T05:26:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.815+0000 7f8eb2ffd640 1 -- 172.21.15.45:0/1552441842 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc010bb0 con 0x555687d955e0 2024-03-12T05:26:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 -- 172.21.15.45:0/1552441842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 msgr2=0x555687da6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00ac20 tx=0x7f8ebc00f790 comp rx=0 tx=0).stop 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 -- 172.21.15.45:0/1552441842 shutdown_connections 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da6a20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/1552441842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687d95010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.816+0000 7f8ed5d68740 1 -- 172.21.15.45:0/1552441842 >> 172.21.15.45:0/1552441842 conn(0x555687d8f980 msgr2=0x555687d8fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.817+0000 7f8ed5d68740 1 -- 172.21.15.45:0/1552441842 shutdown_connections 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.817+0000 7f8ed5d68740 1 -- 172.21.15.45:0/1552441842 wait complete. 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.817+0000 7f8ed5d68740 1 -- start start 2024-03-12T05:26:25.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.817+0000 7f8ed5d68740 1 Processor -- start 2024-03-12T05:26:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687da3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687cfd020 con 0x555687da6f60 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556879d0430 con 0x555687d955e0 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55568798ce00 con 0x555687d94c30 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed5d68740 10 client.0 _ll_register_callbacks cb 0x555687ca4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43844/0 (socket says 172.21.15.45:43844) 2024-03-12T05:26:25.813 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.818+0000 7f8ed0f35640 1 -- 172.21.15.45:0/3607405170 learned_addr learned my addr 172.21.15.45:0/3607405170 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 -- 172.21.15.45:0/3607405170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 msgr2=0x555687da3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687da3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 -- 172.21.15.45:0/3607405170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 msgr2=0x555687da4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 -- 172.21.15.45:0/3607405170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc00a880 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0186a0 tx=0x7f8ebc0186d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ccb17f8bffe55e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.819+0000 7f8eabfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00a720 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8ebc010b00 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556879fc7e0 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ebc0105b0 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8ed5d68740 10 client.28841 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555687cfb6a0 con 0x555687d955e0 2024-03-12T05:26:25.815 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.820+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5556879fc7e0 con 0x555687d955e0 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8ebc004770 con 0x555687d955e0 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 10 client.28841 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 10 client.28841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 20 client.28841 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687db7330 0x555687db9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 4 client.28841 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555687a39720 con 0x555687db7330 2024-03-12T05:26:25.816 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.821+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687db7330 0x555687db9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.822+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687db7330 0x555687db9750 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e0f7914db70889f server_cookie=e1ddf9d16103e33b in_seq=0 out_seq=0 2024-03-12T05:26:25.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.822+0000 7f8eabfff640 10 client.28841 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:25.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.823+0000 7f8eabfff640 1 -- 172.21.15.45:0/3607405170 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555687a39720 con 0x555687db7330 2024-03-12T05:26:25.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.823+0000 7f8eabfff640 10 client.28841 handle_command_reply: tid=0 2024-03-12T05:26:25.817 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.823+0000 7f8ed5d68740 1 client.28841 shutdown 2024-03-12T05:26:25.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.826+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 msgr2=0x555687da4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.826+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0186a0 tx=0x7f8ebc0186d0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 shutdown_connections 2024-03-12T05:26:25.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687db7330 0x555687db9750 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d955e0 0x555687da4500 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/3607405170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d94c30 0x555687da3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 >> 172.21.15.45:0/3607405170 conn(0x555687d8f980 msgr2=0x555687d916d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.827+0000 7f8eabfff640 0 client.28841 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.828+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 shutdown_connections 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.828+0000 7f8ed5d68740 1 -- 172.21.15.45:0/3607405170 wait complete. 2024-03-12T05:26:25.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.828+0000 7f8ed5d68740 20 client.28841 trim_cache size 0 max 16384 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.854+0000 7f8ed5d68740 1 Processor -- start 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.854+0000 7f8ed5d68740 1 -- start start 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.854+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687da66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.854+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687c82130 con 0x555687da6f60 2024-03-12T05:26:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687d94340 con 0x555687d94c30 2024-03-12T05:26:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687d8b2b0 con 0x555687d955e0 2024-03-12T05:26:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36670/0 (socket says 172.21.15.45:36670) 2024-03-12T05:26:25.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 -- 172.21.15.45:0/2646229607 learned_addr learned my addr 172.21.15.45:0/2646229607 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687da66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 -- 172.21.15.45:0/2646229607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 msgr2=0x555687da66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687da66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 -- 172.21.15.45:0/2646229607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 msgr2=0x555687da1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ed0f35640 1 -- 172.21.15.45:0/2646229607 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555687c97c20 con 0x555687d955e0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.855+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.856+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc000c00 tx=0x7f8ebc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4d79da8bc04269b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.856+0000 7f8eb27fc640 1 -- 172.21.15.45:0/2646229607 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc018940 con 0x555687d955e0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.856+0000 7f8eb27fc640 1 -- 172.21.15.45:0/2646229607 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ebc00a3c0 con 0x555687d955e0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.856+0000 7f8eb27fc640 1 -- 172.21.15.45:0/2646229607 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc005000 con 0x555687d955e0 2024-03-12T05:26:25.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 -- 172.21.15.45:0/2646229607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 msgr2=0x555687da4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc000c00 tx=0x7f8ebc00f790 comp rx=0 tx=0).stop 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 -- 172.21.15.45:0/2646229607 shutdown_connections 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687da1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687d955e0 0x555687da4860 secure :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc000c00 tx=0x7f8ebc00f790 comp rx=0 tx=0).stop 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/2646229607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687da66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.857+0000 7f8ed5d68740 1 -- 172.21.15.45:0/2646229607 >> 172.21.15.45:0/2646229607 conn(0x555687d8f980 msgr2=0x555687dbe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.858+0000 7f8ed5d68740 1 -- 172.21.15.45:0/2646229607 shutdown_connections 2024-03-12T05:26:25.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.858+0000 7f8ed5d68740 1 -- 172.21.15.45:0/2646229607 wait complete. 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.858+0000 7f8ed5d68740 1 -- start start 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.858+0000 7f8ed5d68740 1 Processor -- start 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687dac6b0 0x555687dacb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687dae590 con 0x555687da6f60 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687dae700 con 0x555687d94c30 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555687e0d870 con 0x555687dac6b0 2024-03-12T05:26:25.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.859+0000 7f8ed5d68740 10 client.0 _ll_register_callbacks cb 0x7f8ebc00a670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ed0f35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54782/0 (socket says 172.21.15.45:54782) 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43864/0 (socket says 172.21.15.45:43864) 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 -- 172.21.15.45:0/998990769 learned_addr learned my addr 172.21.15.45:0/998990769 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687dac6b0 0x555687dacb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 -- 172.21.15.45:0/998990769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687dac6b0 msgr2=0x555687dacb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687dac6b0 0x555687dacb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 -- 172.21.15.45:0/998990769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 msgr2=0x555687dac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.860+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8ecbfff640 1 -- 172.21.15.45:0/998990769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555687cfb6a0 con 0x555687d94c30 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:25.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8ecbfff640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0004cc0 tx=0x7f8ec002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d5bc0090fcacf6d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8eabfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0004050 con 0x555687d94c30 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8ec00041b0 con 0x555687d94c30 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.861+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ebc018af0 con 0x555687d94c30 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.862+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ec0004370 con 0x555687d94c30 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.862+0000 7f8ed5d68740 10 client.28844 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.862+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555687cbd8c0 con 0x555687d94c30 2024-03-12T05:26:25.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.862+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8ec005b030 con 0x555687d94c30 2024-03-12T05:26:25.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.863+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8ec002e070 con 0x555687d94c30 2024-03-12T05:26:25.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 10 client.28844 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 10 client.28844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 20 client.28844 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687e0efc0 0x555687e113e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 4 client.28844 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.867+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555687a88890 con 0x555687e0efc0 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.868+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687e0efc0 0x555687e113e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:25.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.868+0000 7f8eabfff640 10 client.28844 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:25.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.868+0000 7f8ed0f35640 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687e0efc0 0x555687e113e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3be3a396cc844f61 server_cookie=c04fc9086af4140 in_seq=0 out_seq=0 2024-03-12T05:26:25.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8eabfff640 1 -- 172.21.15.45:0/998990769 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555687a88890 con 0x555687e0efc0 2024-03-12T05:26:25.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8eabfff640 10 client.28844 handle_command_reply: tid=0 2024-03-12T05:26:25.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 client.28844 shutdown 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 msgr2=0x555687dade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0004cc0 tx=0x7f8ec002b9c0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 shutdown_connections 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x555687e0efc0 0x555687e113e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555687da6f60 0x555687dac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555687dac6b0 0x555687dacb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 --2- 172.21.15.45:0/998990769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555687d94c30 0x555687dade70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 >> 172.21.15.45:0/998990769 conn(0x555687d8f980 msgr2=0x555687d8e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.869+0000 7f8eabfff640 0 client.28844 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:25.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.870+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 shutdown_connections 2024-03-12T05:26:25.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.870+0000 7f8ed5d68740 1 -- 172.21.15.45:0/998990769 wait complete. 2024-03-12T05:26:25.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:25 smithi045 ceph-mon[32174]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-03-12T05:26:25.869 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.871+0000 7f8ed5d68740 20 client.28844 trim_cache size 0 max 16384 2024-03-12T05:26:25.870 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.876+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 msgr2=0x7f8eb40a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.876+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 0x7f8eb40a41a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40d1a90 tx=0x7f8ec40d1a00 comp rx=0 tx=0).stop 2024-03-12T05:26:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.876+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 msgr2=0x7f8ecc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:25.872 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.876+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400edb0 tx=0x7f8ec4012490 comp rx=0 tx=0).stop 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 shutdown_connections 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8eb40a1ce0 0x7f8eb40a41a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ecc199d80 0x7f8ecc19a230 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ecc105330 0x7f8ecc195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 --2- 172.21.15.45:0/1524376227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ecc100260 0x7f8ecc1954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 >> 172.21.15.45:0/1524376227 conn(0x7f8ecc077180 msgr2=0x7f8ecc0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 shutdown_connections 2024-03-12T05:26:25.873 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:25.877+0000 7f8ed3199640 1 -- 172.21.15.45:0/1524376227 wait complete. 2024-03-12T05:26:26.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:25 smithi130 ceph-mon[35398]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-03-12T05:26:26.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:25 smithi116 ceph-mon[35486]: pgmap v489: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 17 MiB/s wr, 1.36k op/s 2024-03-12T05:26:26.893 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:26:26.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:26:26.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:26:27.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.073+0000 7ff7b559a640 1 -- 172.21.15.45:0/2957837129 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac02cb70 con 0x7ff7b810d720 2024-03-12T05:26:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/2957837129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b810d720 msgr2=0x7ff7b810db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/2957837129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b810d720 0x7ff7b810db00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009930 tx=0x7ff7ac02bc30 comp rx=0 tx=0).stop 2024-03-12T05:26:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/2957837129 shutdown_connections 2024-03-12T05:26:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/2957837129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b81145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.070 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/2957837129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b8107710 0x7ff7b8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/2957837129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b810d720 0x7ff7b810db00 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.074 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/2957837129 >> 172.21.15.45:0/2957837129 conn(0x7ff7b8075620 msgr2=0x7ff7b8077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/2957837129 shutdown_connections 2024-03-12T05:26:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.074+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/2957837129 wait complete. 2024-03-12T05:26:27.075 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.075+0000 7ff7bcb4b640 1 Processor -- start 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 -- start start 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 0x7ff7b80728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b810d720 0x7ff7b806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b806c2e0 con 0x7ff7b81080d0 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b806c420 con 0x7ff7b810d720 2024-03-12T05:26:27.077 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.076+0000 7ff7bcb4b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b806c5e0 con 0x7ff7b8107710 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.078+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.078+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54798/0 (socket says 172.21.15.45:54798) 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.078+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/3305546736 learned_addr learned my addr 172.21.15.45:0/3305546736 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.078+0000 7ff7b659c640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 0x7ff7b80728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b810d720 0x7ff7b806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/3305546736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 msgr2=0x7ff7b80728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 0x7ff7b80728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/3305546736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b810d720 msgr2=0x7ff7b806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b810d720 0x7ff7b806bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/3305546736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac009560 con 0x7ff7b81080d0 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b659c640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 0x7ff7b80728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:27.078 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800e9f0 tx=0x7ff7a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.079+0000 7ff7a77fe640 1 -- 172.21.15.45:0/3305546736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8009700 con 0x7ff7b81080d0 2024-03-12T05:26:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.080+0000 7ff7a77fe640 1 -- 172.21.15.45:0/3305546736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7a800cd80 con 0x7ff7b81080d0 2024-03-12T05:26:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.080+0000 7ff7a77fe640 1 -- 172.21.15.45:0/3305546736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8018840 con 0x7ff7b81080d0 2024-03-12T05:26:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.080+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b8070410 con 0x7ff7b81080d0 2024-03-12T05:26:27.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.080+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b80708d0 con 0x7ff7b81080d0 2024-03-12T05:26:27.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.084+0000 7ff7a77fe640 1 -- 172.21.15.45:0/3305546736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff7a80189a0 con 0x7ff7b81080d0 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.084+0000 7ff7a77fe640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 0x7ff7a00a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7b659c640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 0x7ff7a00a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 Processor -- start 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7a77fe640 1 -- 172.21.15.45:0/3305546736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff7a8014070 con 0x7ff7b81080d0 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 -- start start 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78380 0x55c4a2e78760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e83d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2de6f90 con 0x55c4a2e818b0 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2de44c0 con 0x55c4a2e78380 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.085+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2ddfca0 con 0x55c4a2e78d30 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.081 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b659c640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 0x7ff7a00a41f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac036040 tx=0x7ff7ac005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36704/0 (socket says 172.21.15.45:36704) 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2839391974 learned_addr learned my addr 172.21.15.45:0/2839391974 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e83d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b659c640 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78380 0x55c4a2e78760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2839391974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78380 msgr2=0x55c4a2e78760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78380 0x55c4a2e78760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2839391974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 msgr2=0x55c4a2e83d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e83d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2839391974 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4a2da20b0 con 0x55c4a2e78d30 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.086+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8097180 tx=0x7ff7a8010b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28e3fb3dfb78072a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.087+0000 7ff7a4ff9640 1 -- 172.21.15.45:0/2839391974 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a809e900 con 0x55c4a2e78d30 2024-03-12T05:26:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.087+0000 7ff7a4ff9640 1 -- 172.21.15.45:0/2839391974 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7a80a2f50 con 0x55c4a2e78d30 2024-03-12T05:26:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.087+0000 7ff7a4ff9640 1 -- 172.21.15.45:0/2839391974 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80ba350 con 0x55c4a2e78d30 2024-03-12T05:26:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2839391974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 msgr2=0x55c4a2e812e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8097180 tx=0x7ff7a8010b50 comp rx=0 tx=0).stop 2024-03-12T05:26:27.085 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2839391974 shutdown_connections 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e83d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e812e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2839391974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78380 0x55c4a2e78760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2839391974 >> 172.21.15.45:0/2839391974 conn(0x55c4a2e72840 msgr2=0x55c4a2e72c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2839391974 shutdown_connections 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2839391974 wait complete. 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.088+0000 7ff7bf7f5740 1 -- start start 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 Processor -- start 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2e7e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e7ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 0x55c4a2e7f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2d84f40 con 0x55c4a2e78380 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2ab34b0 con 0x55c4a2e818b0 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2d65670 con 0x55c4a2e78d30 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7bf7f5740 10 client.0 _ll_register_callbacks cb 0x55c4a2dd2f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2e7e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2e7e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54814/0 (socket says 172.21.15.45:54814) 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7b659c640 1 -- 172.21.15.45:0/4137679698 learned_addr learned my addr 172.21.15.45:0/4137679698 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 0x55c4a2e7f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.089+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e7ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 -- 172.21.15.45:0/4137679698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 msgr2=0x55c4a2e7ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e7ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 -- 172.21.15.45:0/4137679698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 msgr2=0x55c4a2e7e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2e7e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 -- 172.21.15.45:0/4137679698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b80708d0 con 0x55c4a2e818b0 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 0x55c4a2e7f360 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009c10 tx=0x7ff7b0004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f62950304d2f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff79dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0011850 con 0x55c4a2e818b0 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff7b0009e20 con 0x55c4a2e818b0 2024-03-12T05:26:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.090+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4a2db9f40 con 0x55c4a2e818b0 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.091+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7b00119b0 con 0x55c4a2e818b0 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.091+0000 7ff7bf7f5740 10 client.28853 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.091+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c4a2dde320 con 0x55c4a2e818b0 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c4a2db9f40 con 0x55c4a2e818b0 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff7b001bce0 con 0x55c4a2e818b0 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 10 client.28853 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 10 client.28853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 20 client.28853 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2e91f20 0x55c4a2e94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 4 client.28853 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:27.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4a2dd6310 con 0x55c4a2e91f20 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.092+0000 7ff7b659c640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2e91f20 0x55c4a2e94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.093+0000 7ff7b659c640 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2e91f20 0x55c4a2e94340 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c350035c6a67ae server_cookie=bea77fac52c2fe1e in_seq=0 out_seq=0 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.093+0000 7ff79dffb640 10 client.28853 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.094+0000 7ff79dffb640 1 -- 172.21.15.45:0/4137679698 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c4a2dd6310 con 0x55c4a2e91f20 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.094+0000 7ff79dffb640 10 client.28853 handle_command_reply: tid=0 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.094+0000 7ff7bf7f5740 1 client.28853 shutdown 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 msgr2=0x55c4a2e7f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 0x55c4a2e7f360 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009c10 tx=0x7ff7b0004410 comp rx=0 tx=0).stop 2024-03-12T05:26:27.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 shutdown_connections 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2e91f20 0x55c4a2e94340 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e818b0 0x55c4a2e7f360 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78d30 0x55c4a2e7ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/4137679698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2e7e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 >> 172.21.15.45:0/4137679698 conn(0x55c4a2e72840 msgr2=0x55c4a2e747a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff79dffb640 0 client.28853 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 shutdown_connections 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/4137679698 wait complete. 2024-03-12T05:26:27.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.095+0000 7ff7bf7f5740 20 client.28853 trim_cache size 0 max 16384 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 Processor -- start 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 -- start start 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 0x55c4a2e7f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2e744e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e88880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e91d70 con 0x55c4a2e818b0 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e8e930 con 0x55c4a2e78d30 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7bf7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e7ce50 con 0x55c4a2e78380 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7b5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2e744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 0x55c4a2e7f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 0x55c4a2e7f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36718/0 (socket says 172.21.15.45:36718) 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7b659c640 1 -- 172.21.15.45:0/2844807017 learned_addr learned my addr 172.21.15.45:0/2844807017 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.124+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e88880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2844807017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 msgr2=0x55c4a2e7f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 0x55c4a2e7f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2844807017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 msgr2=0x55c4a2e88880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e88880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2844807017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4a2d7ac00 con 0x55c4a2e78d30 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2e744e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x55c4a2e844f0 tx=0x7ff7a8010d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=915f2f2ebaab113f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.125+0000 7ff79ffff640 1 -- 172.21.15.45:0/2844807017 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a809e900 con 0x55c4a2e78d30 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.126+0000 7ff79ffff640 1 -- 172.21.15.45:0/2844807017 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7a809a7b0 con 0x55c4a2e78d30 2024-03-12T05:26:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.126+0000 7ff79ffff640 1 -- 172.21.15.45:0/2844807017 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8099c00 con 0x55c4a2e78d30 2024-03-12T05:26:27.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2844807017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 msgr2=0x55c4a2e744e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2e744e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x55c4a2e844f0 tx=0x7ff7a8010d40 comp rx=0 tx=0).stop 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2844807017 shutdown_connections 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e818b0 0x55c4a2e88880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2e744e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2844807017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e78380 0x55c4a2e7f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2844807017 >> 172.21.15.45:0/2844807017 conn(0x55c4a2e72840 msgr2=0x55c4a2e73dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2844807017 shutdown_connections 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.127+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2844807017 wait complete. 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.128+0000 7ff7bf7f5740 1 -- start start 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.128+0000 7ff7bf7f5740 1 Processor -- start 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.128+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2ee2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2ee3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2ee2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7b659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2ee2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54820/0 (socket says 172.21.15.45:54820) 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7b659c640 1 -- 172.21.15.45:0/2328950004 learned_addr learned my addr 172.21.15.45:0/2328950004 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e818b0 0x55c4a2e87fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e88570 con 0x55c4a2e78380 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e886e0 con 0x55c4a2e78d30 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2ee3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4a2e888a0 con 0x55c4a2e818b0 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7bf7f5740 10 client.0 _ll_register_callbacks cb 0x7ff7a8096390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.129+0000 7ff7b6d9d640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e818b0 0x55c4a2e87fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2328950004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e818b0 msgr2=0x55c4a2e87fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e818b0 0x55c4a2e87fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2328950004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 msgr2=0x55c4a2ee2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2ee2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.130 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4a2da20b0 con 0x55c4a2e78d30 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7b5d9b640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2ee3270 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55c4a2e7fe60 tx=0x7ff7a800cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1917bb1a65c48ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff79dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80ba0b0 con 0x55c4a2e78d30 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4a2e05900 con 0x55c4a2e78d30 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.130+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff7a80ba210 con 0x55c4a2e78d30 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7a80ba3d0 con 0x55c4a2e78d30 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff7bf7f5740 10 client.28859 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:27.131 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c4a2da0910 con 0x55c4a2e78d30 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c4a2e05900 con 0x55c4a2e78d30 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff7a809bca0 con 0x55c4a2e78d30 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff7bf7f5740 10 client.28859 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff7bf7f5740 10 client.28859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.131+0000 7ff7bf7f5740 20 client.28859 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2eea660 0x55c4a2eeca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff7bf7f5740 4 client.28859 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c4a2adf860 con 0x55c4a2eea660 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff7b659c640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2eea660 0x55c4a2eeca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff7b659c640 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2eea660 0x55c4a2eeca50 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533fcb02b55493dc server_cookie=6d00874bc7647660 in_seq=0 out_seq=0 2024-03-12T05:26:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.132+0000 7ff79dffb640 10 client.28859 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.133+0000 7ff79dffb640 1 -- 172.21.15.45:0/2328950004 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c4a2adf860 con 0x55c4a2eea660 2024-03-12T05:26:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.133+0000 7ff79dffb640 10 client.28859 handle_command_reply: tid=0 2024-03-12T05:26:27.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.133+0000 7ff7bf7f5740 1 client.28859 shutdown 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 msgr2=0x55c4a2ee3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2ee3270 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55c4a2e7fe60 tx=0x7ff7a800cef0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 shutdown_connections 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c4a2eea660 0x55c4a2eeca50 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a2e818b0 0x55c4a2e87fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4a2e78d30 0x55c4a2ee3270 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff79dffb640 0 client.28859 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 --2- 172.21.15.45:0/2328950004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c4a2e78380 0x55c4a2ee2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 >> 172.21.15.45:0/2328950004 conn(0x55c4a2e72840 msgr2=0x55c4a2e718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 shutdown_connections 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.134+0000 7ff7bf7f5740 1 -- 172.21.15.45:0/2328950004 wait complete. 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.135+0000 7ff7bf7f5740 20 client.28859 trim_cache size 0 max 16384 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "1ac5e9bd-4514-4dba-a480-031522e8b62d", 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:26:27.135 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:26:27.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.135+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 msgr2=0x7ff7a00a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.135+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 0x7ff7a00a41f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac036040 tx=0x7ff7ac005fd0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 msgr2=0x7ff7b8072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800e9f0 tx=0x7ff7a800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 shutdown_connections 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7a00a1d30 0x7ff7a00a41f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7b81080d0 0x7ff7b8072e30 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b810d720 0x7ff7b806bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 --2- 172.21.15.45:0/3305546736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7b8107710 0x7ff7b80728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 >> 172.21.15.45:0/3305546736 conn(0x7ff7b8075620 msgr2=0x7ff7b8075f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 shutdown_connections 2024-03-12T05:26:27.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.136+0000 7ff7bcb4b640 1 -- 172.21.15.45:0/3305546736 wait complete. 2024-03-12T05:26:27.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:26:27.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.333+0000 7f373146f640 1 -- 172.21.15.45:0/2682400752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 msgr2=0x7f372c10dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.333+0000 7f373146f640 1 --2- 172.21.15.45:0/2682400752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c10dba0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3724009cd0 tx=0x7f372402c170 comp rx=0 tx=0).stop 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 -- 172.21.15.45:0/2682400752 shutdown_connections 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 --2- 172.21.15.45:0/2682400752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 --2- 172.21.15.45:0/2682400752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 0x7f372c114610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 --2- 172.21.15.45:0/2682400752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c10dba0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 -- 172.21.15.45:0/2682400752 >> 172.21.15.45:0/2682400752 conn(0x7f372c076ff0 msgr2=0x7f372c105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.334+0000 7f373146f640 1 -- 172.21.15.45:0/2682400752 shutdown_connections 2024-03-12T05:26:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.335+0000 7f373146f640 1 -- 172.21.15.45:0/2682400752 wait complete. 2024-03-12T05:26:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.336+0000 7f373146f640 1 Processor -- start 2024-03-12T05:26:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.337+0000 7f373146f640 1 -- start start 2024-03-12T05:26:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.338+0000 7f373146f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c1a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.338+0000 7f373146f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 0x7f372c1a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.338+0000 7f372affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c1a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.338+0000 7f372affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c1a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54846/0 (socket says 172.21.15.45:54846) 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.338+0000 7f372affd640 1 -- 172.21.15.45:0/2441859824 learned_addr learned my addr 172.21.15.45:0/2441859824 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.339+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.339+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e140 con 0x7f372c1078a0 2024-03-12T05:26:27.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.339+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e2b0 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.339+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c19e470 con 0x7f372c108240 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.339+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372a7fc640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 0x7f372c1a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372b7fe640 1 -- 172.21.15.45:0/2441859824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 msgr2=0x7f372c1a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 0x7f372c1a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372b7fe640 1 -- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 msgr2=0x7f372c1a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c1a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.340+0000 7f372b7fe640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c009490 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.341+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c19daf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f371400d600 tx=0x7f371400dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.341+0000 7f3723fff640 1 -- 172.21.15.45:0/2441859824 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714004220 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.341+0000 7f3723fff640 1 -- 172.21.15.45:0/2441859824 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3714004380 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.341+0000 7f3723fff640 1 -- 172.21.15.45:0/2441859824 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714003cc0 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.341+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3724009900 con 0x7f372c10d7c0 2024-03-12T05:26:27.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.342+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c1aa800 con 0x7f372c10d7c0 2024-03-12T05:26:27.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.345+0000 7f3723fff640 1 -- 172.21.15.45:0/2441859824 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3714009780 con 0x7f372c10d7c0 2024-03-12T05:26:27.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.345+0000 7f3734042740 1 Processor -- start 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.345+0000 7f3734042740 1 -- start start 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3a4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3afba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff313280 con 0x558dff3a4bd0 2024-03-12T05:26:27.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3107b0 con 0x558dff3ad7b0 2024-03-12T05:26:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3734042740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff30bf90 con 0x558dff3a42b0 2024-03-12T05:26:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3723fff640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 0x7f37180a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.341 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f372affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f372a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f372affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54848/0 (socket says 172.21.15.45:54848) 2024-03-12T05:26:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f372affd640 1 -- 172.21.15.45:0/350586081 learned_addr learned my addr 172.21.15.45:0/350586081 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f372a7fc640 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.346+0000 7f3723fff640 1 -- 172.21.15.45:0/2441859824 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f37140ccee0 con 0x7f372c10d7c0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 -- 172.21.15.45:0/350586081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 msgr2=0x558dff3a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 -- 172.21.15.45:0/350586081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 msgr2=0x558dff3ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 -- 172.21.15.45:0/350586081 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dff2ca570 con 0x558dff3ad7b0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 0x7f37180a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372affd640 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f372a7fc640 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3afba0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f371c0097e0 tx=0x7f371c009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff7846ffe51d5e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f37217fa640 1 -- 172.21.15.45:0/350586081 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c004050 con 0x558dff3ad7b0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f37217fa640 1 -- 172.21.15.45:0/350586081 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f371c005020 con 0x558dff3ad7b0 2024-03-12T05:26:27.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.347+0000 7f37217fa640 1 -- 172.21.15.45:0/350586081 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c004bc0 con 0x558dff3ad7b0 2024-03-12T05:26:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 -- 172.21.15.45:0/350586081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 msgr2=0x558dff3afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3afba0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f371c0097e0 tx=0x7f371c009770 comp rx=0 tx=0).stop 2024-03-12T05:26:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f372b7fe640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 0x7f37180a41b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3714099b20 tx=0x7f37140035f0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 -- 172.21.15.45:0/350586081 shutdown_connections 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3afba0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 --2- 172.21.15.45:0/350586081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.348+0000 7f3734042740 1 -- 172.21.15.45:0/350586081 >> 172.21.15.45:0/350586081 conn(0x558dff39e990 msgr2=0x558dff39ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.349+0000 7f3734042740 1 -- 172.21.15.45:0/350586081 shutdown_connections 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.349+0000 7f3734042740 1 -- 172.21.15.45:0/350586081 wait complete. 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 -- start start 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 Processor -- start 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 0x558dff3ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 0x558dff3aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dfefdf430 con 0x558dff3a42b0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.350+0000 7f3734042740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dfef9be00 con 0x558dff3a4bd0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f3734042740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff290fd0 con 0x558dff3ad7b0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 0x558dff3ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f3734042740 10 client.0 _ll_register_callbacks cb 0x558dff2e6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 0x558dff3ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54858/0 (socket says 172.21.15.45:54858) 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 -- 172.21.15.45:0/3010475212 learned_addr learned my addr 172.21.15.45:0/3010475212 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 0x558dff3aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 -- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 msgr2=0x558dff3aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3aa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 -- 172.21.15.45:0/3010475212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 msgr2=0x558dff3ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 0x558dff3ad220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372a7fc640 1 -- 172.21.15.45:0/3010475212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c019040 con 0x558dff3ad7b0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.351+0000 7f372affd640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.352+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 0x558dff3aacf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f371c0047e0 tx=0x7f371c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2247455361142973 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.352+0000 7f37127fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.352+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c004bc0 con 0x558dff3ad7b0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.352+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558dff00b7e0 con 0x558dff3ad7b0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.352+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f371c000e40 con 0x558dff3ad7b0 2024-03-12T05:26:27.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.353+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f371c004d20 con 0x558dff3ad7b0 2024-03-12T05:26:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.353+0000 7f3734042740 10 client.28828 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.353+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558dff30a610 con 0x558dff3ad7b0 2024-03-12T05:26:27.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.353+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f371c004e80 con 0x558dff3ad7b0 2024-03-12T05:26:27.353 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.353+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f371c02c070 con 0x558dff3ad7b0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 10 client.28828 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 10 client.28828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 20 client.28828 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff3bd7f0 0x558dff3bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 4 client.28828 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558dff00b7e0 con 0x558dff3bd7f0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.354+0000 7f372affd640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff3bd7f0 0x558dff3bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.355+0000 7f372affd640 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff3bd7f0 0x558dff3bfbe0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f182570382580b2 server_cookie=b79e88bb33ce83f in_seq=0 out_seq=0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.355+0000 7f37127fc640 10 client.28828 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.356+0000 7f37127fc640 1 -- 172.21.15.45:0/3010475212 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558dff00b7e0 con 0x558dff3bd7f0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.356+0000 7f37127fc640 10 client.28828 handle_command_reply: tid=0 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.356+0000 7f3734042740 1 client.28828 shutdown 2024-03-12T05:26:27.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.357+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 msgr2=0x558dff3aacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.357+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 0x558dff3aacf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f371c0047e0 tx=0x7f371c005850 comp rx=0 tx=0).stop 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 shutdown_connections 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff3bd7f0 0x558dff3bfbe0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3ad7b0 0x558dff3aacf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 --2- 172.21.15.45:0/3010475212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a42b0 0x558dff3ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 >> 172.21.15.45:0/3010475212 conn(0x558dff39e990 msgr2=0x558dff3a0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f37127fc640 0 client.28828 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 shutdown_connections 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 1 -- 172.21.15.45:0/3010475212 wait complete. 2024-03-12T05:26:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.358+0000 7f3734042740 20 client.28828 trim_cache size 0 max 16384 2024-03-12T05:26:27.380 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 Processor -- start 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 -- start start 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3ad7b0 0x558dff3a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3a3f10 con 0x558dff3ad7b0 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff39f9e0 con 0x558dff3a4bd0 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f3734042740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3a3680 con 0x558dff3a42b0 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.385+0000 7f372affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:43972/0 (socket says 172.21.15.45:43972) 2024-03-12T05:26:27.381 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 -- 172.21.15.45:0/1855394774 learned_addr learned my addr 172.21.15.45:0/1855394774 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372a7fc640 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3ad7b0 0x558dff3a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372a7fc640 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 -- 172.21.15.45:0/1855394774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 msgr2=0x558dff3acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 -- 172.21.15.45:0/1855394774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3ad7b0 msgr2=0x558dff3a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3ad7b0 0x558dff3a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 -- 172.21.15.45:0/1855394774 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dff2a6ac0 con 0x558dff3a4bd0 2024-03-12T05:26:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f372affd640 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f372402e010 tx=0x7f37240095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ccb8f203d16c926 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f3720ff9640 1 -- 172.21.15.45:0/1855394774 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724045040 con 0x558dff3a4bd0 2024-03-12T05:26:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f3720ff9640 1 -- 172.21.15.45:0/1855394774 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f372404b070 con 0x558dff3a4bd0 2024-03-12T05:26:27.383 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.386+0000 7f3720ff9640 1 -- 172.21.15.45:0/1855394774 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724034440 con 0x558dff3a4bd0 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.387+0000 7f3734042740 1 -- 172.21.15.45:0/1855394774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 msgr2=0x558dff3ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.387+0000 7f3734042740 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f372402e010 tx=0x7f37240095c0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.387+0000 7f3734042740 1 -- 172.21.15.45:0/1855394774 shutdown_connections 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3ad7b0 0x558dff3a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3a4bd0 0x558dff3ad220 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 --2- 172.21.15.45:0/1855394774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 -- 172.21.15.45:0/1855394774 >> 172.21.15.45:0/1855394774 conn(0x558dff39e990 msgr2=0x558dff3a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 -- 172.21.15.45:0/1855394774 shutdown_connections 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 -- 172.21.15.45:0/1855394774 wait complete. 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 -- start start 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.388+0000 7f3734042740 1 Processor -- start 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.389 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3b1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3b4f80 con 0x558dff3a4bd0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3b16b0 con 0x558dff3ad7b0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dff3b1870 con 0x558dff3a42b0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f3734042740 10 client.0 _ll_register_callbacks cb 0x7f371c009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:36762/0 (socket says 172.21.15.45:36762) 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 -- 172.21.15.45:0/3689406125 learned_addr learned my addr 172.21.15.45:0/3689406125 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372affd640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 -- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 msgr2=0x558dff3b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3b1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.390 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 -- 172.21.15.45:0/3689406125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 msgr2=0x558dff3b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 -- 172.21.15.45:0/3689406125 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c114e80 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f372a7fc640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f371c0058c0 tx=0x7f371c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2467b9d59a14a0db server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.389+0000 7f37127fc640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c00fa70 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f371c000e40 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f371c005020 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f371c004810 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f3734042740 10 client.28834 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558dff2cc6a0 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.390+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f371c004e20 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f371c02c040 con 0x558dff3a42b0 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 10 client.28834 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 10 client.28834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:27.391 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 20 client.28834 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff414de0 0x558dff417200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 4 client.28834 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f371c005020 con 0x558dff414de0 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f372affd640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff414de0 0x558dff417200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f372affd640 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff414de0 0x558dff417200 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d089790ca0b6259 server_cookie=675414db64fc0c2f in_seq=0 out_seq=0 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.391+0000 7f37127fc640 10 client.28834 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f37127fc640 1 -- 172.21.15.45:0/3689406125 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f371c005020 con 0x558dff414de0 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f37127fc640 10 client.28834 handle_command_reply: tid=0 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f3734042740 1 client.28834 shutdown 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 msgr2=0x558dff3b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f371c0058c0 tx=0x7f371c00f790 comp rx=0 tx=0).stop 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 shutdown_connections 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.392+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x558dff414de0 0x558dff417200 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x558dff3a4bd0 0x558dff3b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dff3ad7b0 0x558dff3b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f37127fc640 0 client.28834 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 --2- 172.21.15.45:0/3689406125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558dff3a42b0 0x558dff3b46b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 >> 172.21.15.45:0/3689406125 conn(0x558dff39e990 msgr2=0x558dff39fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 shutdown_connections 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 1 -- 172.21.15.45:0/3689406125 wait complete. 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.393+0000 7f3734042740 20 client.28834 trim_cache size 0 max 16384 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stdout: "1ac5e9bd-4514-4dba-a480-031522e8b62d": { 2024-03-12T05:26:27.393 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:26:27.394 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "1ac5e9bd-4514-4dba-a480-031522e8b62d", 2024-03-12T05:26:27.394 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:26:27.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:26:27.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:26:27.394 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 msgr2=0x7f37180a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 0x7f37180a41b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3714099b20 tx=0x7f37140035f0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 msgr2=0x7f372c19daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c19daf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f371400d600 tx=0x7f371400dac0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 shutdown_connections 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37180a1cf0 0x7f37180a41b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f372c10d7c0 0x7f372c19daf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f372c108240 0x7f372c1a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 --2- 172.21.15.45:0/2441859824 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f372c1078a0 0x7f372c1a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.394+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 >> 172.21.15.45:0/2441859824 conn(0x7f372c076ff0 msgr2=0x7f372c077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.395+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 shutdown_connections 2024-03-12T05:26:27.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:27.395+0000 7f373146f640 1 -- 172.21.15.45:0/2441859824 wait complete. 2024-03-12T05:26:27.405 INFO:tasks.cephfs.filesystem:scrub status for tag:1ac5e9bd-4514-4dba-a480-031522e8b62d - {'path': '/', 'tag': '1ac5e9bd-4514-4dba-a480-031522e8b62d', 'options': 'recursive,force'} 2024-03-12T05:26:28.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:27 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:27 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: pgmap v490: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:27 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:29.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:28 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:26:29.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:28 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:29.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:28 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:26:29.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:28 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:29.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:28 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:26:29.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:28 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:26:29.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 865.0 s, 1395.2 tps, lat 72.184 ms stddev 29.416 2024-03-12T05:26:30.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:30 smithi045 ceph-mon[32174]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 18 MiB/s wr, 1.33k op/s 2024-03-12T05:26:30.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:30 smithi130 ceph-mon[35398]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 18 MiB/s wr, 1.33k op/s 2024-03-12T05:26:30.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:30 smithi116 ceph-mon[35486]: pgmap v491: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 18 MiB/s wr, 1.33k op/s 2024-03-12T05:26:32.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:32 smithi045 ceph-mon[32174]: pgmap v492: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-03-12T05:26:32.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:32 smithi130 ceph-mon[35398]: pgmap v492: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-03-12T05:26:32.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:32 smithi116 ceph-mon[35486]: pgmap v492: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-03-12T05:26:33.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:33 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:26:33.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:33 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:26:33.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:33 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:26:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:34 smithi045 ceph-mon[32174]: pgmap v493: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-03-12T05:26:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:34 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:26:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:34 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:26:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:34 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:26:34.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 870.0 s, 1257.2 tps, lat 79.595 ms stddev 45.664 2024-03-12T05:26:34.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:34 smithi130 ceph-mon[35398]: pgmap v493: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-03-12T05:26:34.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:34 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:26:34.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:34 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:26:34.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:34 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:26:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:34 smithi116 ceph-mon[35486]: pgmap v493: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-03-12T05:26:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:34 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:26:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:34 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:26:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:34 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:26:35.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:35 smithi045 ceph-mon[32174]: pgmap v494: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-03-12T05:26:35.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:35 smithi130 ceph-mon[35398]: pgmap v494: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-03-12T05:26:35.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:35 smithi116 ceph-mon[35486]: pgmap v494: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 26 MiB/s wr, 1.64k op/s 2024-03-12T05:26:38.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:37 smithi045 ceph-mon[32174]: pgmap v495: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-12T05:26:38.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:37 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:38.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:37 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:37 smithi130 ceph-mon[35398]: pgmap v495: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-12T05:26:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:37 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:37 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:38.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:37 smithi116 ceph-mon[35486]: pgmap v495: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-12T05:26:38.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:37 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:38.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:37 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:39.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:39.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:38 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:26:39.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:39.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:38 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:26:39.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:39.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:38 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:26:39.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 875.0 s, 1398.2 tps, lat 71.390 ms stddev 33.927 2024-03-12T05:26:40.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:39 smithi045 ceph-mon[32174]: pgmap v496: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 20 MiB/s wr, 1.25k op/s 2024-03-12T05:26:40.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:39 smithi130 ceph-mon[35398]: pgmap v496: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 20 MiB/s wr, 1.25k op/s 2024-03-12T05:26:40.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:39 smithi116 ceph-mon[35486]: pgmap v496: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 20 MiB/s wr, 1.25k op/s 2024-03-12T05:26:42.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:41 smithi045 ceph-mon[32174]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 21 MiB/s wr, 1.37k op/s 2024-03-12T05:26:42.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:41 smithi130 ceph-mon[35398]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 21 MiB/s wr, 1.37k op/s 2024-03-12T05:26:42.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:41 smithi116 ceph-mon[35486]: pgmap v497: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 21 MiB/s wr, 1.37k op/s 2024-03-12T05:26:44.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:43 smithi045 ceph-mon[32174]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-03-12T05:26:44.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:43 smithi130 ceph-mon[35398]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-03-12T05:26:44.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:43 smithi116 ceph-mon[35486]: pgmap v498: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 16 MiB/s wr, 1.06k op/s 2024-03-12T05:26:44.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 880.0 s, 1303.6 tps, lat 76.802 ms stddev 41.400 2024-03-12T05:26:46.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:45 smithi045 ceph-mon[32174]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-03-12T05:26:46.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:45 smithi130 ceph-mon[35398]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-03-12T05:26:46.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:45 smithi116 ceph-mon[35486]: pgmap v499: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-03-12T05:26:48.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:47 smithi045 ceph-mon[32174]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-03-12T05:26:48.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:47 smithi130 ceph-mon[35398]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-03-12T05:26:48.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:47 smithi116 ceph-mon[35486]: pgmap v500: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-03-12T05:26:49.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 885.0 s, 1416.6 tps, lat 70.457 ms stddev 34.016 2024-03-12T05:26:50.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:49 smithi045 ceph-mon[32174]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-03-12T05:26:50.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:49 smithi130 ceph-mon[35398]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-03-12T05:26:50.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:49 smithi116 ceph-mon[35486]: pgmap v501: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-03-12T05:26:52.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:51 smithi045 ceph-mon[32174]: pgmap v502: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-03-12T05:26:52.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:51 smithi130 ceph-mon[35398]: pgmap v502: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-03-12T05:26:52.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:51 smithi116 ceph-mon[35486]: pgmap v502: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 19 MiB/s wr, 1.45k op/s 2024-03-12T05:26:54.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:53 smithi045 ceph-mon[32174]: pgmap v503: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-03-12T05:26:54.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:53 smithi130 ceph-mon[35398]: pgmap v503: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-03-12T05:26:54.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:53 smithi116 ceph-mon[35486]: pgmap v503: 97 pgs: 97 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 15 MiB/s wr, 1.14k op/s 2024-03-12T05:26:54.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 890.0 s, 1422.4 tps, lat 70.389 ms stddev 32.485 2024-03-12T05:26:56.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:55 smithi045 ceph-mon[32174]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-03-12T05:26:56.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:55 smithi130 ceph-mon[35398]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-03-12T05:26:56.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:55 smithi116 ceph-mon[35486]: pgmap v504: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-03-12T05:26:57.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:26:57.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:26:57.578 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.583+0000 7f95a5ffb640 1 -- 172.21.15.45:0/1722337138 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598034ea0 con 0x7f95a8108660 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 -- 172.21.15.45:0/1722337138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8108660 msgr2=0x7f95a8110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1722337138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8108660 0x7f95a8110230 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9598009860 tx=0x7f959802bb70 comp rx=0 tx=0).stop 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 -- 172.21.15.45:0/1722337138 shutdown_connections 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1722337138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8103f40 0x7f95a8107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1722337138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8108660 0x7f95a8110230 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1722337138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 -- 172.21.15.45:0/1722337138 >> 172.21.15.45:0/1722337138 conn(0x7f95a80fee30 msgr2=0x7f95a8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 -- 172.21.15.45:0/1722337138 shutdown_connections 2024-03-12T05:26:57.579 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.584+0000 7f95ad66f640 1 -- 172.21.15.45:0/1722337138 wait complete. 2024-03-12T05:26:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 Processor -- start 2024-03-12T05:26:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 -- start start 2024-03-12T05:26:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108660 0x7f95a81960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.582 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 0x7f95a819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819a7d0 con 0x7f95a8108660 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819a940 con 0x7f95a8103590 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95ad66f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819aab0 con 0x7f95a8199de0 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 0x7f95a819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55774/0 (socket says 172.21.15.45:55774) 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 0x7f95a819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:32940/0 (socket says 172.21.15.45:32940) 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a6ffd640 1 -- 172.21.15.45:0/1616800878 learned_addr learned my addr 172.21.15.45:0/1616800878 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.585+0000 7f95a67fc640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108660 0x7f95a81960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 -- 172.21.15.45:0/1616800878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 msgr2=0x7f95a819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 0x7f95a819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 -- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108660 msgr2=0x7f95a81960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108660 0x7f95a81960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.583 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 -- 172.21.15.45:0/1616800878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009490 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.586+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009fa0 tx=0x7f95a0007970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.587+0000 7f958ffff640 1 -- 172.21.15.45:0/1616800878 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a002e790 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.587+0000 7f958ffff640 1 -- 172.21.15.45:0/1616800878 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95a0029040 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.587+0000 7f958ffff640 1 -- 172.21.15.45:0/1616800878 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a002fd40 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.587+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a806aba0 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.587+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a806b000 con 0x7f95a8103590 2024-03-12T05:26:57.584 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.590+0000 7f958ffff640 1 -- 172.21.15.45:0/1616800878 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f95a0039460 con 0x7f95a8103590 2024-03-12T05:26:57.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.591+0000 7f958ffff640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 0x7f95900a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.591+0000 7f958ffff640 1 -- 172.21.15.45:0/1616800878 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f95a00e8330 con 0x7f95a8103590 2024-03-12T05:26:57.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.592+0000 7f95a67fc640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 0x7f95900a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.593+0000 7f95a67fc640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 0x7f95900a4230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95a8108100 tx=0x7f959c009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.593+0000 7f95b0240740 1 Processor -- start 2024-03-12T05:26:57.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.594+0000 7f95b0240740 1 -- start start 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.594+0000 7f95b0240740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa654690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.594+0000 7f95b0240740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa65d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95b0240740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa65fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa65fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a77fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa65fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38374/0 (socket says 172.21.15.45:38374) 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a77fe640 1 -- 172.21.15.45:0/3528135480 learned_addr learned my addr 172.21.15.45:0/3528135480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa654690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa5c3280 con 0x564cfa65d7b0 2024-03-12T05:26:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa5c07b0 con 0x564cfa654bd0 2024-03-12T05:26:57.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa5bbf90 con 0x564cfa6542b0 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 -- 172.21.15.45:0/3528135480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 msgr2=0x564cfa65d270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa65d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 -- 172.21.15.45:0/3528135480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 msgr2=0x564cfa65fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa65fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.595+0000 7f95a6ffd640 1 -- 172.21.15.45:0/3528135480 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cfa57a570 con 0x564cfa6542b0 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.596+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa654690 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f95a00e59f0 tx=0x7f95a00b5720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca44ded635f34bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.596+0000 7f958d7fa640 1 -- 172.21.15.45:0/3528135480 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a002fea0 con 0x564cfa6542b0 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.596+0000 7f958d7fa640 1 -- 172.21.15.45:0/3528135480 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f95a00e7050 con 0x564cfa6542b0 2024-03-12T05:26:57.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.597+0000 7f958d7fa640 1 -- 172.21.15.45:0/3528135480 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00bb0b0 con 0x564cfa6542b0 2024-03-12T05:26:57.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 msgr2=0x564cfa654690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa654690 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f95a00e59f0 tx=0x7f95a00b5720 comp rx=0 tx=0).stop 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 shutdown_connections 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa65fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa65d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 --2- 172.21.15.45:0/3528135480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa654690 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 >> 172.21.15.45:0/3528135480 conn(0x564cfa64e990 msgr2=0x564cfa64ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 shutdown_connections 2024-03-12T05:26:57.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.599+0000 7f95b0240740 1 -- 172.21.15.45:0/3528135480 wait complete. 2024-03-12T05:26:57.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.600+0000 7f95b0240740 1 -- start start 2024-03-12T05:26:57.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.600+0000 7f95b0240740 1 Processor -- start 2024-03-12T05:26:57.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa65ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 0x564cfa65b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:32946/0 (socket says 172.21.15.45:32946) 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a6ffd640 1 -- 172.21.15.45:0/2785621082 learned_addr learned my addr 172.21.15.45:0/2785621082 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa28f430 con 0x564cfa654bd0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa24be00 con 0x564cfa65d7b0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa540fd0 con 0x564cfa6542b0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95b0240740 10 client.0 _ll_register_callbacks cb 0x564cfa596fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a67fc640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa65ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a77fe640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 0x564cfa65b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.601+0000 7f95a6ffd640 1 -- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 msgr2=0x564cfa65b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 0x564cfa65b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a6ffd640 1 -- 172.21.15.45:0/2785621082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 msgr2=0x564cfa65ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa65ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a6ffd640 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a806b000 con 0x564cfa6542b0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a77fe640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 0x564cfa65b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564cfa2bb7e0 con 0x564cfa6542b0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f95a00b1930 tx=0x7f95a00b1330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5007ec5b3071027 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f958aff5640 10 client.28846 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f958aff5640 1 -- 172.21.15.45:0/2785621082 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a002fea0 con 0x564cfa6542b0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f958aff5640 1 -- 172.21.15.45:0/2785621082 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f95a00bb300 con 0x564cfa6542b0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f958aff5640 1 -- 172.21.15.45:0/2785621082 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95a00bd670 con 0x564cfa6542b0 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95b0240740 10 client.28846 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:57.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.602+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x564cfa5ba610 con 0x564cfa6542b0 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f958aff5640 1 -- 172.21.15.45:0/2785621082 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f95a002c070 con 0x564cfa6542b0 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 10 client.28846 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 10 client.28846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 20 client.28846 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa66d260 0x564cfa66f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 4 client.28846 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564cfa2f8720 con 0x564cfa66d260 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.603+0000 7f95a67fc640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa66d260 0x564cfa66f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.604+0000 7f958aff5640 10 client.28846 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.604+0000 7f95a67fc640 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa66d260 0x564cfa66f6a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f01289a6e1e23cc server_cookie=cd102d8c8dff72f2 in_seq=0 out_seq=0 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.604+0000 7f958aff5640 1 -- 172.21.15.45:0/2785621082 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564cfa2f8720 con 0x564cfa66d260 2024-03-12T05:26:57.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.604+0000 7f958aff5640 10 client.28846 handle_command_reply: tid=0 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.604+0000 7f95b0240740 1 client.28846 shutdown 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 msgr2=0x564cfa65a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f95a00b1930 tx=0x7f95a00b1330 comp rx=0 tx=0).stop 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 shutdown_connections 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa66d260 0x564cfa66f6a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa65ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa65d7b0 0x564cfa65b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f958aff5640 0 client.28846 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 --2- 172.21.15.45:0/2785621082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa65a630 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 >> 172.21.15.45:0/2785621082 conn(0x564cfa64e990 msgr2=0x564cfa650730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 shutdown_connections 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 1 -- 172.21.15.45:0/2785621082 wait complete. 2024-03-12T05:26:57.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.605+0000 7f95b0240740 20 client.28846 trim_cache size 0 max 16384 2024-03-12T05:26:57.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.638+0000 7f95b0240740 1 Processor -- start 2024-03-12T05:26:57.633 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 -- start start 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 0x564cfa65cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa659790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 0x564cfa65ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa64a0f0 con 0x564cfa654bd0 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa64f9e0 con 0x564cfa6542b0 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95b0240740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa653630 con 0x564cfa65d7b0 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 0x564cfa65cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 0x564cfa65cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55812/0 (socket says 172.21.15.45:55812) 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95a6ffd640 1 -- 172.21.15.45:0/3579160392 learned_addr learned my addr 172.21.15.45:0/3579160392 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.634 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 0x564cfa65ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.639+0000 7f95a67fc640 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa659790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 -- 172.21.15.45:0/3579160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 msgr2=0x564cfa65cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 0x564cfa65cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 -- 172.21.15.45:0/3579160392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 msgr2=0x564cfa659790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa659790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 -- 172.21.15.45:0/3579160392 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cfa556ac0 con 0x564cfa65d7b0 2024-03-12T05:26:57.635 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f95a77fe640 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 0x564cfa65ad00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x564cfa64fb70 tx=0x7f9598034b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df4dcd4f5c907d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f958cff9640 1 -- 172.21.15.45:0/3579160392 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598004030 con 0x564cfa65d7b0 2024-03-12T05:26:57.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f958cff9640 1 -- 172.21.15.45:0/3579160392 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f959802ce90 con 0x564cfa65d7b0 2024-03-12T05:26:57.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.640+0000 7f958cff9640 1 -- 172.21.15.45:0/3579160392 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959802c910 con 0x564cfa65d7b0 2024-03-12T05:26:57.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.641+0000 7f95b0240740 1 -- 172.21.15.45:0/3579160392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 msgr2=0x564cfa65ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.641+0000 7f95b0240740 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 0x564cfa65ad00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x564cfa64fb70 tx=0x7f9598034b70 comp rx=0 tx=0).stop 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 -- 172.21.15.45:0/3579160392 shutdown_connections 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa654bd0 0x564cfa659790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa65d7b0 0x564cfa65ad00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 --2- 172.21.15.45:0/3579160392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa6542b0 0x564cfa65cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 -- 172.21.15.45:0/3579160392 >> 172.21.15.45:0/3579160392 conn(0x564cfa64e990 msgr2=0x564cfa65a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 -- 172.21.15.45:0/3579160392 shutdown_connections 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 -- 172.21.15.45:0/3579160392 wait complete. 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 -- start start 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.642+0000 7f95b0240740 1 Processor -- start 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95b0240740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa6bde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95b0240740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95b0240740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa663be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa6bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a6ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa6bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:32958/0 (socket says 172.21.15.45:32958) 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55816/0 (socket says 172.21.15.45:55816) 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a6ffd640 1 -- 172.21.15.45:0/204389499 learned_addr learned my addr 172.21.15.45:0/204389499 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95a77fe640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa663be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa6641a0 con 0x564cfa65d7b0 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.643+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa664310 con 0x564cfa654bd0 2024-03-12T05:26:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cfa6644d0 con 0x564cfa6542b0 2024-03-12T05:26:57.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95b0240740 10 client.0 _ll_register_callbacks cb 0x7f95a00b1280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:57.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 -- 172.21.15.45:0/204389499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 msgr2=0x564cfa6bde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa6bde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 -- 172.21.15.45:0/204389499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 msgr2=0x564cfa663be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa663be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a8110a40 con 0x564cfa654bd0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a77fe640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa663be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95a00e7050 con 0x564cfa654bd0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f95a67fc640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x564cfa659e90 tx=0x7f959c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690c6cd7394b2f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.644+0000 7f958aff5640 10 client.28889 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f958aff5640 1 -- 172.21.15.45:0/204389499 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c024040 con 0x564cfa654bd0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f958aff5640 1 -- 172.21.15.45:0/204389499 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f959c017070 con 0x564cfa654bd0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f958aff5640 1 -- 172.21.15.45:0/204389499 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f959c011b10 con 0x564cfa654bd0 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 10 client.28889 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:57.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x564cfa57c6a0 con 0x564cfa654bd0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f958aff5640 1 -- 172.21.15.45:0/204389499 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f959c011c70 con 0x564cfa654bd0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 10 client.28889 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 10 client.28889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 20 client.28889 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa6c5bb0 0x564cfa6c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 4 client.28889 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.645+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564cfa347890 con 0x564cfa6c5bb0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa6c5bb0 0x564cfa6c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95a6ffd640 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa6c5bb0 0x564cfa6c7fd0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd727957f7327eaa server_cookie=2f49468d2a493f8f in_seq=0 out_seq=0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f958aff5640 10 client.28889 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f958aff5640 1 -- 172.21.15.45:0/204389499 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564cfa347890 con 0x564cfa6c5bb0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f958aff5640 10 client.28889 handle_command_reply: tid=0 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95b0240740 1 client.28889 shutdown 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 msgr2=0x564cfa6be3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.643 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x564cfa659e90 tx=0x7f959c004730 comp rx=0 tx=0).stop 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 shutdown_connections 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.646+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x564cfa6c5bb0 0x564cfa6c7fd0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f958aff5640 0 client.28889 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x564cfa65d7b0 0x564cfa663be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564cfa654bd0 0x564cfa6be3c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 --2- 172.21.15.45:0/204389499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564cfa6542b0 0x564cfa6bde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 >> 172.21.15.45:0/204389499 conn(0x564cfa64e990 msgr2=0x564cfa64da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 shutdown_connections 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 1 -- 172.21.15.45:0/204389499 wait complete. 2024-03-12T05:26:57.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.647+0000 7f95b0240740 20 client.28889 trim_cache size 0 max 16384 2024-03-12T05:26:57.645 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:26:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:26:57.645 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:26:57.645 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.648+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 msgr2=0x7f95900a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.648+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 0x7f95900a4230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95a8108100 tx=0x7f959c009260 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.648+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 msgr2=0x7f95a8195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.648+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009fa0 tx=0x7f95a0007970 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 shutdown_connections 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f95900a1d70 0x7f95900a4230 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95a8108660 0x7f95a81960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95a8199de0 0x7f95a819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 --2- 172.21.15.45:0/1616800878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8103590 0x7f95a8195b80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 >> 172.21.15.45:0/1616800878 conn(0x7f95a80fee30 msgr2=0x7f95a8106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 shutdown_connections 2024-03-12T05:26:57.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.649+0000 7f95ad66f640 1 -- 172.21.15.45:0/1616800878 wait complete. 2024-03-12T05:26:57.661 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:26:57.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 -- 172.21.15.45:0/104183489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f012809acf0 msgr2=0x7f012809b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 --2- 172.21.15.45:0/104183489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f012809acf0 0x7f012809b0d0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0124009930 tx=0x7f012402bc70 comp rx=0 tx=0).stop 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 -- 172.21.15.45:0/104183489 shutdown_connections 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 --2- 172.21.15.45:0/104183489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f01280968f0 0x7f01280a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 --2- 172.21.15.45:0/104183489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0128095f30 0x7f01280963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 --2- 172.21.15.45:0/104183489 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f012809acf0 0x7f012809b0d0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 -- 172.21.15.45:0/104183489 >> 172.21.15.45:0/104183489 conn(0x7f012800cb40 msgr2=0x7f012800cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 -- 172.21.15.45:0/104183489 shutdown_connections 2024-03-12T05:26:57.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.844+0000 7f0134d31640 1 -- 172.21.15.45:0/104183489 wait complete. 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.845+0000 7f0134d31640 1 Processor -- start 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.845+0000 7f0134d31640 1 -- start start 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 0x7f0128006e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 0x7f0128007380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 0x7f01280078c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f012effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 0x7f0128007380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 0x7f0128006e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f012f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 0x7f0128006e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:32974/0 (socket says 172.21.15.45:32974) 2024-03-12T05:26:57.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f012effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 0x7f0128007380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38428/0 (socket says 172.21.15.45:38428) 2024-03-12T05:26:57.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012813a550 con 0x7f01280968f0 2024-03-12T05:26:57.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f012f7fe640 1 -- 172.21.15.45:0/2565755897 learned_addr learned my addr 172.21.15.45:0/2565755897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:57.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012813a6c0 con 0x7f012809acf0 2024-03-12T05:26:57.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.846+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012813a880 con 0x7f0128095f30 2024-03-12T05:26:57.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 0x7f01280078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 -- 172.21.15.45:0/2565755897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 msgr2=0x7f0128006e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 0x7f0128006e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 -- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 msgr2=0x7f0128007380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 0x7f0128007380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124009560 con 0x7f012809acf0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.847+0000 7f012ffff640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 0x7f01280078c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0130066e50 tx=0x7f0130074cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.848+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013006a5f0 con 0x7f012809acf0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.848+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f013006a750 con 0x7f012809acf0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.848+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130080420 con 0x7f012809acf0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.848+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012813ab60 con 0x7f012809acf0 2024-03-12T05:26:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.848+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012813aff0 con 0x7f012809acf0 2024-03-12T05:26:57.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.850+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128004830 con 0x7f012809acf0 2024-03-12T05:26:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.850+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0130080600 con 0x7f012809acf0 2024-03-12T05:26:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.851+0000 7f012cff9640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 0x7f01200a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.851+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f01301384f0 con 0x7f012809acf0 2024-03-12T05:26:57.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.853+0000 7f012f7fe640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 0x7f01200a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:57.847 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.853+0000 7f012f7fe640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 0x7f01200a41b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0124006010 tx=0x7f0124005a20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:57.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:57.858+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0130101900 con 0x7f012809acf0 2024-03-12T05:26:58.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:57 smithi045 ceph-mon[32174]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-03-12T05:26:58.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:57 smithi130 ceph-mon[35398]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-03-12T05:26:58.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:57 smithi116 ceph-mon[35486]: pgmap v505: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 14 MiB/s wr, 1.12k op/s 2024-03-12T05:26:58.252 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.258+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01280a5dd0 con 0x7f012809acf0 2024-03-12T05:26:58.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.261+0000 7f012cff9640 1 -- 172.21.15.45:0/2565755897 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f0130104fc0 con 0x7f012809acf0 2024-03-12T05:26:58.255 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:26:58.255 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:26:58.255 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:26:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.264+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 msgr2=0x7f01200a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.264+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 0x7f01200a41b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0124006010 tx=0x7f0124005a20 comp rx=0 tx=0).stop 2024-03-12T05:26:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.264+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 msgr2=0x7f01280078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.264+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 0x7f01280078c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0130066e50 tx=0x7f0130074cf0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 shutdown_connections 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f01200a1cf0 0x7f01200a41b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f01280968f0 0x7f0128007380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012809acf0 0x7f01280078c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 --2- 172.21.15.45:0/2565755897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0128095f30 0x7f0128006e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 >> 172.21.15.45:0/2565755897 conn(0x7f012800cb40 msgr2=0x7f0128094a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 shutdown_connections 2024-03-12T05:26:58.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.265+0000 7f0134d31640 1 -- 172.21.15.45:0/2565755897 wait complete. 2024-03-12T05:26:58.278 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:58.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:26:58.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.454+0000 7fc5d8d61640 1 -- 172.21.15.45:0/2260085817 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc033750 con 0x7fc5d4106450 2024-03-12T05:26:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.455+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/2260085817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4106450 msgr2=0x7fc5d410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.455+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/2260085817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4106450 0x7fc5d410e020 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009c10 tx=0x7fc5cc02bef0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/2260085817 shutdown_connections 2024-03-12T05:26:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/2260085817 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4106450 0x7fc5d410e020 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/2260085817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4100240 0x7fc5d4105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/2260085817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d40ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/2260085817 >> 172.21.15.45:0/2260085817 conn(0x7fc5d40765a0 msgr2=0x7fc5d40fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/2260085817 shutdown_connections 2024-03-12T05:26:58.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.456+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/2260085817 wait complete. 2024-03-12T05:26:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 Processor -- start 2024-03-12T05:26:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 -- start start 2024-03-12T05:26:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 0x7fc5d419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 0x7fc5d4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d419a1f0 con 0x7fc5d4199720 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d419a360 con 0x7fc5d40ff890 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5dbfc7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d419a520 con 0x7fc5d4106450 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5da564640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 0x7fc5d4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d419f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55848/0 (socket says 172.21.15.45:55848) 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5d9d63640 1 -- 172.21.15.45:0/878488257 learned_addr learned my addr 172.21.15.45:0/878488257 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.457+0000 7fc5d9562640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 0x7fc5d419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 -- 172.21.15.45:0/878488257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 msgr2=0x7fc5d419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 0x7fc5d419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 -- 172.21.15.45:0/878488257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 msgr2=0x7fc5d419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.454 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 -- 172.21.15.45:0/878488257 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc009840 con 0x7fc5d4199720 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5d9562640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 0x7fc5d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5da564640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 0x7fc5d4199ba0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc036040 tx=0x7fc5cc033fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.458+0000 7fc5caffd640 1 -- 172.21.15.45:0/878488257 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc003b80 con 0x7fc5d4199720 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.459+0000 7fc5caffd640 1 -- 172.21.15.45:0/878488257 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc5cc003ce0 con 0x7fc5d4199720 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.459+0000 7fc5caffd640 1 -- 172.21.15.45:0/878488257 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc0347f0 con 0x7fc5d4199720 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.459+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d41a6460 con 0x7fc5d4199720 2024-03-12T05:26:58.455 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.459+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d41a6940 con 0x7fc5d4199720 2024-03-12T05:26:58.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.462+0000 7fc5caffd640 1 -- 172.21.15.45:0/878488257 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc5cc004580 con 0x7fc5d4199720 2024-03-12T05:26:58.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.463+0000 7fc5caffd640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 0x7fc5c00a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.463+0000 7fc5caffd640 1 -- 172.21.15.45:0/878488257 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc5cc0f0530 con 0x7fc5d4199720 2024-03-12T05:26:58.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.463+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 0x7fc5c00a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.464+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 0x7fc5c00a4270 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00058c0 tx=0x7fc5d0009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.464+0000 7fc5deb9a740 1 Processor -- start 2024-03-12T05:26:58.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.464+0000 7fc5deb9a740 1 -- start start 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c98210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38458/0 (socket says 172.21.15.45:38458) 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 -- 172.21.15.45:0/3768205269 learned_addr learned my addr 172.21.15.45:0/3768205269 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874c4a960 con 0x555874c8f250 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874bfe310 con 0x555874c8fb70 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874bfb840 con 0x555874c98750 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5da564640 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 -- 172.21.15.45:0/3768205269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 msgr2=0x555874c9ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 -- 172.21.15.45:0/3768205269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 msgr2=0x555874c98210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c98210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.465+0000 7fc5d9d63640 1 -- 172.21.15.45:0/3768205269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555874bb9060 con 0x555874c8f250 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.466+0000 7fc5da564640 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:58.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.466+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009fc0 tx=0x7fc5d0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace789c2766f1845 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.466+0000 7fc5bffff640 1 -- 172.21.15.45:0/3768205269 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0026620 con 0x555874c8f250 2024-03-12T05:26:58.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.466+0000 7fc5bffff640 1 -- 172.21.15.45:0/3768205269 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc5d0026c30 con 0x555874c8f250 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.466+0000 7fc5bffff640 1 -- 172.21.15.45:0/3768205269 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d000ec50 con 0x555874c8f250 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.467+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 msgr2=0x555874c8f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.467+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009fc0 tx=0x7fc5d0014000 comp rx=0 tx=0).stop 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 shutdown_connections 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c98210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/3768205269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c8f630 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 >> 172.21.15.45:0/3768205269 conn(0x555874c89980 msgr2=0x555874c89d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 shutdown_connections 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 -- 172.21.15.45:0/3768205269 wait complete. 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 -- start start 2024-03-12T05:26:58.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.468+0000 7fc5deb9a740 1 Processor -- start 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c96290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5da564640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c96290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5d9562640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55864/0 (socket says 172.21.15.45:55864) 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38462/0 (socket says 172.21.15.45:38462) 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5da564640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c96290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33012/0 (socket says 172.21.15.45:33012) 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5d9562640 1 -- 172.21.15.45:0/98112132 learned_addr learned my addr 172.21.15.45:0/98112132 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874bf7020 con 0x555874c8f250 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558748ca430 con 0x555874c8fb70 2024-03-12T05:26:58.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874886e00 con 0x555874c98750 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.469+0000 7fc5deb9a740 10 client.0 _ll_register_callbacks cb 0x555874b9ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 -- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 msgr2=0x555874c96290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c96290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 -- 172.21.15.45:0/98112132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 msgr2=0x555874c95810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d0017050 con 0x555874c8fb70 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5d9562640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4010680 tx=0x7fc5c4010b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47cb2ad337a5aa3b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400eab0 con 0x555874c8fb70 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc5c4017070 con 0x555874c8fb70 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.470+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558748f67e0 con 0x555874c8fb70 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5c4010e80 con 0x555874c8fb70 2024-03-12T05:26:58.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5deb9a740 10 client.28901 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555874bf56a0 con 0x555874c8fb70 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5558748f67e0 con 0x555874c8fb70 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc5c401b430 con 0x555874c8fb70 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5deb9a740 10 client.28901 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.471+0000 7fc5deb9a740 10 client.28901 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.472+0000 7fc5deb9a740 20 client.28901 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.472+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874ca88a0 0x555874caacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.472+0000 7fc5deb9a740 4 client.28901 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:58.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.472+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555874933720 con 0x555874ca88a0 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.472+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874ca88a0 0x555874caacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.473+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874ca88a0 0x555874caacc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=16b9949a08b01b7a server_cookie=9e19ce7e6822d9c8 in_seq=0 out_seq=0 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.473+0000 7fc5bd7fa640 10 client.28901 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.473+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/98112132 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555874933720 con 0x555874ca88a0 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.473+0000 7fc5bd7fa640 10 client.28901 handle_command_reply: tid=0 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.473+0000 7fc5deb9a740 1 client.28901 shutdown 2024-03-12T05:26:58.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 msgr2=0x555874c95d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4010680 tx=0x7fc5c4010b40 comp rx=0 tx=0).stop 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 shutdown_connections 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874ca88a0 0x555874caacc0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c96290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8fb70 0x555874c95d50 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/98112132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8f250 0x555874c95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 >> 172.21.15.45:0/98112132 conn(0x555874c89980 msgr2=0x555874c8b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5bd7fa640 0 client.28901 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 shutdown_connections 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 1 -- 172.21.15.45:0/98112132 wait complete. 2024-03-12T05:26:58.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.474+0000 7fc5deb9a740 20 client.28901 trim_cache size 0 max 16384 2024-03-12T05:26:58.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 Processor -- start 2024-03-12T05:26:58.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 -- start start 2024-03-12T05:26:58.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c92ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5d9d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55872/0 (socket says 172.21.15.45:55872) 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5d9d63640 1 -- 172.21.15.45:0/2862756810 learned_addr learned my addr 172.21.15.45:0/2862756810 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5d9562640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874b7c130 con 0x555874c8fb70 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874c96b60 con 0x555874c8f250 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.494+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874c8abe0 con 0x555874c98750 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5da564640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c92ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 -- 172.21.15.45:0/2862756810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 msgr2=0x555874c92ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c92ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 -- 172.21.15.45:0/2862756810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 msgr2=0x555874c960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 -- 172.21.15.45:0/2862756810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555874b91c20 con 0x555874c8f250 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9562640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.495+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009740 tx=0x7fc5d0011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6fcb96737f0574f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.496+0000 7fc5bf7fe640 1 -- 172.21.15.45:0/2862756810 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0036ea0 con 0x555874c8f250 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.496+0000 7fc5bf7fe640 1 -- 172.21.15.45:0/2862756810 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc5d0036070 con 0x555874c8f250 2024-03-12T05:26:58.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.496+0000 7fc5bf7fe640 1 -- 172.21.15.45:0/2862756810 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0036440 con 0x555874c8f250 2024-03-12T05:26:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.497+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 msgr2=0x555874c97f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.497+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009740 tx=0x7fc5d0011d70 comp rx=0 tx=0).stop 2024-03-12T05:26:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.497+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 shutdown_connections 2024-03-12T05:26:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.497+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c92ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2862756810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c97f40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 >> 172.21.15.45:0/2862756810 conn(0x555874c89980 msgr2=0x555874c90b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 shutdown_connections 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2862756810 wait complete. 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.498+0000 7fc5deb9a740 1 -- start start 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.499+0000 7fc5deb9a740 1 Processor -- start 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.499+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.499+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5deb9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5deb9a740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874cfef60 con 0x555874c8fb70 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5deb9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874cff0d0 con 0x555874c8f250 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5deb9a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555874cff290 con 0x555874c98750 2024-03-12T05:26:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5deb9a740 10 client.0 _ll_register_callbacks cb 0x7fc5d0009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5d9562640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5d9562640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38494/0 (socket says 172.21.15.45:38494) 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5d9562640 1 -- 172.21.15.45:0/2828673089 learned_addr learned my addr 172.21.15.45:0/2828673089 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5da564640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.500+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 -- 172.21.15.45:0/2828673089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 msgr2=0x555874c9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 -- 172.21.15.45:0/2828673089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 msgr2=0x555874c9e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 -- 172.21.15.45:0/2828673089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555874bf56a0 con 0x555874c98750 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5d9562640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5da564640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9eda0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0edd00 tx=0x7fc5cc0bd8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19d40a4af88fb6d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc0ee070 con 0x555874c98750 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc5cc0c1330 con 0x555874c98750 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5d0026c30 con 0x555874c98750 2024-03-12T05:26:58.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.501+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5cc0c5980 con 0x555874c98750 2024-03-12T05:26:58.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.502+0000 7fc5deb9a740 10 client.28870 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:58.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.502+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555874bb9060 con 0x555874c98750 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.502+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc5cc0c9080 con 0x555874c98750 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc5cc03c070 con 0x555874c98750 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 10 client.28870 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 10 client.28870 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 20 client.28870 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874d008c0 0x555874d02cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 4 client.28870 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.503+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555874982890 con 0x555874d008c0 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.504+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874d008c0 0x555874d02cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.505+0000 7fc5d9d63640 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874d008c0 0x555874d02cb0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2dec7c2749e4cc53 server_cookie=31039d7704a01ef9 in_seq=0 out_seq=0 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.505+0000 7fc5bd7fa640 10 client.28870 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.505+0000 7fc5bd7fa640 1 -- 172.21.15.45:0/2828673089 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555874982890 con 0x555874d008c0 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.505+0000 7fc5bd7fa640 10 client.28870 handle_command_reply: tid=0 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.505+0000 7fc5deb9a740 1 client.28870 shutdown 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 msgr2=0x555874c9eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9eda0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc0edd00 tx=0x7fc5cc0bd8e0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 shutdown_connections 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x555874d008c0 0x555874d02cb0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x555874c8fb70 0x555874c9e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555874c98750 0x555874c9eda0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 --2- 172.21.15.45:0/2828673089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555874c8f250 0x555874c9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 >> 172.21.15.45:0/2828673089 conn(0x555874c89980 msgr2=0x555874c8adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5bd7fa640 0 client.28870 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 shutdown_connections 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 1 -- 172.21.15.45:0/2828673089 wait complete. 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.506+0000 7fc5deb9a740 20 client.28870 trim_cache size 0 max 16384 2024-03-12T05:26:58.504 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.507+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 msgr2=0x7fc5c00a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.507+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 0x7fc5c00a4270 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00058c0 tx=0x7fc5d0009260 comp rx=0 tx=0).stop 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 msgr2=0x7fc5d4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 0x7fc5d4199ba0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc036040 tx=0x7fc5cc033fb0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 shutdown_connections 2024-03-12T05:26:58.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc5c00a1db0 0x7fc5c00a4270 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d4199720 0x7fc5d4199ba0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5d4106450 0x7fc5d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 --2- 172.21.15.45:0/878488257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d40ff890 0x7fc5d419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 >> 172.21.15.45:0/878488257 conn(0x7fc5d40765a0 msgr2=0x7fc5d41047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 shutdown_connections 2024-03-12T05:26:58.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.508+0000 7fc5dbfc7640 1 -- 172.21.15.45:0/878488257 wait complete. 2024-03-12T05:26:58.521 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:58.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:26:58.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.711+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/3939681473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c41009e0 msgr2=0x7fc9c410cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.711+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/3939681473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c41009e0 0x7fc9c410cea0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009be0 tx=0x7fc9bc02bf80 comp rx=0 tx=0).stop 2024-03-12T05:26:58.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.711+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/3939681473 shutdown_connections 2024-03-12T05:26:58.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.711+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/3939681473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c41009e0 0x7fc9c410cea0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.712+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/3939681473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c4100040 0x7fc9c41004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.712+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/3939681473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4106100 0x7fc9c41064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.712+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/3939681473 >> 172.21.15.45:0/3939681473 conn(0x7fc9c4077110 msgr2=0x7fc9c4077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.713+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/3939681473 shutdown_connections 2024-03-12T05:26:58.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.714+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/3939681473 wait complete. 2024-03-12T05:26:58.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.715+0000 7fc9ca6ae640 1 Processor -- start 2024-03-12T05:26:58.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.715+0000 7fc9ca6ae640 1 -- start start 2024-03-12T05:26:58.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c41009e0 0x7fc9c406c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c4106100 0x7fc9c41b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41b2610 con 0x7fc9c4106100 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41b2750 con 0x7fc9c41009e0 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9ca6ae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41b28e0 con 0x7fc9c4100040 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33052/0 (socket says 172.21.15.45:33052) 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 -- 172.21.15.45:0/517022697 learned_addr learned my addr 172.21.15.45:0/517022697 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c41009e0 0x7fc9c406c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c4106100 0x7fc9c41b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 -- 172.21.15.45:0/517022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c41009e0 msgr2=0x7fc9c406c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c41009e0 0x7fc9c406c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 -- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c4106100 msgr2=0x7fc9c41b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c4106100 0x7fc9c41b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.716+0000 7fc9c3fff640 1 -- 172.21.15.45:0/517022697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009560 con 0x7fc9c4100040 2024-03-12T05:26:58.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.717+0000 7fc9c3fff640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400e9c0 tx=0x7fc9b400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.717+0000 7fc9c17fa640 1 -- 172.21.15.45:0/517022697 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b400cde0 con 0x7fc9c4100040 2024-03-12T05:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.717+0000 7fc9c17fa640 1 -- 172.21.15.45:0/517022697 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9b40044d0 con 0x7fc9c4100040 2024-03-12T05:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.717+0000 7fc9c17fa640 1 -- 172.21.15.45:0/517022697 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4010430 con 0x7fc9c4100040 2024-03-12T05:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.717+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc009840 con 0x7fc9c4100040 2024-03-12T05:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.718+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c41b2e90 con 0x7fc9c4100040 2024-03-12T05:26:58.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 Processor -- start 2024-03-12T05:26:58.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 -- start start 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b9a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b9cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8b4c960 con 0x563ff8b91b70 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8b00310 con 0x563ff8b91250 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.721+0000 7fc9cd281740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8afd840 con 0x563ff8b9a750 2024-03-12T05:26:58.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.722+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55910/0 (socket says 172.21.15.45:55910) 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1607334605 learned_addr learned my addr 172.21.15.45:0/1607334605 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1607334605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 msgr2=0x563ff8b9cb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b9cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1607334605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 msgr2=0x563ff8b9a210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:58.717 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b9a210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.723+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1607334605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ff8abb060 con 0x563ff8b91250 2024-03-12T05:26:58.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.724+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009be0 tx=0x7fc9bc02c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b86632b3999518e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.718 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.724+0000 7fc9c17fa640 1 -- 172.21.15.45:0/517022697 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc9b4004000 con 0x7fc9c4100040 2024-03-12T05:26:58.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.724+0000 7fc9aa7fc640 1 -- 172.21.15.45:0/1607334605 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc003970 con 0x563ff8b91250 2024-03-12T05:26:58.719 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.725+0000 7fc9aa7fc640 1 -- 172.21.15.45:0/1607334605 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9bc004040 con 0x563ff8b91250 2024-03-12T05:26:58.720 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.725+0000 7fc9c17fa640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 0x7fc9ac0a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.725+0000 7fc9aa7fc640 1 -- 172.21.15.45:0/1607334605 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc034430 con 0x563ff8b91250 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.725+0000 7fc9c17fa640 1 -- 172.21.15.45:0/517022697 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc9b4014070 con 0x7fc9c4100040 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9c3fff640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 0x7fc9ac0a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 -- 172.21.15.45:0/1607334605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 msgr2=0x563ff8b91630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009be0 tx=0x7fc9bc02c9d0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 -- 172.21.15.45:0/1607334605 shutdown_connections 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b9a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b9cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1607334605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b91630 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 -- 172.21.15.45:0/1607334605 >> 172.21.15.45:0/1607334605 conn(0x563ff8b8b980 msgr2=0x563ff8b8bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.722 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.726+0000 7fc9cd281740 1 -- 172.21.15.45:0/1607334605 shutdown_connections 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.727+0000 7fc9cd281740 1 -- 172.21.15.45:0/1607334605 wait complete. 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.727+0000 7fc9c3fff640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 0x7fc9ac0a41a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x563ff8b90d20 tx=0x7fc9b4005c80 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.728+0000 7fc9cd281740 1 -- start start 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.728+0000 7fc9cd281740 1 Processor -- start 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.728+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.728+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b97c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55912/0 (socket says 172.21.15.45:55912) 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1234890765 learned_addr learned my addr 172.21.15.45:0/1234890765 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b97c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b981c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8af9020 con 0x563ff8b91b70 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff87cc430 con 0x563ff8b91250 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8788e00 con 0x563ff8b9a750 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.729+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1234890765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 msgr2=0x563ff8b981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1234890765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 msgr2=0x563ff8b97c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b97c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c410d7a0 con 0x563ff8b91250 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b97c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9cd281740 10 client.0 _ll_register_callbacks cb 0x563ff8aa0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ff87f87e0 con 0x563ff8b91250 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc0048b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=528d86f7c8fb46f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.730+0000 7fc9a37fe640 10 client.28919 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.731+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1234890765 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc034700 con 0x563ff8b91250 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.731+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1234890765 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc9bc034860 con 0x563ff8b91250 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.731+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1234890765 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9bc034a20 con 0x563ff8b91250 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.731+0000 7fc9cd281740 10 client.28919 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:58.729 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.731+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x563ff8af76a0 con 0x563ff8b91250 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1234890765 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc9bc03c050 con 0x563ff8b91250 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 10 client.28919 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 10 client.28919 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 20 client.28919 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8baa630 0x563ff8baca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 4 client.28919 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ff8835720 con 0x563ff8baa630 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.732+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8baa630 0x563ff8baca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.733+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8baa630 0x563ff8baca70 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a6db773509734caa server_cookie=b63e55a793dc410b in_seq=0 out_seq=0 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.733+0000 7fc9a37fe640 10 client.28919 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:58.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1234890765 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563ff8835720 con 0x563ff8baa630 2024-03-12T05:26:58.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9a37fe640 10 client.28919 handle_command_reply: tid=0 2024-03-12T05:26:58.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 client.28919 shutdown 2024-03-12T05:26:58.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 msgr2=0x563ff8b97740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc036040 tx=0x7fc9bc0048b0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 shutdown_connections 2024-03-12T05:26:58.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8baa630 0x563ff8baca70 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b97c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.732 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1234890765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b97740 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 >> 172.21.15.45:0/1234890765 conn(0x563ff8b8b980 msgr2=0x563ff8b8d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.734+0000 7fc9a37fe640 0 client.28919 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.735+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 shutdown_connections 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.735+0000 7fc9cd281740 1 -- 172.21.15.45:0/1234890765 wait complete. 2024-03-12T05:26:58.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.735+0000 7fc9cd281740 20 client.28919 trim_cache size 0 max 16384 2024-03-12T05:26:58.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.757+0000 7fc9cd281740 1 Processor -- start 2024-03-12T05:26:58.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.757+0000 7fc9cd281740 1 -- start start 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.757+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.757+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b96720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b97fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9cd281740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8a7e130 con 0x563ff8b91b70 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9cd281740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8835720 con 0x563ff8b91250 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9cd281740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8b8c980 con 0x563ff8b9a750 2024-03-12T05:26:58.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55920/0 (socket says 172.21.15.45:55920) 2024-03-12T05:26:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/382782194 learned_addr learned my addr 172.21.15.45:0/382782194 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b97fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b96720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/382782194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 msgr2=0x563ff8b97fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b97fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/382782194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 msgr2=0x563ff8b96720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b96720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.758+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/382782194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ff8a93c20 con 0x563ff8b91250 2024-03-12T05:26:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.759+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005cc0 tx=0x7fc9bc042070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a342744632d1c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.759+0000 7fc9a9ffb640 1 -- 172.21.15.45:0/382782194 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc02ca40 con 0x563ff8b91250 2024-03-12T05:26:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.760+0000 7fc9a9ffb640 1 -- 172.21.15.45:0/382782194 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc9bc066070 con 0x563ff8b91250 2024-03-12T05:26:58.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.760+0000 7fc9a9ffb640 1 -- 172.21.15.45:0/382782194 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc034970 con 0x563ff8b91250 2024-03-12T05:26:58.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 -- 172.21.15.45:0/382782194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 msgr2=0x563ff8b99e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc005cc0 tx=0x7fc9bc042070 comp rx=0 tx=0).stop 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 -- 172.21.15.45:0/382782194 shutdown_connections 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b91b70 0x563ff8b96720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b9a750 0x563ff8b97fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 --2- 172.21.15.45:0/382782194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91250 0x563ff8b99e70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 -- 172.21.15.45:0/382782194 >> 172.21.15.45:0/382782194 conn(0x563ff8b8b980 msgr2=0x563ff8b96010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 -- 172.21.15.45:0/382782194 shutdown_connections 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.761+0000 7fc9cd281740 1 -- 172.21.15.45:0/382782194 wait complete. 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.762+0000 7fc9cd281740 1 -- start start 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.762+0000 7fc9cd281740 1 Processor -- start 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 0x563ff8b9f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 0x563ff8b9fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 0x563ff8bffb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 0x563ff8b9f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9c8c4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 0x563ff8b9f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33086/0 (socket says 172.21.15.45:33086) 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 0x563ff8b9fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8ba03d0 con 0x563ff8b9a750 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 0x563ff8b9fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55928/0 (socket says 172.21.15.45:55928) 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8c00090 con 0x563ff8b91b70 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9c37fe640 1 -- 172.21.15.45:0/1013265456 learned_addr learned my addr 172.21.15.45:0/1013265456 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.763+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff8c00200 con 0x563ff8b91250 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9cd281740 10 client.0 _ll_register_callbacks cb 0x7fc9b4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 0x563ff8bffb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1013265456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 msgr2=0x563ff8b9f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 0x563ff8b9f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.763 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 msgr2=0x563ff8b9fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 0x563ff8b9fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.764+0000 7fc9c8c4b640 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ff8af76a0 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9c8c4b640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 0x563ff8bffb50 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc05f620 tx=0x7fc9bc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661e30c0f7ae8bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0048a0 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc9bc034700 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9bc004040 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9bc0348c0 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9cd281740 10 client.19473 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.765+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563ff8ab98c0 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.766+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x563ff87f87e0 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.766+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc9bc03c070 con 0x563ff8b9a750 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 10 client.19473 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 10 client.19473 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:26:58.764 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 20 client.19473 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:58.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8c014d0 0x563ff8c038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 4 client.19473 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:58.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fc9bc004040 con 0x563ff8c014d0 2024-03-12T05:26:58.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.767+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8c014d0 0x563ff8c038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.768+0000 7fc9c37fe640 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8c014d0 0x563ff8c038c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=38c72144cf96aa7c server_cookie=b3b29bd2d760c982 in_seq=0 out_seq=0 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.768+0000 7fc9a37fe640 10 client.19473 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.768+0000 7fc9a37fe640 1 -- 172.21.15.45:0/1013265456 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fc9bc004040 con 0x563ff8c014d0 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.768+0000 7fc9a37fe640 10 client.19473 handle_command_reply: tid=0 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.768+0000 7fc9cd281740 1 client.19473 shutdown 2024-03-12T05:26:58.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 msgr2=0x563ff8bffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 0x563ff8bffb50 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc05f620 tx=0x7fc9bc000f40 comp rx=0 tx=0).stop 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 shutdown_connections 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x563ff8c014d0 0x563ff8c038c0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x563ff8b9a750 0x563ff8bffb50 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff8b91b70 0x563ff8b9fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 --2- 172.21.15.45:0/1013265456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ff8b91250 0x563ff8b9f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9a37fe640 0 client.19473 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 >> 172.21.15.45:0/1013265456 conn(0x563ff8b8b980 msgr2=0x563ff8b8a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 shutdown_connections 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.769+0000 7fc9cd281740 1 -- 172.21.15.45:0/1013265456 wait complete. 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.770+0000 7fc9cd281740 20 client.19473 trim_cache size 0 max 16384 2024-03-12T05:26:58.767 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.770+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 msgr2=0x7fc9ac0a41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.770+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 0x7fc9ac0a41a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x563ff8b90d20 tx=0x7fc9b4005c80 comp rx=0 tx=0).stop 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.770+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 msgr2=0x7fc9c406bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.770+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400e9c0 tx=0x7fc9b400ee80 comp rx=0 tx=0).stop 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 shutdown_connections 2024-03-12T05:26:58.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc9ac0a1ce0 0x7fc9ac0a41a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9c4106100 0x7fc9c41b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9c41009e0 0x7fc9c406c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 --2- 172.21.15.45:0/517022697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9c4100040 0x7fc9c406bb30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 >> 172.21.15.45:0/517022697 conn(0x7fc9c4077110 msgr2=0x7fc9c4069640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 shutdown_connections 2024-03-12T05:26:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.771+0000 7fc9ca6ae640 1 -- 172.21.15.45:0/517022697 wait complete. 2024-03-12T05:26:58.787 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:58.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:26:58.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:58 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2565755897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.972+0000 7fd5b69a1640 1 -- 172.21.15.45:0/3449206184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b00685d0 msgr2=0x7fd5b00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.972+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/3449206184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b00685d0 0x7fd5b00689b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009cd0 tx=0x7fd5a802c0e0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.972+0000 7fd5b69a1640 1 -- 172.21.15.45:0/3449206184 shutdown_connections 2024-03-12T05:26:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/3449206184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b0106430 0x7fd5b010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/3449206184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0068f80 0x7fd5b0105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/3449206184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b00685d0 0x7fd5b00689b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 -- 172.21.15.45:0/3449206184 >> 172.21.15.45:0/3449206184 conn(0x7fd5b0077180 msgr2=0x7fd5b0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 -- 172.21.15.45:0/3449206184 shutdown_connections 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 -- 172.21.15.45:0/3449206184 wait complete. 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.973+0000 7fd5b69a1640 1 Processor -- start 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 -- start start 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 0x7fd5b019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b0199730 0x7fd5b0199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019a200 con 0x7fd5b00685d0 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019a370 con 0x7fd5b0199730 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5b69a1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019a530 con 0x7fd5b0106430 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33106/0 (socket says 172.21.15.45:33106) 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.974+0000 7fd5af7fe640 1 -- 172.21.15.45:0/2985806375 learned_addr learned my addr 172.21.15.45:0/2985806375 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 -- 172.21.15.45:0/2985806375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b0199730 msgr2=0x7fd5b0199bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b0199730 0x7fd5b0199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b0199730 0x7fd5b0199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 -- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 msgr2=0x7fd5b019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5affff640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 0x7fd5b019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 0x7fd5b019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 -- 172.21.15.45:0/2985806375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8009900 con 0x7fd5b0106430 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5affff640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 0x7fd5b019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.975+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400e9c0 tx=0x7fd5a400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.976+0000 7fd5ad7fa640 1 -- 172.21.15.45:0/2985806375 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4009700 con 0x7fd5b0106430 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.976+0000 7fd5ad7fa640 1 -- 172.21.15.45:0/2985806375 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5a400cd80 con 0x7fd5b0106430 2024-03-12T05:26:58.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.976+0000 7fd5ad7fa640 1 -- 172.21.15.45:0/2985806375 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4010430 con 0x7fd5b0106430 2024-03-12T05:26:58.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.976+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b01a64d0 con 0x7fd5b0106430 2024-03-12T05:26:58.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.976+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b01a6960 con 0x7fd5b0106430 2024-03-12T05:26:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.979+0000 7fd5ad7fa640 1 -- 172.21.15.45:0/2985806375 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd5a4004f50 con 0x7fd5b0106430 2024-03-12T05:26:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.979+0000 7fd5b9579740 1 Processor -- start 2024-03-12T05:26:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.979+0000 7fd5b9579740 1 -- start start 2024-03-12T05:26:58.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.979+0000 7fd5ad7fa640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 0x7fd5980a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6055e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e605ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6061350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e6008960 con 0x55a9e605ef60 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5fbc310 con 0x55a9e6056380 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b9579740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5fb9840 con 0x55a9e6055a60 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 0x7fd5980a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5ad7fa640 1 -- 172.21.15.45:0/2985806375 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd5a4014070 con 0x7fd5b0106430 2024-03-12T05:26:58.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6061350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6061350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38580/0 (socket says 172.21.15.45:38580) 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2509940208 learned_addr learned my addr 172.21.15.45:0/2509940208 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e605ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.980+0000 7fd5affff640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 -- 172.21.15.45:0/2509940208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 msgr2=0x55a9e605ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e605ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 -- 172.21.15.45:0/2509940208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 msgr2=0x55a9e6061350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6061350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 -- 172.21.15.45:0/2509940208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9e5f77060 con 0x55a9e6055a60 2024-03-12T05:26:58.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e605ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5affff640 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6055e40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009bf0 tx=0x7fd5a8032340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc63a07c8fef7ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 0x7fd5980a4160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd5a00147d0 tx=0x7fd5a0005c90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5967fc640 1 -- 172.21.15.45:0/2509940208 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8032e90 con 0x55a9e6055a60 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5967fc640 1 -- 172.21.15.45:0/2509940208 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5a8032430 con 0x55a9e6055a60 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.981+0000 7fd5967fc640 1 -- 172.21.15.45:0/2509940208 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8003970 con 0x55a9e6055a60 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 -- 172.21.15.45:0/2509940208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 msgr2=0x55a9e6055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6055e40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009bf0 tx=0x7fd5a8032340 comp rx=0 tx=0).stop 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 -- 172.21.15.45:0/2509940208 shutdown_connections 2024-03-12T05:26:58.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6061350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e605ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2509940208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6055e40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 -- 172.21.15.45:0/2509940208 >> 172.21.15.45:0/2509940208 conn(0x55a9e6050190 msgr2=0x55a9e6050570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 -- 172.21.15.45:0/2509940208 shutdown_connections 2024-03-12T05:26:58.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.982+0000 7fd5b9579740 1 -- 172.21.15.45:0/2509940208 wait complete. 2024-03-12T05:26:58.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.983+0000 7fd5b9579740 1 -- start start 2024-03-12T05:26:58.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.983+0000 7fd5b9579740 1 Processor -- start 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 0x55a9e605c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5fb5020 con 0x55a9e605ef60 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5c88430 con 0x55a9e6055a60 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5c44e00 con 0x55a9e6056380 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b9579740 10 client.0 _ll_register_callbacks cb 0x55a9e5f5ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38586/0 (socket says 172.21.15.45:38586) 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.984+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2147602819 learned_addr learned my addr 172.21.15.45:0/2147602819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 0x55a9e605c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5affff640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 msgr2=0x55a9e605c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 0x55a9e605c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2147602819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 msgr2=0x55a9e605bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2147602819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b010e810 con 0x55a9e605ef60 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 0x55a9e605c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0014fe0 tx=0x7fd5a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120833dee8ea6f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.985+0000 7fd58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0005220 con 0x55a9e605ef60 2024-03-12T05:26:58.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9e5cb47e0 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd5a0005380 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a0005580 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd5b9579740 10 client.19482 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a9e5fb36a0 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a9e5cb47e0 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.986+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5a0012a00 con 0x55a9e605ef60 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 10 client.19482 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 10 client.19482 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 20 client.19482 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e606f260 0x55a9e6071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 4 client.19482 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9e5cf1720 con 0x55a9e606f260 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5affff640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e606f260 0x55a9e6071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd5affff640 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e606f260 0x55a9e6071680 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=974c9292eb25fe3b server_cookie=ff2bfd5f4562f3fc in_seq=0 out_seq=0 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.987+0000 7fd58f7fe640 10 client.19482 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2147602819 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a9e5cf1720 con 0x55a9e606f260 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd58f7fe640 10 client.19482 handle_command_reply: tid=0 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 client.19482 shutdown 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 msgr2=0x55a9e605c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0014fe0 tx=0x7fd5a0005f90 comp rx=0 tx=0).stop 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 shutdown_connections 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e606f260 0x55a9e6071680 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd58f7fe640 0 client.19482 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e605c960 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6056380 0x55a9e605c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2147602819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.988+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 >> 172.21.15.45:0/2147602819 conn(0x55a9e6050190 msgr2=0x55a9e6051650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.989+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 shutdown_connections 2024-03-12T05:26:58.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.989+0000 7fd5b9579740 1 -- 172.21.15.45:0/2147602819 wait complete. 2024-03-12T05:26:58.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:58.989+0000 7fd5b9579740 20 client.19482 trim_cache size 0 max 16384 2024-03-12T05:26:59.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 Processor -- start 2024-03-12T05:26:59.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 -- start start 2024-03-12T05:26:59.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e605ef60 0x55a9e6066320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5f3a130 con 0x55a9e6056380 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e5cf1750 con 0x55a9e6055a60 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5b9579740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e6055700 con 0x55a9e605ef60 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5af7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38588/0 (socket says 172.21.15.45:38588) 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.011+0000 7fd5af7fe640 1 -- 172.21.15.45:0/1044205302 learned_addr learned my addr 172.21.15.45:0/1044205302 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e605ef60 0x55a9e6066320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 -- 172.21.15.45:0/1044205302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e605ef60 msgr2=0x55a9e6066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e605ef60 0x55a9e6066320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 -- 172.21.15.45:0/1044205302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 msgr2=0x55a9e605e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 -- 172.21.15.45:0/1044205302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9e5f4fc20 con 0x55a9e6056380 2024-03-12T05:26:59.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.012+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd5a409a560 tx=0x7fd5a40ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee767b248f470cca server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.013+0000 7fd595ffb640 1 -- 172.21.15.45:0/1044205302 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a409a860 con 0x55a9e6056380 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.013+0000 7fd595ffb640 1 -- 172.21.15.45:0/1044205302 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5a409e9b0 con 0x55a9e6056380 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.013+0000 7fd595ffb640 1 -- 172.21.15.45:0/1044205302 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a40a2b70 con 0x55a9e6056380 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- 172.21.15.45:0/1044205302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 msgr2=0x55a9e6065de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd5a409a560 tx=0x7fd5a40ba040 comp rx=0 tx=0).stop 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- 172.21.15.45:0/1044205302 shutdown_connections 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e6056380 0x55a9e6065de0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e605ef60 0x55a9e6066320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 --2- 172.21.15.45:0/1044205302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6055a60 0x55a9e605e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- 172.21.15.45:0/1044205302 >> 172.21.15.45:0/1044205302 conn(0x55a9e6050190 msgr2=0x55a9e6042a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- 172.21.15.45:0/1044205302 shutdown_connections 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- 172.21.15.45:0/1044205302 wait complete. 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.014+0000 7fd5b9579740 1 -- start start 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 Processor -- start 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6066240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e60645e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e6065300 con 0x55a9e605ef60 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e6065470 con 0x55a9e6056380 2024-03-12T05:26:59.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38592/0 (socket says 172.21.15.45:38592) 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2933966731 learned_addr learned my addr 172.21.15.45:0/2933966731 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.015+0000 7fd5b9579740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9e60c5800 con 0x55a9e6055a60 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5affff640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6066240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5af7fe640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e60645e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b9579740 10 client.0 _ll_register_callbacks cb 0x7fd5a8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 msgr2=0x55a9e6066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6066240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2933966731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 msgr2=0x55a9e60645e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.014 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e60645e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 -- 172.21.15.45:0/2933966731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b01a64d0 con 0x55a9e605ef60 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5affff640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6066240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd5b4f3e640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0012e30 tx=0x7fd5a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e24f7cba97d137af server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.016+0000 7fd58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0009be0 con 0x55a9e605ef60 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd5a00144f0 con 0x55a9e605ef60 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5a8032430 con 0x55a9e605ef60 2024-03-12T05:26:59.015 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a0005850 con 0x55a9e605ef60 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd5b9579740 10 client.19488 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.017+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a9e5f758c0 con 0x55a9e605ef60 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fd5a0046020 con 0x55a9e605ef60 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd5a0038030 con 0x55a9e605ef60 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 10 client.19488 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 10 client.19488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 20 client.19488 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:59.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e60c6fc0 0x55a9e60c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 4 client.19488 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a9e5d40890 con 0x55a9e60c6fc0 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.018+0000 7fd5affff640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e60c6fc0 0x55a9e60c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.019+0000 7fd58f7fe640 10 client.19488 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.019+0000 7fd5affff640 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e60c6fc0 0x55a9e60c93e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82966498b4f56f13 server_cookie=e2c15b8c7dc116d8 in_seq=0 out_seq=0 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.019+0000 7fd58f7fe640 1 -- 172.21.15.45:0/2933966731 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a9e5d40890 con 0x55a9e60c6fc0 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.019+0000 7fd58f7fe640 10 client.19488 handle_command_reply: tid=0 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.019+0000 7fd5b9579740 1 client.19488 shutdown 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 msgr2=0x55a9e6064c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0012e30 tx=0x7fd5a0004270 comp rx=0 tx=0).stop 2024-03-12T05:26:59.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 shutdown_connections 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55a9e60c6fc0 0x55a9e60c93e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a9e605ef60 0x55a9e6064c60 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e6056380 0x55a9e60645e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd58f7fe640 0 client.19488 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 --2- 172.21.15.45:0/2933966731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9e6055a60 0x55a9e6066240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 >> 172.21.15.45:0/2933966731 conn(0x55a9e6050190 msgr2=0x55a9e60462f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 shutdown_connections 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 1 -- 172.21.15.45:0/2933966731 wait complete. 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.020+0000 7fd5b9579740 20 client.19488 trim_cache size 0 max 16384 2024-03-12T05:26:59.019 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.021+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 msgr2=0x7fd5980a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.021+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 0x7fd5980a4160 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd5a00147d0 tx=0x7fd5a0005c90 comp rx=0 tx=0).stop 2024-03-12T05:26:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.021+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 msgr2=0x7fd5b019fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.021+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400e9c0 tx=0x7fd5a400ee80 comp rx=0 tx=0).stop 2024-03-12T05:26:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 shutdown_connections 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fd5980a1ca0 0x7fd5980a4160 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b0199730 0x7fd5b0199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5b0106430 0x7fd5b019fbb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 --2- 172.21.15.45:0/2985806375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5b00685d0 0x7fd5b019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 >> 172.21.15.45:0/2985806375 conn(0x7fd5b0077180 msgr2=0x7fd5b010c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 shutdown_connections 2024-03-12T05:26:59.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.022+0000 7fd5b69a1640 1 -- 172.21.15.45:0/2985806375 wait complete. 2024-03-12T05:26:59.034 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:59.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:26:59.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:58 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2565755897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:59.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:58 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2565755897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 -- 172.21.15.45:0/2259176247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546010d9d0 msgr2=0x7f546010de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 --2- 172.21.15.45:0/2259176247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546010d9d0 0x7f546010de50 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5458009900 tx=0x7f545802bc40 comp rx=0 tx=0).stop 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 -- 172.21.15.45:0/2259176247 shutdown_connections 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 --2- 172.21.15.45:0/2259176247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546010d9d0 0x7f546010de50 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 --2- 172.21.15.45:0/2259176247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546011a960 0x7f546011cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 --2- 172.21.15.45:0/2259176247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f546010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 -- 172.21.15.45:0/2259176247 >> 172.21.15.45:0/2259176247 conn(0x7f546006c260 msgr2=0x7f546006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 -- 172.21.15.45:0/2259176247 shutdown_connections 2024-03-12T05:26:59.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.256+0000 7f546524e640 1 -- 172.21.15.45:0/2259176247 wait complete. 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.257+0000 7f546524e640 1 Processor -- start 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.257+0000 7f546524e640 1 -- start start 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546011a960 0x7f54601aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54601ab440 con 0x7f546011a960 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54601ab5b0 con 0x7f546010d9d0 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f546524e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54601ab770 con 0x7f546010f9a0 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545e59c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33170/0 (socket says 172.21.15.45:33170) 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545f59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546011a960 0x7f54601aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:55992/0 (socket says 172.21.15.45:55992) 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.258+0000 7f545e59c640 1 -- 172.21.15.45:0/3743031803 learned_addr learned my addr 172.21.15.45:0/3743031803 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 -- 172.21.15.45:0/3743031803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 msgr2=0x7f54601a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 -- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546011a960 msgr2=0x7f54601aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546011a960 0x7f54601aae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 -- 172.21.15.45:0/3743031803 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009560 con 0x7f546010f9a0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545ed9d640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f545e59c640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5458000c00 tx=0x7f54580040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f5447fff640 1 -- 172.21.15.45:0/3743031803 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580035f0 con 0x7f546010f9a0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f5447fff640 1 -- 172.21.15.45:0/3743031803 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5458003c00 con 0x7f546010f9a0 2024-03-12T05:26:59.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f5447fff640 1 -- 172.21.15.45:0/3743031803 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54580054f0 con 0x7f546010f9a0 2024-03-12T05:26:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.259+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54601ab9f0 con 0x7f546010f9a0 2024-03-12T05:26:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.260+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601b32d0 con 0x7f546010f9a0 2024-03-12T05:26:59.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.263+0000 7f5447fff640 1 -- 172.21.15.45:0/3743031803 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5458003750 con 0x7f546010f9a0 2024-03-12T05:26:59.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.263+0000 7f5467e29740 1 Processor -- start 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.263+0000 7f5467e29740 1 -- start start 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa676630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa681b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa631960 con 0x55e3fa67f750 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa5e5310 con 0x55e3fa676250 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5467e29740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa5e2840 con 0x55e3fa676b70 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5447fff640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 0x7f54480a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa681b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33186/0 (socket says 172.21.15.45:33186) 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 -- 172.21.15.45:0/2917394480 learned_addr learned my addr 172.21.15.45:0/2917394480 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545f59e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 0x7f54480a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545f59e640 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa676630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 -- 172.21.15.45:0/2917394480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 msgr2=0x55e3fa676630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa676630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f545ed9d640 1 -- 172.21.15.45:0/2917394480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 msgr2=0x55e3fa681b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.264+0000 7f5447fff640 1 -- 172.21.15.45:0/3743031803 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54580f0120 con 0x7f546010f9a0 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa681b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f545ed9d640 1 -- 172.21.15.45:0/2917394480 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3fa5a0060 con 0x55e3fa676b70 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f545e59c640 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa681b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.260 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f545f59e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 0x7f54480a4060 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5454006e10 tx=0x7f54540098a0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5450010750 tx=0x7f5450010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1297b6dd02968a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f54457fa640 1 -- 172.21.15.45:0/2917394480 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545000ead0 con 0x55e3fa676b70 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f54457fa640 1 -- 172.21.15.45:0/2917394480 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f545000ec30 con 0x55e3fa676b70 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.265+0000 7f54457fa640 1 -- 172.21.15.45:0/2917394480 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54500113e0 con 0x55e3fa676b70 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.266+0000 7f5467e29740 1 -- 172.21.15.45:0/2917394480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 msgr2=0x55e3fa67f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.266+0000 7f5467e29740 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5450010750 tx=0x7f5450010c10 comp rx=0 tx=0).stop 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.266+0000 7f5467e29740 1 -- 172.21.15.45:0/2917394480 shutdown_connections 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.266+0000 7f5467e29740 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa681b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676b70 0x55e3fa67f210 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 --2- 172.21.15.45:0/2917394480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa676630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 -- 172.21.15.45:0/2917394480 >> 172.21.15.45:0/2917394480 conn(0x55e3fa670980 msgr2=0x55e3fa670d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 -- 172.21.15.45:0/2917394480 shutdown_connections 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 -- 172.21.15.45:0/2917394480 wait complete. 2024-03-12T05:26:59.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.267+0000 7f5467e29740 1 -- start start 2024-03-12T05:26:59.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 Processor -- start 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676250 0x55e3fa67c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa67d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa5de020 con 0x55e3fa67f750 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa2b1430 con 0x55e3fa676b70 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa26de00 con 0x55e3fa676250 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f5467e29740 10 client.0 _ll_register_callbacks cb 0x55e3fa585cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:59.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f545e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f545e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56008/0 (socket says 172.21.15.45:56008) 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.268+0000 7f545e59c640 1 -- 172.21.15.45:0/3690993699 learned_addr learned my addr 172.21.15.45:0/3690993699 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 -- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676250 msgr2=0x55e3fa67c770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676250 0x55e3fa67c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 -- 172.21.15.45:0/3690993699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 msgr2=0x55e3fa67d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa67d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 -- 172.21.15.45:0/3690993699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450010430 con 0x55e3fa676b70 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f545e59c640 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f54580b9450 tx=0x7f54580b26c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33ca5b9b85a12b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.269+0000 7f5442ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545808c070 con 0x55e3fa676b70 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3fa2dd7e0 con 0x55e3fa676b70 2024-03-12T05:26:59.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f54580bcff0 con 0x55e3fa676b70 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54580c1140 con 0x55e3fa676b70 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5467e29740 10 client.28949 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55e3fa5dc6a0 con 0x55e3fa676b70 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.270+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f54580c5790 con 0x55e3fa676b70 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f545803c070 con 0x55e3fa676b70 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 10 client.28949 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 10 client.28949 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 20 client.28949 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa68f860 0x55e3fa691cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 4 client.28949 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:59.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3fa2dd7e0 con 0x55e3fa68f860 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.271+0000 7f545ed9d640 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa68f860 0x55e3fa691cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.272+0000 7f5442ff5640 10 client.28949 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.273+0000 7f545ed9d640 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa68f860 0x55e3fa691cf0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dae4f49f6f5fc5a0 server_cookie=a63fa2669e821cd9 in_seq=0 out_seq=0 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.273+0000 7f5442ff5640 1 -- 172.21.15.45:0/3690993699 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3fa2dd7e0 con 0x55e3fa68f860 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.273+0000 7f5442ff5640 10 client.28949 handle_command_reply: tid=0 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.273+0000 7f5467e29740 1 client.28949 shutdown 2024-03-12T05:26:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 msgr2=0x55e3fa67ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f54580b9450 tx=0x7f54580b26c0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 shutdown_connections 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa67f750 0x55e3fa67d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa68f860 0x55e3fa691cf0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa67ccb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 --2- 172.21.15.45:0/3690993699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa676250 0x55e3fa67c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 >> 172.21.15.45:0/3690993699 conn(0x55e3fa670980 msgr2=0x55e3fa6726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5442ff5640 0 client.28949 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.274+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 shutdown_connections 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.275+0000 7f5467e29740 1 -- 172.21.15.45:0/3690993699 wait complete. 2024-03-12T05:26:59.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.275+0000 7f5467e29740 20 client.28949 trim_cache size 0 max 16384 2024-03-12T05:26:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.295+0000 7f5467e29740 1 Processor -- start 2024-03-12T05:26:59.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.295+0000 7f5467e29740 1 -- start start 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.295+0000 7f5467e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f5467e29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676b70 0x55e3fa677a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f5467e29740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa677f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f5467e29740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa563130 con 0x55e3fa676b70 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f5467e29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa6704b0 con 0x55e3fa676250 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f5467e29740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa2dd7e0 con 0x55e3fa67f750 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56014/0 (socket says 172.21.15.45:56014) 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 -- 172.21.15.45:0/279841185 learned_addr learned my addr 172.21.15.45:0/279841185 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545e59c640 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676b70 0x55e3fa677a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545f59e640 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa677f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 -- 172.21.15.45:0/279841185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 msgr2=0x55e3fa677f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa677f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 -- 172.21.15.45:0/279841185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676b70 msgr2=0x55e3fa677a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676b70 0x55e3fa677a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.296+0000 7f545ed9d640 1 -- 172.21.15.45:0/279841185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3fa578c20 con 0x55e3fa676250 2024-03-12T05:26:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.297+0000 7f545ed9d640 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5450010720 tx=0x7f5450009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64fc03e4a6d8bd61 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.297+0000 7f5444ff9640 1 -- 172.21.15.45:0/279841185 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545001f040 con 0x55e3fa676250 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.297+0000 7f5444ff9640 1 -- 172.21.15.45:0/279841185 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5450020040 con 0x55e3fa676250 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.297+0000 7f5444ff9640 1 -- 172.21.15.45:0/279841185 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450012870 con 0x55e3fa676250 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 -- 172.21.15.45:0/279841185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 msgr2=0x55e3fa67eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5450010720 tx=0x7f5450009f60 comp rx=0 tx=0).stop 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 -- 172.21.15.45:0/279841185 shutdown_connections 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676b70 0x55e3fa677a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa677f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 --2- 172.21.15.45:0/279841185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676250 0x55e3fa67eea0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.298+0000 7f5467e29740 1 -- 172.21.15.45:0/279841185 >> 172.21.15.45:0/279841185 conn(0x55e3fa670980 msgr2=0x55e3fa696660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.299+0000 7f5467e29740 1 -- 172.21.15.45:0/279841185 shutdown_connections 2024-03-12T05:26:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.299+0000 7f5467e29740 1 -- 172.21.15.45:0/279841185 wait complete. 2024-03-12T05:26:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.300+0000 7f5467e29740 1 -- start start 2024-03-12T05:26:59.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.300+0000 7f5467e29740 1 Processor -- start 2024-03-12T05:26:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.301+0000 7f5467e29740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 0x55e3fa686aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.301+0000 7f5467e29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.301+0000 7f5467e29740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa685470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56024/0 (socket says 172.21.15.45:56024) 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545e59c640 1 -- 172.21.15.45:0/2575502230 learned_addr learned my addr 172.21.15.45:0/2575502230 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa685b40 con 0x55e3fa676250 2024-03-12T05:26:59.296 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa685cb0 con 0x55e3fa676b70 2024-03-12T05:26:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3fa6e5e20 con 0x55e3fa67f750 2024-03-12T05:26:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa685470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 0x55e3fa686aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 -- 172.21.15.45:0/2575502230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 msgr2=0x55e3fa684e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 -- 172.21.15.45:0/2575502230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 msgr2=0x55e3fa686aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 0x55e3fa686aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.302+0000 7f545f59e640 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3fa5a0060 con 0x55e3fa67f750 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f545e59c640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 0x55e3fa686aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f545f59e640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa685470 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5454009b80 tx=0x7f54540090e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beaf3d7bcb210aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.298 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f5467e29740 10 client.0 _ll_register_callbacks cb 0x7f5450009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f5442ff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f5442ff5640 1 -- 172.21.15.45:0/2575502230 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454018920 con 0x55e3fa67f750 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f5442ff5640 1 -- 172.21.15.45:0/2575502230 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f5454018a80 con 0x55e3fa67f750 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.303+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f545000ec30 con 0x55e3fa67f750 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5442ff5640 1 -- 172.21.15.45:0/2575502230 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5454018c40 con 0x55e3fa67f750 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 10 client.28918 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:59.299 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=33}) v3 -- 0x55e3fa59e8c0 con 0x55e3fa67f750 2024-03-12T05:26:59.300 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5442ff5640 1 -- 172.21.15.45:0/2575502230 <== mon.1 v2:172.21.15.130:3300/0 4 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5454020050 con 0x55e3fa67f750 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 10 client.28918 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 10 client.28918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 20 client.28918 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa6e7350 0x55e3fa6e97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 4 client.28918 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:59.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.304+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e3fa31a720 con 0x55e3fa6e7350 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.305+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa6e7350 0x55e3fa6e97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.305+0000 7f545ed9d640 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa6e7350 0x55e3fa6e97d0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a60039161952b53 server_cookie=f4d0038935f42f80 in_seq=0 out_seq=0 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.305+0000 7f5442ff5640 10 client.28918 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5442ff5640 1 -- 172.21.15.45:0/2575502230 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e3fa31a720 con 0x55e3fa6e7350 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5442ff5640 10 client.28918 handle_command_reply: tid=0 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 client.28918 shutdown 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 msgr2=0x55e3fa685470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa685470 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5454009b80 tx=0x7f54540090e0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 shutdown_connections 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55e3fa6e7350 0x55e3fa6e97d0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3fa67f750 0x55e3fa685470 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e3fa676b70 0x55e3fa684e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 --2- 172.21.15.45:0/2575502230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55e3fa676250 0x55e3fa686aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 >> 172.21.15.45:0/2575502230 conn(0x55e3fa670980 msgr2=0x55e3fa677a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5442ff5640 0 client.28918 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:26:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 shutdown_connections 2024-03-12T05:26:59.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 1 -- 172.21.15.45:0/2575502230 wait complete. 2024-03-12T05:26:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.306+0000 7f5467e29740 20 client.28918 trim_cache size 0 max 16384 2024-03-12T05:26:59.305 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.308+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 msgr2=0x7f54480a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.308+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 0x7f54480a4060 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5454006e10 tx=0x7f54540098a0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 msgr2=0x7f54601a6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5458000c00 tx=0x7f54580040c0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 shutdown_connections 2024-03-12T05:26:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f54480a1ba0 0x7f54480a4060 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f546011a960 0x7f54601aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f546010f9a0 0x7f54601a6b60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 --2- 172.21.15.45:0/3743031803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546010d9d0 0x7f54601a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.309+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 >> 172.21.15.45:0/3743031803 conn(0x7f546006c260 msgr2=0x7f546011c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.310+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 shutdown_connections 2024-03-12T05:26:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.310+0000 7f546524e640 1 -- 172.21.15.45:0/3743031803 wait complete. 2024-03-12T05:26:59.324 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:26:59.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:26:59.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 -- 172.21.15.45:0/1473931749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8041015b0 msgr2=0x7ff80410cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 --2- 172.21.15.45:0/1473931749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8041015b0 0x7ff80410cf10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009c10 tx=0x7ff7fc02bee0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 -- 172.21.15.45:0/1473931749 shutdown_connections 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 --2- 172.21.15.45:0/1473931749 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 0x7ff804101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 --2- 172.21.15.45:0/1473931749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8041015b0 0x7ff80410cf10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 --2- 172.21.15.45:0/1473931749 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff804105080 0x7ff804105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 -- 172.21.15.45:0/1473931749 >> 172.21.15.45:0/1473931749 conn(0x7ff8040761d0 msgr2=0x7ff8040fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 -- 172.21.15.45:0/1473931749 shutdown_connections 2024-03-12T05:26:59.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.501+0000 7ff80c120640 1 -- 172.21.15.45:0/1473931749 wait complete. 2024-03-12T05:26:59.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.502+0000 7ff80c120640 1 Processor -- start 2024-03-12T05:26:59.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.502+0000 7ff80c120640 1 -- start start 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 0x7ff80419d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8041015b0 0x7ff80419d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804195e40 con 0x7ff804100bf0 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804195fb0 con 0x7ff804105080 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80c120640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804196170 con 0x7ff8041015b0 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80a6bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80a6bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56052/0 (socket says 172.21.15.45:56052) 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff80a6bd640 1 -- 172.21.15.45:0/3881354633 learned_addr learned my addr 172.21.15.45:0/3881354633 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.503+0000 7ff809ebc640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 0x7ff80419d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff8096bb640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8041015b0 0x7ff80419d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 -- 172.21.15.45:0/3881354633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8041015b0 msgr2=0x7ff80419d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8041015b0 0x7ff80419d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 -- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 msgr2=0x7ff80419d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 0x7ff80419d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 -- 172.21.15.45:0/3881354633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc009840 con 0x7ff804105080 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.504+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0045e0 tx=0x7ff7fc004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.505+0000 7ff7faffd640 1 -- 172.21.15.45:0/3881354633 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc02ca80 con 0x7ff804105080 2024-03-12T05:26:59.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.505+0000 7ff7faffd640 1 -- 172.21.15.45:0/3881354633 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc004830 con 0x7ff804105080 2024-03-12T05:26:59.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.505+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8041963f0 con 0x7ff804105080 2024-03-12T05:26:59.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.505+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8041b30d0 con 0x7ff804105080 2024-03-12T05:26:59.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.506+0000 7ff7faffd640 1 -- 172.21.15.45:0/3881354633 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc033810 con 0x7ff804105080 2024-03-12T05:26:59.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.509+0000 7ff7faffd640 1 -- 172.21.15.45:0/3881354633 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff7fc033970 con 0x7ff804105080 2024-03-12T05:26:59.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.510+0000 7ff7faffd640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 0x7ff7f00a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff809ebc640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 0x7ff7f00a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff7faffd640 1 -- 172.21.15.45:0/3881354633 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff7fc0f0430 con 0x7ff804105080 2024-03-12T05:26:59.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 Processor -- start 2024-03-12T05:26:59.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 -- start start 2024-03-12T05:26:59.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923858d960 con 0x5592385d2250 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x559238541310 con 0x5592385db750 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff80edcb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923853e840 con 0x5592385d2b70 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.511+0000 7ff809ebc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff809ebc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38662/0 (socket says 172.21.15.45:38662) 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56056/0 (socket says 172.21.15.45:56056) 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff809ebc640 1 -- 172.21.15.45:0/1779956631 learned_addr learned my addr 172.21.15.45:0/1779956631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff8096bb640 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff809ebc640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 0x7ff7f00a4230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4005fd0 tx=0x7ff7f4009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.507 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 -- 172.21.15.45:0/1779956631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 msgr2=0x5592385db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385db210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 -- 172.21.15.45:0/1779956631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 msgr2=0x5592385d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 -- 172.21.15.45:0/1779956631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592384fc060 con 0x5592385db750 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0edc00 tx=0x7ff7fc0bcf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=137c238ea8148854 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff7effff640 1 -- 172.21.15.45:0/1779956631 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0d2020 con 0x5592385db750 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff7effff640 1 -- 172.21.15.45:0/1779956631 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc0bd0e0 con 0x5592385db750 2024-03-12T05:26:59.508 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.512+0000 7ff7effff640 1 -- 172.21.15.45:0/1779956631 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0edd70 con 0x5592385db750 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 -- 172.21.15.45:0/1779956631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 msgr2=0x5592385ddb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0edc00 tx=0x7ff7fc0bcf70 comp rx=0 tx=0).stop 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 -- 172.21.15.45:0/1779956631 shutdown_connections 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385ddb40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385db210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 --2- 172.21.15.45:0/1779956631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 -- 172.21.15.45:0/1779956631 >> 172.21.15.45:0/1779956631 conn(0x5592385cc980 msgr2=0x5592385ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 -- 172.21.15.45:0/1779956631 shutdown_connections 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.514+0000 7ff80edcb740 1 -- 172.21.15.45:0/1779956631 wait complete. 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 -- start start 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 Processor -- start 2024-03-12T05:26:59.510 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385d90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923853a020 con 0x5592385d2250 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55923820d430 con 0x5592385db750 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592381c9e00 con 0x5592385d2b70 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff809ebc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.515+0000 7ff80edcb740 10 client.0 _ll_register_callbacks cb 0x5592384e1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33222/0 (socket says 172.21.15.45:33222) 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 -- 172.21.15.45:0/844229303 learned_addr learned my addr 172.21.15.45:0/844229303 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.511 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385d90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 -- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 msgr2=0x5592385d90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385d90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.513 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 -- 172.21.15.45:0/844229303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 msgr2=0x5592385d8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 -- 172.21.15.45:0/844229303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8041b30d0 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff8096bb640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff800009d80 tx=0x7ff8000117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c70d79df6f48c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.516+0000 7ff7ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800012860 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592382397e0 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff8000129c0 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff800009e70 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff80edcb740 10 client.28927 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.517+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592385386a0 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff800026020 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff80000c070 con 0x5592385d2b70 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff80edcb740 10 client.28927 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:59.514 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff80edcb740 10 client.28927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff80edcb740 20 client.28927 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.518+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592385eb990 0x5592385eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.519+0000 7ff809ebc640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592385eb990 0x5592385eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.519+0000 7ff80edcb740 4 client.28927 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.519+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559238276720 con 0x5592385eb990 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.519+0000 7ff809ebc640 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592385eb990 0x5592385eddb0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6870cd6d69bc1d87 server_cookie=7704a16bc5f2af8c in_seq=0 out_seq=0 2024-03-12T05:26:59.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.519+0000 7ff7ed7fa640 10 client.28927 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:59.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.520+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/844229303 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559238276720 con 0x5592385eb990 2024-03-12T05:26:59.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.520+0000 7ff7ed7fa640 10 client.28927 handle_command_reply: tid=0 2024-03-12T05:26:59.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.520+0000 7ff80edcb740 1 client.28927 shutdown 2024-03-12T05:26:59.516 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.520+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 msgr2=0x5592385d8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.520+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff800009d80 tx=0x7ff8000117b0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 shutdown_connections 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x5592385eb990 0x5592385eddb0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385d90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385d8bb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 --2- 172.21.15.45:0/844229303 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385d8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff7ed7fa640 0 client.28927 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 >> 172.21.15.45:0/844229303 conn(0x5592385cc980 msgr2=0x5592385ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 shutdown_connections 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 1 -- 172.21.15.45:0/844229303 wait complete. 2024-03-12T05:26:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.521+0000 7ff80edcb740 20 client.28927 trim_cache size 0 max 16384 2024-03-12T05:26:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.547+0000 7ff80edcb740 1 Processor -- start 2024-03-12T05:26:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.547+0000 7ff80edcb740 1 -- start start 2024-03-12T05:26:59.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.547+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff809ebc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff809ebc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38680/0 (socket says 172.21.15.45:38680) 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56070/0 (socket says 172.21.15.45:56070) 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff809ebc640 1 -- 172.21.15.45:0/2074837288 learned_addr learned my addr 172.21.15.45:0/2074837288 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385db750 0x5592385d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592384bf130 con 0x5592385d2250 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592385c85c0 con 0x5592385d2b70 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592385db5a0 con 0x5592385db750 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.548+0000 7ff80a6bd640 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385db750 0x5592385d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 -- 172.21.15.45:0/2074837288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385db750 msgr2=0x5592385d5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385db750 0x5592385d5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 -- 172.21.15.45:0/2074837288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 msgr2=0x5592385dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 -- 172.21.15.45:0/2074837288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592384d4c20 con 0x5592385d2b70 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff809ebc640 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff80001ae10 tx=0x7ff8000099f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b6a1e3a3a33668 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff7ef7fe640 1 -- 172.21.15.45:0/2074837288 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800005230 con 0x5592385d2b70 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff7ef7fe640 1 -- 172.21.15.45:0/2074837288 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff800026040 con 0x5592385d2b70 2024-03-12T05:26:59.545 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.549+0000 7ff7ef7fe640 1 -- 172.21.15.45:0/2074837288 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8000055f0 con 0x5592385d2b70 2024-03-12T05:26:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 msgr2=0x5592385d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff80001ae10 tx=0x7ff8000099f0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 shutdown_connections 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385db750 0x5592385d5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385d2b70 0x5592385d58d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2074837288 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385dada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 >> 172.21.15.45:0/2074837288 conn(0x5592385cc980 msgr2=0x5592385d3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 shutdown_connections 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.550+0000 7ff80edcb740 1 -- 172.21.15.45:0/2074837288 wait complete. 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.551+0000 7ff80edcb740 1 -- start start 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.551+0000 7ff80edcb740 1 Processor -- start 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385e2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592385e2d00 con 0x5592385d2250 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592385e12a0 con 0x5592385db750 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592385e1460 con 0x5592385d2b70 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff80edcb740 10 client.0 _ll_register_callbacks cb 0x7ff7fc009280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff8096bb640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33232/0 (socket says 172.21.15.45:33232) 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.552+0000 7ff8096bb640 1 -- 172.21.15.45:0/2030350726 learned_addr learned my addr 172.21.15.45:0/2030350726 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 -- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 msgr2=0x5592385e0d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385e0d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 -- 172.21.15.45:0/2030350726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 msgr2=0x5592385e2430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385e2430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 -- 172.21.15.45:0/2030350726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f4017070 con 0x5592385d2b70 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff8096bb640 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x5592385d9b20 tx=0x7ff80001ac10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c878691f94298 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:26:59.549 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff7ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-03-12T05:26:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.553+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800011cb0 con 0x5592385d2b70 2024-03-12T05:26:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7ff800027080 con 0x5592385d2b70 2024-03-12T05:26:59.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff7fc0bd0e0 con 0x5592385d2b70 2024-03-12T05:26:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff800009e70 con 0x5592385d2b70 2024-03-12T05:26:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff80edcb740 10 client.28933 fetch_fsmap learned FSMap version 33 2024-03-12T05:26:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5592385386a0 con 0x5592385d2b70 2024-03-12T05:26:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc0bd0e0 con 0x5592385d2b70 2024-03-12T05:26:59.552 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.554+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff80000c070 con 0x5592385d2b70 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 10 client.28933 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 10 client.28933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 20 client.28933 populate_metadata read hostname 'smithi045' 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559238642f20 0x5592386453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 4 client.28933 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.555+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592382c5890 con 0x559238642f20 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.556+0000 7ff809ebc640 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559238642f20 0x5592386453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.556+0000 7ff809ebc640 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559238642f20 0x5592386453e0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=433b711c5a1151a7 server_cookie=2f4ed71384d9d9f6 in_seq=0 out_seq=0 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.556+0000 7ff7ed7fa640 10 client.28933 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff7ed7fa640 1 -- 172.21.15.45:0/2030350726 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592382c5890 con 0x559238642f20 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff7ed7fa640 10 client.28933 handle_command_reply: tid=0 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 client.28933 shutdown 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 msgr2=0x5592385e0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.553 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x5592385d9b20 tx=0x7ff80001ac10 comp rx=0 tx=0).stop 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 shutdown_connections 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x559238642f20 0x5592386453e0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5592385db750 0x5592385e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.557+0000 7ff7ed7fa640 0 client.28933 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.558+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5592385d2b70 0x5592385e0820 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.558+0000 7ff80edcb740 1 --2- 172.21.15.45:0/2030350726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5592385d2250 0x5592385e2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.558+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 >> 172.21.15.45:0/2030350726 conn(0x5592385cc980 msgr2=0x5592385d6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.558+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 shutdown_connections 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.558+0000 7ff80edcb740 1 -- 172.21.15.45:0/2030350726 wait complete. 2024-03-12T05:26:59.554 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80edcb740 20 client.28933 trim_cache size 0 max 16384 2024-03-12T05:26:59.555 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:26:59.555 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 msgr2=0x7ff7f00a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 0x7ff7f00a4230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4005fd0 tx=0x7ff7f4009260 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 msgr2=0x7ff8041956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0045e0 tx=0x7ff7fc004610 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 shutdown_connections 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7ff7f00a1d70 0x7ff7f00a4230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff804105080 0x7ff8041956b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8041015b0 0x7ff80419d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.559+0000 7ff80c120640 1 --2- 172.21.15.45:0/3881354633 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff804100bf0 0x7ff80419d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.560+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 >> 172.21.15.45:0/3881354633 conn(0x7ff8040761d0 msgr2=0x7ff8040ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.560+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 shutdown_connections 2024-03-12T05:26:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:26:59.560+0000 7ff80c120640 1 -- 172.21.15.45:0/3881354633 wait complete. 2024-03-12T05:26:59.809 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:26:59 smithi045 ceph-mon[32174]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-03-12T05:26:59.949 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 895.0 s, 1284.7 tps, lat 77.629 ms stddev 39.194 2024-03-12T05:27:00.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:26:59 smithi130 ceph-mon[35398]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-03-12T05:27:00.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:26:59 smithi116 ceph-mon[35486]: pgmap v506: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 17 MiB/s wr, 1.32k op/s 2024-03-12T05:27:00.571 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:27:00.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:27:00.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 -- 172.21.15.45:0/3813734935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 msgr2=0x7f3718103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 --2- 172.21.15.45:0/3813734935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f3718103960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3714009930 tx=0x7f371402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 -- 172.21.15.45:0/3813734935 shutdown_connections 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 --2- 172.21.15.45:0/3813734935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3718108650 0x7f3718110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 --2- 172.21.15.45:0/3813734935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718103f30 0x7f3718107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 --2- 172.21.15.45:0/3813734935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f3718103960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 -- 172.21.15.45:0/3813734935 >> 172.21.15.45:0/3813734935 conn(0x7f37180feda0 msgr2=0x7f37181011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 -- 172.21.15.45:0/3813734935 shutdown_connections 2024-03-12T05:27:00.740 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.744+0000 7f371ed0c640 1 -- 172.21.15.45:0/3813734935 wait complete. 2024-03-12T05:27:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.745+0000 7f371ed0c640 1 Processor -- start 2024-03-12T05:27:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 -- start start 2024-03-12T05:27:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f37180729e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718108650 0x7f371806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180731c0 con 0x7f371806bff0 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3718070500 con 0x7f3718103580 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371ed0c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180706c0 con 0x7f3718108650 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38694/0 (socket says 172.21.15.45:38694) 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371caa8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f37180729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371d2a9640 1 -- 172.21.15.45:0/4280040728 learned_addr learned my addr 172.21.15.45:0/4280040728 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f370ffff640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718108650 0x7f371806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371caa8640 1 -- 172.21.15.45:0/4280040728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718108650 msgr2=0x7f371806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.746+0000 7f371caa8640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718108650 0x7f371806bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371caa8640 1 -- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 msgr2=0x7f371806c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371caa8640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371caa8640 1 -- 172.21.15.45:0/4280040728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3714009560 con 0x7f3718103580 2024-03-12T05:27:00.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371d2a9640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371caa8640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f37180729e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3714009230 tx=0x7f3714005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f370dffb640 1 -- 172.21.15.45:0/4280040728 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714004350 con 0x7f3718103580 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f370dffb640 1 -- 172.21.15.45:0/4280040728 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f371402de90 con 0x7f3718103580 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3718070940 con 0x7f3718103580 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f370dffb640 1 -- 172.21.15.45:0/4280040728 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3714005220 con 0x7f3718103580 2024-03-12T05:27:00.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.747+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3718070e20 con 0x7f3718103580 2024-03-12T05:27:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f370dffb640 1 -- 172.21.15.45:0/4280040728 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3714003c30 con 0x7f3718103580 2024-03-12T05:27:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 Processor -- start 2024-03-12T05:27:00.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 -- start start 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 0x55c7f737b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 0x55c7f73842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f73848b0 0x55c7f7386d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f72e9f90 con 0x55c7f737bd30 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f72e74c0 con 0x55c7f73848b0 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.751+0000 7f37219bf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f72e2ca0 con 0x55c7f737b380 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371caa8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 0x55c7f73842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371caa8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 0x55c7f73842e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38698/0 (socket says 172.21.15.45:38698) 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371caa8640 1 -- 172.21.15.45:0/766835770 learned_addr learned my addr 172.21.15.45:0/766835770 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 0x55c7f737b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 -- 172.21.15.45:0/766835770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f73848b0 msgr2=0x55c7f7386d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f73848b0 0x55c7f7386d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 -- 172.21.15.45:0/766835770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 msgr2=0x55c7f73842e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 0x55c7f73842e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 -- 172.21.15.45:0/766835770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7f72a50b0 con 0x55c7f737b380 2024-03-12T05:27:00.747 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f370dffb640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 0x7f37000a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f370dffb640 1 -- 172.21.15.45:0/4280040728 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f37140ef550 con 0x7f3718103580 2024-03-12T05:27:00.748 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371d2a9640 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 0x55c7f737b760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3710011760 tx=0x7f3710011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b5702ac0d9c8de server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.752+0000 7f371caa8640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 0x7f37000a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.753+0000 7f36feffd640 1 -- 172.21.15.45:0/766835770 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710004d20 con 0x55c7f737b380 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.753+0000 7f36feffd640 1 -- 172.21.15.45:0/766835770 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3710004e80 con 0x55c7f737b380 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.753+0000 7f36feffd640 1 -- 172.21.15.45:0/766835770 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710005990 con 0x55c7f737b380 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- 172.21.15.45:0/766835770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 msgr2=0x55c7f737b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 0x55c7f737b760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3710011760 tx=0x7f3710011c20 comp rx=0 tx=0).stop 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- 172.21.15.45:0/766835770 shutdown_connections 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737bd30 0x55c7f73842e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f73848b0 0x55c7f7386d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 --2- 172.21.15.45:0/766835770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737b380 0x55c7f737b760 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- 172.21.15.45:0/766835770 >> 172.21.15.45:0/766835770 conn(0x55c7f7375840 msgr2=0x55c7f7375c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f371caa8640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 0x7f37000a41b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3714034040 tx=0x7f3714009260 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- 172.21.15.45:0/766835770 shutdown_connections 2024-03-12T05:27:00.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- 172.21.15.45:0/766835770 wait complete. 2024-03-12T05:27:00.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 -- start start 2024-03-12T05:27:00.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.754+0000 7f37219bf740 1 Processor -- start 2024-03-12T05:27:00.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f37219bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737b380 0x55c7f7381840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f37219bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 0x55c7f7381d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f37219bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737b380 0x55c7f7381840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38702/0 (socket says 172.21.15.45:38702) 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f371d2a9640 1 -- 172.21.15.45:0/3281128386 learned_addr learned my addr 172.21.15.45:0/3281128386 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f7287f40 con 0x55c7f73848b0 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f6fb64b0 con 0x55c7f737b380 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.755+0000 7f370ffff640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 0x55c7f7381d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f7268670 con 0x55c7f737bd30 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f37219bf740 10 client.0 _ll_register_callbacks cb 0x55c7f72d5f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f371d2a9640 1 -- 172.21.15.45:0/3281128386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 msgr2=0x55c7f7381d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f371d2a9640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 0x55c7f7381d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f371d2a9640 1 -- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737b380 msgr2=0x55c7f7381840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f371d2a9640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737b380 0x55c7f7381840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.756+0000 7f371d2a9640 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3710011440 con 0x55c7f73848b0 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.757+0000 7f370ffff640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 0x55c7f7381d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.757+0000 7f36f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.757+0000 7f371d2a9640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f37100055f0 tx=0x7f3710000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85faa91dff0191a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.757+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710028650 con 0x55c7f73848b0 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3710004cc0 con 0x55c7f73848b0 2024-03-12T05:27:00.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7f72bcf40 con 0x55c7f73848b0 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37100287b0 con 0x55c7f73848b0 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 10 client.19530 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c7f72e1320 con 0x55c7f73848b0 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c7f72bcf40 con 0x55c7f73848b0 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f371002f060 con 0x55c7f73848b0 2024-03-12T05:27:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 10 client.19530 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 10 client.19530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 20 client.19530 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.758+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f7394db0 0x55c7f73971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.759+0000 7f37219bf740 4 client.19530 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.759+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7f72d9310 con 0x55c7f7394db0 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.759+0000 7f370ffff640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f7394db0 0x55c7f73971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.759+0000 7f370ffff640 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f7394db0 0x55c7f73971d0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=699e0e7869198cea server_cookie=17e9a22c7786008b in_seq=0 out_seq=0 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.760+0000 7f36f7fff640 10 client.19530 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.760+0000 7f36f7fff640 1 -- 172.21.15.45:0/3281128386 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c7f72d9310 con 0x55c7f7394db0 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.760+0000 7f36f7fff640 10 client.19530 handle_command_reply: tid=0 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.760+0000 7f37219bf740 1 client.19530 shutdown 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 msgr2=0x55c7f73822c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f37100055f0 tx=0x7f3710000f40 comp rx=0 tx=0).stop 2024-03-12T05:27:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 shutdown_connections 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f7394db0 0x55c7f73971d0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f73848b0 0x55c7f73822c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f737bd30 0x55c7f7381d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 --2- 172.21.15.45:0/3281128386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737b380 0x55c7f7381840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 >> 172.21.15.45:0/3281128386 conn(0x55c7f7375840 msgr2=0x55c7f73777a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f36f7fff640 0 client.19530 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 shutdown_connections 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 1 -- 172.21.15.45:0/3281128386 wait complete. 2024-03-12T05:27:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.761+0000 7f37219bf740 20 client.19530 trim_cache size 0 max 16384 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.788+0000 7f37219bf740 1 Processor -- start 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.788+0000 7f37219bf740 1 -- start start 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.788+0000 7f37219bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f37219bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73775b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f37219bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f37219bf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f737acd0 con 0x55c7f737b380 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f37219bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f737a410 con 0x55c7f737bd30 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f37219bf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f7391900 con 0x55c7f73848b0 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38714/0 (socket says 172.21.15.45:38714) 2024-03-12T05:27:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 -- 172.21.15.45:0/4150483572 learned_addr learned my addr 172.21.15.45:0/4150483572 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f370ffff640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73775b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 -- 172.21.15.45:0/4150483572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 msgr2=0x55c7f738b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 -- 172.21.15.45:0/4150483572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 msgr2=0x55c7f73775b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73775b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f371d2a9640 1 -- 172.21.15.45:0/4150483572 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7f727dc00 con 0x55c7f737b380 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.789+0000 7f370ffff640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73775b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.790+0000 7f371d2a9640 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3710004a10 tx=0x7f3710005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e591a34162a73167 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.790+0000 7f36fe7fc640 1 -- 172.21.15.45:0/4150483572 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37100056f0 con 0x55c7f737b380 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.790+0000 7f36fe7fc640 1 -- 172.21.15.45:0/4150483572 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3710005430 con 0x55c7f737b380 2024-03-12T05:27:00.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.790+0000 7f36fe7fc640 1 -- 172.21.15.45:0/4150483572 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710019720 con 0x55c7f737b380 2024-03-12T05:27:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.791+0000 7f37219bf740 1 -- 172.21.15.45:0/4150483572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 msgr2=0x55c7f73820e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.791+0000 7f37219bf740 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3710004a10 tx=0x7f3710005330 comp rx=0 tx=0).stop 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 -- 172.21.15.45:0/4150483572 shutdown_connections 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73775b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 --2- 172.21.15.45:0/4150483572 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73820e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 -- 172.21.15.45:0/4150483572 >> 172.21.15.45:0/4150483572 conn(0x55c7f7375840 msgr2=0x55c7f7376ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 -- 172.21.15.45:0/4150483572 shutdown_connections 2024-03-12T05:27:00.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.792+0000 7f37219bf740 1 -- 172.21.15.45:0/4150483572 wait complete. 2024-03-12T05:27:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.793+0000 7f37219bf740 1 -- start start 2024-03-12T05:27:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.793+0000 7f37219bf740 1 Processor -- start 2024-03-12T05:27:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73e5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f738b470 con 0x55c7f737b380 2024-03-12T05:27:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f738b5e0 con 0x55c7f737bd30 2024-03-12T05:27:00.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7f738b7a0 con 0x55c7f73848b0 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f37219bf740 10 client.0 _ll_register_callbacks cb 0x7f3710011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f370ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f370ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56126/0 (socket says 172.21.15.45:56126) 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f370ffff640 1 -- 172.21.15.45:0/756284819 learned_addr learned my addr 172.21.15.45:0/756284819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.794+0000 7f371d2a9640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f371d2a9640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f370ffff640 1 -- 172.21.15.45:0/756284819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 msgr2=0x55c7f738aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f370ffff640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f370ffff640 1 -- 172.21.15.45:0/756284819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 msgr2=0x55c7f73e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f370ffff640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f370ffff640 1 -- 172.21.15.45:0/756284819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7f72e1320 con 0x55c7f737bd30 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.795+0000 7f371d2a9640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f370ffff640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f371806d220 tx=0x7f3708004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd249307a3dbc3b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f36f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3708009af0 con 0x55c7f737bd30 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f370801ad70 con 0x55c7f737bd30 2024-03-12T05:27:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7f7308900 con 0x55c7f737bd30 2024-03-12T05:27:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f370801a900 con 0x55c7f737bd30 2024-03-12T05:27:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f37219bf740 10 client.28982 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:00.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.796+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55c7f72a3910 con 0x55c7f737bd30 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55c7f7308900 con 0x55c7f737bd30 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f37080237e0 con 0x55c7f737bd30 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 10 client.28982 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 10 client.28982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 20 client.28982 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f73ed900 0x55c7f73efd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 4 client.28982 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.797+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c7f6fe2860 con 0x55c7f73ed900 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.798+0000 7f371d2a9640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f73ed900 0x55c7f73efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.798+0000 7f371d2a9640 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f73ed900 0x55c7f73efd20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2604151213f6f77f server_cookie=5e3d3e141479d6b1 in_seq=0 out_seq=0 2024-03-12T05:27:00.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.798+0000 7f36f7fff640 10 client.28982 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.800+0000 7f36f7fff640 1 -- 172.21.15.45:0/756284819 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c7f6fe2860 con 0x55c7f73ed900 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.800+0000 7f36f7fff640 10 client.28982 handle_command_reply: tid=0 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.800+0000 7f37219bf740 1 client.28982 shutdown 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.800+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 msgr2=0x55c7f73e5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f371806d220 tx=0x7f3708004410 comp rx=0 tx=0).stop 2024-03-12T05:27:00.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 shutdown_connections 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55c7f73ed900 0x55c7f73efd20 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c7f73848b0 0x55c7f738aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7f737bd30 0x55c7f73e5ec0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f36f7fff640 0 client.28982 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 --2- 172.21.15.45:0/756284819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55c7f737b380 0x55c7f73e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 >> 172.21.15.45:0/756284819 conn(0x55c7f7375840 msgr2=0x55c7f7377730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 shutdown_connections 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 1 -- 172.21.15.45:0/756284819 wait complete. 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.801+0000 7f37219bf740 20 client.28982 trim_cache size 0 max 16384 2024-03-12T05:27:00.796 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:00.797 INFO:teuthology.orchestra.run.smithi045.stdout: "return_code": 0, 2024-03-12T05:27:00.797 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_tag": "482f662d-c62f-4fc1-b342-2e04b4051971", 2024-03-12T05:27:00.797 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": "asynchronous" 2024-03-12T05:27:00.797 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 msgr2=0x7f37000a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 0x7f37000a41b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3714034040 tx=0x7f3714009260 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 msgr2=0x7f37180729e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f37180729e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3714009230 tx=0x7f3714005dc0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 shutdown_connections 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f37000a1cf0 0x7f37000a41b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f371806bff0 0x7f371806c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3718108650 0x7f371806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 --2- 172.21.15.45:0/4280040728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3718103580 0x7f37180729e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.802+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 >> 172.21.15.45:0/4280040728 conn(0x7f37180feda0 msgr2=0x7f37181001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.803+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 shutdown_connections 2024-03-12T05:27:00.798 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.803+0000 7f371ed0c640 1 -- 172.21.15.45:0/4280040728 wait complete. 2024-03-12T05:27:00.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:27:00.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:27:00.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 -- 172.21.15.45:0/2544806991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c1119e0 msgr2=0x7f061c111dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 --2- 172.21.15.45:0/2544806991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c1119e0 0x7f061c111dc0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0604009900 tx=0x7f060402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 -- 172.21.15.45:0/2544806991 shutdown_connections 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 --2- 172.21.15.45:0/2544806991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c10ba20 0x7f061c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 --2- 172.21.15.45:0/2544806991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c1188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 --2- 172.21.15.45:0/2544806991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c1119e0 0x7f061c111dc0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 -- 172.21.15.45:0/2544806991 >> 172.21.15.45:0/2544806991 conn(0x7f061c06bfb0 msgr2=0x7f061c10a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 -- 172.21.15.45:0/2544806991 shutdown_connections 2024-03-12T05:27:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.992+0000 7f0621dde640 1 -- 172.21.15.45:0/2544806991 wait complete. 2024-03-12T05:27:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.993+0000 7f0621dde640 1 Processor -- start 2024-03-12T05:27:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.993+0000 7f0621dde640 1 -- start start 2024-03-12T05:27:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.993+0000 7f0621dde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c10ba20 0x7f061c118670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.993+0000 7f0621dde640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c115760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f0621dde640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 0x7f061c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f0621dde640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c116430 con 0x7f061c1119e0 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f0621dde640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c1165a0 con 0x7f061c10ba20 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f0621dde640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c1b28c0 con 0x7f061c10c3e0 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 0x7f061c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f061bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c115760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f061bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c115760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33322/0 (socket says 172.21.15.45:33322) 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f061bfff640 1 -- 172.21.15.45:0/2200415244 learned_addr learned my addr 172.21.15.45:0/2200415244 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f0620ddc640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c10ba20 0x7f061c118670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 -- 172.21.15.45:0/2200415244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 msgr2=0x7f061c115760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c115760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 -- 172.21.15.45:0/2200415244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c10ba20 msgr2=0x7f061c118670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c10ba20 0x7f061c118670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.994+0000 7f06215dd640 1 -- 172.21.15.45:0/2200415244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0604009560 con 0x7f061c1119e0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.995+0000 7f06215dd640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 0x7f061c115ca0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f061400ea80 tx=0x7f061400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.995+0000 7f0619ffb640 1 -- 172.21.15.45:0/2200415244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400cea0 con 0x7f061c1119e0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.995+0000 7f0619ffb640 1 -- 172.21.15.45:0/2200415244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0614005020 con 0x7f061c1119e0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.995+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c1b2b20 con 0x7f061c1119e0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.995+0000 7f0619ffb640 1 -- 172.21.15.45:0/2200415244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614010690 con 0x7f061c1119e0 2024-03-12T05:27:00.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.996+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c1b3000 con 0x7f061c1119e0 2024-03-12T05:27:00.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:00.998+0000 7f0619ffb640 1 -- 172.21.15.45:0/2200415244 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0614005190 con 0x7f061c1119e0 2024-03-12T05:27:00.994 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.000+0000 7f0625c13740 1 Processor -- start 2024-03-12T05:27:00.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.000+0000 7f0625c13740 1 -- start start 2024-03-12T05:27:00.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.000+0000 7f0625c13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.000+0000 7f0625c13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.995 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0625c13740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc21d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0625c13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fb85280 con 0x55d36fc16db0 2024-03-12T05:27:00.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0625c13740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fb827b0 con 0x55d36fc1f990 2024-03-12T05:27:00.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0625c13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fb7df90 con 0x55d36fc16490 2024-03-12T05:27:00.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.996 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33326/0 (socket says 172.21.15.45:33326) 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 -- 172.21.15.45:0/2993953857 learned_addr learned my addr 172.21.15.45:0/2993953857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0619ffb640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 0x7f06080a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f0620ddc640 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc21d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 -- 172.21.15.45:0/2993953857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 msgr2=0x55d36fc21d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc21d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 -- 172.21.15.45:0/2993953857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 msgr2=0x55d36fc1f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f061bfff640 1 -- 172.21.15.45:0/2993953857 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d36fb3c570 con 0x55d36fc16490 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.001+0000 7f06215dd640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 0x7f06080a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.002+0000 7f061bfff640 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f060c009f80 tx=0x7f060c009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d31d06d6e35e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.002+0000 7f0619ffb640 1 -- 172.21.15.45:0/2200415244 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0614014070 con 0x7f061c1119e0 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.002+0000 7f0612ffd640 1 -- 172.21.15.45:0/2993953857 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c004050 con 0x55d36fc16490 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.002+0000 7f0612ffd640 1 -- 172.21.15.45:0/2993953857 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f060c0035f0 con 0x55d36fc16490 2024-03-12T05:27:00.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.002+0000 7f0612ffd640 1 -- 172.21.15.45:0/2993953857 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c00f890 con 0x55d36fc16490 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 -- 172.21.15.45:0/2993953857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 msgr2=0x55d36fc16870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f060c009f80 tx=0x7f060c009910 comp rx=0 tx=0).stop 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f06215dd640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 0x7f06080a40e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f061409a730 tx=0x7f0614087040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 -- 172.21.15.45:0/2993953857 shutdown_connections 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc21d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 --2- 172.21.15.45:0/2993953857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc16870 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 -- 172.21.15.45:0/2993953857 >> 172.21.15.45:0/2993953857 conn(0x55d36fc10bc0 msgr2=0x55d36fc10fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:00.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.004+0000 7f0625c13740 1 -- 172.21.15.45:0/2993953857 shutdown_connections 2024-03-12T05:27:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.005+0000 7f0625c13740 1 -- 172.21.15.45:0/2993953857 wait complete. 2024-03-12T05:27:01.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.005+0000 7f0625c13740 1 -- start start 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.005+0000 7f0625c13740 1 Processor -- start 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc1f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 0x55d36fc1c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 0x55d36fc1cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36f851430 con 0x55d36fc1f990 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36f80de00 con 0x55d36fc16db0 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fb02fd0 con 0x55d36fc16490 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0625c13740 10 client.0 _ll_register_callbacks cb 0x55d36fb58fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0620ddc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 0x55d36fc1c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0620ddc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 0x55d36fc1c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56150/0 (socket says 172.21.15.45:56150) 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f0620ddc640 1 -- 172.21.15.45:0/3100727792 learned_addr learned my addr 172.21.15.45:0/3100727792 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:01.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.006+0000 7f061bfff640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 0x55d36fc1cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc1f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 -- 172.21.15.45:0/3100727792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 msgr2=0x55d36fc1f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc1f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 -- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 msgr2=0x55d36fc1c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 0x55d36fc1c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 -- 172.21.15.45:0/3100727792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c009df0 con 0x55d36fc1f990 2024-03-12T05:27:01.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.007+0000 7f061bfff640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 0x55d36fc1cf10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f060c009fd0 tx=0x7f060c003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8973be3b75cfc83c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f05fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c004430 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f060c005050 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d36f87d7e0 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f060c005210 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f0625c13740 10 client.19548 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.008+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d36fb7c610 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55d36f87d7e0 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f060c00f630 con 0x55d36fc1f990 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 10 client.19548 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 10 client.19548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 20 client.19548 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc2faf0 0x55d36fc31f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 4 client.19548 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:01.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d36f8ba720 con 0x55d36fc2faf0 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.009+0000 7f0620ddc640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc2faf0 0x55d36fc31f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.010+0000 7f05fbfff640 10 client.19548 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.010+0000 7f0620ddc640 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc2faf0 0x55d36fc31f10 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64eca30543d45e36 server_cookie=e0eb686dfa7ff0ff in_seq=0 out_seq=0 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f05fbfff640 1 -- 172.21.15.45:0/3100727792 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d36f8ba720 con 0x55d36fc2faf0 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f05fbfff640 10 client.19548 handle_command_reply: tid=0 2024-03-12T05:27:01.006 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 client.19548 shutdown 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 msgr2=0x55d36fc1cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 0x55d36fc1cf10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f060c009fd0 tx=0x7f060c003f20 comp rx=0 tx=0).stop 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 shutdown_connections 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc2faf0 0x55d36fc31f10 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc1f990 0x55d36fc1cf10 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16db0 0x55d36fc1c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f05fbfff640 0 client.19548 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 --2- 172.21.15.45:0/3100727792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16490 0x55d36fc1f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.011+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 >> 172.21.15.45:0/3100727792 conn(0x55d36fc10bc0 msgr2=0x55d36fc12910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.012+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 shutdown_connections 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.012+0000 7f0625c13740 1 -- 172.21.15.45:0/3100727792 wait complete. 2024-03-12T05:27:01.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.012+0000 7f0625c13740 20 client.19548 trim_cache size 0 max 16384 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.038+0000 7f0625c13740 1 Processor -- start 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.038+0000 7f0625c13740 1 -- start start 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc1f990 0x55d36fc1b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc0b010 con 0x55d36fc16db0 2024-03-12T05:27:01.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc15ed0 con 0x55d36fc16490 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0625c13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc2b780 con 0x55d36fc1f990 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f0620ddc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc1f990 0x55d36fc1b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.45:56158/0 (socket says 172.21.15.45:56158) 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 -- 172.21.15.45:0/2199059159 learned_addr learned my addr 172.21.15.45:0/2199059159 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 -- 172.21.15.45:0/2199059159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc1f990 msgr2=0x55d36fc1b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc1f990 0x55d36fc1b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 -- 172.21.15.45:0/2199059159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 msgr2=0x55d36fc1f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 -- 172.21.15.45:0/2199059159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d36fb18ac0 con 0x55d36fc16490 2024-03-12T05:27:01.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.039+0000 7f061bfff640 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f060c003ef0 tx=0x7f060c009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72ceeffa94a04beb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:01.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.040+0000 7f06127fc640 1 -- 172.21.15.45:0/2199059159 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c003b90 con 0x55d36fc16490 2024-03-12T05:27:01.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.040+0000 7f06127fc640 1 -- 172.21.15.45:0/2199059159 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f060c013170 con 0x55d36fc16490 2024-03-12T05:27:01.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.040+0000 7f06127fc640 1 -- 172.21.15.45:0/2199059159 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c01f690 con 0x55d36fc16490 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.040+0000 7f0625c13740 1 -- 172.21.15.45:0/2199059159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 msgr2=0x55d36fc1ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.040+0000 7f0625c13740 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f060c003ef0 tx=0x7f060c009910 comp rx=0 tx=0).stop 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 -- 172.21.15.45:0/2199059159 shutdown_connections 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16db0 0x55d36fc1f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc1f990 0x55d36fc1b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 --2- 172.21.15.45:0/2199059159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc16490 0x55d36fc1ef00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 -- 172.21.15.45:0/2199059159 >> 172.21.15.45:0/2199059159 conn(0x55d36fc10bc0 msgr2=0x55d36fc17a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 -- 172.21.15.45:0/2199059159 shutdown_connections 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 -- 172.21.15.45:0/2199059159 wait complete. 2024-03-12T05:27:01.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 -- start start 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.041+0000 7f0625c13740 1 Processor -- start 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 0x55d36fc230e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc23ed0 con 0x55d36fc16490 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc85fb0 con 0x55d36fc1f990 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36fc860f0 con 0x55d36fc16db0 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0625c13740 10 client.0 _ll_register_callbacks cb 0x7f060c009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0620ddc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 0x55d36fc230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f061bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f061bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38780/0 (socket says 172.21.15.45:38780) 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f061bfff640 1 -- 172.21.15.45:0/3926677865 learned_addr learned my addr 172.21.15.45:0/3926677865 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f0620ddc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 0x55d36fc230e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:33350/0 (socket says 172.21.15.45:33350) 2024-03-12T05:27:01.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.042+0000 7f061bfff640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 -- 172.21.15.45:0/3926677865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 msgr2=0x55d36fc230e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 0x55d36fc230e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 -- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 msgr2=0x55d36fc237d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc237d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 -- 172.21.15.45:0/3926677865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d36fb7c610 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f061bfff640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f060c00fb30 tx=0x7f060c003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665ae21417f74704 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f05fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060c043070 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f060c000e40 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.043+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f060c0035f0 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.044+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f060c003c20 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.044+0000 7f0625c13740 10 client.19554 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.044+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d36fb3e6a0 con 0x55d36fc16490 2024-03-12T05:27:01.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.044+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f060c0035f0 con 0x55d36fc16490 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f060c01f850 con 0x55d36fc16490 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 10 client.19554 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 10 client.19554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 20 client.19554 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc873c0 0x55d36fc897b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 4 client.19554 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d36f909890 con 0x55d36fc873c0 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.045+0000 7f0620ddc640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc873c0 0x55d36fc897b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.046+0000 7f0620ddc640 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc873c0 0x55d36fc897b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=658a477e1c8fd4d2 server_cookie=c5e11c1564743888 in_seq=0 out_seq=0 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.046+0000 7f05fbfff640 10 client.19554 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.046+0000 7f05fbfff640 1 -- 172.21.15.45:0/3926677865 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d36f909890 con 0x55d36fc873c0 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.046+0000 7f05fbfff640 10 client.19554 handle_command_reply: tid=0 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.046+0000 7f0625c13740 1 client.19554 shutdown 2024-03-12T05:27:01.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 msgr2=0x55d36fc26d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f060c00fb30 tx=0x7f060c003f20 comp rx=0 tx=0).stop 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 shutdown_connections 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55d36fc873c0 0x55d36fc897b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d36fc1f990 0x55d36fc237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f05fbfff640 0 client.19554 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36fc16db0 0x55d36fc230e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 --2- 172.21.15.45:0/3926677865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55d36fc16490 0x55d36fc26d60 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 >> 172.21.15.45:0/3926677865 conn(0x55d36fc10bc0 msgr2=0x55d36fc1c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.047+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 shutdown_connections 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.048+0000 7f0625c13740 1 -- 172.21.15.45:0/3926677865 wait complete. 2024-03-12T05:27:01.046 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.048+0000 7f0625c13740 20 client.19554 trim_cache size 0 max 16384 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": { 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "482f662d-c62f-4fc1-b342-2e04b4051971": { 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "path": "/", 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "tag": "482f662d-c62f-4fc1-b342-2e04b4051971", 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: "options": "recursive,force" 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:01.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:01.048 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 msgr2=0x7f06080a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 0x7f06080a40e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f061409a730 tx=0x7f0614087040 comp rx=0 tx=0).stop 2024-03-12T05:27:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 msgr2=0x7f061c115ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:01.048 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 0x7f061c115ca0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f061400ea80 tx=0x7f061400ef40 comp rx=0 tx=0).stop 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 shutdown_connections 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f06080a1c20 0x7f06080a40e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f061c1119e0 0x7f061c115ca0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f061c10c3e0 0x7f061c115760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 --2- 172.21.15.45:0/2200415244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f061c10ba20 0x7f061c118670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 >> 172.21.15.45:0/2200415244 conn(0x7f061c06bfb0 msgr2=0x7f061c10e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 shutdown_connections 2024-03-12T05:27:01.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:01.049+0000 7f0621dde640 1 -- 172.21.15.45:0/2200415244 wait complete. 2024-03-12T05:27:01.061 INFO:tasks.cephfs.filesystem:scrub status for tag:482f662d-c62f-4fc1-b342-2e04b4051971 - {'path': '/', 'tag': '482f662d-c62f-4fc1-b342-2e04b4051971', 'options': 'recursive,force'} 2024-03-12T05:27:02.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 19 MiB/s wr, 1.46k op/s 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub queued for path: / 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: active paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:01 smithi045 ceph-mon[32174]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 19 MiB/s wr, 1.46k op/s 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub queued for path: / 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:27:02.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: active paths [/] 2024-03-12T05:27:02.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:01 smithi130 ceph-mon[35398]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: pgmap v507: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 19 MiB/s wr, 1.46k op/s 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub queued for path: / 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: active paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:02.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:01 smithi116 ceph-mon[35486]: scrub summary: idle+waiting paths [/] 2024-03-12T05:27:04.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:03 smithi045 ceph-mon[32174]: pgmap v508: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-03-12T05:27:04.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:03 smithi045 ceph-mon[32174]: osdmap e109: 12 total, 12 up, 12 in 2024-03-12T05:27:04.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:03 smithi130 ceph-mon[35398]: pgmap v508: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-03-12T05:27:04.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:03 smithi130 ceph-mon[35398]: osdmap e109: 12 total, 12 up, 12 in 2024-03-12T05:27:04.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:03 smithi116 ceph-mon[35486]: pgmap v508: 97 pgs: 97 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-03-12T05:27:04.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:03 smithi116 ceph-mon[35486]: osdmap e109: 12 total, 12 up, 12 in 2024-03-12T05:27:04.948 INFO:teuthology.orchestra.run.smithi045.stderr:progress: 900.0 s, 1030.3 tps, lat 96.931 ms stddev 88.425 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:transaction type: 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:scaling factor: 32 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:query mode: simple 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:number of clients: 100 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:number of threads: 4 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:duration: 900 s 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:number of transactions actually processed: 1215874 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:latency average = 73.992 ms 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:latency stddev = 94.499 ms 2024-03-12T05:27:05.099 INFO:teuthology.orchestra.run.smithi045.stdout:tps = 1350.744704 (including connections establishing) 2024-03-12T05:27:05.100 INFO:teuthology.orchestra.run.smithi045.stdout:tps = 1350.761257 (excluding connections establishing) 2024-03-12T05:27:05.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-03-12T05:27:06.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:05 smithi045 ceph-mon[32174]: pgmap v510: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:06.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:05 smithi130 ceph-mon[35398]: pgmap v510: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:06.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:05 smithi116 ceph-mon[35486]: pgmap v510: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:08.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:07 smithi045 ceph-mon[32174]: pgmap v511: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:08.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:07 smithi130 ceph-mon[35398]: pgmap v511: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:08.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:07 smithi116 ceph-mon[35486]: pgmap v511: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-12T05:27:10.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:10 smithi045 ceph-mon[32174]: pgmap v512: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:27:10.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:10 smithi045 ceph-mon[32174]: osdmap e110: 12 total, 12 up, 12 in 2024-03-12T05:27:10.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:10 smithi130 ceph-mon[35398]: pgmap v512: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:27:10.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:10 smithi130 ceph-mon[35398]: osdmap e110: 12 total, 12 up, 12 in 2024-03-12T05:27:10.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:10 smithi116 ceph-mon[35486]: pgmap v512: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-12T05:27:10.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:10 smithi116 ceph-mon[35486]: osdmap e110: 12 total, 12 up, 12 in 2024-03-12T05:27:12.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:12 smithi045 ceph-mon[32174]: pgmap v514: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 15 MiB/s wr, 1.34k op/s 2024-03-12T05:27:12.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:12 smithi130 ceph-mon[35398]: pgmap v514: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 15 MiB/s wr, 1.34k op/s 2024-03-12T05:27:12.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:12 smithi116 ceph-mon[35486]: pgmap v514: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 15 MiB/s wr, 1.34k op/s 2024-03-12T05:27:13.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-12T05:27:13.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:27:13.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:27:13.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:27:13.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:13 smithi045 ceph-mon[32174]: scrub summary: idle 2024-03-12T05:27:13.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:27:13.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:27:13.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:27:13.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:13 smithi130 ceph-mon[35398]: scrub summary: idle 2024-03-12T05:27:13.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:27:13.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:27:13.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:27:13.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:13 smithi116 ceph-mon[35486]: scrub summary: idle 2024-03-12T05:27:14.085 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/: 2024-03-12T05:27:14.085 INFO:teuthology.orchestra.run.smithi045.stdout:total 2 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1655468868 Mar 12 05:10 . 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1655468868 Mar 12 05:10 .. 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 data 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data: 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:total 46 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 . 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1655468868 Mar 12 05:10 .. 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 base 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Mar 12 05:11 current_logfiles 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 572520 Mar 12 05:11 global 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 716 Mar 12 05:11 log 2024-03-12T05:27:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_commit_ts 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_dynshmem 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Mar 12 05:10 pg_hba.conf 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Mar 12 05:10 pg_ident.conf 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:27 pg_logical 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 pg_multixact 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_notify 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_replslot 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_serial 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_snapshots 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 14082 Mar 12 05:27 pg_stat 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:27 pg_stat_tmp 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:27 pg_subtrans 2024-03-12T05:27:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_tblspc 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 pg_twophase 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:10 PG_VERSION 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1040187392 Mar 12 05:27 pg_wal 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 311296 Mar 12 05:26 pg_xact 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Mar 12 05:10 postgresql.auto.conf 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Mar 12 05:10 postgresql.conf 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Mar 12 05:11 postmaster.opts 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base: 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:total 3 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 . 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.088 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 1 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 13493 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 596939311 Mar 12 05:17 13494 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:12 pgsql_tmp 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/1: 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:total 7778 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 . 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 .. 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 112 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 113 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 1247 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:27:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1247_vm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:11 1249 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1249_fsm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1249_vm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:11 1259 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1259_vm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:27:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:27:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 1417 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 1418 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 174 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 175 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2187 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2224 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2328 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2336 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2579 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:27:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2604 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606 2024-03-12T05:27:14.093 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606_vm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:11 2608 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2608_vm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2610 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:27:14.094 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2610_vm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2611 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2613 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:27:14.095 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:11 2619 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2619_fsm 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2619_vm 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2620 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:27:14.096 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:11 2658 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 2659 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2662 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2663 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2664 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2665 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2666 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2667 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2668 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2669 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2670 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:11 2673 2024-03-12T05:27:14.097 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:11 2674 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2678 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2679 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2680 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2683 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2686 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2687 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:10 2688 2024-03-12T05:27:14.098 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:10 2689 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2696 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2699 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2701 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2702 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2703 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2704 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2754 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2755 2024-03-12T05:27:14.099 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2830 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2832 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2834 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2836 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:27:14.100 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2995 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2996 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:27:14.101 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3118 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3119 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3256 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3257 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3258 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3350 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3351 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3379 2024-03-12T05:27:14.102 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3380 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3381 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3429 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3430 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3433 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3439 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3455 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:27:14.103 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3466 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3467 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3468 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3501 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3502 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3503 2024-03-12T05:27:14.104 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3534 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3542 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3574 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3575 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3576 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3596 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3597 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3598 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:27:14.105 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:27:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3606 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 3607 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3997 2024-03-12T05:27:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4143 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4145 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4147 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4149 2024-03-12T05:27:14.108 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:27:14.109 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4151 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4153 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4155 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4157 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4159 2024-03-12T05:27:14.311 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4161 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4163 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4165 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4167 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4169 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4171 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4173 2024-03-12T05:27:14.312 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 5002 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 548 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 549 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6102 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6104 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6106 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6110 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6111 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6112 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6113 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6117 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 826 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 827 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 828 2024-03-12T05:27:14.313 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:total 7778 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Mar 12 05:11 . 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 .. 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 112 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 113 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 1247 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1247_vm 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:11 1249 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1249_fsm 2024-03-12T05:27:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1249_vm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:11 1259 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1259_vm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:27:14.315 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1417 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1418 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 174 2024-03-12T05:27:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 175 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2187 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2224 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2328 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2336 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2579 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:27:14.317 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2604 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:27:14.318 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2606_vm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:11 2608 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2608_vm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2610 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2610_vm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2611 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:27:14.319 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2613 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:27:14.320 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:11 2619 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2619_fsm 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2619_vm 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2620 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:27:14.321 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:11 2658 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:11 2659 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2662 2024-03-12T05:27:14.322 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2663 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2664 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2665 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2666 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2667 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2668 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2669 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2670 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:11 2673 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:11 2674 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:27:14.323 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2678 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2679 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2680 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2683 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2686 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2687 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2688 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2689 2024-03-12T05:27:14.324 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2696 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2699 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2701 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2702 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2703 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2704 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:27:14.325 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2754 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2755 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2830 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2832 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2834 2024-03-12T05:27:14.326 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2836 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2995 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2996 2024-03-12T05:27:14.327 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3118 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3119 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3256 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3257 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3258 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3350 2024-03-12T05:27:14.328 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3351 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3379 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3380 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3381 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3429 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3430 2024-03-12T05:27:14.329 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3433 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3439 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3455 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3466 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3467 2024-03-12T05:27:14.330 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3468 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3501 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3502 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3503 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3534 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3542 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3574 2024-03-12T05:27:14.331 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3575 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3576 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3596 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3597 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3598 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:27:14.332 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3606 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3607 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:27:14.333 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3997 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4143 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4145 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4147 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:27:14.334 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4149 2024-03-12T05:27:14.335 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:27:14.335 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4151 2024-03-12T05:27:14.335 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:27:14.335 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4153 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4155 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4157 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4159 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4161 2024-03-12T05:27:14.597 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4163 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4165 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4167 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4169 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4171 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4173 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:27:14.598 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 5002 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 548 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 549 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6102 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6104 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6106 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6110 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6111 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6112 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6113 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6117 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 826 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 827 2024-03-12T05:27:14.599 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 828 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:total 582951 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 596939311 Mar 12 05:17 . 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 .. 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 112 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 113 2024-03-12T05:27:14.600 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:12 1247 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1247_fsm 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 1247_vm 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 12 05:12 1249 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:12 1249_fsm 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 1249_vm 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 12 05:11 1255 2024-03-12T05:27:14.601 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1255_fsm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1255_vm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 12 05:26 1259 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1259_fsm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 1259_vm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 12 05:11 13344 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13344_fsm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13344_vm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13346 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13348 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13349_fsm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13349_vm 2024-03-12T05:27:14.602 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13351 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13353 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13354_fsm 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13354_vm 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13356 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13358 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 13359_fsm 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13359_vm 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 13361 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 13363 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1417 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 1418 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 443875328 Mar 12 05:27 16396 2024-03-12T05:27:14.603 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Mar 12 05:26 16396_fsm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:19 16396_vm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 2449408 Mar 12 05:27 16397 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:26 16397_fsm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:26 16397_vm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 6225920 Mar 12 05:27 16399 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:26 16399_fsm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:26 16399_vm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:27 16400 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:27 16402 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Mar 12 05:27 16404 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 64012288 Mar 12 05:27 16406 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:27 16406_fsm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:25 16406_vm 2024-03-12T05:27:14.604 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 174 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 175 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2187 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2224 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2328 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2336 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2337 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2579 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2600 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2600_fsm 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2600_vm 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2601_fsm 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2601_vm 2024-03-12T05:27:14.605 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2602 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2602_fsm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2602_vm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2603 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2603_fsm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2603_vm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2604 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2605 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2605_fsm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2605_vm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 2606 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2606_fsm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 2606_vm 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2607 2024-03-12T05:27:14.606 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2607_fsm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2607_vm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 12 05:12 2608 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2608_fsm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 2608_vm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 12 05:11 2609 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2609_fsm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2609_vm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:12 2610 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2610_fsm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 2610_vm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2611 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2612_fsm 2024-03-12T05:27:14.607 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2612_vm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2613 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2615_fsm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2615_vm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2616 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2616_fsm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2616_vm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 12 05:11 2617 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2617_fsm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2617_vm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 12 05:11 2618 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2618_fsm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2618_vm 2024-03-12T05:27:14.608 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Mar 12 05:26 2619 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:12 2619_fsm 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:12 2619_vm 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2620 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2650 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2651 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2652 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2653 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2654 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2655 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2656 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2657 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 12 05:12 2658 2024-03-12T05:27:14.609 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 12 05:12 2659 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2660 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2661 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:12 2662 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:12 2663 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2664 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2665 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2666 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2667 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2668 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2669 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2670 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 12 05:12 2673 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 12 05:12 2674 2024-03-12T05:27:14.610 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 12 05:11 2675 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2678 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:12 2679 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2680 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2681 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2682 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2683 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2684 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2685 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2686 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2687 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 2688 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:11 2689 2024-03-12T05:27:14.611 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 12 05:11 2690 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 12 05:11 2691 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2692 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2693 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:14 2696 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2699 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2701 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2702 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:12 2703 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:12 2704 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2753 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2753_fsm 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2753_vm 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2754 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2755 2024-03-12T05:27:14.612 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2756 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 2757 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2830 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2831 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2832 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2833 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2834 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2835 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2836 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2837 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 12 05:11 2838 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2838_fsm 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2838_vm 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2839 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840 2024-03-12T05:27:14.613 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2840_fsm 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2840_vm 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2841 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 2995 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2996 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3079_fsm 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3079_vm 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3080 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3081 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:11 3085 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3118 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3119 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 12 05:11 3164 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3256 2024-03-12T05:27:14.614 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3257 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3258 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3350 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3351 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3379 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3380 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3381 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3394 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3394_fsm 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3394_vm 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3395 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3429 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3430 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3431 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3433 2024-03-12T05:27:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3439 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3440 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 12 05:12 3455 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 12 05:11 3456 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3456_fsm 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3456_vm 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3466 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3467 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3468 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3501 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3502 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3503 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3534 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3541_fsm 2024-03-12T05:27:14.616 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3541_vm 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3542 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3574 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3575 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3576 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3596 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3597 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 3598 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3599 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3600_fsm 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3600_vm 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3601_fsm 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3601_vm 2024-03-12T05:27:14.617 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3602_fsm 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3602_vm 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3603_fsm 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3603_vm 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3604 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3605 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3606 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3607 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3608 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 12 05:11 3609 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3712 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764 2024-03-12T05:27:14.618 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 3764_fsm 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3764_vm 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3766 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 3767 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 3997 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4143 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4144 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4145 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4146 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4147 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4148 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4149 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4150 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4151 2024-03-12T05:27:14.619 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4152 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4153 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4154 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4155 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4156 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4157 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4158 2024-03-12T05:27:14.620 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4159 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4160 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4161 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4162 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4163 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4164 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4165 2024-03-12T05:27:14.621 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4166 2024-03-12T05:27:14.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4167 2024-03-12T05:27:14.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4168 2024-03-12T05:27:14.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4169 2024-03-12T05:27:14.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4170 2024-03-12T05:27:14.622 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4171 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4172 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 4173 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4174 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 5002 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 548 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 549 2024-03-12T05:27:14.623 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6102 2024-03-12T05:27:14.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6104 2024-03-12T05:27:14.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 6106 2024-03-12T05:27:14.624 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6110 2024-03-12T05:27:14.647 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6111 2024-03-12T05:27:14.647 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6112 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6113 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 6117 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:11 826 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 827 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 828 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Mar 12 05:11 pg_internal.init 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 12 05:11 PG_VERSION 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:12 . 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 612865589 Mar 12 05:11 .. 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.648 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/global: 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:total 561 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 572520 Mar 12 05:11 . 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1213 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1213_fsm 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1213_vm 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1214 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1214_fsm 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1214_vm 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 1232 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 1233 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1260 2024-03-12T05:27:14.649 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1260_fsm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1260_vm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1261 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1261_fsm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1261_vm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1262 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 1262_fsm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 1262_vm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2396 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 12 05:11 2396_fsm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2396_vm 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2397 2024-03-12T05:27:14.650 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2671 2024-03-12T05:27:14.651 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2672 2024-03-12T05:27:14.651 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2676 2024-03-12T05:27:14.651 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2677 2024-03-12T05:27:14.651 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2694 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:11 2695 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2697 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 2698 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2846 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2847 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2964 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 2965 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 2966 2024-03-12T05:27:14.663 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 2967 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 3592 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 3593 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4060 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4061 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4175 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4176 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4177 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4178 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4181 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4182 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4183 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4184 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 4185 2024-03-12T05:27:14.664 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:11 4186 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6000 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6001 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6002 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 6100 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6114 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 6115 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:27 pg_control 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 12 05:11 pg_filenode.map 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Mar 12 05:11 pg_internal.init 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/log: 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:total 3 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 716 Mar 12 05:11 . 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.665 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1118 Mar 12 05:27 postgresql-Tue.log 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-03-12T05:27:14.666 INFO:teuthology.orchestra.run.smithi045.stdout:total 3 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:27 . 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 mappings 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:27 replorigin_checkpoint 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 snapshots 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:27 .. 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.704 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 12 05:27 .. 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:total 2 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 . 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 members 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 offsets 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:27:14.705 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 .. 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 0000 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:total 9 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:10 . 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 12 05:10 .. 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 12 05:16 0000 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.706 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.707 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:total 16 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 14082 Mar 12 05:27 . 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Mar 12 05:27 db_0.stat 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Mar 12 05:27 db_13494.stat 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Mar 12 05:27 global.stat 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:27 . 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-03-12T05:27:14.708 INFO:teuthology.orchestra.run.smithi045.stdout:total 145 2024-03-12T05:27:14.709 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:27 . 2024-03-12T05:27:14.709 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.760 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 12 05:27 0012 2024-03-12T05:27:14.760 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.760 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-03-12T05:27:14.760 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.760 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:total 1015810 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1040187392 Mar 12 05:27 . 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:27 000000010000000000000086 2024-03-12T05:27:14.761 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:17 000000010000000000000087 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:16 000000010000000000000088 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:16 000000010000000000000089 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000008A 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000008B 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 00000001000000000000008C 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 00000001000000000000008D 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000008E 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000008F 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 000000010000000000000090 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 000000010000000000000091 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:23 000000010000000000000092 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 000000010000000000000093 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 000000010000000000000094 2024-03-12T05:27:14.762 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:23 000000010000000000000095 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:25 000000010000000000000096 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 000000010000000000000097 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 000000010000000000000098 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 000000010000000000000099 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 00000001000000000000009A 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 00000001000000000000009B 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:25 00000001000000000000009C 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000009D 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:24 00000001000000000000009E 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 00000001000000000000009F 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:23 0000000100000000000000A0 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000A1 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 0000000100000000000000A2 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000A3 2024-03-12T05:27:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:23 0000000100000000000000A4 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000A5 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:24 0000000100000000000000A6 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 0000000100000000000000A7 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000A8 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:24 0000000100000000000000A9 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000AA 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000AB 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000AC 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:22 0000000100000000000000AD 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000AE 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:21 0000000100000000000000AF 2024-03-12T05:27:14.764 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B0 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B1 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B2 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B3 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B4 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B5 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B6 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B7 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B8 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000B9 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000BA 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000BB 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000BC 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:27 0000000100000000000000BD 2024-03-12T05:27:14.765 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000BE 2024-03-12T05:27:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000BF 2024-03-12T05:27:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000C0 2024-03-12T05:27:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000C1 2024-03-12T05:27:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000C2 2024-03-12T05:27:14.766 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 12 05:26 0000000100000000000000C3 2024-03-12T05:27:14.768 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 archive_status 2024-03-12T05:27:14.768 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.768 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:total 1 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 12 05:10 . 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1040187392 Mar 12 05:27 .. 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:total 305 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 311296 Mar 12 05:26 . 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1654149849 Mar 12 05:27 .. 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Mar 12 05:26 0000 2024-03-12T05:27:14.769 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 12 05:27 0001 2024-03-12T05:27:14.771 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-12T05:27:14.783 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-12T05:27:14.783 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:27:14.783 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:27:14.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-12T05:27:14.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-12T05:27:14.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:14 smithi045 ceph-mon[32174]: pgmap v515: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-03-12T05:27:14.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-12T05:27:14.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:14 smithi130 ceph-mon[35398]: pgmap v515: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-03-12T05:27:14.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:14 smithi116 ceph-mon[35486]: pgmap v515: 97 pgs: 97 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 -- 172.21.15.45:0/2690725203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018100c90 msgr2=0x7f501810ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 --2- 172.21.15.45:0/2690725203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018100c90 0x7f501810ceb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f5010009ca0 tx=0x7f501002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 -- 172.21.15.45:0/2690725203 shutdown_connections 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 --2- 172.21.15.45:0/2690725203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018100c90 0x7f501810ceb0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 --2- 172.21.15.45:0/2690725203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f50181002d0 0x7f5018100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 --2- 172.21.15.45:0/2690725203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5018105020 0x7f5018105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 -- 172.21.15.45:0/2690725203 >> 172.21.15.45:0/2690725203 conn(0x7f5018076ed0 msgr2=0x7f50180772e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 -- 172.21.15.45:0/2690725203 shutdown_connections 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.024+0000 7f502022c640 1 -- 172.21.15.45:0/2690725203 wait complete. 2024-03-12T05:27:15.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 Processor -- start 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 -- start start 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 0x7f5018101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 0x7f5018102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181a59b0 con 0x7f5018105020 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181a5b20 con 0x7f50181002d0 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f502022c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181a5ce0 con 0x7f5018100c90 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43674/0 (socket says 172.21.15.45:43674) 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 -- 172.21.15.45:0/1373214106 learned_addr learned my addr 172.21.15.45:0/1373214106 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:15.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501dfc8640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 0x7f5018101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501d7c7640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 0x7f5018102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 -- 172.21.15.45:0/1373214106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 msgr2=0x7f5018102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 0x7f5018102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 -- 172.21.15.45:0/1373214106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 msgr2=0x7f5018101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 0x7f5018101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501e7c9640 1 -- 172.21.15.45:0/1373214106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010009900 con 0x7f5018105020 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501dfc8640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 0x7f5018101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.025+0000 7f501d7c7640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 0x7f5018102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f501e7c9640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f501002c700 tx=0x7f5010034ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010032820 con 0x7f5018105020 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f5010003e20 con 0x7f5018105020 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100047d0 con 0x7f5018105020 2024-03-12T05:27:15.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50181a5f60 con 0x7f5018105020 2024-03-12T05:27:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.026+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50181a6430 con 0x7f5018105020 2024-03-12T05:27:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.027+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501804eee0 con 0x7f5018105020 2024-03-12T05:27:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.028+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f501000a040 con 0x7f5018105020 2024-03-12T05:27:15.024 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.028+0000 7f500effd640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 0x7f50040a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.028+0000 7f501dfc8640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 0x7f50040a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.028+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f50100f0420 con 0x7f5018105020 2024-03-12T05:27:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.029+0000 7f501dfc8640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 0x7f50040a41b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5014005e00 tx=0x7f5014005c70 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.033+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f50100b93d0 con 0x7f5018105020 2024-03-12T05:27:15.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.236+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f50181100f0 con 0x7f50040a1cf0 2024-03-12T05:27:15.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.244+0000 7f500effd640 1 -- 172.21.15.45:0/1373214106 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f50181100f0 con 0x7f50040a1cf0 2024-03-12T05:27:15.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-12T00:00:00", "created": "2024-03-12T05:10:53", "first": "2024-03-12T05:11:00", "last": "2024-03-12T05:27:00", "last_pruned": "2024-03-12T05:27:00", "created_count": 17, "pruned_count": 11, "active": true} 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 msgr2=0x7f50040a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 0x7f50040a41b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5014005e00 tx=0x7f5014005c70 comp rx=0 tx=0).stop 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 msgr2=0x7f5018102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f501002c700 tx=0x7f5010034ae0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 shutdown_connections 2024-03-12T05:27:15.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f50040a1cf0 0x7f50040a41b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5018105020 0x7f5018102770 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5018100c90 0x7f5018102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 --2- 172.21.15.45:0/1373214106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50181002d0 0x7f5018101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 >> 172.21.15.45:0/1373214106 conn(0x7f5018076ed0 msgr2=0x7f5018068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 shutdown_connections 2024-03-12T05:27:15.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.246+0000 7f502022c640 1 -- 172.21.15.45:0/1373214106 wait complete. 2024-03-12T05:27:15.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-12T05:27:15.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.361+0000 7f65133a0640 1 -- 172.21.15.45:0/4126520793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c10b550 msgr2=0x7f650c10d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.361+0000 7f65133a0640 1 --2- 172.21.15.45:0/4126520793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c10b550 0x7f650c10d9f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009ca0 tx=0x7f64fc02c180 comp rx=0 tx=0).stop 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- 172.21.15.45:0/4126520793 shutdown_connections 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- 172.21.15.45:0/4126520793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c10b550 0x7f650c10d9f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- 172.21.15.45:0/4126520793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 0x7f650c10af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- 172.21.15.45:0/4126520793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c102130 0x7f650c102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- 172.21.15.45:0/4126520793 >> 172.21.15.45:0/4126520793 conn(0x7f650c0fb990 msgr2=0x7f650c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- 172.21.15.45:0/4126520793 shutdown_connections 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- 172.21.15.45:0/4126520793 wait complete. 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 Processor -- start 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- start start 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 0x7f650c1002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c10b550 0x7f650c100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1a58b0 con 0x7f650c102130 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1a5a20 con 0x7f650c102a50 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f65133a0640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1a5be0 con 0x7f650c10b550 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.362+0000 7f651239e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43686/0 (socket says 172.21.15.45:43686) 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 -- 172.21.15.45:0/155591747 learned_addr learned my addr 172.21.15.45:0/155591747 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 -- 172.21.15.45:0/155591747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c10b550 msgr2=0x7f650c100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f6512b9f640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c10b550 0x7f650c100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c10b550 0x7f650c100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 -- 172.21.15.45:0/155591747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 msgr2=0x7f650c1002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f6511b9d640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 0x7f650c1002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 0x7f650c1002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 -- 172.21.15.45:0/155591747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc009900 con 0x7f650c102130 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f651239e640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f650400b4a0 tx=0x7f650400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6504010610 con 0x7f650c102130 2024-03-12T05:27:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6504010c20 con 0x7f650c102130 2024-03-12T05:27:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6504017d00 con 0x7f650c102130 2024-03-12T05:27:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c1a5ec0 con 0x7f650c102130 2024-03-12T05:27:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.363+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c1a62f0 con 0x7f650c102130 2024-03-12T05:27:15.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.364+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c04f730 con 0x7f650c102130 2024-03-12T05:27:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.365+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6504010770 con 0x7f650c102130 2024-03-12T05:27:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.366+0000 7f650b7fe640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 0x7f64f40a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.366+0000 7f6511b9d640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 0x7f64f40a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.366+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f65040cd410 con 0x7f650c102130 2024-03-12T05:27:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.366+0000 7f6511b9d640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 0x7f64f40a40d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6500005fd0 tx=0x7f6500005d00 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:15.363 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.369+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6504096360 con 0x7f650c102130 2024-03-12T05:27:15.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.572+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 --> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f650c110fc0 con 0x7f64f40a1c10 2024-03-12T05:27:15.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.579+0000 7f650b7fe640 1 -- 172.21.15.45:0/155591747 <== mgr.25024 v2:172.21.15.45:6800/869096728 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f650c110fc0 con 0x7f64f40a1c10 2024-03-12T05:27:15.573 INFO:teuthology.orchestra.run.smithi045.stdout:/ 1m 6m3h 2024-03-12T05:27:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 msgr2=0x7f64f40a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 0x7f64f40a40d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6500005fd0 tx=0x7f6500005d00 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 msgr2=0x7f650c0ffd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f650400b4a0 tx=0x7f650400b960 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 shutdown_connections 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f64f40a1c10 0x7f64f40a40d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f650c10b550 0x7f650c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c102a50 0x7f650c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 --2- 172.21.15.45:0/155591747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f650c102130 0x7f650c0ffd80 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 >> 172.21.15.45:0/155591747 conn(0x7f650c0fb990 msgr2=0x7f650c0fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 shutdown_connections 2024-03-12T05:27:15.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:15.581+0000 7f65133a0640 1 -- 172.21.15.45:0/155591747 wait complete. 2024-03-12T05:27:15.586 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-12T05:27:15.597 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-12T05:27:15.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:15 smithi045 ceph-mon[32174]: pgmap v516: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:15.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:15 smithi045 ceph-mon[32174]: from='client.19560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:15.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:15 smithi045 ceph-mon[32174]: from='client.19566 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:15 smithi130 ceph-mon[35398]: pgmap v516: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:15 smithi130 ceph-mon[35398]: from='client.19560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:15 smithi130 ceph-mon[35398]: from='client.19566 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:15.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:15 smithi116 ceph-mon[35486]: pgmap v516: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:15.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:15 smithi116 ceph-mon[35486]: from='client.19560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:15.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:15 smithi116 ceph-mon[35486]: from='client.19566 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:27:18.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:17 smithi045 ceph-mon[32174]: pgmap v517: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:18.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:17 smithi130 ceph-mon[35398]: pgmap v517: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:18.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:17 smithi116 ceph-mon[35486]: pgmap v517: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-12T05:27:20.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:19 smithi045 ceph-mon[32174]: pgmap v518: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-03-12T05:27:20.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:19 smithi130 ceph-mon[35398]: pgmap v518: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-03-12T05:27:20.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:19 smithi116 ceph-mon[35486]: pgmap v518: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-03-12T05:27:22.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:21 smithi045 ceph-mon[32174]: pgmap v519: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 5.1 MiB/s wr, 517 op/s 2024-03-12T05:27:22.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:21 smithi130 ceph-mon[35398]: pgmap v519: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 5.1 MiB/s wr, 517 op/s 2024-03-12T05:27:22.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:21 smithi116 ceph-mon[35486]: pgmap v519: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 5.1 MiB/s wr, 517 op/s 2024-03-12T05:27:24.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:23 smithi045 ceph-mon[32174]: pgmap v520: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.8 MiB/s wr, 479 op/s 2024-03-12T05:27:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:23 smithi130 ceph-mon[35398]: pgmap v520: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.8 MiB/s wr, 479 op/s 2024-03-12T05:27:24.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:23 smithi116 ceph-mon[35486]: pgmap v520: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.8 MiB/s wr, 479 op/s 2024-03-12T05:27:26.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:25 smithi045 ceph-mon[32174]: pgmap v521: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-12T05:27:26.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:25 smithi130 ceph-mon[35398]: pgmap v521: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-12T05:27:26.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:25 smithi116 ceph-mon[35486]: pgmap v521: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-12T05:27:28.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:27 smithi045 ceph-mon[32174]: pgmap v522: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-03-12T05:27:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:27 smithi130 ceph-mon[35398]: pgmap v522: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-03-12T05:27:28.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:27 smithi116 ceph-mon[35486]: pgmap v522: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-03-12T05:27:30.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:30 smithi045 ceph-mon[32174]: pgmap v523: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 35 KiB/s wr, 17 op/s 2024-03-12T05:27:30.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:30 smithi130 ceph-mon[35398]: pgmap v523: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 35 KiB/s wr, 17 op/s 2024-03-12T05:27:30.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:30 smithi116 ceph-mon[35486]: pgmap v523: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 35 KiB/s wr, 17 op/s 2024-03-12T05:27:31.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:27:31.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 -- 172.21.15.45:0/3685238746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728100c90 msgr2=0x7fc72810ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/3685238746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728100c90 0x7fc72810ce40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ca0 tx=0x7fc72002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 -- 172.21.15.45:0/3685238746 shutdown_connections 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/3685238746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728100c90 0x7fc72810ce40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/3685238746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7281002d0 0x7fc728100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/3685238746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc728106110 0x7fc7281064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 -- 172.21.15.45:0/3685238746 >> 172.21.15.45:0/3685238746 conn(0x7fc728076e20 msgr2=0x7fc728077230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 -- 172.21.15.45:0/3685238746 shutdown_connections 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.185+0000 7fc72f9e9640 1 -- 172.21.15.45:0/3685238746 wait complete. 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 Processor -- start 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 -- start start 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 0x7fc728197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc728100c90 0x7fc728197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72819c5a0 con 0x7fc728106110 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72819c710 con 0x7fc7281002d0 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72f9e9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72819c8d0 con 0x7fc728100c90 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42248/0 (socket says 172.21.15.45:42248) 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 -- 172.21.15.45:0/107947936 learned_addr learned my addr 172.21.15.45:0/107947936 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72cf84640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc728100c90 0x7fc728197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72d785640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 0x7fc728197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 -- 172.21.15.45:0/107947936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc728100c90 msgr2=0x7fc728197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc728100c90 0x7fc728197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 -- 172.21.15.45:0/107947936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 msgr2=0x7fc728197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 0x7fc728197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.186+0000 7fc72df86640 1 -- 172.21.15.45:0/107947936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc720009900 con 0x7fc728106110 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc72d785640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 0x7fc728197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc72df86640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc72002c700 tx=0x7fc720035d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc71e7fc640 1 -- 172.21.15.45:0/107947936 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7200329e0 con 0x7fc728106110 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc71e7fc640 1 -- 172.21.15.45:0/107947936 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc720032410 con 0x7fc728106110 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72819cb50 con 0x7fc728106110 2024-03-12T05:27:31.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc71e7fc640 1 -- 172.21.15.45:0/107947936 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720004840 con 0x7fc728106110 2024-03-12T05:27:31.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.187+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7281a4430 con 0x7fc728106110 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.189+0000 7fc71e7fc640 1 -- 172.21.15.45:0/107947936 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc72003b040 con 0x7fc728106110 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.189+0000 7fc7325c4740 1 Processor -- start 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 -- start start 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bdf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0beaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0b4e280 con 0x55fef0bdfbd0 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0b4b7b0 con 0x55fef0be87b0 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc7325c4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0b46f90 con 0x55fef0bdf2b0 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc71e7fc640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 0x7fc7140a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42252/0 (socket says 172.21.15.45:42252) 2024-03-12T05:27:31.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 -- 172.21.15.45:0/328916544 learned_addr learned my addr 172.21.15.45:0/328916544 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72cf84640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bdf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72df86640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 0x7fc7140a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72cf84640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0beaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 -- 172.21.15.45:0/328916544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 msgr2=0x55fef0bdf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bdf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 -- 172.21.15.45:0/328916544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 msgr2=0x55fef0beaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0beaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72d785640 1 -- 172.21.15.45:0/328916544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fef0b05570 con 0x55fef0bdfbd0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc71e7fc640 1 -- 172.21.15.45:0/107947936 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc7200f02a0 con 0x7fc728106110 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.190+0000 7fc72cf84640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0beaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc72d785640 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc724009c50 tx=0x7fc72400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343fe5824cead81e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc72df86640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 0x7fc7140a40e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc7200b93b0 tx=0x7fc7200b8b50 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7137fe640 1 -- 172.21.15.45:0/328916544 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724005020 con 0x55fef0bdfbd0 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7137fe640 1 -- 172.21.15.45:0/328916544 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc724005630 con 0x55fef0bdfbd0 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7137fe640 1 -- 172.21.15.45:0/328916544 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724011670 con 0x55fef0bdfbd0 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 -- 172.21.15.45:0/328916544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 msgr2=0x55fef0be8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc724009c50 tx=0x7fc72400e6c0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 -- 172.21.15.45:0/328916544 shutdown_connections 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be8270 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0beaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.191+0000 7fc7325c4740 1 --2- 172.21.15.45:0/328916544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bdf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- 172.21.15.45:0/328916544 >> 172.21.15.45:0/328916544 conn(0x55fef0bd9990 msgr2=0x55fef0bd9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- 172.21.15.45:0/328916544 shutdown_connections 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- 172.21.15.45:0/328916544 wait complete. 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- start start 2024-03-12T05:27:31.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 Processor -- start 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0be8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0be5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef081a430 con 0x55fef0bdfbd0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef07d6e00 con 0x55fef0be87b0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0acbfd0 con 0x55fef0bdf2b0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.192+0000 7fc7325c4740 10 client.0 _ll_register_callbacks cb 0x55fef0b21fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42266/0 (socket says 172.21.15.45:42266) 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 -- 172.21.15.45:0/963772419 learned_addr learned my addr 172.21.15.45:0/963772419 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72cf84640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0be8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72cf84640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0be5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 -- 172.21.15.45:0/963772419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 msgr2=0x55fef0be8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0be8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 -- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 msgr2=0x55fef0be5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0be5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 -- 172.21.15.45:0/963772419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc724009880 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72cf84640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0be5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc72d785640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc7240057c0 tx=0x7fc724009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e310378cf2761e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724021070 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc724005060 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fef08467e0 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc724004b10 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc7325c4740 10 client.19578 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.193+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fef0b45610 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc724031020 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc72400c070 con 0x55fef0bdfbd0 2024-03-12T05:27:31.196 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 10 client.19578 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 10 client.19578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 20 client.19578 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0bf8990 0x55fef0bfadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 4 client.19578 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fef0883720 con 0x55fef0bf8990 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc72cf84640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0bf8990 0x55fef0bfadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc72cf84640 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0bf8990 0x55fef0bfadb0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf39608b032c68f server_cookie=26cd1229a7932798 in_seq=0 out_seq=0 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.194+0000 7fc710ff9640 10 client.19578 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc710ff9640 1 -- 172.21.15.45:0/963772419 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fef0883720 con 0x55fef0bf8990 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc710ff9640 10 client.19578 handle_command_reply: tid=0 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 client.19578 shutdown 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 msgr2=0x55fef0be55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc7240057c0 tx=0x7fc724009f70 comp rx=0 tx=0).stop 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 shutdown_connections 2024-03-12T05:27:31.197 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0bf8990 0x55fef0bfadb0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0bdfbd0 0x55fef0be55e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc710ff9640 0 client.19578 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0be87b0 0x55fef0be5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 --2- 172.21.15.45:0/963772419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0be8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 >> 172.21.15.45:0/963772419 conn(0x55fef0bd9990 msgr2=0x55fef0bdb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 shutdown_connections 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.195+0000 7fc7325c4740 1 -- 172.21.15.45:0/963772419 wait complete. 2024-03-12T05:27:31.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.196+0000 7fc7325c4740 20 client.19578 trim_cache size 0 max 16384 2024-03-12T05:27:31.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 Processor -- start 2024-03-12T05:27:31.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 -- start start 2024-03-12T05:27:31.210 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdf2b0 0x55fef0bef660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdfbd0 0x55fef0befba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0bdef50 con 0x55fef0be87b0 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0be65a0 con 0x55fef0bdf2b0 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc7325c4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0beaf60 con 0x55fef0bdfbd0 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42272/0 (socket says 172.21.15.45:42272) 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 -- 172.21.15.45:0/3489936552 learned_addr learned my addr 172.21.15.45:0/3489936552 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 -- 172.21.15.45:0/3489936552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdfbd0 msgr2=0x55fef0befba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdfbd0 0x55fef0befba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 -- 172.21.15.45:0/3489936552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdf2b0 msgr2=0x55fef0bef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdf2b0 0x55fef0bef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.211 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 -- 172.21.15.45:0/3489936552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fef0ae1ac0 con 0x55fef0be87b0 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.211+0000 7fc72cf84640 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc718008010 tx=0x7fc71800a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955197ec6a565492 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc712ffd640 1 -- 172.21.15.45:0/3489936552 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718016a20 con 0x55fef0be87b0 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc712ffd640 1 -- 172.21.15.45:0/3489936552 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc718022070 con 0x55fef0be87b0 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc712ffd640 1 -- 172.21.15.45:0/3489936552 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718018450 con 0x55fef0be87b0 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 -- 172.21.15.45:0/3489936552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 msgr2=0x55fef0be7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc718008010 tx=0x7fc71800a230 comp rx=0 tx=0).stop 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 -- 172.21.15.45:0/3489936552 shutdown_connections 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0be7d10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdfbd0 0x55fef0befba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 --2- 172.21.15.45:0/3489936552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdf2b0 0x55fef0bef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 -- 172.21.15.45:0/3489936552 >> 172.21.15.45:0/3489936552 conn(0x55fef0bd9990 msgr2=0x55fef0be3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 -- 172.21.15.45:0/3489936552 shutdown_connections 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.212+0000 7fc7325c4740 1 -- 172.21.15.45:0/3489936552 wait complete. 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 -- start start 2024-03-12T05:27:31.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 Processor -- start 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdfbd0 0x55fef0bee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0c4ef30 con 0x55fef0be87b0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0c4f0a0 con 0x55fef0bdfbd0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef0c4f260 con 0x55fef0bdf2b0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc7325c4740 10 client.0 _ll_register_callbacks cb 0x7fc724009400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42274/0 (socket says 172.21.15.45:42274) 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 -- 172.21.15.45:0/1031014420 learned_addr learned my addr 172.21.15.45:0/1031014420 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72d785640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdfbd0 0x55fef0bee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 -- 172.21.15.45:0/1031014420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 msgr2=0x55fef0bee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.213+0000 7fc72cf84640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bee1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc72cf84640 1 -- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdfbd0 msgr2=0x55fef0bee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc72cf84640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdfbd0 0x55fef0bee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc72cf84640 1 -- 172.21.15.45:0/1031014420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71800e040 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc72cf84640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc718006ee0 tx=0x7fc71800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1372ebc76ddf93c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718016cd0 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc718016e30 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc724005630 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc71801e150 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 10 client.19584 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fef0b076a0 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc724005630 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc718018b20 con 0x55fef0be87b0 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 10 client.19584 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 10 client.19584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 20 client.19584 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0c50960 0x55fef0c52d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 4 client.19584 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.214+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fef08d2890 con 0x55fef0c50960 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc72d785640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0c50960 0x55fef0c52d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc72d785640 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0c50960 0x55fef0c52d80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d088b6ae3d1c604 server_cookie=b5bbef3633bed0dc in_seq=0 out_seq=0 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc710ff9640 10 client.19584 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc710ff9640 1 -- 172.21.15.45:0/1031014420 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fef08d2890 con 0x55fef0c50960 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc710ff9640 10 client.19584 handle_command_reply: tid=0 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 client.19584 shutdown 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 msgr2=0x55fef0beec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc718006ee0 tx=0x7fc71800af70 comp rx=0 tx=0).stop 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 shutdown_connections 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55fef0c50960 0x55fef0c52d80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55fef0be87b0 0x55fef0beec70 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fef0bdfbd0 0x55fef0bee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc710ff9640 0 client.19584 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 --2- 172.21.15.45:0/1031014420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fef0bdf2b0 0x55fef0bee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.215+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 >> 172.21.15.45:0/1031014420 conn(0x55fef0bd9990 msgr2=0x55fef0be3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 shutdown_connections 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc7325c4740 1 -- 172.21.15.45:0/1031014420 wait complete. 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc7325c4740 20 client.19584 trim_cache size 0 max 16384 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "no active scrubs running", 2024-03-12T05:27:31.216 INFO:teuthology.orchestra.run.smithi045.stdout: "scrubs": {} 2024-03-12T05:27:31.217 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:31.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 msgr2=0x7fc7140a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 0x7fc7140a40e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc7200b93b0 tx=0x7fc7200b8b50 comp rx=0 tx=0).stop 2024-03-12T05:27:31.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 msgr2=0x7fc72819bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc72002c700 tx=0x7fc720035d20 comp rx=0 tx=0).stop 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 shutdown_connections 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc7140a1c20 0x7fc7140a40e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc728106110 0x7fc72819bfa0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc728100c90 0x7fc728197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 --2- 172.21.15.45:0/107947936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7281002d0 0x7fc728197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 >> 172.21.15.45:0/107947936 conn(0x7fc728076e20 msgr2=0x7fc7280699c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 shutdown_connections 2024-03-12T05:27:31.218 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.216+0000 7fc72f9e9640 1 -- 172.21.15.45:0/107947936 wait complete. 2024-03-12T05:27:31.222 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:27:31.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:27:31.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 -- 172.21.15.45:0/122676968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c01023f0 msgr2=0x7f22c01027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 --2- 172.21.15.45:0/122676968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c01023f0 0x7f22c01027d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009900 tx=0x7f22b802bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:31.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 -- 172.21.15.45:0/122676968 shutdown_connections 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 --2- 172.21.15.45:0/122676968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 0x7f22c010dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 --2- 172.21.15.45:0/122676968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c0102d10 0x7f22c010b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 --2- 172.21.15.45:0/122676968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c01023f0 0x7f22c01027d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 -- 172.21.15.45:0/122676968 >> 172.21.15.45:0/122676968 conn(0x7f22c0077180 msgr2=0x7f22c0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 -- 172.21.15.45:0/122676968 shutdown_connections 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.331+0000 7f22c5556640 1 -- 172.21.15.45:0/122676968 wait complete. 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 Processor -- start 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 -- start start 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c01023f0 0x7f22c01976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 0x7f22c019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c019c560 con 0x7f22c0102d10 2024-03-12T05:27:31.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c019c6d0 con 0x7f22c01023f0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22c5556640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c019c890 con 0x7f22c010b7c0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22be7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42292/0 (socket says 172.21.15.45:42292) 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.332+0000 7f22be7fc640 1 -- 172.21.15.45:0/233389797 learned_addr learned my addr 172.21.15.45:0/233389797 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22bf7fe640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 0x7f22c019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 -- 172.21.15.45:0/233389797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 msgr2=0x7f22c019bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22beffd640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c01023f0 0x7f22c01976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 0x7f22c019bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 -- 172.21.15.45:0/233389797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c01023f0 msgr2=0x7f22c01976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c01023f0 0x7f22c01976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 -- 172.21.15.45:0/233389797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b8009560 con 0x7f22c0102d10 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22be7fc640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c920 tx=0x7f22b400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4007c70 con 0x7f22c0102d10 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f22b4007dd0 con 0x7f22c0102d10 2024-03-12T05:27:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b40187c0 con 0x7f22c0102d10 2024-03-12T05:27:31.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c019cb70 con 0x7f22c0102d10 2024-03-12T05:27:31.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.333+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c01a43f0 con 0x7f22c0102d10 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.334+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c0067f70 con 0x7f22c0102d10 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.335+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f22b400f450 con 0x7f22c0102d10 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.336+0000 7f22a7fff640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 0x7f22a80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.336+0000 7f22beffd640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 0x7f22a80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.336+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f22b40cda30 con 0x7f22c0102d10 2024-03-12T05:27:31.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.336+0000 7f22beffd640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 0x7f22a80a40e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036040 tx=0x7f22b8005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.340+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f22b4096870 con 0x7f22c0102d10 2024-03-12T05:27:31.557 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.563+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22c01989f0 con 0x7f22c0102d10 2024-03-12T05:27:31.558 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.564+0000 7f22a7fff640 1 -- 172.21.15.45:0/233389797 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f22b4099f30 con 0x7f22c0102d10 2024-03-12T05:27:31.558 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:31.558 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:27:31.559 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 msgr2=0x7f22a80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 0x7f22a80a40e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036040 tx=0x7f22b8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 msgr2=0x7f22c0197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c920 tx=0x7f22b400cde0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 shutdown_connections 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f22a80a1c20 0x7f22a80a40e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22c0102d10 0x7f22c0197c30 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22c010b7c0 0x7f22c019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 --2- 172.21.15.45:0/233389797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22c01023f0 0x7f22c01976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.566+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 >> 172.21.15.45:0/233389797 conn(0x7f22c0077180 msgr2=0x7f22c00fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.567+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 shutdown_connections 2024-03-12T05:27:31.561 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.567+0000 7f22c5556640 1 -- 172.21.15.45:0/233389797 wait complete. 2024-03-12T05:27:31.572 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:27:31.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-12T05:27:31.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 -- 172.21.15.45:0/1086948836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc100c10 msgr2=0x7f3ffc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 --2- 172.21.15.45:0/1086948836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc100c10 0x7f3ffc104c50 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009930 tx=0x7f3fe402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:31.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 -- 172.21.15.45:0/1086948836 shutdown_connections 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 --2- 172.21.15.45:0/1086948836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc100c10 0x7f3ffc104c50 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 --2- 172.21.15.45:0/1086948836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc105330 0x7f3ffc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 --2- 172.21.15.45:0/1086948836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ffc100260 0x7f3ffc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 -- 172.21.15.45:0/1086948836 >> 172.21.15.45:0/1086948836 conn(0x7f3ffc077180 msgr2=0x7f3ffc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 -- 172.21.15.45:0/1086948836 shutdown_connections 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 -- 172.21.15.45:0/1086948836 wait complete. 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.684+0000 7f4000ef4640 1 Processor -- start 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 -- start start 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc100260 0x7f3ffc196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ffc19aee0 0x7f3ffc19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc19b8d0 con 0x7f3ffc105330 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc19ba40 con 0x7f3ffc100260 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc19bbb0 con 0x7f3ffc19aee0 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42310/0 (socket says 172.21.15.45:42310) 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 -- 172.21.15.45:0/4131942560 learned_addr learned my addr 172.21.15.45:0/4131942560 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc100260 0x7f3ffc196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 -- 172.21.15.45:0/4131942560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ffc19aee0 msgr2=0x7f3ffc19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ffc19aee0 0x7f3ffc19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 -- 172.21.15.45:0/4131942560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc100260 msgr2=0x7f3ffc196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc100260 0x7f3ffc196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 -- 172.21.15.45:0/4131942560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4009560 con 0x7f3ffc105330 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009230 tx=0x7f3fe4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff37fe640 1 -- 172.21.15.45:0/4131942560 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4004440 con 0x7f3ffc105330 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff37fe640 1 -- 172.21.15.45:0/4131942560 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3fe402de90 con 0x7f3ffc105330 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f3ff37fe640 1 -- 172.21.15.45:0/4131942560 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4005380 con 0x7f3ffc105330 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.685+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc1a3190 con 0x7f3ffc105330 2024-03-12T05:27:31.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.686+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc1a3640 con 0x7f3ffc105330 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.688+0000 7f3ff37fe640 1 -- 172.21.15.45:0/4131942560 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3fe40054e0 con 0x7f3ffc105330 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.688+0000 7f4003b9b740 1 Processor -- start 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.688+0000 7f4003b9b740 1 -- start start 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b37210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b39b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7ae9960 con 0x5617a7b2e250 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7a9d310 con 0x5617a7b37750 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f4003b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7a9a840 con 0x5617a7b2eb70 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ff37fe640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 0x7f3fe80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42326/0 (socket says 172.21.15.45:42326) 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 -- 172.21.15.45:0/2564214504 learned_addr learned my addr 172.21.15.45:0/2564214504 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 0x7f3fe80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ff37fe640 1 -- 172.21.15.45:0/4131942560 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3fe40ef8e0 con 0x7f3ffc105330 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b37210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b39b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 -- 172.21.15.45:0/2564214504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 msgr2=0x5617a7b37210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b37210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 -- 172.21.15.45:0/2564214504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 msgr2=0x5617a7b39b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b39b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 -- 172.21.15.45:0/2564214504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617a7a58060 con 0x5617a7b2e250 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b39b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40148f0 tx=0x7f3ff4014db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd71f5b1b3ae3d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.689+0000 7f3ff0ff9640 1 -- 172.21.15.45:0/2564214504 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4014070 con 0x5617a7b2e250 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f3ff0ff9640 1 -- 172.21.15.45:0/2564214504 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3ff40141d0 con 0x5617a7b2e250 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f3ff0ff9640 1 -- 172.21.15.45:0/2564214504 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff401d6c0 con 0x5617a7b2e250 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 -- 172.21.15.45:0/2564214504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 msgr2=0x5617a7b2e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 0x7f3fe80a40e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff401e000 tx=0x7f3ff401b000 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40148f0 tx=0x7f3ff4014db0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 -- 172.21.15.45:0/2564214504 shutdown_connections 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b39b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b37210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2564214504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b2e630 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 -- 172.21.15.45:0/2564214504 >> 172.21.15.45:0/2564214504 conn(0x5617a7b28980 msgr2=0x5617a7b28d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 -- 172.21.15.45:0/2564214504 shutdown_connections 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.690+0000 7f4003b9b740 1 -- 172.21.15.45:0/2564214504 wait complete. 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 -- start start 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 Processor -- start 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b34cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b351f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7a96020 con 0x5617a7b2e250 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7769430 con 0x5617a7b37750 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7725e00 con 0x5617a7b2eb70 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f4003b9b740 10 client.0 _ll_register_callbacks cb 0x5617a7a3dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42328/0 (socket says 172.21.15.45:42328) 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 -- 172.21.15.45:0/1919040370 learned_addr learned my addr 172.21.15.45:0/1919040370 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b34cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 -- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 msgr2=0x5617a7b34cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b34cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 -- 172.21.15.45:0/1919040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 msgr2=0x5617a7b351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffa59c640 1 -- 172.21.15.45:0/1919040370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff40144d0 con 0x5617a7b2e250 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b34cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.691+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0138a0 tx=0x7f3fec013d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3ac577520e25f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec018070 con 0x5617a7b2e250 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3fec00fb30 con 0x5617a7b2e250 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617a77957e0 con 0x5617a7b2e250 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fec009780 con 0x5617a7b2e250 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 10 client.19602 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5617a7a946a0 con 0x5617a7b2e250 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5617a77957e0 con 0x5617a7b2e250 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3fec024430 con 0x5617a7b2e250 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 10 client.19602 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 10 client.19602 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 20 client.19602 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b47b50 0x5617a7b49f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 4 client.19602 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.692+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617a77d2720 con 0x5617a7b47b50 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.693+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b47b50 0x5617a7b49f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.693+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b47b50 0x5617a7b49f40 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=690bd70d6c79c666 server_cookie=afae2627e5e040f5 in_seq=0 out_seq=0 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.693+0000 7f3fe1ffb640 10 client.19602 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:27:31.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1919040370 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5617a77d2720 con 0x5617a7b47b50 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f3fe1ffb640 10 client.19602 handle_command_reply: tid=0 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 client.19602 shutdown 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 msgr2=0x5617a7b34770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0138a0 tx=0x7f3fec013d60 comp rx=0 tx=0).stop 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 shutdown_connections 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b47b50 0x5617a7b49f40 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f3fe1ffb640 0 client.19602 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2eb70 0x5617a7b34cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1919040370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2e250 0x5617a7b34770 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 >> 172.21.15.45:0/1919040370 conn(0x5617a7b28980 msgr2=0x5617a7b2a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.698 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 shutdown_connections 2024-03-12T05:27:31.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 1 -- 172.21.15.45:0/1919040370 wait complete. 2024-03-12T05:27:31.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.694+0000 7f4003b9b740 20 client.19602 trim_cache size 0 max 16384 2024-03-12T05:27:31.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 Processor -- start 2024-03-12T05:27:31.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 -- start start 2024-03-12T05:27:31.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b36ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b2fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7a1b130 con 0x5617a7b2eb70 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7b32cc0 con 0x5617a7b37750 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f4003b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7b35550 con 0x5617a7b2e250 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42334/0 (socket says 172.21.15.45:42334) 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 -- 172.21.15.45:0/2794089183 learned_addr learned my addr 172.21.15.45:0/2794089183 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 -- 172.21.15.45:0/2794089183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 msgr2=0x5617a7b36ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b36ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b36ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 -- 172.21.15.45:0/2794089183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 msgr2=0x5617a7b2fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b2fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.706+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b2fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3ff3fff640 1 -- 172.21.15.45:0/2794089183 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617a7a30c20 con 0x5617a7b2eb70 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b2fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40ed170 tx=0x7f3fe408f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c3db9f78037801 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3fe3fff640 1 -- 172.21.15.45:0/2794089183 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40bc5f0 con 0x5617a7b2eb70 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3fe3fff640 1 -- 172.21.15.45:0/2794089183 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3fe40c0740 con 0x5617a7b2eb70 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f3fe3fff640 1 -- 172.21.15.45:0/2794089183 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40a70b0 con 0x5617a7b2eb70 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 -- 172.21.15.45:0/2794089183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 msgr2=0x5617a7b35010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40ed170 tx=0x7f3fe408f8a0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 -- 172.21.15.45:0/2794089183 shutdown_connections 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b2eb70 0x5617a7b35010 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b37750 0x5617a7b2fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 --2- 172.21.15.45:0/2794089183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b36ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 -- 172.21.15.45:0/2794089183 >> 172.21.15.45:0/2794089183 conn(0x5617a7b28980 msgr2=0x5617a7b2ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 -- 172.21.15.45:0/2794089183 shutdown_connections 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.707+0000 7f4003b9b740 1 -- 172.21.15.45:0/2794089183 wait complete. 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 -- start start 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 Processor -- start 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b3d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b2eb70 0x5617a7b3d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7b9e160 con 0x5617a7b37750 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7b9e2d0 con 0x5617a7b2eb70 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617a7b9e490 con 0x5617a7b2e250 2024-03-12T05:27:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f4003b9b740 10 client.0 _ll_register_callbacks cb 0x7f3ff4009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42338/0 (socket says 172.21.15.45:42338) 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 -- 172.21.15.45:0/1827855430 learned_addr learned my addr 172.21.15.45:0/1827855430 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ff3fff640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b2eb70 0x5617a7b3d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b3d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 -- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 msgr2=0x5617a7b3d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b3d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 -- 172.21.15.45:0/1827855430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b2eb70 msgr2=0x5617a7b3d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b2eb70 0x5617a7b3d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffad9d640 1 -- 172.21.15.45:0/1827855430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc1a3640 con 0x5617a7b37750 2024-03-12T05:27:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.708+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b3d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3ffad9d640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4012010 tx=0x7f3ff4004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6de48e11bcd4d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff401dda0 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3ff4009df0 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ff40141d0 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ff4009750 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 10 client.19608 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5617a7a568c0 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3ff403c020 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3ff4005040 con 0x5617a7b37750 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 10 client.19608 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 10 client.19608 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-12T05:27:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 20 client.19608 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b9fb90 0x5617a7ba1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 4 client.19608 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 --> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5617a7821890 con 0x5617a7b9fb90 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.709+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b9fb90 0x5617a7ba1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f3fe1ffb640 10 client.19608 ms_handle_connect on v2:172.21.15.130:6834/1592150426 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f3ffa59c640 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b9fb90 0x5617a7ba1f80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fceb4178fe85ad8b server_cookie=7a76a0209a3cf267 in_seq=0 out_seq=0 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f3fe1ffb640 1 -- 172.21.15.45:0/1827855430 <== mds.4 v2:172.21.15.130:6834/1592150426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5617a7821890 con 0x5617a7b9fb90 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f3fe1ffb640 10 client.19608 handle_command_reply: tid=0 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 client.19608 shutdown 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 msgr2=0x5617a7b3dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4012010 tx=0x7f3ff4004dd0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 shutdown_connections 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:6834/1592150426,v1:172.21.15.130:6835/1592150426] conn(0x5617a7b9fb90 0x5617a7ba1f80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5617a7b37750 0x5617a7b3dcc0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617a7b2eb70 0x5617a7b3d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f3fe1ffb640 0 client.19608 ms_handle_reset on v2:172.21.15.130:6834/1592150426 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 --2- 172.21.15.45:0/1827855430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617a7b2e250 0x5617a7b3d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 >> 172.21.15.45:0/1827855430 conn(0x5617a7b28980 msgr2=0x5617a7b2fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 shutdown_connections 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 1 -- 172.21.15.45:0/1827855430 wait complete. 2024-03-12T05:27:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.710+0000 7f4003b9b740 20 client.19608 trim_cache size 0 max 16384 2024-03-12T05:27:31.714 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 msgr2=0x7f3fe80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 0x7f3fe80a40e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff401e000 tx=0x7f3ff401b000 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 msgr2=0x7f3ffc196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4009230 tx=0x7f3fe4004040 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 shutdown_connections 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3fe80a1c20 0x7f3fe80a40e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ffc105330 0x7f3ffc196b60 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ffc19aee0 0x7f3ffc19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 --2- 172.21.15.45:0/4131942560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ffc100260 0x7f3ffc196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 >> 172.21.15.45:0/4131942560 conn(0x7f3ffc077180 msgr2=0x7f3ffc0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 shutdown_connections 2024-03-12T05:27:31.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.711+0000 7f4000ef4640 1 -- 172.21.15.45:0/4131942560 wait complete. 2024-03-12T05:27:31.717 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:27:31.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-12T05:27:31.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- 172.21.15.45:0/3233783781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100c30 msgr2=0x7f0830104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3233783781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100c30 0x7f0830104c70 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0824009930 tx=0x7f082402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:31.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- 172.21.15.45:0/3233783781 shutdown_connections 2024-03-12T05:27:31.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3233783781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100c30 0x7f0830104c70 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3233783781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0830105350 0x7f083010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3233783781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830100280 0x7f0830100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- 172.21.15.45:0/3233783781 >> 172.21.15.45:0/3233783781 conn(0x7f0830077150 msgr2=0x7f0830077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- 172.21.15.45:0/3233783781 shutdown_connections 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- 172.21.15.45:0/3233783781 wait complete. 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 Processor -- start 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.828+0000 7f0836d4b640 1 -- start start 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 0x7f0830196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 0x7f083019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083019b8b0 con 0x7f0830100280 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083019ba20 con 0x7f0830105350 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083019bb90 con 0x7f083019aec0 2024-03-12T05:27:31.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42360/0 (socket says 172.21.15.45:42360) 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 -- 172.21.15.45:0/3188501696 learned_addr learned my addr 172.21.15.45:0/3188501696 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f082ffff640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 0x7f0830196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f08352e8640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 0x7f083019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 -- 172.21.15.45:0/3188501696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 msgr2=0x7f083019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 0x7f083019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 -- 172.21.15.45:0/3188501696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 msgr2=0x7f0830196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 0x7f0830196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 -- 172.21.15.45:0/3188501696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824009560 con 0x7f0830100280 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f08352e8640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 0x7f083019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f082ffff640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 0x7f0830196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f082800eab0 tx=0x7f082800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f082dffb640 1 -- 172.21.15.45:0/3188501696 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082800ce40 con 0x7f0830100280 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f082dffb640 1 -- 172.21.15.45:0/3188501696 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0828011920 con 0x7f0830100280 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f082dffb640 1 -- 172.21.15.45:0/3188501696 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280109d0 con 0x7f0830100280 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.829+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08301a3170 con 0x7f0830100280 2024-03-12T05:27:31.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.830+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08301a3640 con 0x7f0830100280 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f082dffb640 1 -- 172.21.15.45:0/3188501696 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0828011a90 con 0x7f0830100280 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f08399f5740 1 Processor -- start 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f08399f5740 1 -- start start 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f082dffb640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 0x7f08180a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f08399f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f08399f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a5287210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.832+0000 7f08399f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5289b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08399f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a5239960 con 0x5605a527e250 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08399f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a51ed310 con 0x5605a5287750 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08399f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a51ea840 con 0x5605a527eb70 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08352e8640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 0x7f08180a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42370/0 (socket says 172.21.15.45:42370) 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 -- 172.21.15.45:0/923225807 learned_addr learned my addr 172.21.15.45:0/923225807 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f082ffff640 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a5287210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f082dffb640 1 -- 172.21.15.45:0/3188501696 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f08280cdef0 con 0x7f0830100280 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08352e8640 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5289b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 -- 172.21.15.45:0/923225807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 msgr2=0x5605a5287210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a5287210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 -- 172.21.15.45:0/923225807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 msgr2=0x5605a5289b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5289b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 -- 172.21.15.45:0/923225807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a51a8060 con 0x5605a527e250 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f08352e8640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 0x7f08180a40e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0820005d80 tx=0x7f08200157f0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0834ae7640 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f082809aa20 tx=0x7f08280094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a33e5870176c754 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0816ffd640 1 -- 172.21.15.45:0/923225807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280aa0b0 con 0x5605a527e250 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0816ffd640 1 -- 172.21.15.45:0/923225807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0828011de0 con 0x5605a527e250 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.833+0000 7f0816ffd640 1 -- 172.21.15.45:0/923225807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828097100 con 0x5605a527e250 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- 172.21.15.45:0/923225807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 msgr2=0x5605a527e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f082809aa20 tx=0x7f08280094a0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- 172.21.15.45:0/923225807 shutdown_connections 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5289b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a5287210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 --2- 172.21.15.45:0/923225807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a527e630 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- 172.21.15.45:0/923225807 >> 172.21.15.45:0/923225807 conn(0x5605a5278980 msgr2=0x5605a5278d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- 172.21.15.45:0/923225807 shutdown_connections 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- 172.21.15.45:0/923225807 wait complete. 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 -- start start 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.834+0000 7f08399f5740 1 Processor -- start 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a52956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a51e6020 con 0x5605a527e250 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a4eb9430 con 0x5605a5287750 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a4e75e00 con 0x5605a527eb70 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08399f5740 10 client.0 _ll_register_callbacks cb 0x5605a518dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42384/0 (socket says 172.21.15.45:42384) 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 -- 172.21.15.45:0/3294530631 learned_addr learned my addr 172.21.15.45:0/3294530631 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f082ffff640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f08352e8640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a52956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 -- 172.21.15.45:0/3294530631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 msgr2=0x5605a528ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 -- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 msgr2=0x5605a52956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a52956d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 -- 172.21.15.45:0/3294530631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08301a3640 con 0x5605a527e250 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f082ffff640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0828096ef0 tx=0x7f0828098fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4647c2430bc1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.835+0000 7f080ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082809aef0 con 0x5605a527e250 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f082809aa90 con 0x5605a527e250 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605a4ee57e0 con 0x5605a527e250 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08280aa0b0 con 0x5605a527e250 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 10 client.19620 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5605a51e46a0 con 0x5605a527e250 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x5605a4ee57e0 con 0x5605a527e250 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0828021070 con 0x5605a527e250 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 10 client.19620 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 10 client.19620 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 20 client.19620 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a5299970 0x5605a529bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 4 client.19620 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605a4f22720 con 0x5605a5299970 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.836+0000 7f082ffff640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a5299970 0x5605a529bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f080ffff640 10 client.19620 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f082ffff640 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a5299970 0x5605a529bd60 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a1ea3b342a036c4e server_cookie=e1935e1d9dc6068c in_seq=0 out_seq=0 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f080ffff640 1 -- 172.21.15.45:0/3294530631 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5605a4f22720 con 0x5605a5299970 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f080ffff640 10 client.19620 handle_command_reply: tid=0 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f08399f5740 1 client.19620 shutdown 2024-03-12T05:27:31.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 msgr2=0x5605a528a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.837+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0828096ef0 tx=0x7f0828098fa0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 shutdown_connections 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a5299970 0x5605a529bd60 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a52956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f080ffff640 0 client.19620 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 --2- 172.21.15.45:0/3294530631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528a650 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 >> 172.21.15.45:0/3294530631 conn(0x5605a5278980 msgr2=0x5605a527a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 shutdown_connections 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 1 -- 172.21.15.45:0/3294530631 wait complete. 2024-03-12T05:27:31.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.838+0000 7f08399f5740 20 client.19620 trim_cache size 0 max 16384 2024-03-12T05:27:31.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.849+0000 7f08399f5740 1 Processor -- start 2024-03-12T05:27:31.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.849+0000 7f08399f5740 1 -- start start 2024-03-12T05:27:31.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.851 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5294fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a516b130 con 0x5605a527e250 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a5286b50 con 0x5605a5287750 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08399f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a52a0f70 con 0x5605a527eb70 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42392/0 (socket says 172.21.15.45:42392) 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 -- 172.21.15.45:0/1398753082 learned_addr learned my addr 172.21.15.45:0/1398753082 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.852 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f082ffff640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 -- 172.21.15.45:0/1398753082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 msgr2=0x5605a528fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08352e8640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5294fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 -- 172.21.15.45:0/1398753082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 msgr2=0x5605a5294fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5294fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 -- 172.21.15.45:0/1398753082 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a5180c20 con 0x5605a527e250 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f082ffff640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08352e8640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5294fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f0834ae7640 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0828097410 tx=0x7f082809a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec63481ca29d3259 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.853 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08167fc640 1 -- 172.21.15.45:0/1398753082 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280117c0 con 0x5605a527e250 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08167fc640 1 -- 172.21.15.45:0/1398753082 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f08280cba40 con 0x5605a527e250 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.850+0000 7f08167fc640 1 -- 172.21.15.45:0/1398753082 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082809af50 con 0x5605a527e250 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- 172.21.15.45:0/1398753082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 msgr2=0x5605a528f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0828097410 tx=0x7f082809a970 comp rx=0 tx=0).stop 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- 172.21.15.45:0/1398753082 shutdown_connections 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a5294fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527eb70 0x5605a528fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 --2- 172.21.15.45:0/1398753082 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527e250 0x5605a528f4f0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- 172.21.15.45:0/1398753082 >> 172.21.15.45:0/1398753082 conn(0x5605a5278980 msgr2=0x5605a528a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- 172.21.15.45:0/1398753082 shutdown_connections 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- 172.21.15.45:0/1398753082 wait complete. 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 -- start start 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.851+0000 7f08399f5740 1 Processor -- start 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 0x5605a528f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a528e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a528ea60 con 0x5605a527eb70 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a528ebd0 con 0x5605a5287750 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a52efed0 con 0x5605a527e250 2024-03-12T05:27:31.855 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 10 client.0 _ll_register_callbacks cb 0x7f08280966f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42398/0 (socket says 172.21.15.45:42398) 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 -- 172.21.15.45:0/3116156575 learned_addr learned my addr 172.21.15.45:0/3116156575 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08352e8640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a528e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 0x5605a528f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 -- 172.21.15.45:0/3116156575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 msgr2=0x5605a528f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 0x5605a528f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 -- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 msgr2=0x5605a528e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a528e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 -- 172.21.15.45:0/3116156575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605a51e46a0 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 0x5605a528f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f082ffff640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f082402bc70 tx=0x7f0824004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197ed30b44a83fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f080ffff640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08240042c0 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f0824044040 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.852+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0828011de0 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0824004420 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 10 client.19626 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5605a51a68c0 con 0x5605a527eb70 2024-03-12T05:27:31.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f0828011de0 con 0x5605a527eb70 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f082404f020 con 0x5605a527eb70 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 10 client.19626 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 10 client.19626 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 20 client.19626 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a52f1710 0x5605a52f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 4 client.19626 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 --> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5605a4f71890 con 0x5605a52f1710 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.853+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a52f1710 0x5605a52f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f080ffff640 10 client.19626 ms_handle_connect on v2:172.21.15.116:6836/2581569461 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f0834ae7640 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a52f1710 0x5605a52f3b30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=578aa8114216e9f4 server_cookie=6e01ba962efc6a71 in_seq=0 out_seq=0 2024-03-12T05:27:31.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f080ffff640 1 -- 172.21.15.45:0/3116156575 <== mds.3 v2:172.21.15.116:6836/2581569461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5605a4f71890 con 0x5605a52f1710 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f080ffff640 10 client.19626 handle_command_reply: tid=0 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 client.19626 shutdown 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 msgr2=0x5605a528dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f082402bc70 tx=0x7f0824004b00 comp rx=0 tx=0).stop 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 shutdown_connections 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:6836/2581569461,v1:172.21.15.116:6837/2581569461] conn(0x5605a52f1710 0x5605a52f3b30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x5605a527eb70 0x5605a528dd10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605a5287750 0x5605a528e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 --2- 172.21.15.45:0/3116156575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605a527e250 0x5605a528f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f080ffff640 0 client.19626 ms_handle_reset on v2:172.21.15.116:6836/2581569461 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 >> 172.21.15.45:0/3116156575 conn(0x5605a5278980 msgr2=0x5605a52830b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 shutdown_connections 2024-03-12T05:27:31.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 1 -- 172.21.15.45:0/3116156575 wait complete. 2024-03-12T05:27:31.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.854+0000 7f08399f5740 20 client.19626 trim_cache size 0 max 16384 2024-03-12T05:27:31.860 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:27:31.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 msgr2=0x7f08180a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 0x7f08180a40e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0820005d80 tx=0x7f08200157f0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 msgr2=0x7f0830196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f082800eab0 tx=0x7f082800ef70 comp rx=0 tx=0).stop 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 shutdown_connections 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f08180a1c20 0x7f08180a40e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f083019aec0 0x7f083019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0830105350 0x7f0830196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 --2- 172.21.15.45:0/3188501696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0830100280 0x7f0830196600 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 >> 172.21.15.45:0/3188501696 conn(0x7f0830077150 msgr2=0x7f08300fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 shutdown_connections 2024-03-12T05:27:31.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.855+0000 7f0836d4b640 1 -- 172.21.15.45:0/3188501696 wait complete. 2024-03-12T05:27:31.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:31 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/233389797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:27:31.863 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:27:31.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:27:31.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:31 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/233389797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:27:31.968 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/3527788841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c102d10 msgr2=0x7f3b9c10b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/3527788841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c102d10 0x7f3b9c10b1f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009900 tx=0x7f3b8402bcd0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/3527788841 shutdown_connections 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/3527788841 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c102d10 0x7f3b9c10b1f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/3527788841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 0x7f3b9c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/3527788841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/3527788841 >> 172.21.15.45:0/3527788841 conn(0x7f3b9c0771e0 msgr2=0x7f3b9c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/3527788841 shutdown_connections 2024-03-12T05:27:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.974+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/3527788841 wait complete. 2024-03-12T05:27:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 Processor -- start 2024-03-12T05:27:31.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 -- start start 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 0x7f3b9c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 0x7f3b9c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19c5a0 con 0x7f3b9c1023f0 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19c710 con 0x7f3b9c10b7c0 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3ba0f4f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c19c8d0 con 0x7f3b9c102d10 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42428/0 (socket says 172.21.15.45:42428) 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 -- 172.21.15.45:0/705743019 learned_addr learned my addr 172.21.15.45:0/705743019 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b93fff640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 0x7f3b9c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 0x7f3b9c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 -- 172.21.15.45:0/705743019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 msgr2=0x7f3b9c197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 0x7f3b9c197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 -- 172.21.15.45:0/705743019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 msgr2=0x7f3b9c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 0x7f3b9c19bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b9a59c640 1 -- 172.21.15.45:0/705743019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b84009560 con 0x7f3b9c1023f0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.975+0000 7f3b93fff640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 0x7f3b9c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c970 tx=0x7f3b8c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3b937fe640 1 -- 172.21.15.45:0/705743019 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c007bb0 con 0x7f3b9c1023f0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3b937fe640 1 -- 172.21.15.45:0/705743019 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b8c007d10 con 0x7f3b9c1023f0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c19cbb0 con 0x7f3b9c1023f0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3b937fe640 1 -- 172.21.15.45:0/705743019 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c004e80 con 0x7f3b9c1023f0 2024-03-12T05:27:31.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.976+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c1a4430 con 0x7f3b9c1023f0 2024-03-12T05:27:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.978+0000 7f3b937fe640 1 -- 172.21.15.45:0/705743019 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3b8c005020 con 0x7f3b9c1023f0 2024-03-12T05:27:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.978+0000 7f3ba3c04740 1 Processor -- start 2024-03-12T05:27:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.978+0000 7f3ba3c04740 1 -- start start 2024-03-12T05:27:31.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a51e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a527210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a4d9960 con 0x55621a527750 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a48d310 con 0x55621a51eb70 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a48a840 con 0x55621a51e250 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a51e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42444/0 (socket says 172.21.15.45:42444) 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 -- 172.21.15.45:0/2658034128 learned_addr learned my addr 172.21.15.45:0/2658034128 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b937fe640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 0x7f3b880a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a527210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 -- 172.21.15.45:0/2658034128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 msgr2=0x55621a51e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a51e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 0x7f3b880a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 -- 172.21.15.45:0/2658034128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 msgr2=0x55621a527210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a527210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 -- 172.21.15.45:0/2658034128 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55621a448060 con 0x55621a527750 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b937fe640 1 -- 172.21.15.45:0/705743019 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3b8c0cd6c0 con 0x7f3b9c1023f0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3b84036040 tx=0x7f3b840047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cffdf4ab14cf41 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 0x7f3b880a4060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3b94005cc0 tx=0x7f3b94004ab0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.979+0000 7f3b90ff9640 1 -- 172.21.15.45:0/2658034128 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8402c810 con 0x55621a527750 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3b90ff9640 1 -- 172.21.15.45:0/2658034128 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b840044f0 con 0x55621a527750 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3b90ff9640 1 -- 172.21.15.45:0/2658034128 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84003b90 con 0x55621a527750 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2658034128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 msgr2=0x55621a529b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3b84036040 tx=0x7f3b840047f0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2658034128 shutdown_connections 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a529b40 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a527210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2658034128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a51e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2658034128 >> 172.21.15.45:0/2658034128 conn(0x55621a518980 msgr2=0x55621a518d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2658034128 shutdown_connections 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.980+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2658034128 wait complete. 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 -- start start 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 Processor -- start 2024-03-12T05:27:31.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a524770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a524cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a486020 con 0x55621a527750 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a159430 con 0x55621a51eb70 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a115e00 con 0x55621a51e250 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3ba3c04740 10 client.0 _ll_register_callbacks cb 0x55621a42dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42446/0 (socket says 172.21.15.45:42446) 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 -- 172.21.15.45:0/128175341 learned_addr learned my addr 172.21.15.45:0/128175341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a524770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b93fff640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a524cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 -- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 msgr2=0x55621a524770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a524770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 -- 172.21.15.45:0/128175341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 msgr2=0x55621a524cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a524cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b9ad9d640 1 -- 172.21.15.45:0/128175341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c10e8b0 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.981+0000 7f3b93fff640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a524cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3b94011f30 tx=0x7f3b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121aa6595ed43698 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9401d810 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3b9401d970 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55621a1857e0 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b9401db30 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 10 client.19638 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55621a4846a0 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55621a1857e0 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3b94027cd0 con 0x55621a527750 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 10 client.19638 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 10 client.19638 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:27:31.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 20 client.19638 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a537ba0 0x55621a539fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 4 client.19638 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.982+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55621a1c2720 con 0x55621a537ba0 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.983+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a537ba0 0x55621a539fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.983+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a537ba0 0x55621a539fc0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dedc2a66155a2a38 server_cookie=40211e13fbf7f483 in_seq=0 out_seq=0 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.983+0000 7f3b81ffb640 10 client.19638 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3b81ffb640 1 -- 172.21.15.45:0/128175341 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55621a1c2720 con 0x55621a537ba0 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3b81ffb640 10 client.19638 handle_command_reply: tid=0 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 client.19638 shutdown 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 msgr2=0x55621a5251f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3b94011f30 tx=0x7f3b94004a40 comp rx=0 tx=0).stop 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 shutdown_connections 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a537ba0 0x55621a539fc0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a527750 0x55621a5251f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a51eb70 0x55621a524cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/128175341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a524770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3b81ffb640 0 client.19638 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 >> 172.21.15.45:0/128175341 conn(0x55621a518980 msgr2=0x55621a51a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 shutdown_connections 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 1 -- 172.21.15.45:0/128175341 wait complete. 2024-03-12T05:27:31.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.984+0000 7f3ba3c04740 20 client.19638 trim_cache size 0 max 16384 2024-03-12T05:27:31.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:31 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/233389797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 Processor -- start 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 -- start start 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51eb70 0x55621a517a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a52e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a40b130 con 0x55621a51e250 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a51d5d0 con 0x55621a527750 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a53f210 con 0x55621a51eb70 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3b9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.996+0000 7f3b9a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42456/0 (socket says 172.21.15.45:42456) 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 -- 172.21.15.45:0/2792010750 learned_addr learned my addr 172.21.15.45:0/2792010750 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:31.997 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a52e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51eb70 0x55621a517a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 -- 172.21.15.45:0/2792010750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51eb70 msgr2=0x55621a517a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51eb70 0x55621a517a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 -- 172.21.15.45:0/2792010750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 msgr2=0x55621a52e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a52e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 -- 172.21.15.45:0/2792010750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55621a420c20 con 0x55621a51e250 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c099f60 tx=0x7f3b8c0cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426411f1e61d0f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b83fff640 1 -- 172.21.15.45:0/2792010750 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c09e3b0 con 0x55621a51e250 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b83fff640 1 -- 172.21.15.45:0/2792010750 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b8c0a2a00 con 0x55621a51e250 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.997+0000 7f3b83fff640 1 -- 172.21.15.45:0/2792010750 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c0052d0 con 0x55621a51e250 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2792010750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 msgr2=0x55621a525010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c099f60 tx=0x7f3b8c0cf040 comp rx=0 tx=0).stop 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2792010750 shutdown_connections 2024-03-12T05:27:31.998 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a52e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51eb70 0x55621a517a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2792010750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51e250 0x55621a525010 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2792010750 >> 172.21.15.45:0/2792010750 conn(0x55621a518980 msgr2=0x55621a5172f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2792010750 shutdown_connections 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2792010750 wait complete. 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- start start 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 Processor -- start 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a52d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a588130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a588670 con 0x55621a51eb70 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a5887b0 con 0x55621a527750 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621a588970 con 0x55621a51e250 2024-03-12T05:27:31.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3ba3c04740 10 client.0 _ll_register_callbacks cb 0x7f3b84009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3b93fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42468/0 (socket says 172.21.15.45:42468) 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.998+0000 7f3b93fff640 1 -- 172.21.15.45:0/2274035725 learned_addr learned my addr 172.21.15.45:0/2274035725 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a588130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a52d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 -- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 msgr2=0x55621a52d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a52d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 -- 172.21.15.45:0/2274035725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 msgr2=0x55621a588130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a588130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 -- 172.21.15.45:0/2274035725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c05fe70 con 0x55621a51eb70 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b9ad9d640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a588130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b93fff640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3b84005ab0 tx=0x7f3b8402c290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ba49a5b98b0011 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8404c070 con 0x55621a51eb70 2024-03-12T05:27:32.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f3b8402ce40 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b840044f0 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b8402c350 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3ba3c04740 10 client.19644 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55621a4468c0 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:31.999+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3b840044f0 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3b8403c040 con 0x55621a51eb70 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 10 client.19644 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 10 client.19644 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 20 client.19644 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a58ff00 0x55621a592320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 4 client.19644 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 --> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55621a211890 con 0x55621a58ff00 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a58ff00 0x55621a592320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3b81ffb640 10 client.19644 ms_handle_connect on v2:172.21.15.130:6836/2852741522 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.000+0000 7f3b9a59c640 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a58ff00 0x55621a592320 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da1da3903747a97e server_cookie=de1c4eb530791549 in_seq=0 out_seq=0 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3b81ffb640 1 -- 172.21.15.45:0/2274035725 <== mds.2 v2:172.21.15.130:6836/2852741522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55621a211890 con 0x55621a58ff00 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3b81ffb640 10 client.19644 handle_command_reply: tid=0 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 client.19644 shutdown 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 msgr2=0x55621a52de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3b84005ab0 tx=0x7f3b8402c290 comp rx=0 tx=0).stop 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 shutdown_connections 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:6836/2852741522,v1:172.21.15.130:6837/2852741522] conn(0x55621a58ff00 0x55621a592320 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55621a51eb70 0x55621a52de30 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55621a527750 0x55621a588130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3b81ffb640 0 client.19644 ms_handle_reset on v2:172.21.15.130:6836/2852741522 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 --2- 172.21.15.45:0/2274035725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55621a51e250 0x55621a52d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 >> 172.21.15.45:0/2274035725 conn(0x55621a518980 msgr2=0x55621a517b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 shutdown_connections 2024-03-12T05:27:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 1 -- 172.21.15.45:0/2274035725 wait complete. 2024-03-12T05:27:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba3c04740 20 client.19644 trim_cache size 0 max 16384 2024-03-12T05:27:32.003 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:27:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 msgr2=0x7f3b880a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 0x7f3b880a4060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3b94005cc0 tx=0x7f3b94004ab0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 msgr2=0x7f3b9c1976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c970 tx=0x7f3b8c00ce30 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 shutdown_connections 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3b880a1ba0 0x7f3b880a4060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9c10b7c0 0x7f3b9c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b9c102d10 0x7f3b9c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 --2- 172.21.15.45:0/705743019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b9c1023f0 0x7f3b9c1976e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 >> 172.21.15.45:0/705743019 conn(0x7f3b9c0771e0 msgr2=0x7f3b9c0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 shutdown_connections 2024-03-12T05:27:32.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.001+0000 7f3ba0f4f640 1 -- 172.21.15.45:0/705743019 wait complete. 2024-03-12T05:27:32.006 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:27:32.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:27:32.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 -- 172.21.15.45:0/3440858800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4101d30 msgr2=0x7f07e4105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 --2- 172.21.15.45:0/3440858800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4101d30 0x7f07e4105d70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009930 tx=0x7f07d402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 -- 172.21.15.45:0/3440858800 shutdown_connections 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 --2- 172.21.15.45:0/3440858800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4101d30 0x7f07e4105d70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 --2- 172.21.15.45:0/3440858800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e4106450 0x7f07e410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 --2- 172.21.15.45:0/3440858800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07e4101380 0x7f07e4101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 -- 172.21.15.45:0/3440858800 >> 172.21.15.45:0/3440858800 conn(0x7f07e4078580 msgr2=0x7f07e4078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 -- 172.21.15.45:0/3440858800 shutdown_connections 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.117+0000 7f07ec24e640 1 -- 172.21.15.45:0/3440858800 wait complete. 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 Processor -- start 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 -- start start 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e4101380 0x7f07e4073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07e4073f00 0x7f07e41a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e41a5e80 con 0x7f07e4106450 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e41a5ff0 con 0x7f07e4101380 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07ec24e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e41a61b0 con 0x7f07e4073f00 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42480/0 (socket says 172.21.15.45:42480) 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 -- 172.21.15.45:0/1911167789 learned_addr learned my addr 172.21.15.45:0/1911167789 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 -- 172.21.15.45:0/1911167789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07e4073f00 msgr2=0x7f07e41a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07e4073f00 0x7f07e41a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 -- 172.21.15.45:0/1911167789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e4101380 msgr2=0x7f07e4073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e4101380 0x7f07e4073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 -- 172.21.15.45:0/1911167789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4009560 con 0x7f07e4106450 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.118+0000 7f07e97e9640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f07d4000c00 tx=0x7f07d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.119+0000 7f07daffd640 1 -- 172.21.15.45:0/1911167789 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d402cb70 con 0x7f07e4106450 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.119+0000 7f07daffd640 1 -- 172.21.15.45:0/1911167789 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07d402de90 con 0x7f07e4106450 2024-03-12T05:27:32.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.119+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e41a6430 con 0x7f07e4106450 2024-03-12T05:27:32.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.119+0000 7f07daffd640 1 -- 172.21.15.45:0/1911167789 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d40053c0 con 0x7f07e4106450 2024-03-12T05:27:32.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.119+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e41a6990 con 0x7f07e4106450 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.121+0000 7f07daffd640 1 -- 172.21.15.45:0/1911167789 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f07d4005520 con 0x7f07e4106450 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.121+0000 7f07eee27740 1 Processor -- start 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.121+0000 7f07eee27740 1 -- start start 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e17210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07daffd640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 0x7f07d00a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211dc9960 con 0x55a211e0e250 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211d7d310 con 0x55a211e17750 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07eee27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211d7a840 con 0x55a211e0eb70 2024-03-12T05:27:32.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07e97e9640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 0x7f07d00a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42482/0 (socket says 172.21.15.45:42482) 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 -- 172.21.15.45:0/315528936 learned_addr learned my addr 172.21.15.45:0/315528936 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07e9fea640 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e17210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07daffd640 1 -- 172.21.15.45:0/1911167789 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f07d40ef910 con 0x7f07e4106450 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e19b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 -- 172.21.15.45:0/315528936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 msgr2=0x55a211e17210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e17210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 -- 172.21.15.45:0/315528936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 msgr2=0x55a211e19b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e19b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 -- 172.21.15.45:0/315528936 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a211d38060 con 0x55a211e0e250 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f07dc014ad0 tx=0x7f07dc009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e80ec090a0ddaa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.122+0000 7f07e97e9640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 0x7f07d00a4110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07d40ed1f0 tx=0x7f07d40a7040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07cffff640 1 -- 172.21.15.45:0/315528936 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc003b70 con 0x55a211e0e250 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07cffff640 1 -- 172.21.15.45:0/315528936 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07dc003cd0 con 0x55a211e0e250 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07cffff640 1 -- 172.21.15.45:0/315528936 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0044e0 con 0x55a211e0e250 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 -- 172.21.15.45:0/315528936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 msgr2=0x55a211e0e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f07dc014ad0 tx=0x7f07dc009650 comp rx=0 tx=0).stop 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 -- 172.21.15.45:0/315528936 shutdown_connections 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e17210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 --2- 172.21.15.45:0/315528936 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e0e630 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 -- 172.21.15.45:0/315528936 >> 172.21.15.45:0/315528936 conn(0x55a211e08980 msgr2=0x55a211e08d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 -- 172.21.15.45:0/315528936 shutdown_connections 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.123+0000 7f07eee27740 1 -- 172.21.15.45:0/315528936 wait complete. 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 -- start start 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 Processor -- start 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0e250 0x55a211e14700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.124 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e14c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211d76020 con 0x55a211e17750 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211a49430 con 0x55a211e0e250 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211a05e00 con 0x55a211e0eb70 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07eee27740 10 client.0 _ll_register_callbacks cb 0x55a211d1dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42494/0 (socket says 172.21.15.45:42494) 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07ea7eb640 1 -- 172.21.15.45:0/3419933635 learned_addr learned my addr 172.21.15.45:0/3419933635 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0e250 0x55a211e14700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.124+0000 7f07e9fea640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e14c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 -- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 msgr2=0x55a211e14c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e14c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 -- 172.21.15.45:0/3419933635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0e250 msgr2=0x55a211e14700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0e250 0x55a211e14700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 -- 172.21.15.45:0/3419933635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc0147b0 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009750 tx=0x7f07dc004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c11c53f9514de13 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc004670 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f07dc0047d0 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a211a757e0 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07dc00b720 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 10 client.19656 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a211d746a0 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x55a211a757e0 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f07dc01c070 con 0x55a211e17750 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 10 client.19656 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:32.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 10 client.19656 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 20 client.19656 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.125+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e27a30 0x55a211e29e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.126+0000 7f07eee27740 4 client.19656 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.126+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a211ab2720 con 0x55a211e27a30 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.126+0000 7f07e9fea640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e27a30 0x55a211e29e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.126+0000 7f07cd7fa640 10 client.19656 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.126+0000 7f07e9fea640 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e27a30 0x55a211e29e50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad03dca9fd0b85cd server_cookie=aaeb66d9ddb00587 in_seq=0 out_seq=0 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07cd7fa640 1 -- 172.21.15.45:0/3419933635 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a211ab2720 con 0x55a211e27a30 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07cd7fa640 10 client.19656 handle_command_reply: tid=0 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 client.19656 shutdown 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 msgr2=0x55a211e15180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009750 tx=0x7f07dc004020 comp rx=0 tx=0).stop 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 shutdown_connections 2024-03-12T05:27:32.127 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e15180 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e27a30 0x55a211e29e50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e14c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 --2- 172.21.15.45:0/3419933635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0e250 0x55a211e14700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07cd7fa640 0 client.19656 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 >> 172.21.15.45:0/3419933635 conn(0x55a211e08980 msgr2=0x55a211e0a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 shutdown_connections 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 1 -- 172.21.15.45:0/3419933635 wait complete. 2024-03-12T05:27:32.128 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.127+0000 7f07eee27740 20 client.19656 trim_cache size 0 max 16384 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 Processor -- start 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 -- start start 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0e250 0x55a211e16e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0eb70 0x55a211e11960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211cfb130 con 0x55a211e17750 2024-03-12T05:27:32.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211e24580 con 0x55a211e0eb70 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07eee27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211e19b40 con 0x55a211e0e250 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42508/0 (socket says 172.21.15.45:42508) 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1869532501 learned_addr learned my addr 172.21.15.45:0/1869532501 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0e250 0x55a211e16e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1869532501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0e250 msgr2=0x55a211e16e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0e250 0x55a211e16e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1869532501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0eb70 msgr2=0x55a211e11960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0eb70 0x55a211e11960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1869532501 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a211d10c20 con 0x55a211e17750 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.139+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004a80 tx=0x7f07dc009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21a8db67234d2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07cf7fe640 1 -- 172.21.15.45:0/1869532501 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc009810 con 0x55a211e17750 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07cf7fe640 1 -- 172.21.15.45:0/1869532501 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07dc020e90 con 0x55a211e17750 2024-03-12T05:27:32.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07cf7fe640 1 -- 172.21.15.45:0/1869532501 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc020ce0 con 0x55a211e17750 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- 172.21.15.45:0/1869532501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 msgr2=0x55a211e11ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004a80 tx=0x7f07dc009650 comp rx=0 tx=0).stop 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- 172.21.15.45:0/1869532501 shutdown_connections 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e17750 0x55a211e11ea0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e0eb70 0x55a211e11960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 --2- 172.21.15.45:0/1869532501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0e250 0x55a211e16e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- 172.21.15.45:0/1869532501 >> 172.21.15.45:0/1869532501 conn(0x55a211e08980 msgr2=0x55a211e0f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- 172.21.15.45:0/1869532501 shutdown_connections 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- 172.21.15.45:0/1869532501 wait complete. 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.140+0000 7f07eee27740 1 -- start start 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 Processor -- start 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e1d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.140 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e1da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211e1eed0 con 0x55a211e0e250 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211e7df00 con 0x55a211e17750 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a211e7e070 con 0x55a211e0eb70 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07eee27740 10 client.0 _ll_register_callbacks cb 0x7f07dc009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42524/0 (socket says 172.21.15.45:42524) 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1037135875 learned_addr learned my addr 172.21.15.45:0/1037135875 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e1da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 msgr2=0x55a211e1d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e1d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1037135875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 msgr2=0x55a211e1da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e1da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 -- 172.21.15.45:0/1037135875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a211d746a0 con 0x55a211e0e250 2024-03-12T05:27:32.141 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07ea7eb640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f07dc014780 tx=0x7f07dc014710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f427f8c287dd3c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.141+0000 7f07cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc004670 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f07dc03e040 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07dc003cd0 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07dc009810 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 10 client.19662 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a211d368c0 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f07dc003cd0 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f07dc01c070 con 0x55a211e0e250 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 10 client.19662 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 10 client.19662 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 20 client.19662 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e7f390 0x55a211e81780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 4 client.19662 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 --> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a211b01890 con 0x55a211e7f390 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.142+0000 7f07e9fea640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e7f390 0x55a211e81780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07cd7fa640 10 client.19662 ms_handle_connect on v2:172.21.15.130:6838/2939102833 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07e9fea640 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e7f390 0x55a211e81780 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c3a822231815204d server_cookie=d47f2aa9ff9abd99 in_seq=0 out_seq=0 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07cd7fa640 1 -- 172.21.15.45:0/1037135875 <== mds.1 v2:172.21.15.130:6838/2939102833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a211b01890 con 0x55a211e7f390 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07cd7fa640 10 client.19662 handle_command_reply: tid=0 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 client.19662 shutdown 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 msgr2=0x55a211e1cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f07dc014780 tx=0x7f07dc014710 comp rx=0 tx=0).stop 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 shutdown_connections 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:6838/2939102833,v1:172.21.15.130:6839/2939102833] conn(0x55a211e7f390 0x55a211e81780 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a211e17750 0x55a211e1da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a211e0eb70 0x55a211e1d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07cd7fa640 0 client.19662 ms_handle_reset on v2:172.21.15.130:6838/2939102833 2024-03-12T05:27:32.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 --2- 172.21.15.45:0/1037135875 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55a211e0e250 0x55a211e1cf80 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 >> 172.21.15.45:0/1037135875 conn(0x55a211e08980 msgr2=0x55a211e146c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 shutdown_connections 2024-03-12T05:27:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 1 -- 172.21.15.45:0/1037135875 wait complete. 2024-03-12T05:27:32.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.143+0000 7f07eee27740 20 client.19662 trim_cache size 0 max 16384 2024-03-12T05:27:32.144 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 msgr2=0x7f07d00a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 0x7f07d00a4110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07d40ed1f0 tx=0x7f07d40a7040 comp rx=0 tx=0).stop 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 msgr2=0x7f07e40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f07d4000c00 tx=0x7f07d4004270 comp rx=0 tx=0).stop 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 shutdown_connections 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f07d00a1c50 0x7f07d00a4110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07e4106450 0x7f07e40739c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f07e4073f00 0x7f07e41a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 --2- 172.21.15.45:0/1911167789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07e4101380 0x7f07e4073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 >> 172.21.15.45:0/1911167789 conn(0x7f07e4078580 msgr2=0x7f07e40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 shutdown_connections 2024-03-12T05:27:32.146 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.144+0000 7f07ec24e640 1 -- 172.21.15.45:0/1911167789 wait complete. 2024-03-12T05:27:32.148 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:27:32.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:27:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 -- 172.21.15.45:0/2951001218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f24100c30 msgr2=0x7f8f24104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/2951001218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f24100c30 0x7f8f24104c70 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009930 tx=0x7f8f1802bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:32.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 -- 172.21.15.45:0/2951001218 shutdown_connections 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/2951001218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f24100c30 0x7f8f24104c70 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/2951001218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 0x7f8f2410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/2951001218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 0x7f8f24100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 -- 172.21.15.45:0/2951001218 >> 172.21.15.45:0/2951001218 conn(0x7f8f24077150 msgr2=0x7f8f24077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 -- 172.21.15.45:0/2951001218 shutdown_connections 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.259+0000 7f8f29fb5640 1 -- 172.21.15.45:0/2951001218 wait complete. 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 Processor -- start 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 -- start start 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 0x7f8f24197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 0x7f8f24197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2419c9e0 con 0x7f8f2419bff0 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2419cb50 con 0x7f8f24100280 2024-03-12T05:27:32.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f29fb5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2419ccc0 con 0x7f8f24105350 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42544/0 (socket says 172.21.15.45:42544) 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 -- 172.21.15.45:0/399720083 learned_addr learned my addr 172.21.15.45:0/399720083 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 0x7f8f24197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f237fe640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 0x7f8f24197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 -- 172.21.15.45:0/399720083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 msgr2=0x7f8f24197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 0x7f8f24197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 -- 172.21.15.45:0/399720083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 msgr2=0x7f8f24197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 0x7f8f24197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.260+0000 7f8f23fff640 1 -- 172.21.15.45:0/399720083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f18009560 con 0x7f8f2419bff0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f23fff640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00cad0 tx=0x7f8f1c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f20ff9640 1 -- 172.21.15.45:0/399720083 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c00ae20 con 0x7f8f2419bff0 2024-03-12T05:27:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f241a42a0 con 0x7f8f2419bff0 2024-03-12T05:27:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f20ff9640 1 -- 172.21.15.45:0/399720083 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8f1c005020 con 0x7f8f2419bff0 2024-03-12T05:27:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f20ff9640 1 -- 172.21.15.45:0/399720083 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c00f690 con 0x7f8f2419bff0 2024-03-12T05:27:32.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.261+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f241a46d0 con 0x7f8f2419bff0 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.263+0000 7f8f20ff9640 1 -- 172.21.15.45:0/399720083 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8f1c005190 con 0x7f8f2419bff0 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.263+0000 7f8f2cb82740 1 Processor -- start 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.263+0000 7f8f2cb82740 1 -- start start 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f20ff9640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 0x7f8f0c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f20f250 0x55691f20f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f218750 0x55691f21ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f1ca960 con 0x55691f20fb70 2024-03-12T05:27:32.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f17e310 con 0x55691f20f250 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f17b840 con 0x55691f218750 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 0x7f8f0c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f20ff9640 1 -- 172.21.15.45:0/399720083 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8f1c0cd7e0 con 0x7f8f2419bff0 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42550/0 (socket says 172.21.15.45:42550) 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 -- 172.21.15.45:0/3733328840 learned_addr learned my addr 172.21.15.45:0/3733328840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f237fe640 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f20f250 0x55691f20f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f23fff640 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f218750 0x55691f21ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 -- 172.21.15.45:0/3733328840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f218750 msgr2=0x55691f21ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f218750 0x55691f21ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 -- 172.21.15.45:0/3733328840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f20f250 msgr2=0x55691f20f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f20f250 0x55691f20f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 -- 172.21.15.45:0/3733328840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55691f139060 con 0x55691f20fb70 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 0x7f8f0c0a4060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8f1802c680 tx=0x7f8f18004850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009900 tx=0x7f8f18048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a24e1838e51d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.264+0000 7f8f09ffb640 1 -- 172.21.15.45:0/3733328840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18048070 con 0x55691f20fb70 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f09ffb640 1 -- 172.21.15.45:0/3733328840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8f1802de90 con 0x55691f20fb70 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f09ffb640 1 -- 172.21.15.45:0/3733328840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f180037b0 con 0x55691f20fb70 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- 172.21.15.45:0/3733328840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 msgr2=0x55691f218210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009900 tx=0x7f8f18048000 comp rx=0 tx=0).stop 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- 172.21.15.45:0/3733328840 shutdown_connections 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f218210 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f218750 0x55691f21ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/3733328840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f20f250 0x55691f20f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- 172.21.15.45:0/3733328840 >> 172.21.15.45:0/3733328840 conn(0x55691f209980 msgr2=0x55691f209d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- 172.21.15.45:0/3733328840 shutdown_connections 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- 172.21.15.45:0/3733328840 wait complete. 2024-03-12T05:27:32.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.265+0000 7f8f2cb82740 1 -- start start 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 Processor -- start 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20fb70 0x55691f215cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f216210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f177020 con 0x55691f20f250 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691ee4a430 con 0x55691f218750 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691ee06e00 con 0x55691f20fb70 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f2cb82740 10 client.0 _ll_register_callbacks cb 0x55691f11ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42556/0 (socket says 172.21.15.45:42556) 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 -- 172.21.15.45:0/4264448052 learned_addr learned my addr 172.21.15.45:0/4264448052 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.268 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f23fff640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f216210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 -- 172.21.15.45:0/4264448052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20fb70 msgr2=0x55691f215cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20fb70 0x55691f215cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 -- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 msgr2=0x55691f216210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f216210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.266+0000 7f8f237fe640 1 -- 172.21.15.45:0/4264448052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2410d730 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f237fe640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400bab0 tx=0x7f8f1400bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4766a313a91f6df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14016070 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8f14011440 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55691ee767e0 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f140041f0 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 10 client.19674 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55691f1756a0 con 0x55691f20f250 2024-03-12T05:27:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8f14027080 con 0x55691f20f250 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8f14023760 con 0x55691f20f250 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 10 client.19674 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 10 client.19674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 20 client.19674 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f22d0a0 0x55691f22f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 4 client.19674 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.267+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55691eeb3720 con 0x55691f22d0a0 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.268+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f22d0a0 0x55691f22f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.268+0000 7f8f02ffd640 10 client.19674 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.268+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f22d0a0 0x55691f22f4c0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c222e3d5f8b9742 server_cookie=8d2c52177cd41da5 in_seq=0 out_seq=0 2024-03-12T05:27:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.268+0000 7f8f02ffd640 1 -- 172.21.15.45:0/4264448052 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55691eeb3720 con 0x55691f22d0a0 2024-03-12T05:27:32.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.268+0000 7f8f02ffd640 10 client.19674 handle_command_reply: tid=0 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 client.19674 shutdown 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 msgr2=0x55691f215790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400bab0 tx=0x7f8f1400bf70 comp rx=0 tx=0).stop 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 shutdown_connections 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f22d0a0 0x55691f22f4c0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f216210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20fb70 0x55691f215cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f02ffd640 0 client.19674 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/4264448052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20f250 0x55691f215790 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 >> 172.21.15.45:0/4264448052 conn(0x55691f209980 msgr2=0x55691f20b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 shutdown_connections 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 1 -- 172.21.15.45:0/4264448052 wait complete. 2024-03-12T05:27:32.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.269+0000 7f8f2cb82740 20 client.19674 trim_cache size 0 max 16384 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.280+0000 7f8f2cb82740 1 Processor -- start 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 -- start start 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f217ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f212a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f0fc130 con 0x55691f20fb70 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f229a30 con 0x55691f218750 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f21ae70 con 0x55691f20f250 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.281 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42568/0 (socket says 172.21.15.45:42568) 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 -- 172.21.15.45:0/484808726 learned_addr learned my addr 172.21.15.45:0/484808726 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f23fff640 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f212a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 -- 172.21.15.45:0/484808726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 msgr2=0x55691f217ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f217ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 -- 172.21.15.45:0/484808726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 msgr2=0x55691f212a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f212a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 -- 172.21.15.45:0/484808726 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55691f111c20 con 0x55691f20fb70 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009930 tx=0x7f8f180035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb70c536c389b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f097fa640 1 -- 172.21.15.45:0/484808726 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f180585c0 con 0x55691f20fb70 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f097fa640 1 -- 172.21.15.45:0/484808726 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8f18003e10 con 0x55691f20fb70 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.281+0000 7f8f097fa640 1 -- 172.21.15.45:0/484808726 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18005020 con 0x55691f20fb70 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- 172.21.15.45:0/484808726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 msgr2=0x55691f216030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009930 tx=0x7f8f180035f0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- 172.21.15.45:0/484808726 shutdown_connections 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f216030 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f212a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/484808726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f217ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- 172.21.15.45:0/484808726 >> 172.21.15.45:0/484808726 conn(0x55691f209980 msgr2=0x55691f233e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- 172.21.15.45:0/484808726 shutdown_connections 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- 172.21.15.45:0/484808726 wait complete. 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 -- start start 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.282+0000 7f8f2cb82740 1 Processor -- start 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f222610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f223090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f283710 con 0x55691f20fb70 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f283880 con 0x55691f218750 2024-03-12T05:27:32.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691f283a40 con 0x55691f20f250 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 10 client.0 _ll_register_callbacks cb 0x7f8f1802c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42572/0 (socket says 172.21.15.45:42572) 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 -- 172.21.15.45:0/216859222 learned_addr learned my addr 172.21.15.45:0/216859222 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f237fe640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f222610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f23fff640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f223090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 -- 172.21.15.45:0/216859222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 msgr2=0x55691f222610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f222610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 -- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 msgr2=0x55691f223090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f223090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 -- 172.21.15.45:0/216859222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55691f139060 con 0x55691f20fb70 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f23fff640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f223090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f22ffd640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f18003760 tx=0x7f8f1802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb1fe0ce07475d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:32.284 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18067070 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f8f18003960 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.283+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f1802de90 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f18048040 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 10 client.19680 fetch_fsmap learned FSMap version 33 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55691f1378c0 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f8f1802de90 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 33) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8f1806a020 con 0x55691f20fb70 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 10 client.19680 fetch_fsmap finished waiting for FSMap version 33 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 10 client.19680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 20 client.19680 populate_metadata read hostname 'smithi045' 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f284da0 0x55691f2871c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 4 client.19680 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:27:32.285 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 --> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55691ef02890 con 0x55691f284da0 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.284+0000 7f8f237fe640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f284da0 0x55691f2871c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f237fe640 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f284da0 0x55691f2871c0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf45c99500565575 server_cookie=6db48e751b18c598 in_seq=0 out_seq=0 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f02ffd640 10 client.19680 ms_handle_connect on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f02ffd640 1 -- 172.21.15.45:0/216859222 <== mds.0 v2:172.21.15.116:6838/397791649 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55691ef02890 con 0x55691f284da0 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f02ffd640 10 client.19680 handle_command_reply: tid=0 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 client.19680 shutdown 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 msgr2=0x55691f222b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f18003760 tx=0x7f8f1802bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 shutdown_connections 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:6838/397791649,v1:172.21.15.116:6839/397791649] conn(0x55691f284da0 0x55691f2871c0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55691f20fb70 0x55691f222b50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55691f218750 0x55691f223090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f02ffd640 0 client.19680 ms_handle_reset on v2:172.21.15.116:6838/397791649 2024-03-12T05:27:32.286 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 --2- 172.21.15.45:0/216859222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55691f20f250 0x55691f222610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 >> 172.21.15.45:0/216859222 conn(0x55691f209980 msgr2=0x55691f2082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 shutdown_connections 2024-03-12T05:27:32.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 1 -- 172.21.15.45:0/216859222 wait complete. 2024-03-12T05:27:32.287 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.285+0000 7f8f2cb82740 20 client.19680 trim_cache size 0 max 16384 2024-03-12T05:27:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 msgr2=0x7f8f0c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 0x7f8f0c0a4060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8f1802c680 tx=0x7f8f18004850 comp rx=0 tx=0).stop 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 msgr2=0x7f8f2419c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00cad0 tx=0x7f8f1c00cf90 comp rx=0 tx=0).stop 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 shutdown_connections 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f8f0c0a1ba0 0x7f8f0c0a4060 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f2419bff0 0x7f8f2419c4a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.288 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f24105350 0x7f8f24197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 --2- 172.21.15.45:0/399720083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f24100280 0x7f8f24197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 >> 172.21.15.45:0/399720083 conn(0x7f8f24077150 msgr2=0x7f8f24069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 shutdown_connections 2024-03-12T05:27:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:32.286+0000 7f8f29fb5640 1 -- 172.21.15.45:0/399720083 wait complete. 2024-03-12T05:27:32.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:32 smithi045 ceph-mon[32174]: pgmap v524: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-03-12T05:27:32.866 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:32 smithi130 ceph-mon[35398]: pgmap v524: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-03-12T05:27:32.891 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:32 smithi116 ceph-mon[35486]: pgmap v524: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-03-12T05:27:33.291 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-12T05:27:33.292 INFO:tasks.fwd_scrub:done joining 2024-03-12T05:27:33.292 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-12T05:27:33.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 -- 172.21.15.45:0/3102309780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034101380 msgr2=0x7fe034101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 --2- 172.21.15.45:0/3102309780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034101380 0x7fe034101760 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe024009930 tx=0x7fe02402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 -- 172.21.15.45:0/3102309780 shutdown_connections 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 --2- 172.21.15.45:0/3102309780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034106450 0x7fe03410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 --2- 172.21.15.45:0/3102309780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe034101d30 0x7fe034105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 --2- 172.21.15.45:0/3102309780 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034101380 0x7fe034101760 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 -- 172.21.15.45:0/3102309780 >> 172.21.15.45:0/3102309780 conn(0x7fe0340fcc20 msgr2=0x7fe0340ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 -- 172.21.15.45:0/3102309780 shutdown_connections 2024-03-12T05:27:33.420 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.425+0000 7fe038d72640 1 -- 172.21.15.45:0/3102309780 wait complete. 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 Processor -- start 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 -- start start 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034101380 0x7fe034197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe03419c000 0x7fe03419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03419c9f0 con 0x7fe034106450 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03419cb60 con 0x7fe03419c000 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.426+0000 7fe038d72640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03419ccd0 con 0x7fe034101380 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.424 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42578/0 (socket says 172.21.15.45:42578) 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 -- 172.21.15.45:0/2812450733 learned_addr learned my addr 172.21.15.45:0/2812450733 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe032d9d640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe03419c000 0x7fe03419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe03259c640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034101380 0x7fe034197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 -- 172.21.15.45:0/2812450733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034101380 msgr2=0x7fe034197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034101380 0x7fe034197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 -- 172.21.15.45:0/2812450733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe03419c000 msgr2=0x7fe03419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe03419c000 0x7fe03419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 -- 172.21.15.45:0/2812450733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024009560 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02bfff640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00e9c0 tx=0x7fe01c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c009700 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe01c00cd80 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0341a42b0 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01c010430 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.427+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0341a4760 con 0x7fe034106450 2024-03-12T05:27:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.429+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0340683c0 con 0x7fe034106450 2024-03-12T05:27:33.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.429+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe01c004f50 con 0x7fe034106450 2024-03-12T05:27:33.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.430+0000 7fe02b7fe640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 0x7fe0200a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.430+0000 7fe03259c640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 0x7fe0200a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.430+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fe01c014070 con 0x7fe034106450 2024-03-12T05:27:33.426 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.431+0000 7fe03259c640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 0x7fe0200a4060 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe024005ab0 tx=0x7fe024005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:33.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.435+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe01c096a50 con 0x7fe034106450 2024-03-12T05:27:33.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.622+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe034109680 con 0x7fe034106450 2024-03-12T05:27:33.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.623+0000 7fe02b7fe640 1 -- 172.21.15.45:0/2812450733 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+23168 (secure 0 0 0) 0x7fe01c09a110 con 0x7fe034106450 2024-03-12T05:27:33.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:33.618 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":110,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:27:09.587612+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3527414001":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/1231700554":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/2345425176":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/4188215761":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/759469548":"2024-03-13T05:10:52.434958+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.625+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 msgr2=0x7fe0200a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.625+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 0x7fe0200a4060 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe024005ab0 tx=0x7fe024005fd0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.625+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 msgr2=0x7fe034197cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.625+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00e9c0 tx=0x7fe01c00ee80 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 shutdown_connections 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe0200a1ba0 0x7fe0200a4060 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe034106450 0x7fe034197cd0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe03419c000 0x7fe03419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 --2- 172.21.15.45:0/2812450733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe034101380 0x7fe034197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.620 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 >> 172.21.15.45:0/2812450733 conn(0x7fe0340fcc20 msgr2=0x7fe0340fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:33.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 shutdown_connections 2024-03-12T05:27:33.621 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.626+0000 7fe038d72640 1 -- 172.21.15.45:0/2812450733 wait complete. 2024-03-12T05:27:33.631 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:27:33.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 -- 172.21.15.45:0/3872981799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4101d30 msgr2=0x7f9df4105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/3872981799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4101d30 0x7f9df4105d70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009930 tx=0x7f9ddc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 -- 172.21.15.45:0/3872981799 shutdown_connections 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/3872981799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4101d30 0x7f9df4105d70 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/3872981799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4106450 0x7f9df410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/3872981799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df4101380 0x7f9df4101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 -- 172.21.15.45:0/3872981799 >> 172.21.15.45:0/3872981799 conn(0x7f9df4078580 msgr2=0x7f9df4078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 -- 172.21.15.45:0/3872981799 shutdown_connections 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.744+0000 7f9df8e9d640 1 -- 172.21.15.45:0/3872981799 wait complete. 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 Processor -- start 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 -- start start 2024-03-12T05:27:33.741 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101380 0x7f9df4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 0x7f9df419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419c9f0 con 0x7f9df4106450 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419cb60 con 0x7f9df4101380 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9df8e9d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419ccd0 con 0x7f9df419c000 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9debfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9debfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42598/0 (socket says 172.21.15.45:42598) 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.745+0000 7f9debfff640 1 -- 172.21.15.45:0/2242084255 learned_addr learned my addr 172.21.15.45:0/2242084255 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9df259c640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101380 0x7f9df4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.742 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9df2d9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 0x7f9df419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 -- 172.21.15.45:0/2242084255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 msgr2=0x7f9df419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 0x7f9df419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 -- 172.21.15.45:0/2242084255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101380 msgr2=0x7f9df4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101380 0x7f9df4197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 -- 172.21.15.45:0/2242084255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc009560 con 0x7f9df4106450 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9df2d9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 0x7f9df419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9debfff640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000c00 tx=0x7f9ddc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc02cb70 con 0x7f9df4106450 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9ddc02de90 con 0x7f9df4106450 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0053c0 con 0x7f9df4106450 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df41a42b0 con 0x7f9df4106450 2024-03-12T05:27:33.743 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.746+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df41a4760 con 0x7f9df4106450 2024-03-12T05:27:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.747+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df40683c0 con 0x7f9df4106450 2024-03-12T05:27:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.748+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ddc0035f0 con 0x7f9df4106450 2024-03-12T05:27:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.749+0000 7f9deb7fe640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 0x7f9de00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:33.744 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.749+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9ddc0f0280 con 0x7f9df4106450 2024-03-12T05:27:33.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.749+0000 7f9df259c640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 0x7f9de00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:33.745 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.749+0000 7f9df259c640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 0x7f9de00a40e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9de4005fd0 tx=0x7f9de400b390 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:33.746 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.753+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ddc0b91d0 con 0x7f9df4106450 2024-03-12T05:27:33.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.939+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9df4109710 con 0x7f9df4106450 2024-03-12T05:27:33.933 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.939+0000 7f9deb7fe640 1 -- 172.21.15.45:0/2242084255 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9df4109710 con 0x7f9df4106450 2024-03-12T05:27:33.933 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 msgr2=0x7f9de00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 0x7f9de00a40e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9de4005fd0 tx=0x7f9de400b390 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 msgr2=0x7f9df4197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc000c00 tx=0x7f9ddc004270 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 shutdown_connections 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9de00a1c20 0x7f9de00a40e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df4106450 0x7f9df4197c80 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df419c000 0x7f9df419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 --2- 172.21.15.45:0/2242084255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101380 0x7f9df4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.941+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 >> 172.21.15.45:0/2242084255 conn(0x7f9df4078580 msgr2=0x7f9df40ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:33.936 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.942+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 shutdown_connections 2024-03-12T05:27:33.937 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:33.942+0000 7f9df8e9d640 1 -- 172.21.15.45:0/2242084255 wait complete. 2024-03-12T05:27:33.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 -- 172.21.15.45:0/3399194447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100c10 msgr2=0x7f9d94104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/3399194447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100c10 0x7f9d94104c50 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009930 tx=0x7f9d8802bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 -- 172.21.15.45:0/3399194447 shutdown_connections 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/3399194447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100c10 0x7f9d94104c50 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/3399194447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/3399194447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d94100260 0x7f9d94100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 -- 172.21.15.45:0/3399194447 >> 172.21.15.45:0/3399194447 conn(0x7f9d94077180 msgr2=0x7f9d94077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 -- 172.21.15.45:0/3399194447 shutdown_connections 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.058+0000 7f9d9b12c640 1 -- 172.21.15.45:0/3399194447 wait complete. 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 Processor -- start 2024-03-12T05:27:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 -- start start 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 0x7f9d94197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94197fb0 con 0x7f9d94100260 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94198120 con 0x7f9d94105330 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d9b12c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941982e0 con 0x7f9d941975c0 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d98ec8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d98ec8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42616/0 (socket says 172.21.15.45:42616) 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d98ec8640 1 -- 172.21.15.45:0/2661414381 learned_addr learned my addr 172.21.15.45:0/2661414381 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d93fff640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.059+0000 7f9d996c9640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 0x7f9d94197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 -- 172.21.15.45:0/2661414381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 msgr2=0x7f9d94197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 0x7f9d94197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 -- 172.21.15.45:0/2661414381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 msgr2=0x7f9d9419d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9419d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 -- 172.21.15.45:0/2661414381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88009560 con 0x7f9d94100260 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d996c9640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 0x7f9d94197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d93fff640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d98ec8640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00eab0 tx=0x7f9d8c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00ce40 con 0x7f9d94100260 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9d8c011920 con 0x7f9d94100260 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d941a42d0 con 0x7f9d94100260 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0109d0 con 0x7f9d94100260 2024-03-12T05:27:34.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.060+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d941a47b0 con 0x7f9d94100260 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.061+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d94067f70 con 0x7f9d94100260 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.062+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9d8c011a90 con 0x7f9d94100260 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.063+0000 7f9d91ffb640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 0x7f9d7c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.063+0000 7f9d93fff640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 0x7f9d7c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.063+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9d8c0cdf90 con 0x7f9d94100260 2024-03-12T05:27:34.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.063+0000 7f9d93fff640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 0x7f9d7c0a4060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009900 tx=0x7f9d88005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.067+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d8c096dd0 con 0x7f9d94100260 2024-03-12T05:27:34.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.250+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d94100c10 con 0x7f9d94100260 2024-03-12T05:27:34.244 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.250+0000 7f9d91ffb640 1 -- 172.21.15.45:0/2661414381 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d8c09a490 con 0x7f9d94100260 2024-03-12T05:27:34.244 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.252+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 msgr2=0x7f9d7c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 0x7f9d7c0a4060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009900 tx=0x7f9d88005fd0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 msgr2=0x7f9d9419d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00eab0 tx=0x7f9d8c00ef70 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 shutdown_connections 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9d7c0a1ba0 0x7f9d7c0a4060 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d941975c0 0x7f9d94197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d94105330 0x7f9d9419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 --2- 172.21.15.45:0/2661414381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d94100260 0x7f9d9419d410 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 >> 172.21.15.45:0/2661414381 conn(0x7f9d94077180 msgr2=0x7f9d94068740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:34.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 shutdown_connections 2024-03-12T05:27:34.248 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.253+0000 7f9d9b12c640 1 -- 172.21.15.45:0/2661414381 wait complete. 2024-03-12T05:27:34.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 -- 172.21.15.45:0/2308363353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e541002a0 msgr2=0x7f9e54105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 --2- 172.21.15.45:0/2308363353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e541002a0 0x7f9e54105dd0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009930 tx=0x7f9e4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 -- 172.21.15.45:0/2308363353 shutdown_connections 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 --2- 172.21.15.45:0/2308363353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e541002a0 0x7f9e54105dd0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 --2- 172.21.15.45:0/2308363353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e54106500 0x7f9e5410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 --2- 172.21.15.45:0/2308363353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 0x7f9e540ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 -- 172.21.15.45:0/2308363353 >> 172.21.15.45:0/2308363353 conn(0x7f9e54076640 msgr2=0x7f9e540fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 -- 172.21.15.45:0/2308363353 shutdown_connections 2024-03-12T05:27:34.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.372+0000 7f9e593e2640 1 -- 172.21.15.45:0/2308363353 wait complete. 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 Processor -- start 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 -- start start 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 0x7f9e54197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e5419c020 0x7f9e5419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5419ca10 con 0x7f9e54106500 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5419cb80 con 0x7f9e540ff8f0 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e593e2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5419ccf0 con 0x7f9e5419c020 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e527fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e527fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42624/0 (socket says 172.21.15.45:42624) 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.373+0000 7f9e527fc640 1 -- 172.21.15.45:0/1760842653 learned_addr learned my addr 172.21.15.45:0/1760842653 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:34.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e537fe640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e5419c020 0x7f9e5419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e52ffd640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 0x7f9e54197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 -- 172.21.15.45:0/1760842653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e5419c020 msgr2=0x7f9e5419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e5419c020 0x7f9e5419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 -- 172.21.15.45:0/1760842653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 msgr2=0x7f9e54197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 0x7f9e54197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 -- 172.21.15.45:0/1760842653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44009560 con 0x7f9e54106500 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e527fc640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9e44000c00 tx=0x7f9e44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4402c5d0 con 0x7f9e54106500 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e541a42d0 con 0x7f9e54106500 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f9e4402de90 con 0x7f9e54106500 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44005320 con 0x7f9e54106500 2024-03-12T05:27:34.371 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.374+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e541a4750 con 0x7f9e54106500 2024-03-12T05:27:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.375+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e541029a0 con 0x7f9e54106500 2024-03-12T05:27:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.377+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9e440035f0 con 0x7f9e54106500 2024-03-12T05:27:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.378+0000 7f9e3bfff640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 0x7f9e3c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.378+0000 7f9e52ffd640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 0x7f9e3c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.372 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.378+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f9e440effc0 con 0x7f9e54106500 2024-03-12T05:27:34.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.378+0000 7f9e52ffd640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 0x7f9e3c0a4060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005fd0 tx=0x7f9e48005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.380+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9e440b8f10 con 0x7f9e54106500 2024-03-12T05:27:34.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.562+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e541096b0 con 0x7f9e54106500 2024-03-12T05:27:34.556 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.562+0000 7f9e3bfff640 1 -- 172.21.15.45:0/1760842653 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e440bc5d0 con 0x7f9e54106500 2024-03-12T05:27:34.557 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 msgr2=0x7f9e3c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 0x7f9e3c0a4060 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005fd0 tx=0x7f9e48005850 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 msgr2=0x7f9e54197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9e44000c00 tx=0x7f9e44004270 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 shutdown_connections 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f9e3c0a1ba0 0x7f9e3c0a4060 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e54106500 0x7f9e54197ca0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e5419c020 0x7f9e5419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 --2- 172.21.15.45:0/1760842653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e540ff8f0 0x7f9e54197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 >> 172.21.15.45:0/1760842653 conn(0x7f9e54076640 msgr2=0x7f9e54104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 shutdown_connections 2024-03-12T05:27:34.559 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.564+0000 7f9e593e2640 1 -- 172.21.15.45:0/1760842653 wait complete. 2024-03-12T05:27:34.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-12T05:27:34.678 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 -- 172.21.15.45:0/1807661536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741002f0 msgr2=0x7f81741006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 --2- 172.21.15.45:0/1807661536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741002f0 0x7f81741006d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8170009930 tx=0x7f817002bc30 comp rx=0 tx=0).stop 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 -- 172.21.15.45:0/1807661536 shutdown_connections 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 --2- 172.21.15.45:0/1807661536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741053c0 0x7f817410cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 --2- 172.21.15.45:0/1807661536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8174100ca0 0x7f8174104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 --2- 172.21.15.45:0/1807661536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741002f0 0x7f81741006d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 -- 172.21.15.45:0/1807661536 >> 172.21.15.45:0/1807661536 conn(0x7f8174077180 msgr2=0x7f8174077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 -- 172.21.15.45:0/1807661536 shutdown_connections 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.682+0000 7f817beb5640 1 -- 172.21.15.45:0/1807661536 wait complete. 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 Processor -- start 2024-03-12T05:27:34.679 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 -- start start 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 0x7f81741966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f817419af60 0x7f817419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419b950 con 0x7f81741053c0 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419bac0 con 0x7f81741002f0 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817beb5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817419bc30 con 0x7f817419af60 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f8179450640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f8179450640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42634/0 (socket says 172.21.15.45:42634) 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f8179c51640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 0x7f81741966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f8179450640 1 -- 172.21.15.45:0/3807049212 learned_addr learned my addr 172.21.15.45:0/3807049212 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.683+0000 7f817a452640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f817419af60 0x7f817419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 -- 172.21.15.45:0/3807049212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f817419af60 msgr2=0x7f817419b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.680 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f817419af60 0x7f817419b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 -- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 msgr2=0x7f8174196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 -- 172.21.15.45:0/3807049212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c009840 con 0x7f81741002f0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179450640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f8179c51640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 0x7f81741966f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8170009230 tx=0x7f8170005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170004350 con 0x7f81741002f0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8170009560 con 0x7f81741002f0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f817002de90 con 0x7f81741002f0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170005120 con 0x7f81741002f0 2024-03-12T05:27:34.681 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.684+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81741a34a0 con 0x7f81741002f0 2024-03-12T05:27:34.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.685+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8174067f70 con 0x7f81741002f0 2024-03-12T05:27:34.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.686+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8170003b30 con 0x7f81741002f0 2024-03-12T05:27:34.682 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.687+0000 7f816affd640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 0x7f81600a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:27:34.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.687+0000 7f8179450640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 0x7f81600a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:27:34.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.687+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f81700ef620 con 0x7f81741002f0 2024-03-12T05:27:34.683 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.687+0000 7f8179450640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 0x7f81600a4060 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f81740ffcf0 tx=0x7f8164005f90 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:27:34.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.690+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f81700b8570 con 0x7f81741002f0 2024-03-12T05:27:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: pgmap v525: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-12T05:27:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2812450733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:27:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2242084255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:27:34.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:27:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2661414381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:27:34.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:34 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1760842653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:27:34.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.923+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f8174108690 con 0x7f81741002f0 2024-03-12T05:27:34.925 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.931+0000 7f816affd640 1 -- 172.21.15.45:0/3807049212 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+734995 (secure 0 0 0) 0x7f81700bbc30 con 0x7f81741002f0 2024-03-12T05:27:34.937 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:27:34.937 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:34.937 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 27, 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "active_gid": 25024, 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "active_name": "x", 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "active_addrs": { 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6800", 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 869096728 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.938 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6801", 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 869096728 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "active_addr": "172.21.15.45:6801/869096728", 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "active_change": "2024-03-12T05:10:52.435195+0000", 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "active_mgr_features": 4540701547738038271, 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "available": true, 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "standbys": [ 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "gid": 15513, 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "z", 2024-03-12T05:27:34.939 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_features": 4540701547738038271, 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "available_modules": [ 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "alerts", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.940 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.941 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.942 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_destination": { 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_destination", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email address to send alerts to", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_from_name": { 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_from_name", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "Ceph", 2024-03-12T05:27:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email From: name", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_host": { 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_host", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP server", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_password": { 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_password", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Password to authenticate with", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_port": { 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_port", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "465", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP port", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_sender": { 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_sender", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP envelope sender", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_ssl": { 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_ssl", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T05:27:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_user": { 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_user", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "User to authenticate as", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "balancer", 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "active": { 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "active", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_time": { 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_time", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0000", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_weekday": { 2024-03-12T05:27:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_weekday", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_max_iterations": { 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_max_iterations", 2024-03-12T05:27:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "25", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "250", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_metrics": { 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_metrics", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.958 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_step": { 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_step", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.5", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0.001", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "0.999", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "aggressiveness of optimization", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "end_time": { 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_time", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.960 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2359", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "end_weekday": { 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_weekday", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:34.961 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.962 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.963 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.964 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.965 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.966 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.967 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "min_score": { 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_score", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.968 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "upmap", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-compat", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "none", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "read", 2024-03-12T05:27:34.969 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap-read" 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Balancer mode", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_ids": { 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_ids", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.970 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:34.971 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_deviation": { 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_deviation", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:34.972 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_optimizations": { 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_optimizations", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.973 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm", 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:34.974 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_down_multiplier": { 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_down_multiplier", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.975 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_refresh_rate": { 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_refresh_rate", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "20", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_starting_port": { 2024-03-12T05:27:34.976 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_starting_port", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4721", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_ptrace": { 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_ptrace", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.978 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_interval": { 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_interval", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.979 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_memory_target_ratio": { 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.7", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm_log_destination": { 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm_log_destination", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "file", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "file,syslog", 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: "syslog" 2024-03-12T05:27:34.981 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "cgroups_split": { 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cgroups_split", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:34.982 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "config_checks_enabled": { 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_checks_enabled", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:34.983 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "config_dashboard": { 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_dashboard", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.984 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_alertmanager": { 2024-03-12T05:27:34.985 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_alertmanager", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_base": { 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_base", 2024-03-12T05:27:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Container image name, without the tag", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_elasticsearch": { 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_elasticsearch", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.987 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "elasticsearch container image", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_grafana": { 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_grafana", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.988 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_haproxy": { 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_haproxy", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.989 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "HAproxy container image", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_agent": { 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_agent", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.990 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger agent container image", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_collector": { 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_collector", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.991 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger collector container image", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_query": { 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_query", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.992 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger query container image", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_keepalived": { 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_keepalived", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.993 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Keepalived container image", 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.994 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_loki": { 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_loki", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Loki container image", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_node_exporter": { 2024-03-12T05:27:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_node_exporter", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.996 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_nvmeof": { 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_nvmeof", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Nvme-of container image", 2024-03-12T05:27:34.997 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_prometheus": { 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_prometheus", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:34.998 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_promtail": { 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_promtail", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Promtail container image", 2024-03-12T05:27:34.999 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_snmp_gateway": { 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_snmp_gateway", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.000 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SNMP Gateway container image", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "container_init": { 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_init", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.001 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "daemon_cache_timeout": { 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "daemon_cache_timeout", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.002 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "default_cephadm_command_timeout": { 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "900", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.003 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "default_registry": { 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_registry", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.004 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "device_cache_timeout": { 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_cache_timeout", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1800", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.005 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache device inventory", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "device_enhanced_scan": { 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_enhanced_scan", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.006 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "facts_cache_timeout": { 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "facts_cache_timeout", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.007 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache host facts data", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "host_check_interval": { 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "host_check_interval", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.008 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to perform a host check", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "hw_monitoring": { 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hw_monitoring", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.009 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "inventory_list_all": { 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "inventory_list_all", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.010 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.011 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "log_refresh_metadata": { 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_refresh_metadata", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T05:27:35.014 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.015 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.018 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "*", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.019 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "max_count_per_host": { 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_count_per_host", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of daemons per service per host", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.020 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd_draining_count": { 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_osd_draining_count", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "migration_current": { 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "migration_current", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "internal - do not modify", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.022 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "root", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm-package", 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "root" 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.023 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "oob_default_addr": { 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "oob_default_addr", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "169.254.1.1", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T05:27:35.024 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "prometheus_alerts_path": { 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus_alerts_path", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T05:27:35.025 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_insecure": { 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_insecure", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T05:27:35.026 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_password": { 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_password", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_url": { 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_url", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.028 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_username": { 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_username", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.029 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "secure_monitoring_stack": { 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "secure_monitoring_stack", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.030 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "service_discovery_port": { 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "service_discovery_port", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8765", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.031 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "cephadm service discovery port", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "ssh_config_file": { 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssh_config_file", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.032 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "use_agent": { 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_agent", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.033 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "use_repo_digest": { 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_repo_digest", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.034 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_failed_host_check": { 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_failed_host_check", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.035 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_daemons": { 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_daemons", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.036 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_hosts": { 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_hosts", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.037 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crash", 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.038 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.039 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.040 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.041 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.042 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.043 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.044 INFO:teuthology.orchestra.run.smithi045.stdout: "retain_interval": { 2024-03-12T05:27:35.044 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retain_interval", 2024-03-12T05:27:35.044 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "31536000", 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.046 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_recent_interval": { 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_recent_interval", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1209600", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.047 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dashboard", 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T05:27:35.048 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T05:27:35.049 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T05:27:35.050 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_ENABLED": { 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T05:27:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.052 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.053 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.054 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.055 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.056 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.057 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.058 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.059 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.060 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.061 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.062 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.063 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_URL": { 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_URL", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.064 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.065 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.066 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.067 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.068 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.069 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.070 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.071 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.072 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.073 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.074 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.075 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.076 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.077 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.079 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T05:27:35.080 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T05:27:35.081 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "45", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.083 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.084 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.085 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.086 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.087 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "cross_origin_url": { 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cross_origin_url", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.091 INFO:teuthology.orchestra.run.smithi045.stdout: "crt_file": { 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crt_file", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "debug": { 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "debug", 2024-03-12T05:27:35.092 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable/disable debug options", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "jwt_token_ttl": { 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "jwt_token_ttl", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.093 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "28800", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.095 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.096 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.097 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.098 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.099 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "motd": { 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "motd", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.100 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "The message of the day", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_resolve_ip_addr": { 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.101 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.102 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8080", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.103 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.104 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl_server_port": { 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl_server_port", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8443", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.105 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "redirect", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.106 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect" 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.107 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "url_prefix": { 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url_prefix", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.108 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.109 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "devicehealth", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_monitoring": { 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_monitoring", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.110 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "monitor device health metrics", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.111 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.112 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.113 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.114 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.115 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "mark_out_threshold": { 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mark_out_threshold", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.116 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2419200", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": { 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_name", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "device_health_metrics", 2024-03-12T05:27:35.117 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "retention_period": { 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retention_period", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15552000", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.118 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain device health metrics", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_frequency": { 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_frequency", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:35.119 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "self_heal": { 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "self_heal", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.120 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.121 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_threshold": { 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_threshold", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.122 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7257600", 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "diskprediction_local", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.124 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.125 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.126 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.127 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "predict_interval": { 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predict_interval", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.130 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "predictor_model": { 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predictor_model", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "prophetstor", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.131 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "influx", 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": false, 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "influxdb python module not found", 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "batch_size": { 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "batch_size", 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.133 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5000", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "database": { 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "database", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.134 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "ceph", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": { 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hostname", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.135 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server hostname", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.136 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "30", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "5", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.137 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.138 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.139 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.140 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.141 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "password": { 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "password", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.142 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "password of InfluxDB server user", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "port": { 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "port", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8086", 2024-03-12T05:27:35.143 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server port", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "false", 2024-03-12T05:27:35.144 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "threads": { 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threads", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "32", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "username": { 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "username", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "username of InfluxDB server user", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "verify_ssl": { 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "verify_ssl", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "true", 2024-03-12T05:27:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "insights", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "iostat", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "k8sevents", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_event_retention_days": { 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ceph_event_retention_days", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "config_check_secs": { 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_check_secs", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "10", 2024-03-12T05:27:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.162 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.163 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.164 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.165 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "localpool", 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "failure_domain": { 2024-03-12T05:27:35.166 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "failure_domain", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "host", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "failure domain for any created local pool", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.167 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.168 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.169 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.170 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.171 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": { 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_size", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default min_size for any created local pool", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "num_rep": { 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "num_rep", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3", 2024-03-12T05:27:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default replica count for any created local pool", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": { 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_num", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "128", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.174 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "prefix": { 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prefix", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.175 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name prefix for any created local pool", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "subtree": { 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "subtree", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "rack", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.176 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mds_autoscaler", 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.177 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.178 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.179 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.180 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.181 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirroring", 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.182 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.183 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.184 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.185 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.186 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "nfs", 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.187 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.188 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.189 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.190 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.191 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "fail_fs": { 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "fail_fs", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.193 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.194 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.196 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.197 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.198 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator": { 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.199 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "rook", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "test_orchestrator" 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Orchestrator backend", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_perf_query", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.200 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.201 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.202 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.203 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.204 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.205 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_support", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.206 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.207 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.208 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.210 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_autoscaler", 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.211 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.212 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.213 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.214 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.215 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "threshold": { 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threshold", 2024-03-12T05:27:35.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1.0", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "scaling threshold", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "progress", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_pg_recovery_event": { 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_pg_recovery_event", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.219 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.220 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.221 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.222 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.223 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.224 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.225 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "max_completed_events": { 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_completed_events", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "50", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "number of past completed events to remember", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.226 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long the module is going to sleep", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.227 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "cache": { 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cache", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.228 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "exclude_perf_counters": { 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "exclude_perf_counters", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.229 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.230 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.231 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.232 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.233 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.234 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools": { 2024-03-12T05:27:35.235 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T05:27:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "300", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_interval": { 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_interval", 2024-03-12T05:27:35.237 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15.0", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.238 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.239 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "9283", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "stale_cache_strategy": { 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stale_cache_strategy", 2024-03-12T05:27:35.240 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "log", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.241 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "default", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "default", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "error" 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:35.242 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_support", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.244 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.245 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.246 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.247 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.248 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_snap_create": { 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_snap_create", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.249 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "mirror_snapshot_schedule": { 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.250 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "trash_purge_schedule": { 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "trash_purge_schedule", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.251 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "restful", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_auth": { 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_auth", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.252 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.253 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.254 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.255 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.256 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.257 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.258 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.259 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.260 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rgw", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.261 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.262 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.263 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.264 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rook", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.267 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.268 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.269 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.270 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.271 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_class": { 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "storage_class", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "local", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.272 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "selftest", 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.273 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.274 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.275 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.276 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.277 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "roption1": { 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption1", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.278 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "roption2": { 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption2", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "xyz", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption1": { 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption1", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.280 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption2": { 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption2", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.281 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption3": { 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption3", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.282 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption4": { 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption4", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.283 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption5": { 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption5", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.284 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption6": { 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption6", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.285 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption7": { 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption7", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "42", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.286 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "testkey": { 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testkey", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.287 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "testlkey": { 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testlkey", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.288 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "testnewline": { 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testnewline", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snap_schedule", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_m_granularity": { 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_m_granularity", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.290 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "dump_on_update": { 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dump_on_update", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.291 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "dump database to debug log on update", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.292 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.293 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.294 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stats", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.297 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.298 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.299 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.300 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.301 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "status", 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.302 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.303 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.304 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.305 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.306 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telegraf", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "address": { 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "address", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.308 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.309 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.310 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.311 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.312 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.314 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telemetry", 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.315 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_basic": { 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_basic", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.316 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_crash": { 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_crash", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.317 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_device": { 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_device", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_ident": { 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_ident", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_perf": { 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_perf", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "contact": { 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "contact", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.321 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "description": { 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "description", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.322 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "device_url": { 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_url", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.323 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.324 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "24", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "8", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.325 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "last_opt_revision": { 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "last_opt_revision", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.326 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard": { 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.327 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard_description": { 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard_description", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.328 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.329 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.330 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.331 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.333 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.333 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.333 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.333 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.334 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.335 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.336 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "organization": { 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "organization", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.337 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "proxy": { 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "proxy", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.338 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "url": { 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.339 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.340 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test_orchestrator", 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.341 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.342 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.343 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.344 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.345 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.346 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "volumes", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.347 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.348 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.349 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.350 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.351 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.352 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.353 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_clones": { 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_clones", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.354 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "periodic_async_work": { 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "periodic_async_work", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Periodically check for async work", 2024-03-12T05:27:35.355 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "snapshot_clone_delay": { 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snapshot_clone_delay", 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.356 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix", 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "discovery_interval": { 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "discovery_interval", 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.358 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.359 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.359 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "100", 2024-03-12T05:27:35.359 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.359 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.359 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.360 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.360 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.360 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.360 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.361 INFO:teuthology.orchestra.run.smithi045.stdout: "identifier": { 2024-03-12T05:27:35.361 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "identifier", 2024-03-12T05:27:35.361 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.361 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.361 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.363 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.364 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.365 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.366 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.367 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.368 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_host": { 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_host", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_port": { 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_port", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10051", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.371 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_sender": { 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_sender", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.372 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: "gid": 25034, 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "y", 2024-03-12T05:27:35.373 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_features": 4540701547738038271, 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "available_modules": [ 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "alerts", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.374 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.375 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.376 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.377 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.379 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.380 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_destination": { 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_destination", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.381 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email address to send alerts to", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_from_name": { 2024-03-12T05:27:35.382 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_from_name", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "Ceph", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.383 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email From: name", 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.384 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_host": { 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_host", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP server", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.385 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_password": { 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_password", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Password to authenticate with", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.386 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_port": { 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_port", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "465", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP port", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.387 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_sender": { 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_sender", 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.388 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP envelope sender", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_ssl": { 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_ssl", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.389 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_user": { 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_user", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.390 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "User to authenticate as", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "balancer", 2024-03-12T05:27:35.391 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "active": { 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "active", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.392 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_time": { 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_time", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0000", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.393 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_weekday": { 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_weekday", 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.394 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:35.395 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.396 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.396 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_max_iterations": { 2024-03-12T05:27:35.396 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_max_iterations", 2024-03-12T05:27:35.396 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "25", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "250", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_metrics": { 2024-03-12T05:27:35.397 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_metrics", 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.398 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_step": { 2024-03-12T05:27:35.399 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_step", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.5", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0.001", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "0.999", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "aggressiveness of optimization", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.400 INFO:teuthology.orchestra.run.smithi045.stdout: "end_time": { 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_time", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2359", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "end_weekday": { 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_weekday", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.403 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.404 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.405 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.407 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.408 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "min_score": { 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_score", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.409 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:35.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "upmap", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-compat", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "none", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "read", 2024-03-12T05:27:35.411 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap-read" 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Balancer mode", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_ids": { 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_ids", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.412 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.413 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_deviation": { 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_deviation", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.414 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_optimizations": { 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_optimizations", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.415 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.416 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_down_multiplier": { 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_down_multiplier", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.418 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_refresh_rate": { 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_refresh_rate", 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.419 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "20", 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T05:27:35.420 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_starting_port": { 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_starting_port", 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.421 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4721", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.422 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_ptrace": { 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_ptrace", 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.423 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.424 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_interval": { 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_interval", 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.425 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_memory_target_ratio": { 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.7", 2024-03-12T05:27:35.426 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm_log_destination": { 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm_log_destination", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.427 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "file", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "file,syslog", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "syslog" 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.428 INFO:teuthology.orchestra.run.smithi045.stdout: "cgroups_split": { 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cgroups_split", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "config_checks_enabled": { 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_checks_enabled", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.430 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "config_dashboard": { 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_dashboard", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.431 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_alertmanager": { 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_alertmanager", 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.432 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_base": { 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_base", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.433 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Container image name, without the tag", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_elasticsearch": { 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_elasticsearch", 2024-03-12T05:27:35.434 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "elasticsearch container image", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_grafana": { 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_grafana", 2024-03-12T05:27:35.435 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.436 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_haproxy": { 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_haproxy", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "HAproxy container image", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.437 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_agent": { 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_agent", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger agent container image", 2024-03-12T05:27:35.438 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_collector": { 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_collector", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger collector container image", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.439 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_query": { 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_query", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.440 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger query container image", 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_keepalived": { 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_keepalived", 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.441 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Keepalived container image", 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.442 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_loki": { 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_loki", 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.443 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Loki container image", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_node_exporter": { 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_node_exporter", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_nvmeof": { 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_nvmeof", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.445 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Nvme-of container image", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_prometheus": { 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_prometheus", 2024-03-12T05:27:35.446 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_promtail": { 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_promtail", 2024-03-12T05:27:35.447 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Promtail container image", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_snmp_gateway": { 2024-03-12T05:27:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_snmp_gateway", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SNMP Gateway container image", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "container_init": { 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_init", 2024-03-12T05:27:35.449 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.450 INFO:teuthology.orchestra.run.smithi045.stdout: "daemon_cache_timeout": { 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "daemon_cache_timeout", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T05:27:35.451 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "default_cephadm_command_timeout": { 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "900", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T05:27:35.452 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "default_registry": { 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_registry", 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.453 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io", 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.454 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "device_cache_timeout": { 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_cache_timeout", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1800", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache device inventory", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "device_enhanced_scan": { 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_enhanced_scan", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.456 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "facts_cache_timeout": { 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "facts_cache_timeout", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache host facts data", 2024-03-12T05:27:35.457 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "host_check_interval": { 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "host_check_interval", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.458 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to perform a host check", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "hw_monitoring": { 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hw_monitoring", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.459 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "inventory_list_all": { 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "inventory_list_all", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.460 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.461 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.462 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.463 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.464 INFO:teuthology.orchestra.run.smithi045.stdout: "log_refresh_metadata": { 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_refresh_metadata", 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.465 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.466 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.467 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.468 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.470 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.471 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "*", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.472 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "max_count_per_host": { 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_count_per_host", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.473 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of daemons per service per host", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd_draining_count": { 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_osd_draining_count", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.474 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "migration_current": { 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "migration_current", 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.475 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "internal - do not modify", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "root", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.477 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm-package", 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "root" 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "oob_default_addr": { 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "oob_default_addr", 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.478 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "169.254.1.1", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.479 INFO:teuthology.orchestra.run.smithi045.stdout: "prometheus_alerts_path": { 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus_alerts_path", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_insecure": { 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_insecure", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.481 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_password": { 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_password", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.482 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_url": { 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_url", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.483 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_username": { 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_username", 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.484 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "secure_monitoring_stack": { 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "secure_monitoring_stack", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.485 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.486 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "service_discovery_port": { 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "service_discovery_port", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8765", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "cephadm service discovery port", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.487 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "ssh_config_file": { 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssh_config_file", 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.488 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "use_agent": { 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_agent", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.489 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "use_repo_digest": { 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_repo_digest", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.490 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_failed_host_check": { 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_failed_host_check", 2024-03-12T05:27:35.491 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_daemons": { 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_daemons", 2024-03-12T05:27:35.492 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T05:27:35.493 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_hosts": { 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_hosts", 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.494 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.495 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crash", 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.496 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.497 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.498 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.499 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.500 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.501 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.502 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "retain_interval": { 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retain_interval", 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "31536000", 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.503 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.504 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.504 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T05:27:35.504 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.504 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.504 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_recent_interval": { 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_recent_interval", 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.505 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1209600", 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.506 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dashboard", 2024-03-12T05:27:35.507 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.508 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T05:27:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.510 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_ENABLED": { 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.512 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.513 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.514 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.517 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.518 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.519 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T05:27:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T05:27:35.521 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.522 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.523 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.524 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.525 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.526 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.527 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.528 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.529 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.530 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_URL": { 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_URL", 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.531 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.532 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T05:27:35.533 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.534 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.535 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.536 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.537 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.538 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.539 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.540 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.541 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.542 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.542 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.542 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T05:27:35.542 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.543 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.544 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.545 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T05:27:35.546 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.547 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.548 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.549 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.551 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.552 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.553 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.554 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.555 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.556 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T05:27:35.557 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.558 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T05:27:35.559 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.560 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T05:27:35.561 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.562 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.564 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8", 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.565 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "45", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.566 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.567 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.568 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.569 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.570 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.571 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.572 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.572 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.572 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.572 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.572 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.573 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.574 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.575 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.576 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.577 INFO:teuthology.orchestra.run.smithi045.stdout: "cross_origin_url": { 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cross_origin_url", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.578 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "crt_file": { 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crt_file", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.579 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "debug": { 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "debug", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.580 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable/disable debug options", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "jwt_token_ttl": { 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "jwt_token_ttl", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "28800", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.581 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.582 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.583 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.584 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.585 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.586 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.587 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.588 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "motd": { 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "motd", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.589 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "The message of the day", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_resolve_ip_addr": { 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.592 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8080", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.593 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.594 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl_server_port": { 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl_server_port", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.595 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8443", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.596 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "redirect", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect" 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:35.597 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.598 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "url_prefix": { 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url_prefix", 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.599 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.600 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "devicehealth", 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_monitoring": { 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_monitoring", 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.601 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "monitor device health metrics", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.602 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.603 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.604 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.605 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.606 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.607 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "mark_out_threshold": { 2024-03-12T05:27:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mark_out_threshold", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2419200", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": { 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_name", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "device_health_metrics", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.610 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "retention_period": { 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retention_period", 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.611 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15552000", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain device health metrics", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_frequency": { 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_frequency", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.612 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "self_heal": { 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "self_heal", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.613 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.614 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_threshold": { 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_threshold", 2024-03-12T05:27:35.615 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7257600", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.616 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "diskprediction_local", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.617 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.621 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.621 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.621 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.621 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.621 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.622 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.624 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "predict_interval": { 2024-03-12T05:27:35.625 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predict_interval", 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "predictor_model": { 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predictor_model", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.627 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "prophetstor", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.628 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "influx", 2024-03-12T05:27:35.629 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": false, 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "influxdb python module not found", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "batch_size": { 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "batch_size", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5000", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T05:27:35.630 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "database": { 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "database", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "ceph", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.631 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": { 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hostname", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.632 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server hostname", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "30", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "5", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.633 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.634 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.635 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.636 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.637 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.638 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "password": { 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "password", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.639 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "password of InfluxDB server user", 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.640 INFO:teuthology.orchestra.run.smithi045.stdout: "port": { 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "port", 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8086", 2024-03-12T05:27:35.641 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server port", 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.642 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.643 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "false", 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:35.644 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: "threads": { 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threads", 2024-03-12T05:27:35.645 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "32", 2024-03-12T05:27:35.646 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.647 INFO:teuthology.orchestra.run.smithi045.stdout: "username": { 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "username", 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.648 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "username of InfluxDB server user", 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.649 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "verify_ssl": { 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "verify_ssl", 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.650 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "true", 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.651 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "insights", 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.652 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.656 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.657 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "iostat", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.659 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.660 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.661 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.663 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.664 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.665 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.666 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.666 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.666 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.666 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.667 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "k8sevents", 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.668 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_event_retention_days": { 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ceph_event_retention_days", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T05:27:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "config_check_secs": { 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_check_secs", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "10", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.670 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.671 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.672 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.673 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.674 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.675 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.676 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "localpool", 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.677 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "failure_domain": { 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "failure_domain", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "host", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "failure domain for any created local pool", 2024-03-12T05:27:35.678 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.679 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.680 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.681 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.682 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.683 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": { 2024-03-12T05:27:35.684 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_size", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default min_size for any created local pool", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "num_rep": { 2024-03-12T05:27:35.685 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "num_rep", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default replica count for any created local pool", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": { 2024-03-12T05:27:35.686 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_num", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "128", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "prefix": { 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prefix", 2024-03-12T05:27:35.687 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name prefix for any created local pool", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "subtree": { 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "subtree", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.688 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "rack", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mds_autoscaler", 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.689 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.690 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.691 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirroring", 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "nfs", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.703 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.704 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "fail_fs": { 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "fail_fs", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.706 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.707 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.709 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.710 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.711 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator": { 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.712 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "rook", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "test_orchestrator" 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Orchestrator backend", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.713 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_perf_query", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.717 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.718 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_support", 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.719 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.720 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.721 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.724 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_autoscaler", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.725 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.726 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.727 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.728 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.729 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.730 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "threshold": { 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threshold", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:35.731 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1.0", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "scaling threshold", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "progress", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.732 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_pg_recovery_event": { 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_pg_recovery_event", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.733 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.734 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.735 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.736 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.737 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.738 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.739 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "max_completed_events": { 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_completed_events", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "50", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.740 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "number of past completed events to remember", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.741 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long the module is going to sleep", 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus", 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "cache": { 2024-03-12T05:27:35.742 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cache", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "exclude_perf_counters": { 2024-03-12T05:27:35.743 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "exclude_perf_counters", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.745 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.746 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.747 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.748 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools": { 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.749 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T05:27:35.750 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "300", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_interval": { 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_interval", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.751 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15.0", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.752 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.753 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "9283", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "stale_cache_strategy": { 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stale_cache_strategy", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.754 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "log", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.755 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "default", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "default", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "error" 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:35.756 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.757 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_support", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.758 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.759 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.760 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.762 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_snap_create": { 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_snap_create", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.763 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "mirror_snapshot_schedule": { 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.764 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "trash_purge_schedule": { 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "trash_purge_schedule", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.765 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "restful", 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_auth": { 2024-03-12T05:27:35.766 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_auth", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:35.767 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.768 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.769 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.770 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.771 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.772 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.773 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.774 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rgw", 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.776 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.777 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.778 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.779 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rook", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.781 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.782 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.783 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.784 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.785 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_class": { 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "storage_class", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "local", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T05:27:35.786 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "selftest", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.787 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.788 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.789 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.790 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "roption1": { 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption1", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.792 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "roption2": { 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption2", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.793 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "xyz", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption1": { 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption1", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.794 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption2": { 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption2", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.795 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption3": { 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption3", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.796 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption4": { 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption4", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.797 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption5": { 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption5", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.798 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption6": { 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption6", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption7": { 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption7", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.800 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "42", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "testkey": { 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testkey", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "testlkey": { 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testlkey", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.802 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "testnewline": { 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testnewline", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.803 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.804 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snap_schedule", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_m_granularity": { 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_m_granularity", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.805 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "dump_on_update": { 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dump_on_update", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.806 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "dump database to debug log on update", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.807 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.808 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.809 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.810 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.811 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stats", 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.812 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.813 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.814 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.815 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.816 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.817 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "status", 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.818 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.819 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.820 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.821 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.823 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telegraf", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "address": { 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "address", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.824 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.825 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.826 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.827 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.828 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.829 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.830 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telemetry", 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_basic": { 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_basic", 2024-03-12T05:27:35.831 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_crash": { 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_crash", 2024-03-12T05:27:35.832 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_device": { 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_device", 2024-03-12T05:27:35.833 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_ident": { 2024-03-12T05:27:35.834 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_ident", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_perf": { 2024-03-12T05:27:35.835 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_perf", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.836 INFO:teuthology.orchestra.run.smithi045.stdout: "contact": { 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "contact", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.837 INFO:teuthology.orchestra.run.smithi045.stdout: "description": { 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "description", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "device_url": { 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_url", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.840 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "24", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "8", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.841 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "last_opt_revision": { 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "last_opt_revision", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard": { 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.843 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard_description": { 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard_description", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.844 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.845 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.846 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.847 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.848 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "organization": { 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "organization", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.850 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "proxy": { 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "proxy", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "url": { 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test_orchestrator", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.853 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.854 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.855 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.856 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.857 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "volumes", 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.858 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.859 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.860 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.861 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.862 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_clones": { 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_clones", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4", 2024-03-12T05:27:35.863 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "periodic_async_work": { 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "periodic_async_work", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.864 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Periodically check for async work", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "snapshot_clone_delay": { 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snapshot_clone_delay", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.865 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix", 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.866 INFO:teuthology.orchestra.run.smithi045.stdout: "discovery_interval": { 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "discovery_interval", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "100", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.867 INFO:teuthology.orchestra.run.smithi045.stdout: "identifier": { 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "identifier", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.868 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.869 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.871 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.872 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.873 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_host": { 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_host", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.874 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_port": { 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_port", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.875 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10051", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_sender": { 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_sender", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.876 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "modules": [ 2024-03-12T05:27:35.877 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "dashboard", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "iostat", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "nfs", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "restful", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_schedule" 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "available_modules": [ 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "alerts", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:35.878 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.879 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.880 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.881 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.882 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.883 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_destination": { 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_destination", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.884 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email address to send alerts to", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_from_name": { 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_from_name", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.885 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "Ceph", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Email From: name", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_host": { 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_host", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.886 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP server", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_password": { 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_password", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.887 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Password to authenticate with", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_port": { 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_port", 2024-03-12T05:27:35.888 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "465", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP port", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_sender": { 2024-03-12T05:27:35.889 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_sender", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SMTP envelope sender", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.890 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_ssl": { 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_ssl", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "smtp_user": { 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "smtp_user", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "User to authenticate as", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.892 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "balancer", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "active": { 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "active", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.893 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_time": { 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_time", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0000", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.894 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "begin_weekday": { 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "begin_weekday", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.895 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_max_iterations": { 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_max_iterations", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "25", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.896 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "250", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_metrics": { 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_metrics", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T05:27:35.897 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_compat_step": { 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crush_compat_step", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.898 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.5", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0.001", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "0.999", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "aggressiveness of optimization", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "end_time": { 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_time", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.899 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2359", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "end_weekday": { 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "end_weekday", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "0", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "6", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.901 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.902 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.903 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.904 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.905 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "min_score": { 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_score", 2024-03-12T05:27:35.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "upmap", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-compat", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "none", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "read", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap", 2024-03-12T05:27:35.908 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap-read" 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Balancer mode", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_ids": { 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_ids", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.909 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.910 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_deviation": { 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_deviation", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:35.911 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "upmap_max_optimizations": { 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "upmap_max_optimizations", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.912 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm", 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.913 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_down_multiplier": { 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_down_multiplier", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.914 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_refresh_rate": { 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_refresh_rate", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "20", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.915 INFO:teuthology.orchestra.run.smithi045.stdout: "agent_starting_port": { 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "agent_starting_port", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4721", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_ptrace": { 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_ptrace", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.917 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_interval": { 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_interval", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "autotune_memory_target_ratio": { 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0.7", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.919 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm_log_destination": { 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cephadm_log_destination", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "file", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "file,syslog", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "syslog" 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T05:27:35.920 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "cgroups_split": { 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cgroups_split", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.921 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "config_checks_enabled": { 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_checks_enabled", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T05:27:35.922 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "config_dashboard": { 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_dashboard", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.923 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_alertmanager": { 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_alertmanager", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.924 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_base": { 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_base", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Container image name, without the tag", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_elasticsearch": { 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_elasticsearch", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "elasticsearch container image", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_grafana": { 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_grafana", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.927 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_haproxy": { 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_haproxy", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "HAproxy container image", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_agent": { 2024-03-12T05:27:35.928 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_agent", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger agent container image", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_collector": { 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_collector", 2024-03-12T05:27:35.929 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger collector container image", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_jaeger_query": { 2024-03-12T05:27:35.930 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_jaeger_query", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Jaeger query container image", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_keepalived": { 2024-03-12T05:27:35.931 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_keepalived", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Keepalived container image", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_loki": { 2024-03-12T05:27:35.932 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_loki", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Loki container image", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_node_exporter": { 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_node_exporter", 2024-03-12T05:27:35.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_nvmeof": { 2024-03-12T05:27:35.934 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_nvmeof", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Nvme-of container image", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_prometheus": { 2024-03-12T05:27:35.935 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_prometheus", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Prometheus container image", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_promtail": { 2024-03-12T05:27:35.936 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_promtail", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Promtail container image", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image_snmp_gateway": { 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_image_snmp_gateway", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "SNMP Gateway container image", 2024-03-12T05:27:35.938 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "container_init": { 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "container_init", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.939 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "daemon_cache_timeout": { 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "daemon_cache_timeout", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.940 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "default_cephadm_command_timeout": { 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "900", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.941 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "default_registry": { 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default_registry", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "docker.io", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.942 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "device_cache_timeout": { 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_cache_timeout", 2024-03-12T05:27:35.943 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1800", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache device inventory", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.944 INFO:teuthology.orchestra.run.smithi045.stdout: "device_enhanced_scan": { 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_enhanced_scan", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.945 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "facts_cache_timeout": { 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "facts_cache_timeout", 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:35.946 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "seconds to cache host facts data", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "host_check_interval": { 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "host_check_interval", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.947 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to perform a host check", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "hw_monitoring": { 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hw_monitoring", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.948 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "inventory_list_all": { 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "inventory_list_all", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.949 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.950 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.951 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "log_refresh_metadata": { 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_refresh_metadata", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.952 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.953 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.954 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.955 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T05:27:35.956 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.957 INFO:teuthology.orchestra.run.smithi045.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "*", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.958 INFO:teuthology.orchestra.run.smithi045.stdout: "max_count_per_host": { 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_count_per_host", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of daemons per service per host", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.959 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd_draining_count": { 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_osd_draining_count", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.960 INFO:teuthology.orchestra.run.smithi045.stdout: "migration_current": { 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "migration_current", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "internal - do not modify", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.961 INFO:teuthology.orchestra.run.smithi045.stdout: "mode": { 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mode", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "root", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm-package", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "root" 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T05:27:35.962 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "oob_default_addr": { 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "oob_default_addr", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "169.254.1.1", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T05:27:35.963 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "prometheus_alerts_path": { 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus_alerts_path", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.964 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_insecure": { 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_insecure", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.965 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_password": { 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_password", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.966 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_url": { 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_url", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.967 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "registry_username": { 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "registry_username", 2024-03-12T05:27:35.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "secure_monitoring_stack": { 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "secure_monitoring_stack", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T05:27:35.970 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "service_discovery_port": { 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "service_discovery_port", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8765", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.971 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "cephadm service discovery port", 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "ssh_config_file": { 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssh_config_file", 2024-03-12T05:27:35.972 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "use_agent": { 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_agent", 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.974 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "use_repo_digest": { 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "use_repo_digest", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_failed_host_check": { 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_failed_host_check", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_daemons": { 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_daemons", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_on_stray_hosts": { 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_on_stray_hosts", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crash", 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "retain_interval": { 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retain_interval", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "31536000", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_recent_interval": { 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_recent_interval", 2024-03-12T05:27:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1209600", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dashboard", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:35.989 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.990 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_ENABLED": { 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.991 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T05:27:35.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T05:27:35.993 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.997 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.998 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.000 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.001 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.002 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.003 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.004 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_URL": { 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_URL", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.005 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:36.006 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.007 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.008 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.009 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.010 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.011 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.012 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.013 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.014 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.015 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.016 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.017 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.018 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.019 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.020 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.021 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.022 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.023 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T05:27:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "45", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T05:27:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.026 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "admin", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.027 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.028 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.029 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.030 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.031 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.032 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "cross_origin_url": { 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cross_origin_url", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.033 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "crt_file": { 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "crt_file", 2024-03-12T05:27:36.034 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "debug": { 2024-03-12T05:27:36.035 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "debug", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Enable/disable debug options", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.036 INFO:teuthology.orchestra.run.smithi045.stdout: "jwt_token_ttl": { 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "jwt_token_ttl", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "28800", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.037 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.038 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.039 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.040 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "motd": { 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "motd", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "The message of the day", 2024-03-12T05:27:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_resolve_ip_addr": { 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8080", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl_server_port": { 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl_server_port", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8443", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "redirect", 2024-03-12T05:27:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect" 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "url_prefix": { 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url_prefix", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "devicehealth", 2024-03-12T05:27:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_monitoring": { 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_monitoring", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "monitor device health metrics", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.057 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "mark_out_threshold": { 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mark_out_threshold", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "2419200", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.060 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": { 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pool_name", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "device_health_metrics", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "retention_period": { 2024-03-12T05:27:36.061 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "retention_period", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15552000", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long to retain device health metrics", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_frequency": { 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_frequency", 2024-03-12T05:27:36.062 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "self_heal": { 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "self_heal", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.063 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.064 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "warn_threshold": { 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "warn_threshold", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7257600", 2024-03-12T05:27:36.065 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "diskprediction_local", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.066 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.067 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.068 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.069 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.070 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "predict_interval": { 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predict_interval", 2024-03-12T05:27:36.071 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "86400", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "predictor_model": { 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "predictor_model", 2024-03-12T05:27:36.072 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "prophetstor", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:36.073 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "600", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "influx", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": false, 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "influxdb python module not found", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "batch_size": { 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "batch_size", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5000", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T05:27:36.075 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "database": { 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "database", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "ceph", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T05:27:36.076 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": { 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "hostname", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server hostname", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.077 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "30", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "5", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.078 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.079 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.080 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.081 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.082 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.083 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "password": { 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "password", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "password of InfluxDB server user", 2024-03-12T05:27:36.084 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "port": { 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "port", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "8086", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.085 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "InfluxDB server port", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "ssl": { 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ssl", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "false", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.086 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "threads": { 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threads", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:36.087 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "32", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "username": { 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "username", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "username of InfluxDB server user", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "verify_ssl": { 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "verify_ssl", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "true", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.089 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "insights", 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.090 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.091 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.092 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.093 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.095 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "iostat", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.096 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.099 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.100 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "k8sevents", 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.101 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_event_retention_days": { 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "ceph_event_retention_days", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "7", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "config_check_secs": { 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "config_check_secs", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "10", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.103 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.104 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.105 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.106 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.107 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.108 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "localpool", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "failure_domain": { 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "failure_domain", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.109 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "host", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "failure domain for any created local pool", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.110 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.111 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.112 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.113 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.114 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.115 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": { 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "min_size", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default min_size for any created local pool", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.116 INFO:teuthology.orchestra.run.smithi045.stdout: "num_rep": { 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "num_rep", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default replica count for any created local pool", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.117 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": { 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_num", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "128", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.118 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "prefix": { 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prefix", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "name prefix for any created local pool", 2024-03-12T05:27:36.119 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "subtree": { 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "subtree", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "rack", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.120 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mds_autoscaler", 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.121 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.122 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.123 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.124 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.125 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.126 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirroring", 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.127 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.128 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.129 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.130 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.132 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "nfs", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.133 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.136 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.137 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.138 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "fail_fs": { 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "fail_fs", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.139 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.140 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.141 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.142 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.143 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.144 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator": { 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "orchestrator", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm", 2024-03-12T05:27:36.145 INFO:teuthology.orchestra.run.smithi045.stdout: "rook", 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "test_orchestrator" 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Orchestrator backend", 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_perf_query", 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.146 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.147 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.148 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.149 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.150 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.151 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.153 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "osd_support", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.154 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.155 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.156 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.157 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.158 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "pg_autoscaler", 2024-03-12T05:27:36.159 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.160 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.161 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.162 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.163 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.164 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "threshold": { 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "threshold", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "3.0", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1.0", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.166 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "scaling threshold", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "progress", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_pg_recovery_event": { 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_pg_recovery_event", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.167 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.168 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.169 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.170 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.171 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.172 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.173 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "max_completed_events": { 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_completed_events", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "50", 2024-03-12T05:27:36.174 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "number of past completed events to remember", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "sleep_interval": { 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "sleep_interval", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "5", 2024-03-12T05:27:36.175 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "how long the module is going to sleep", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "prometheus", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.176 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "cache": { 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "cache", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.177 INFO:teuthology.orchestra.run.smithi045.stdout: "exclude_perf_counters": { 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "exclude_perf_counters", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.178 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.179 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.180 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.181 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.183 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools": { 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.184 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "300", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.185 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "scrape_interval": { 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "scrape_interval", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15.0", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.186 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "::", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.187 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "9283", 2024-03-12T05:27:36.188 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "stale_cache_strategy": { 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stale_cache_strategy", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "log", 2024-03-12T05:27:36.189 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_behaviour": { 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_behaviour", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "default", 2024-03-12T05:27:36.190 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "default", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "error" 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "standby_error_status_code": { 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "standby_error_status_code", 2024-03-12T05:27:36.191 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "500", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "400", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "599", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_support", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.193 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.195 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.196 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.197 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_snap_create": { 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_snap_create", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.198 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "mirror_snapshot_schedule": { 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.199 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "trash_purge_schedule": { 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "trash_purge_schedule", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.200 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "restful", 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_auth": { 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enable_auth", 2024-03-12T05:27:36.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "key_file": { 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "key_file", 2024-03-12T05:27:36.202 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.203 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.204 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.205 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.206 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.207 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "server_addr": { 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_addr", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.208 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "server_port": { 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "server_port", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.209 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.210 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rgw", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.211 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.212 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.214 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.215 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rook", 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.216 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.217 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.218 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.219 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.220 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_class": { 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "storage_class", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.221 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "local", 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T05:27:36.222 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "selftest", 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.223 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.224 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.225 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.226 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.227 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "roption1": { 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption1", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.228 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "roption2": { 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "roption2", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.229 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "xyz", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption1": { 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption1", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.230 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption2": { 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption2", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.231 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption3": { 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption3", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "float", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.232 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption4": { 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption4", 2024-03-12T05:27:36.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption5": { 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption5", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption6": { 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption6", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "rwoption7": { 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rwoption7", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "1", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "42", 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "testkey": { 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testkey", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.525 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "testlkey": { 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testlkey", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.526 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "testnewline": { 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "testnewline", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.527 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snap_schedule", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_m_granularity": { 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "allow_m_granularity", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.528 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "dump_on_update": { 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "dump_on_update", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.529 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "dump database to debug log on update", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.530 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.532 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.533 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.534 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "stats", 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.535 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.536 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.537 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.538 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.539 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "status", 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.540 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.541 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.542 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.543 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.544 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telegraf", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "address": { 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "address", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.546 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "15", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.547 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.548 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.549 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.550 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "telemetry", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_basic": { 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_basic", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.553 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_crash": { 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_crash", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.554 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_device": { 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_device", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "True", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.555 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_ident": { 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_ident", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.556 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "channel_perf": { 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "channel_perf", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "contact": { 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "contact", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.558 INFO:teuthology.orchestra.run.smithi045.stdout: "description": { 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "description", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "device_url": { 2024-03-12T05:27:36.559 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "device_url", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": { 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "enabled", 2024-03-12T05:27:36.560 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:36.561 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "24", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "8", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "last_opt_revision": { 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "last_opt_revision", 2024-03-12T05:27:36.562 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "1", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard": { 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.563 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "leaderboard_description": { 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "leaderboard_description", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.564 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.565 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.566 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.568 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.569 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "organization": { 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "organization", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.570 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "proxy": { 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "proxy", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.571 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "url": { 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "url", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.572 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test_orchestrator", 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.573 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.575 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.576 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.577 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "volumes", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.578 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.579 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.580 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.581 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.582 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "max_concurrent_clones": { 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "max_concurrent_clones", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "4", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.583 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "periodic_async_work": { 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "periodic_async_work", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Periodically check for async work", 2024-03-12T05:27:36.584 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "snapshot_clone_delay": { 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snapshot_clone_delay", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "0", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.585 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix", 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "can_run": true, 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "error_string": "", 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "module_options": { 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "discovery_interval": { 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "discovery_interval", 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "uint", 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.586 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "100", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "identifier": { 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "identifier", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.587 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "interval": { 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "interval", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "secs", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "60", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.588 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "log_level": { 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_level", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.589 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster": { 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster", 2024-03-12T05:27:36.590 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_cluster_level": { 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_cluster_level", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.591 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "info", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [ 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "critical", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "debug", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "error", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "info", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "warning" 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.592 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": { 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "log_to_file", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "bool", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "False", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.593 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_host": { 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_host", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_port": { 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_port", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "int", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "10051", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.595 INFO:teuthology.orchestra.run.smithi045.stdout: "zabbix_sender": { 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "zabbix_sender", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "str", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "level": "advanced", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 0, 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "min": "", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "max": "", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "enum_allowed": [], 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "desc": "", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "long_desc": "", 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: "see_also": [] 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.596 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "services": { 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "dashboard": "https://172.21.15.45:8443/" 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "always_on_modules": { 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "octopus": [ 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "balancer", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "crash", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "devicehealth", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscaler", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "progress", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_support", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "status", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "telemetry", 2024-03-12T05:27:36.597 INFO:teuthology.orchestra.run.smithi045.stdout: "volumes" 2024-03-12T05:27:36.598 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "pacific": [ 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "balancer", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "crash", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "devicehealth", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscaler", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "progress", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_support", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "status", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "telemetry", 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "volumes" 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.599 INFO:teuthology.orchestra.run.smithi045.stdout: "quincy": [ 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "balancer", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "crash", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "devicehealth", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscaler", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "progress", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_support", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "status", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "telemetry", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "volumes" 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "reef": [ 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "balancer", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "crash", 2024-03-12T05:27:36.600 INFO:teuthology.orchestra.run.smithi045.stdout: "devicehealth", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscaler", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "progress", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_support", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "status", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "telemetry", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "volumes" 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "squid": [ 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "balancer", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "crash", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "devicehealth", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "orchestrator", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscaler", 2024-03-12T05:27:36.601 INFO:teuthology.orchestra.run.smithi045.stdout: "progress", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_support", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "status", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "telemetry", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "volumes" 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "last_failure_osd_epoch": 88, 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "active_clients": [ 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "devicehealth", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.602 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 205813945 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "libcephsqlite", 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 662476641 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.603 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "rbd_support", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1797031607 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snap_schedule", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.604 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 878647285 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "snap_schedule", 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 769859547 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "volumes", 2024-03-12T05:27:36.605 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:0", 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3280403885 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-12T05:27:36.606 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: pgmap v525: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2812450733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2242084255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2661414381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:34 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1760842653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:35 smithi116 ceph-mon[35486]: pgmap v526: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:35 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3807049212' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T05:27:36.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:35 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:35 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:35 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: pgmap v525: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2812450733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2242084255' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2661414381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:34 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1760842653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:35 smithi130 ceph-mon[35398]: pgmap v526: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:35 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3807049212' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:35 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:35 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:27:36.609 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:35 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 msgr2=0x7f81600a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 0x7f81600a4060 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f81740ffcf0 tx=0x7f8164005f90 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 msgr2=0x7f81741966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 0x7f81741966f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8170009230 tx=0x7f8170005dc0 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 shutdown_connections 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f81600a1ba0 0x7f81600a4060 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81741053c0 0x7f8174196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f817419af60 0x7f817419b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 --2- 172.21.15.45:0/3807049212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81741002f0 0x7f81741966f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:27:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 >> 172.21.15.45:0/3807049212 conn(0x7f8174077180 msgr2=0x7f81740fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:27:36.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 shutdown_connections 2024-03-12T05:27:36.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:27:34.936+0000 7f817beb5640 1 -- 172.21.15.45:0/3807049212 wait complete. 2024-03-12T05:27:36.616 DEBUG:tasks.check_counter:Getting stats from a 2024-03-12T05:27:36.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-12T05:27:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:35 smithi045 ceph-mon[32174]: pgmap v526: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-03-12T05:27:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:35 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3807049212' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T05:27:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:35 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:27:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:35 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:27:36.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:35 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' 2024-03-12T05:27:36.884 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:27:38.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:38 smithi045 ceph-mon[32174]: pgmap v527: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-12T05:27:38.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:38 smithi130 ceph-mon[35398]: pgmap v527: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-12T05:27:38.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:38 smithi116 ceph-mon[35486]: pgmap v527: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.3 KiB/s wr, 9 op/s 2024-03-12T05:27:39.033 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 2271, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 2272, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 680998, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 705971, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 6, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 2, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.193197555, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.059194975, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.051404953, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.067329683, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 2270, 2024-03-12T05:27:39.034 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.112464092, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000049543 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 1047, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 1280, 2024-03-12T05:27:39.035 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 258042, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 902535, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 8, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 5, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.114793375, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.037440733, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.023003952, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.041593189, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 1277, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.059776779, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000046810 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:39.036 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 577472 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 363, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 340, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 1819612, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 190727, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 4, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 2, 2024-03-12T05:27:39.037 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.060026511, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.019907780, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.043655057, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.007193061, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 340, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.017203155, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000050597 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.038 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 1819344, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 190416 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "cct": { 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "total_workers": 1, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "unhealthy_workers": 0 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "finisher-MDSRank": { 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "queue_len": 0, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "complete_latency": { 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 5368, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.160793510, 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000029954 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "queue_len": 0, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "complete_latency": { 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": { 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "request": 0, 2024-03-12T05:27:39.040 INFO:teuthology.orchestra.run.smithi045.stdout: "reply": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "reply_latency": { 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "slow_reply": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "forward": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_commit": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_split": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_merge": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes": 0, 2024-03-12T05:27:39.041 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_top": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_bottom": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_pin_tail": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_pinned": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_expired": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_with_caps": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "subtrees": 1, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_hit": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_forward": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_discover": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:39.042 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_lock": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "load_cent": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "q": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "exported": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "exported_inodes": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "imported": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "imported_inodes": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rfiles": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rbytes": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rsnaps": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:39.043 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_file_inodes": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_caps": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_cap_release": 0, 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: "process_request_cap_release": 0 2024-03-12T05:27:39.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache": { 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_update": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_try_discover": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_send_discover": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_handle_discover": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_created": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_enqueued": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_reintegrated": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_migrated": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:39.045 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_started": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_completed": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_exportdir": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_flush": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_inodestats": 0 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log": { 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "evlrg": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "evadd": 0, 2024-03-12T05:27:39.046 INFO:teuthology.orchestra.run.smithi045.stdout: "evex": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "evtrm": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "ev": 7, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "evexg": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "evexd": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segadd": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segex": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segtrm": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "seg": 2, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segmjr": 2, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segexg": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "segexd": 0, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "expos": 4194304, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "wrpos": 4196433, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "rdpos": 4196292, 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "jlat": { 2024-03-12T05:27:39.047 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "replayed": 7 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_mem": { 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ino": 3, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ino+": 3, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "ino-": 0, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dir": 1, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dir+": 1, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dir-": 0, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dn": 0, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dn+": 0, 2024-03-12T05:27:39.048 INFO:teuthology.orchestra.run.smithi045.stdout: "dn-": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "cap": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "cap+": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "cap-": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "rss": 50156, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "heap": 215420 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_server": { 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "dispatch_client_request": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "dispatch_server_request": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_request": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_session": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_peer_request": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "req_create_latency": { 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.049 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getattr_latency": { 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "req_link_latency": { 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.050 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookup_latency": { 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.051 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupino_latency": { 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupname_latency": { 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.052 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lssnap_latency": { 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mkdir_latency": { 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mknod_latency": { 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mksnap_latency": { 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.055 INFO:teuthology.orchestra.run.smithi045.stdout: "req_open_latency": { 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "req_readdir_latency": { 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rename_latency": { 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.056 INFO:teuthology.orchestra.run.smithi045.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmdir_latency": { 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.057 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setattr_latency": { 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setlayout_latency": { 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setxattr_latency": { 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "req_symlink_latency": { 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "req_unlink_latency": { 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.060 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_sessions": { 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "session_count": 1, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "session_add": 3, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "session_remove": 2, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "sessions_open": 2, 2024-03-12T05:27:39.061 INFO:teuthology.orchestra.run.smithi045.stdout: "sessions_stale": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "total_load": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "average_load": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "avg_session_uptime": 3364, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "mempool": { 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:39.062 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:39.063 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_bytes": 190185, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_items": 53, 2024-03-12T05:27:39.064 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_items": 189, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_items": 0, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_bytes": 8320, 2024-03-12T05:27:39.065 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_items": 13, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_items": 0, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_items": 0 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter": { 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_active": 0, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_laggy": 0, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_send": 3316, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_send_bytes": 21, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_resend": 3, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_reply": 3316, 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "op_latency": { 2024-03-12T05:27:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 3316, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 4.488936576, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.001353720 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op_inflight": 0, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "oplen_avg": { 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 3316, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 3317 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op": 3316, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op_r": 3316, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op_w": 0, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op_rmw": 0, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "op_pg": 0, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_stat": 2174, 2024-03-12T05:27:39.067 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_create": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_read": 1140, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_write": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_writefull": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_writesame": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_append": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_zero": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_truncate": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_delete": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_mapext": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_getxattr": 1, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_setxattr": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:39.068 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_call": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_watch": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_notify": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_pgls": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_other": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_active": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_send": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_resend": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_ping": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_active": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_send": 0, 2024-03-12T05:27:39.069 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_resend": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_active": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_send": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_resend": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_active": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_send": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_resend": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "command_active": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "command_send": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "command_resend": 0, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "map_epoch": 110, 2024-03-12T05:27:39.070 INFO:teuthology.orchestra.run.smithi045.stdout: "map_full": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "map_inc": 25, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_sessions": 5, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_session_open": 7, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_session_close": 2, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_laggy": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_wr": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_rd": 2, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_del": 0 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "oft": { 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_objs": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_updates": 0, 2024-03-12T05:27:39.071 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_removes": 0 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "osd-slow-ops": { 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "slow_ops_count": 0 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "purge_queue": { 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_high_water": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executed_ops": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executed": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:39.072 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 3677, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 2406515, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 3677, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3677, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 2406515, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.073 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 3316, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 2134, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3316, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 2134, 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 1024, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 3316, 2024-03-12T05:27:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 3316, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3316, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 3316, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 3758096384, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-write_buf_throttle-0x55f64c8ac960": { 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 3758096384, 2024-03-12T05:27:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:39.078 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:39.992 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-12T05:27:39.992 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-12T05:27:39.992 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-12T05:27:39.992 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:27:39.992 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:27:39.992 DEBUG:tasks.check_counter:Getting stats from d 2024-03-12T05:27:39.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-12T05:27:40.260 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:27:40.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:40 smithi045 ceph-mon[32174]: pgmap v528: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-12T05:27:40.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:40 smithi130 ceph-mon[35398]: pgmap v528: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-12T05:27:40.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:40 smithi116 ceph-mon[35486]: pgmap v528: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-03-12T05:27:42.242 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:42.242 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:42.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 324, 2024-03-12T05:27:42.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 312, 2024-03-12T05:27:42.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 1729104, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 187856, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 4, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 0, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.059565451, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.020443642, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.040434172, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.007149249, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 311, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.013469698, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000043310 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:42.243 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 1729104, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 187856 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 3, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 239, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 480, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 274384, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 6, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 3, 2024-03-12T05:27:42.244 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.018685884, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.013094253, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.002432326, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 236, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.019670590, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000083349 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.245 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 274384 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 4, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 4, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 2992, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 448, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 2, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 1, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.000682164, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.000039398, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.000624639, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T05:27:42.246 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 3, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000748515, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000249505 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "cct": { 2024-03-12T05:27:42.247 INFO:teuthology.orchestra.run.smithi045.stdout: "total_workers": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "unhealthy_workers": 0 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "mempool": { 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:42.248 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:42.249 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_bytes": 148794, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_items": 32, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bytes": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_items": 0, 2024-03-12T05:27:42.250 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_bytes": 928, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_items": 2, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_items": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:42.251 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_items": 0 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 327, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 1683411, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 327, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 327, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 1683411, 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:42.252 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:42.253 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:42.253 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:42.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:42.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:42.253 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:42.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:42 smithi045 ceph-mon[32174]: pgmap v529: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-12T05:27:42.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:42 smithi130 ceph-mon[35398]: pgmap v529: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-12T05:27:42.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:42 smithi116 ceph-mon[35486]: pgmap v529: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-03-12T05:27:43.058 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-12T05:27:43.058 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-12T05:27:43.059 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-12T05:27:43.059 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-12T05:27:43.059 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-12T05:27:43.059 DEBUG:tasks.check_counter:Getting stats from g 2024-03-12T05:27:43.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-12T05:27:43.307 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:27:44.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:44 smithi045 ceph-mon[32174]: pgmap v530: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-12T05:27:44.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:44 smithi130 ceph-mon[35398]: pgmap v530: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-12T05:27:44.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:44 smithi116 ceph-mon[35486]: pgmap v530: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-12T05:27:45.237 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:45.237 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 1501, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 1479, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 2200362, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 545756, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 5, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 1, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.190904702, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.068855031, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.073337370, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.049043365, 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:45.238 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 1478, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.077066046, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000052142 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 1816896, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 191216 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 2188, 2024-03-12T05:27:45.239 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 2420, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 582642, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 1256755, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 7, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 5, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.163060611, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.053982961, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.037680907, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.060654161, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 2418, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.108164943, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000044733 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.240 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 578096 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 4, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 6, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 1629, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 1256, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 7, 2024-03-12T05:27:45.241 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 4, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.002185291, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.000051549, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.001806717, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.000066992, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 4, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.002413279, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000603319 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.242 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "cct": { 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "total_workers": 1, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "unhealthy_workers": 0 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "finisher-MDSRank": { 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "queue_len": 0, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "complete_latency": { 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 5276, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.149771353, 2024-03-12T05:27:45.243 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000028387 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "queue_len": 0, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "complete_latency": { 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": { 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "request": 0, 2024-03-12T05:27:45.244 INFO:teuthology.orchestra.run.smithi045.stdout: "reply": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "reply_latency": { 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "slow_reply": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "forward": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_commit": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_split": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_merge": 0, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes": 9, 2024-03-12T05:27:45.245 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_top": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_bottom": 9, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_pin_tail": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_pinned": 9, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_expired": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "inodes_with_caps": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "caps": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "subtrees": 3, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_hit": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_forward": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_discover": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:45.246 INFO:teuthology.orchestra.run.smithi045.stdout: "traverse_lock": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "load_cent": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "q": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "exported": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "exported_inodes": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "imported": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "imported_inodes": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rfiles": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rbytes": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "root_rsnaps": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:27:45.247 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "scrub_file_inodes": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_caps": 0, 2024-03-12T05:27:45.248 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_cap_release": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "process_request_cap_release": 0 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache": { 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_update": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_try_discover": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_send_discover": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "dir_handle_discover": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_created": 0, 2024-03-12T05:27:45.249 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_enqueued": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_reintegrated": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "strays_migrated": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_started": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_completed": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_exportdir": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_flush": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "ireq_inodestats": 0 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.250 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log": { 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evlrg": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evadd": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evex": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evtrm": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "ev": 23, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evexg": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "evexd": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segadd": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segex": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segtrm": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "seg": 2, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segmjr": 2, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segexg": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "segexd": 0, 2024-03-12T05:27:45.251 INFO:teuthology.orchestra.run.smithi045.stdout: "expos": 4194304, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "wrpos": 4218558, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "rdpos": 4217984, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "jlat": { 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "replayed": 23 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_mem": { 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "ino": 12, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "ino+": 12, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "ino-": 0, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "dir": 9, 2024-03-12T05:27:45.252 INFO:teuthology.orchestra.run.smithi045.stdout: "dir+": 9, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dir-": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dn": 9, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dn+": 9, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dn-": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "cap": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "cap+": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "cap-": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "rss": 50080, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "heap": 215420 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_server": { 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dispatch_client_request": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "dispatch_server_request": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_request": 0, 2024-03-12T05:27:45.253 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_client_session": 0, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "handle_peer_request": 0, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "req_create_latency": { 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getattr_latency": { 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.254 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "req_link_latency": { 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookup_latency": { 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.255 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupino_latency": { 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupname_latency": { 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.256 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "req_lssnap_latency": { 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mkdir_latency": { 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mknod_latency": { 2024-03-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.258 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "req_mksnap_latency": { 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "req_open_latency": { 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "req_readdir_latency": { 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.259 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rename_latency": { 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:45.260 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmdir_latency": { 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.261 INFO:teuthology.orchestra.run.smithi045.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setattr_latency": { 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setlayout_latency": { 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "req_setxattr_latency": { 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "req_symlink_latency": { 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "req_unlink_latency": { 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.264 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_sessions": { 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "session_count": 2, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "session_add": 5, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "session_remove": 3, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "sessions_open": 4, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "sessions_stale": 0, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "total_load": 0, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "average_load": 0, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "avg_session_uptime": 5735, 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:45.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "mempool": { 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:45.266 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:45.267 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_bytes": 198274, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_items": 58, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:45.268 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bytes": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_items": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_items": 189, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_items": 0, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_bytes": 47411, 2024-03-12T05:27:45.269 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_items": 191, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_items": 0, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_items": 0 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter": { 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_active": 0, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_laggy": 0, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_send": 3326, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_send_bytes": 21, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_resend": 3, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_reply": 3326, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "op_latency": { 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 3326, 2024-03-12T05:27:45.270 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 2.721961136, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000818388 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op_inflight": 0, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "oplen_avg": { 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 3326, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 3327 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op": 3326, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op_r": 3326, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op_w": 0, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op_rmw": 0, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "op_pg": 0, 2024-03-12T05:27:45.271 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_stat": 2168, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_create": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_read": 1156, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_write": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_writefull": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_writesame": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_append": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_zero": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_truncate": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_delete": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_mapext": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_getxattr": 1, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_setxattr": 0, 2024-03-12T05:27:45.272 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_call": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_watch": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_notify": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_pgls": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "osdop_other": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_active": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_send": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_resend": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "linger_ping": 0, 2024-03-12T05:27:45.273 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_active": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_send": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "poolop_resend": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_active": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_send": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "poolstat_resend": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_active": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_send": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "statfs_resend": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "command_active": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "command_send": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "command_resend": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "map_epoch": 110, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "map_full": 0, 2024-03-12T05:27:45.274 INFO:teuthology.orchestra.run.smithi045.stdout: "map_inc": 25, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_sessions": 6, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_session_open": 8, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_session_close": 2, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_laggy": 0, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_wr": 0, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_rd": 2, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_del": 0 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "oft": { 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_objs": 0, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_updates": 0, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "omap_total_removes": 0 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.275 INFO:teuthology.orchestra.run.smithi045.stdout: "osd-slow-ops": { 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "slow_ops_count": 0 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "purge_queue": { 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executing_high_water": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executed_ops": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_executed": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:45.276 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 3689, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 2431312, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 3689, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3689, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 2431312, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 3326, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 24259, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3326, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 24259, 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:45.278 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 1024, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 3326, 2024-03-12T05:27:45.279 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 3326, 2024-03-12T05:27:45.280 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 3326, 2024-03-12T05:27:45.280 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 3326, 2024-03-12T05:27:45.280 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 3758096384, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:45.281 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-write_buf_throttle-0x5606e9380960": { 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 3758096384, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:45.282 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:45.283 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:46.143 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-12T05:27:46.144 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-12T05:27:46.144 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-12T05:27:46.144 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:27:46.144 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:27:46.144 DEBUG:tasks.check_counter:Getting stats from j 2024-03-12T05:27:46.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-12T05:27:46.395 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:27:46.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:46 smithi045 ceph-mon[32174]: pgmap v531: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-12T05:27:46.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:46 smithi130 ceph-mon[35398]: pgmap v531: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-12T05:27:46.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:46 smithi116 ceph-mon[35486]: pgmap v531: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-03-12T05:27:48.131 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-12T05:27:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 0, 2024-03-12T05:27:48.131 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 2, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 0, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 0, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 4, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 0, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.001363003, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.000000000, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.000471358, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.132 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 327, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 315, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 1729920, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 189728, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 2, 2024-03-12T05:27:48.133 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 1, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.061409216, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.018209955, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.045046266, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.007749925, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 314, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.019629873, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000062515 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:48.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 1729920, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 189728 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_messages": 7, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages": 243, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_bytes": 3472, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_bytes": 275184, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_created_connections": 6, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_active_connections": 3, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_total_time": 0.018072166, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_send_time": 0.012501950, 2024-03-12T05:27:48.135 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_recv_time": 0.002688407, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 241, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.018554415, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000076989 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: "msgr_send_encrypted_bytes": 275184 2024-03-12T05:27:48.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "cct": { 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "total_workers": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "unhealthy_workers": 0 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "mempool": { 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:48.137 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_items": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:48.139 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_bytes": 148794, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_anon_items": 32, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_bytes": 0, 2024-03-12T05:27:48.140 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_items": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "pgmap_items": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_bytes": 928, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_co_items": 2, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_1_items": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "unittest_2_items": 0 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 0, 2024-03-12T05:27:48.141 INFO:teuthology.orchestra.run.smithi045.stdout: "max": 104857600, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "get_started": 0, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "get": 330, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "get_sum": 1683801, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "get_or_fail_success": 330, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "take": 0, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "take_sum": 0, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "put": 330, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "put_sum": 1683801, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "wait": { 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "avgcount": 0, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "sum": 0.000000000, 2024-03-12T05:27:48.142 INFO:teuthology.orchestra.run.smithi045.stdout: "avgtime": 0.000000000 2024-03-12T05:27:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:48.143 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-12T05:27:48.143 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-12T05:27:48.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:48 smithi130 ceph-mon[35398]: pgmap v532: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:27:48.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:48 smithi116 ceph-mon[35486]: pgmap v532: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:27:48.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:48 smithi045 ceph-mon[32174]: pgmap v532: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:27:48.788 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-12T05:27:48.788 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-12T05:27:48.788 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-12T05:27:48.788 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-12T05:27:48.789 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-12T05:27:48.789 DEBUG:tasks.check_counter:Getting stats from b 2024-03-12T05:27:48.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-12T05:27:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:27:50.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:50 smithi116 ceph-mon[35486]: pgmap v533: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-12T05:27:50.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:50 smithi130 ceph-mon[35398]: pgmap v533: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-12T05:27:50.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:50 smithi045 ceph-mon[32174]: pgmap v533: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-12T05:27:50.949 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-12T05:27:50.949 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:50.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2963, 2024-03-12T05:27:50.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2024, 2024-03-12T05:27:50.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1312986, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 5165520, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 18, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.284487207, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.069434131, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.102975666, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.091366659, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2021, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.425797977, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000210686 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:50.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3368, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000597316, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000177 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 536928, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 169088 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 4582, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 3980, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 2047763, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 3601525, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 247, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 8, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.353325731, 2024-03-12T05:27:50.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.090850879, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.221596008, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.056506321, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3983, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.395383476, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000099267 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 7666, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001109277, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000144 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 45376, 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 135312 2024-03-12T05:27:50.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 8758, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 8346, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 4519133, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 2486006, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 15, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 8, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.643672924, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.213087797, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.431563047, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.087265160, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8341, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.329999382, 2024-03-12T05:27:50.953 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000039563 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 15214, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002292436, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000150 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1920192, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 721216 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.954 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 938, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.387970146, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000413614 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 17, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002318572, 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000136386 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 5844, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 804, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 804, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 18.661777926, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.023211166 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 5040, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 21, 2024-03-12T05:27:50.956 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 1, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 45, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 3, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 18, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 24, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 40, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 205, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 19, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 19, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 3, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 6263, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 1202, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 5039, 2024-03-12T05:27:50.957 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 27, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 224, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 10, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 359, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:50.958 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 1024, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 1654150485, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 6, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 359, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 239, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 30, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 209, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 120, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 329, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 214, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:50.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 364, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 31, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 1, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 247 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 32, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:50.960 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 3, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 2, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 1, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-12T05:27:50.961 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 30, 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 27, 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 1, 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-12T05:27:50.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 601, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 601, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 3, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 3, 2024-03-12T05:27:50.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 8553666, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 0, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 300, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.258461483, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.004194871 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 0 2024-03-12T05:27:50.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 45, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 278, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 233, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 59, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 67, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 8, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 45, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 252, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 207, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 19, 2024-03-12T05:27:50.965 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 255, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 236, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 59820, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 6340, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 1, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 5844, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 46, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 222, 2024-03-12T05:27:50.966 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 204, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.210319278, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001030976 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 205, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.034488562, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000168236 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.003366039, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003366039 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 297, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 18.129344915, 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.061041565 2024-03-12T05:27:50.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.971 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-12T05:27:50.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 17, 2024-03-12T05:27:50.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002568136, 2024-03-12T05:27:50.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000151066 2024-03-12T05:27:50.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 31, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.040961084, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001321325 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 17, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.092253287, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.005426663 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 4, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002147107, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000536776 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-12T05:27:50.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002750713, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001375356 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.072317187, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.024105729 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 11, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.043220564, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003929142 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001247731, 2024-03-12T05:27:50.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001247731 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.977 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.022748720, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002843590 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.978 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.004044603, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001348201 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 5, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 3, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 88, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 44, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 1019, 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:50.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:50.981 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:50.982 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:50.983 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 297495, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 121, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:50.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 189, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 292394, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 1752, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.985 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 1353, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 4481527, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 25, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 1353, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1353, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 21.147669619, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.015630206 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.986 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1353, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1402 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 1353, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 420, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 933, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 0, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 11, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 31, 2024-03-12T05:27:50.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 301, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 66, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 1, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 515, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 389, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 22, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:50.988 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-12T05:27:50.989 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 110, 2024-03-12T05:27:50.990 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 26, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 12, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 12, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 59, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 0, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 7 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 1, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:50.991 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 2, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 2, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 2, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 2, 2024-03-12T05:27:50.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 16299, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 6319578, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 16299, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 16299, 2024-03-12T05:27:50.993 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 6319578, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:50.994 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 1353, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 4481729, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1353, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 4481729, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 1353, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 1353, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1353, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1353, 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:50.996 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 2, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 202, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:50.997 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 2, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 202, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x55bd0e8786e0": { 2024-03-12T05:27:50.998 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 601, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 4359362, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 601, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 300, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 4359362, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:50.999 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:51.000 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:51.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:51.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:51.000 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-12T05:27:51.701 INFO:tasks.check_counter:Daemon mds.b mds.exported=27 2024-03-12T05:27:51.701 INFO:tasks.check_counter:Daemon mds.b mds.imported=10 2024-03-12T05:27:51.701 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-12T05:27:51.701 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=17 2024-03-12T05:27:51.702 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=11 2024-03-12T05:27:51.702 DEBUG:tasks.check_counter:Getting stats from e 2024-03-12T05:27:51.702 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-12T05:27:51.952 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:27:52.554 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:52 smithi116 ceph-mon[35486]: pgmap v534: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.1 KiB/s wr, 10 op/s 2024-03-12T05:27:52.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:52 smithi130 ceph-mon[35398]: pgmap v534: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.1 KiB/s wr, 10 op/s 2024-03-12T05:27:52.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:52 smithi045 ceph-mon[32174]: pgmap v534: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.1 KiB/s wr, 10 op/s 2024-03-12T05:27:53.915 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-12T05:27:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:53.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2449, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2451, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 4918613, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 761649, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.176720289, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.063834934, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.044972084, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.053520604, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2449, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.060847158, 2024-03-12T05:27:53.916 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000024845 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 11, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 248, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 5128, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 671982, 2024-03-12T05:27:53.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 10, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.013015219, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.007728609, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002953785, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000086044, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 246, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.022109880, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000089877 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.918 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 670672 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1536, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1515, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 2272111, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 558680, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-12T05:27:53.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.131625959, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.038296903, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.059190994, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.037913916, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1514, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.037665903, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000024878 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.920 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1818688, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 195120 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 6040, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.128912718, 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000021343 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-12T05:27:53.922 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 17, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 17, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-12T05:27:53.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 17, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 3, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-12T05:27:53.924 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 1024, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 1654150485, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 6, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:53.925 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-12T05:27:53.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:53.927 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-12T05:27:53.928 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 601, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 3, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 8553666, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 8504949, 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-12T05:27:53.929 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 601 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 15, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 239, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 224, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 10, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 42, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 32, 2024-03-12T05:27:53.930 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 17, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 240, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 223, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 53680, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-12T05:27:53.931 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:53.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.935 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-12T05:27:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.936 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.940 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-12T05:27:53.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.942 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 5, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 3, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 3, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-12T05:27:53.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 5143, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:53.944 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:53.945 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:53.946 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 198297, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 58, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 189, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:53.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 126490, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 368, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 3622, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 26, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 4, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 3622, 2024-03-12T05:27:53.948 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3622, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.510976954, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000417166 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3622, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 3623 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 3622, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 3622, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-12T05:27:53.949 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 2278, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 1341, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 2, 2024-03-12T05:27:53.950 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-12T05:27:53.951 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 110, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-12T05:27:53.952 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 25, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 6, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 8, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 2, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-12T05:27:53.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:53.954 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 3992, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 6814860, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 3992, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3992, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 6814860, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3622, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 4359372, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3622, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 4359372, 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:53.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3622, 2024-03-12T05:27:53.957 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 3622, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3622, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3622, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:53.958 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x55a7d2eea960": { 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:53.959 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:53.960 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-12T05:27:54.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:54 smithi116 ceph-mon[35486]: pgmap v535: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-03-12T05:27:54.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:54 smithi130 ceph-mon[35398]: pgmap v535: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-03-12T05:27:54.738 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-12T05:27:54.738 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-12T05:27:54.738 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-12T05:27:54.738 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:27:54.738 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:27:54.738 DEBUG:tasks.check_counter:Getting stats from h 2024-03-12T05:27:54.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-12T05:27:54.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:54 smithi045 ceph-mon[32174]: pgmap v535: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-03-12T05:27:55.032 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:27:56.689 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:56 smithi116 ceph-mon[35486]: pgmap v536: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-12T05:27:56.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:56 smithi130 ceph-mon[35398]: pgmap v536: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 32, 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 241, 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 8123, 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 484423, 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 13, 2024-03-12T05:27:56.720 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.016972629, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.006249140, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.003804550, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000630880, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 237, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.031177341, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000131549 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.721 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000001318, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000164 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 451552 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 687, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 917, 2024-03-12T05:27:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 219284, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 1060280, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 71, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.092597847, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.035417008, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.050526439, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.005098685, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 917, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.044854795, 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000048914 2024-03-12T05:27:56.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1471, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000271098, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000184 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 132448 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 430, 2024-03-12T05:27:56.724 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 461, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1843008, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 244407, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 10, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.058903741, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.018184007, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.044701180, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.006448403, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 460, 2024-03-12T05:27:56.725 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.020161097, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000043828 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 96, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000020614, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000214 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1838240, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 200720 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.726 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 54, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001503530, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000027843 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:56.727 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000007069, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000003534 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 5, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 2, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001155549, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000577774 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.728 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 3, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 11, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 14, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 14, 2024-03-12T05:27:56.729 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 14, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 1, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 1, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 2, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 8, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 3, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 2, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-12T05:27:56.730 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 1030, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 1655469102, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 6, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:56.731 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 1, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 1, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:27:56.732 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 3 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 3, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:56.733 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-12T05:27:56.734 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 7, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 7, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 2, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-12T05:27:56.735 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 2, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 4196433, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 0, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 6, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.018444366, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003074061 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 0 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-12T05:27:56.736 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 17, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 17, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 0, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 15, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 16, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 1, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 14, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 14, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 0, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 1, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 2, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 1, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 54456, 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 207228 2024-03-12T05:27:56.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 7, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 5, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 20, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2, 2024-03-12T05:27:56.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001155549, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000577774 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-12T05:27:56.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-12T05:27:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-12T05:27:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.744 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.745 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.746 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.747 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-12T05:27:56.748 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 1, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 3, 2024-03-12T05:27:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 2, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 1, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 1, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 1, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 1040, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:56.750 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:56.751 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 255887, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 83, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 189, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:56.753 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 66981, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 292, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 44, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 12790, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 33, 2024-03-12T05:27:56.754 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 44, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 44, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.238996244, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.005431732 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 44, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 67 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 44, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 0, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 44, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-12T05:27:56.755 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 10, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 6, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 8, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 6, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:56.756 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 20, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-12T05:27:56.757 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-12T05:27:56.758 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 110, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 25, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 8, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 14, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 6, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 16, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 0, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 1 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 1, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:56.759 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-12T05:27:56.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 1145, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1943782, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 1145, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1145, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1943782, 2024-03-12T05:27:56.761 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:56.762 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 44, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 12790, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 44, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 12790, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:56.763 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 44, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 44, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 44, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 44, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.764 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.765 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x560dfb2c46e0": { 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 7, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 2129, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 7, 2024-03-12T05:27:56.766 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 6, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2129, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:56.767 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-12T05:27:56.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:56 smithi045 ceph-mon[32174]: pgmap v536: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-12T05:27:57.492 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-12T05:27:57.492 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-12T05:27:57.492 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-12T05:27:57.492 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:27:57.492 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:27:57.492 DEBUG:tasks.check_counter:Getting stats from k 2024-03-12T05:27:57.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-12T05:27:57.734 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:27:58.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:27:58 smithi130 ceph-mon[35398]: pgmap v537: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-12T05:27:58.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:27:58 smithi116 ceph-mon[35486]: pgmap v537: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-12T05:27:58.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:27:58 smithi045 ceph-mon[32174]: pgmap v537: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1224, 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1202, 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 44178460, 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 462432, 2024-03-12T05:27:59.419 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.133420199, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.030813779, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.084111614, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.024436072, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1201, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.029315888, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000024409 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.420 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1819344, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 196832 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1276, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1516, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 75919383, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 1067464, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 9, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.133735574, 2024-03-12T05:27:59.421 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.028624584, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.068829793, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.027438476, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1514, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.044889616, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000029649 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-12T05:27:59.422 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 672800 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1828, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1830, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 46791289, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 568566, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.165846708, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.046813689, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.058734720, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.047900739, 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:27:59.423 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1828, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.053017820, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000029003 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-12T05:27:59.424 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 6694, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.140768115, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000021028 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.425 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-12T05:27:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 2486, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 2486, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 2486, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 12, 2024-03-12T05:27:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-12T05:27:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:27:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-12T05:27:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-12T05:27:59.431 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:27:59.432 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 20842, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-12T05:27:59.433 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 46, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 3, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 167909790, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 167909216, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 20842 2024-03-12T05:27:59.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1786, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 2337, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 551, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 102, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 120, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 18, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 2486, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 2788, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 302, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 59184, 2024-03-12T05:27:59.435 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-12T05:27:59.436 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-12T05:27:59.437 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.438 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.439 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-12T05:27:59.440 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.441 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-12T05:27:59.442 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.443 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.444 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 5, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 3, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 4, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 3784, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:27:59.448 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-12T05:27:59.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-12T05:27:59.450 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 232903, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 77, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8264, 2024-03-12T05:27:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 189, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 5804257, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 53165, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 3951, 2024-03-12T05:27:59.452 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 3951, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3951, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 5.244921029, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001327492 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3951, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 3952 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 3951, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 3951, 2024-03-12T05:27:59.453 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 2267, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 1682, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-12T05:27:59.454 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-12T05:27:59.455 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 110, 2024-03-12T05:27:59.456 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 25, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 10, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 10, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:59.458 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 4324, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 166478150, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 4324, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 4324, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 166478150, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.459 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3951, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 163965520, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3951, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 163965520, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.460 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3951, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 3951, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3951, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3951, 2024-03-12T05:27:59.461 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:59.462 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x55ff672c4960": { 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-12T05:27:59.463 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-12T05:27:59.464 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-12T05:28:00.219 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-12T05:28:00.220 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-12T05:28:00.220 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-12T05:28:00.220 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:28:00.220 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:28:00.220 DEBUG:tasks.check_counter:Getting stats from c 2024-03-12T05:28:00.220 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-12T05:28:00.687 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:28:00.715 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:00 smithi130 ceph-mon[35398]: pgmap v538: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:00.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:00 smithi116 ceph-mon[35486]: pgmap v538: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:00.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:00 smithi045 ceph-mon[32174]: pgmap v538: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:02.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:02 smithi130 ceph-mon[35398]: pgmap v539: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:02.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:02 smithi116 ceph-mon[35486]: pgmap v539: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:02.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:02 smithi045 ceph-mon[32174]: pgmap v539: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-12T05:28:03.373 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 35092, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 36054, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 12875498, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 49624309, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 14, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 2.101415629, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.813354566, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 1.513224867, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.111057675, 2024-03-12T05:28:03.374 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 36053, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2.075117329, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000057557 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 57828, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.009138378, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000158 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1854960, 2024-03-12T05:28:03.375 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 237360 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 10044, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 10319, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 2624987, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 67020017, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 71, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.723818219, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.305883429, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.171859928, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.203097786, 2024-03-12T05:28:03.376 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 10316, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.361378300, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000035030 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1536, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000299165, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000194 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2928, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 136752 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.377 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 14129, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 14881, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 3835380, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 70951666, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 11, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.839254609, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.311758656, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.245048518, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.241787457, 2024-03-12T05:28:03.378 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 14880, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.537113986, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000036096 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5341, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000956284, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000179 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 455600 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-12T05:28:03.379 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 22079, 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 9.346409116, 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000423316 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.380 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 941, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.033429880, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000035525 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 9639, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 9620, 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-12T05:28:03.381 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 9620, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 18.635147281, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001937125 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 19, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 71, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 7, 2024-03-12T05:28:03.382 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 2204, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 212, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 1977, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 15, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 1901, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 291, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 761, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 761, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 12, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 16756, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 13848, 2024-03-12T05:28:03.383 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 8, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 21, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 30, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 19, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 290, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 27, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 224, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 21669, 2024-03-12T05:28:03.384 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 1030, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 1655469102, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 21669, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 809, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 809, 2024-03-12T05:28:03.385 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 20860, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 1486, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 2871, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 1, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 19769, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 14229, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 62, 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 2570 2024-03-12T05:28:03.386 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 19, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 4, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 1, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 446, 2024-03-12T05:28:03.387 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 251, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 194, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 26, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 7, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-12T05:28:03.388 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 20842, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 20842, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 46, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 46, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 3, 2024-03-12T05:28:03.389 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 167909790, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2791, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 10.952529134, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003924231 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-03-12T05:28:03.390 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 1821, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 2327, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 506, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 102, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 111, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 9, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 2204, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 2836, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 632, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 761, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 1305, 2024-03-12T05:28:03.391 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 544, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 74456, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207228 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 10801, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 220, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 9639, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 393, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 441, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1072, 2024-03-12T05:28:03.392 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 5.983873043, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.005581971 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 404, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.111878353, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000276926 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-12T05:28:03.393 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 194, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.478507463, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002466533 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5227, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 8.083214303, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001546434 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.394 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.395 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-12T05:28:03.396 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.005362405, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000893734 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.397 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1928, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.385551422, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000718647 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.398 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 136, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.184986880, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001360197 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 421, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.977124934, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004696258 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.399 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.003982629, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001991314 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.400 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002103425, 2024-03-12T05:28:03.401 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002103425 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-12T05:28:03.402 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.403 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 229, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.418562424, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001827783 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.404 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 5, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 3, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 2, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 2575, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 1287, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 1030, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:28:03.405 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-12T05:28:03.406 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-12T05:28:03.407 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 821330, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 251, 2024-03-12T05:28:03.408 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8264, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 189, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:28:03.410 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 6074536, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 66968, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 25741, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 164687363, 2024-03-12T05:28:03.411 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 33, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 25741, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 25741, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 29.921473093, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001162405 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 25741, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 26420 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 25741, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 21785, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 3956, 2024-03-12T05:28:03.412 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 116, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 2907, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 150, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 4, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 334, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-12T05:28:03.413 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 21669, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:28:03.414 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 1, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-12T05:28:03.415 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 110, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 25, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 937, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-03-12T05:28:03.416 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 272 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 2 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.417 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 21, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 6, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 280, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 251, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 59261, 2024-03-12T05:28:03.418 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 13975231, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 59261, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 59261, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 13975231, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.419 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:03.420 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 25741, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 164714042, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 25741, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 164714042, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.421 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 25741, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 25741, 2024-03-12T05:28:03.422 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 25741, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 25741, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:03.423 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 251, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 26679, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 251, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 116, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 26679, 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:03.424 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55b0dff2e6e0": { 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 20842, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 163715486, 2024-03-12T05:28:03.425 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 20842, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2791, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 163715486, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:03.426 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-12T05:28:04.269 INFO:tasks.check_counter:Daemon mds.c mds.exported=19 2024-03-12T05:28:04.269 INFO:tasks.check_counter:Daemon mds.c mds.imported=27 2024-03-12T05:28:04.269 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-12T05:28:04.269 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:28:04.269 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:28:04.269 DEBUG:tasks.check_counter:Getting stats from f 2024-03-12T05:28:04.270 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-12T05:28:04.534 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:28:04.656 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:04 smithi130 ceph-mon[35398]: pgmap v540: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:28:04.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:04 smithi116 ceph-mon[35486]: pgmap v540: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:28:04.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:04 smithi045 ceph-mon[32174]: pgmap v540: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:28:06.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:06 smithi130 ceph-mon[35398]: pgmap v541: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-03-12T05:28:06.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:06 smithi116 ceph-mon[35486]: pgmap v541: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-03-12T05:28:06.748 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-12T05:28:06.748 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:28:06.748 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1027, 2024-03-12T05:28:06.748 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1229, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 316685, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 1088441, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 12, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.100633383, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.039086659, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.051078500, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.008044455, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1227, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.056776295, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000046272 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.749 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1904, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000387759, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000203 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 456864 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 94, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 217, 2024-03-12T05:28:06.750 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 18818, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 614220, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 60, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.027706743, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.005192112, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.020108076, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000357155, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 215, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.016558812, 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000077017 2024-03-12T05:28:06.751 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 201, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000046854, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000233 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 6448, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 142672 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:28:06.752 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 969, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 689, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1999853, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 475328, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 25, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.099240472, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.032997103, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.076897311, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.007383724, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 688, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.045893298, 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000066705 2024-03-12T05:28:06.753 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 705, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000122247, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000173 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1844336, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 231440 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-12T05:28:06.754 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 106, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.006421439, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000060579 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:06.755 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-03-12T05:28:06.756 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000008089, 2024-03-12T05:28:06.756 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000004044 2024-03-12T05:28:06.756 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.756 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.756 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 9, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 7, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 7, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002377652, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000339664 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 2, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-03-12T05:28:06.757 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 19, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 19, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 19, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 1, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 2, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 2, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 4, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 20, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 11, 2024-03-12T05:28:06.758 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 1, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 6, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 1, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 1, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 28, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 1030, 2024-03-12T05:28:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 1655469102, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 28, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 28, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:28:06.760 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 3, 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 3, 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.761 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 8, 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-03-12T05:28:06.762 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-12T05:28:06.763 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-03-12T05:28:06.764 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 23, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 23, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 2, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 4218558, 2024-03-12T05:28:06.765 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.037600510, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002350031 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 22, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 22, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 19, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 20, 2024-03-12T05:28:06.766 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 19, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 20, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 1, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 2, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 7, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 5, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 53148, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207228 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 12, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 9, 2024-03-12T05:28:06.767 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 37, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000454549, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000227274 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.768 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-12T05:28:06.769 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.771 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.771 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.771 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.771 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001617759, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000404439 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.772 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.773 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.774 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.775 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2024-03-12T05:28:06.776 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000305344, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000305344 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-12T05:28:06.777 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.778 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-12T05:28:06.779 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-12T05:28:06.780 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.781 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 5, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 3, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 2, 2024-03-12T05:28:06.782 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 1, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 1037, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:28:06.783 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-12T05:28:06.784 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-12T05:28:06.785 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 280327, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 99, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-12T05:28:06.786 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8264, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 189, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 90331, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 434, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-12T05:28:06.787 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 90, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 36179, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 32, 2024-03-12T05:28:06.788 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 90, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 90, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.286994730, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003188830 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 90, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 119 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 90, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 28, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 62, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-12T05:28:06.789 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 16, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 11, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6, 2024-03-12T05:28:06.790 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 28, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-12T05:28:06.791 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-12T05:28:06.792 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 110, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 25, 2024-03-12T05:28:06.793 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 10, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 14, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 4, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 25, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 3 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:28:06.794 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-03-12T05:28:06.795 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 2086, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 2120651, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 2086, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2086, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 2120651, 2024-03-12T05:28:06.796 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:06.797 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 90, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 36179, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 90, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 36179, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-12T05:28:06.798 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 90, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 90, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 90, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 90, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.799 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-12T05:28:06.800 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x5598ab0606e0": { 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 23, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 24254, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 23, 2024-03-12T05:28:06.801 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 16, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 24254, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:06.802 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-12T05:28:06.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:06 smithi045 ceph-mon[32174]: pgmap v541: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-03-12T05:28:07.875 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-12T05:28:07.875 INFO:tasks.check_counter:Daemon mds.f mds.imported=1 2024-03-12T05:28:07.875 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-12T05:28:07.875 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:28:07.875 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:28:07.875 DEBUG:tasks.check_counter:Getting stats from i 2024-03-12T05:28:07.875 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-12T05:28:08.133 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:28:08.518 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:08 smithi130 ceph-mon[35398]: pgmap v542: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-12T05:28:08.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:08 smithi116 ceph-mon[35486]: pgmap v542: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-12T05:28:08.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:08 smithi045 ceph-mon[32174]: pgmap v542: 97 pgs: 97 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-12T05:28:09.632 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:09 smithi130 ceph-mon[35398]: osdmap e111: 12 total, 12 up, 12 in 2024-03-12T05:28:09.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:09 smithi116 ceph-mon[35486]: osdmap e111: 12 total, 12 up, 12 in 2024-03-12T05:28:09.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:09 smithi045 ceph-mon[32174]: osdmap e111: 12 total, 12 up, 12 in 2024-03-12T05:28:10.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:10 smithi130 ceph-mon[35398]: pgmap v544: 97 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 92 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:10.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:10 smithi116 ceph-mon[35486]: pgmap v544: 97 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 92 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 4011, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 4001, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 2813100, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 2693086, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 9, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-12T05:28:10.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.309778928, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.101485467, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.213707577, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.067105567, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4001, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.172278078, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000043058 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1983, 2024-03-12T05:28:10.838 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000422805, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000213 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1861312, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 235888 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 5273, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 5121, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1725386, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 2600815, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 69, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.330434334, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.117422186, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.137879239, 2024-03-12T05:28:10.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.073034488, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5118, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.238591327, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000046618 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3003, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000556192, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000185 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2016, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 135552 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:28:10.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 2672, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 2882, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 678264, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 1411356, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 16, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.170147052, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.057778233, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.039387053, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.056069239, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2880, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.129342739, 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000044910 2024-03-12T05:28:10.841 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000745, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000093 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 458608 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-12T05:28:10.842 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6763, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.145997995, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000169451 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-03-12T05:28:10.843 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000010794, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000005397 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 327, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 327, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 327, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.155246448, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000474759 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-03-12T05:28:10.844 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 307, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 307, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 307, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 292, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 292, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 11, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 379, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 358, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-03-12T05:28:10.845 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 21, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 8, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 289, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 8381, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 1030, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 1655469102, 2024-03-12T05:28:10.846 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 8381, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 8381, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 2, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 55, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 89, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:28:10.847 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 469, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 26, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 6, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:28:10.848 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-12T05:28:10.849 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 667, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 667, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 2, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-03-12T05:28:10.850 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 5253338, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 307, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.390755221, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001272818 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 310, 2024-03-12T05:28:10.851 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 310, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 27, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 28, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 307, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 307, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 292, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 292, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 60404, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207228 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.852 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 331, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 1, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 327, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 34, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 3, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002139442, 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001069721 2024-03-12T05:28:10.853 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 30, 2024-03-12T05:28:10.854 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.010188069, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000339602 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.855 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-12T05:28:10.856 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.857 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.857 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.857 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.857 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-12T05:28:10.857 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.858 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 285, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.134186545, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000470829 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 10, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.008732392, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000873239 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.859 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.860 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-12T05:28:10.861 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.862 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.863 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:28:10.864 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 4, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 2, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-12T05:28:10.865 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 316, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 105, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 1040, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:28:10.866 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:28:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:28:10.868 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 297957, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 119, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8384, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 192, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:28:10.869 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-12T05:28:10.870 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 1022957, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 7000, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-12T05:28:10.871 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 8733, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 1121840, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 33, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 8733, 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-12T05:28:10.872 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8733, 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 6.248889270, 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000715548 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 8733, 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 8762 2024-03-12T05:28:10.873 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.874 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 8733, 2024-03-12T05:28:10.874 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 8381, 2024-03-12T05:28:10.874 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 352, 2024-03-12T05:28:10.874 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-12T05:28:10.874 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 0, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 307, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 11, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-12T05:28:10.875 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 8381, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:28:10.876 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-12T05:28:10.877 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-12T05:28:10.878 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 111, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 26, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 23, 2024-03-12T05:28:10.879 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 4 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:28:10.880 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 11952, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 4114162, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 11952, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:10.881 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 11952, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 4114162, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:10.882 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 8733, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 1121840, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 8733, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1121840, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.883 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:10.884 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 8733, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 8733, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 8733, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 8733, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:10.885 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.886 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55d7eecf46e0": { 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:10.887 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 667, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1059034, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 667, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 307, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1059034, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:10.888 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-12T05:28:10.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:10 smithi045 ceph-mon[32174]: pgmap v544: 97 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 92 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:11.739 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-12T05:28:11.739 INFO:tasks.check_counter:Daemon mds.i mds.imported=8 2024-03-12T05:28:11.739 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-12T05:28:11.739 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:28:11.740 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:28:11.740 DEBUG:tasks.check_counter:Getting stats from l 2024-03-12T05:28:11.740 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 52aed436-e02d-11ee-95c8-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-12T05:28:11.981 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:28:12.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:12 smithi130 ceph-mon[35398]: pgmap v545: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:12.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:12 smithi116 ceph-mon[35486]: pgmap v545: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:12.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:12 smithi045 ceph-mon[32174]: pgmap v545: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:14.134 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1544, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1523, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 2214352, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 563687, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 1, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.165299724, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.050139059, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.073148895, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.043684967, 2024-03-12T05:28:14.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1522, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.057007256, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000037455 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1821568, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 200128 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 5, 2024-03-12T05:28:14.136 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 250, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1036, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 676764, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.012674776, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.008015501, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.002472445, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000071206, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 247, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.022637078, 2024-03-12T05:28:14.137 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000091648 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 676080 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 2269, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 2269, 2024-03-12T05:28:14.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1638565, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 704870, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 3, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.127532631, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.038238855, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.031991021, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.049580664, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2268, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.057958672, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000025554 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.139 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:14.140 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5747, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.136028611, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000023669 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.141 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-03-12T05:28:14.142 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 297, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 297, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 297, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 10, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-03-12T05:28:14.143 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 0, 2024-03-12T05:28:14.144 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-03-12T05:28:14.145 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-03-12T05:28:14.146 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-03-12T05:28:14.147 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-03-12T05:28:14.148 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 667, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 5253338, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 5253197, 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-03-12T05:28:14.149 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 667 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 300, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 300, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 17, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 17, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 0, 2024-03-12T05:28:14.150 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 297, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 297, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 51160, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 215420 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-03-12T05:28:14.151 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.152 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.153 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.154 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.155 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-03-12T05:28:14.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.156 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.156 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.156 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.157 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-03-12T05:28:14.158 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-03-12T05:28:14.159 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.160 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.161 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.162 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 4, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 3, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-03-12T05:28:14.163 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 3018, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T05:28:14.164 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T05:28:14.165 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 182177, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 51, 2024-03-12T05:28:14.166 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 8384, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 192, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 841149, 2024-03-12T05:28:14.167 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 2512, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 3436, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 21, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 3, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 3436, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3436, 2024-03-12T05:28:14.168 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.884966719, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000548593 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3436, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 3437 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 3436, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 3436, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 2248, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2024-03-12T05:28:14.169 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 1186, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 1, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-03-12T05:28:14.170 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-03-12T05:28:14.171 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 111, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 26, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 4, 2024-03-12T05:28:14.172 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 6, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 2, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-03-12T05:28:14.173 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:14.174 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 3814, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 3488560, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 3814, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3814, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 3488560, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-03-12T05:28:14.175 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 3436, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 1059039, 2024-03-12T05:28:14.176 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3436, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1059039, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:14.177 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 3436, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 3436, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3436, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 3436, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:14.178 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.179 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55ee55996960": { 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-03-12T05:28:14.180 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-03-12T05:28:14.181 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-03-12T05:28:14.181 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.181 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-03-12T05:28:14.181 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-03-12T05:28:14.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:14 smithi116 ceph-mon[35486]: pgmap v546: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:14.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:14 smithi045 ceph-mon[32174]: pgmap v546: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:14.882 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:14 smithi130 ceph-mon[35398]: pgmap v546: 97 pgs: 13 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 77 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-12T05:28:14.909 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-12T05:28:14.909 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-12T05:28:14.909 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-12T05:28:14.909 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-12T05:28:14.909 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T05:28:14.910 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-12T05:28:14.922 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-12T05:28:14.922 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:28:14.922 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T05:28:14.952 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:28:14.952 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T05:28:15.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.171+0000 7f3c65640640 1 -- 172.21.15.45:0/3163493015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c601003d0 msgr2=0x7f3c60100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.171+0000 7f3c65640640 1 --2- 172.21.15.45:0/3163493015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c601003d0 0x7f3c60100850 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009930 tx=0x7f3c4802bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- 172.21.15.45:0/3163493015 shutdown_connections 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- 172.21.15.45:0/3163493015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c601003d0 0x7f3c60100850 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- 172.21.15.45:0/3163493015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60100d90 0x7f3c6010ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- 172.21.15.45:0/3163493015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c60105060 0x7f3c60105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- 172.21.15.45:0/3163493015 >> 172.21.15.45:0/3163493015 conn(0x7f3c60075e80 msgr2=0x7f3c60076290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.168 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- 172.21.15.45:0/3163493015 shutdown_connections 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- 172.21.15.45:0/3163493015 wait complete. 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 Processor -- start 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- start start 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60105060 0x7f3c6019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6019c600 con 0x7f3c60100d90 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6019c770 con 0x7f3c60105060 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.172+0000 7f3c65640640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6019c930 con 0x7f3c601003d0 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5effd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5effd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.45:54340/0 (socket says 172.21.15.45:54340) 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5effd640 1 -- 172.21.15.45:0/3326781900 learned_addr learned my addr 172.21.15.45:0/3326781900 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:15.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54358/0 (socket says 172.21.15.45:54358) 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5f7fe640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60105060 0x7f3c6019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 -- 172.21.15.45:0/3326781900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 msgr2=0x7f3c60197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 -- 172.21.15.45:0/3326781900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60105060 msgr2=0x7f3c6019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60105060 0x7f3c6019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 -- 172.21.15.45:0/3326781900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48009560 con 0x7f3c60100d90 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5effd640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c57fff640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009230 tx=0x7f3c48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4802c5d0 con 0x7f3c60100d90 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f3c48003aa0 con 0x7f3c60100d90 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6019cbb0 con 0x7f3c60100d90 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48005460 con 0x7f3c60100d90 2024-03-12T05:28:15.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.173+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c601a4490 con 0x7f3c60100d90 2024-03-12T05:28:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.174+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c601017a0 con 0x7f3c60100d90 2024-03-12T05:28:15.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.175+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3c48003c10 con 0x7f3c60100d90 2024-03-12T05:28:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.176+0000 7f3c5cff9640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 0x7f3c4c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.176+0000 7f3c5effd640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 0x7f3c4c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.176+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f3c480efa40 con 0x7f3c60100d90 2024-03-12T05:28:15.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.176+0000 7f3c5effd640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 0x7f3c4c0a4060 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3c500104e0 tx=0x7f3c5000b410 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.175 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.181+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c480b8990 con 0x7f3c60100d90 2024-03-12T05:28:15.356 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.362+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c60108500 con 0x7f3c60100d90 2024-03-12T05:28:15.357 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.363+0000 7f3c5cff9640 1 -- 172.21.15.45:0/3326781900 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+23236 (secure 0 0 0) 0x7f3c480dae60 con 0x7f3c60100d90 2024-03-12T05:28:15.357 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:28:15.358 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":111,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:28:08.461475+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3527414001":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/1231700554":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/2345425176":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/4188215761":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/759469548":"2024-03-13T05:10:52.434958+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:28:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 msgr2=0x7f3c4c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 0x7f3c4c0a4060 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3c500104e0 tx=0x7f3c5000b410 comp rx=0 tx=0).stop 2024-03-12T05:28:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 msgr2=0x7f3c60197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009230 tx=0x7f3c48004270 comp rx=0 tx=0).stop 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 shutdown_connections 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f3c4c0a1ba0 0x7f3c4c0a4060 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c60100d90 0x7f3c60197c80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c60105060 0x7f3c6019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 --2- 172.21.15.45:0/3326781900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c601003d0 0x7f3c60197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 >> 172.21.15.45:0/3326781900 conn(0x7f3c60075e80 msgr2=0x7f3c600fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 shutdown_connections 2024-03-12T05:28:15.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.365+0000 7f3c65640640 1 -- 172.21.15.45:0/3326781900 wait complete. 2024-03-12T05:28:15.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 31, 'snap_epoch': 111, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:28:15.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:28:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- 172.21.15.45:0/4073716552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac100260 msgr2=0x7f6cac100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 --2- 172.21.15.45:0/4073716552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac100260 0x7f6cac100640 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009930 tx=0x7f6c9c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- 172.21.15.45:0/4073716552 shutdown_connections 2024-03-12T05:28:15.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 --2- 172.21.15.45:0/4073716552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac105330 0x7f6cac10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 --2- 172.21.15.45:0/4073716552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100c10 0x7f6cac104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 --2- 172.21.15.45:0/4073716552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac100260 0x7f6cac100640 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- 172.21.15.45:0/4073716552 >> 172.21.15.45:0/4073716552 conn(0x7f6cac077180 msgr2=0x7f6cac077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- 172.21.15.45:0/4073716552 shutdown_connections 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- 172.21.15.45:0/4073716552 wait complete. 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 Processor -- start 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.483+0000 7f6cb284d640 1 -- start start 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100260 0x7f6cac196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac19aee0 0x7f6cac19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19b8d0 con 0x7f6cac105330 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19ba40 con 0x7f6cac19aee0 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb284d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac19bbb0 con 0x7f6cac100260 2024-03-12T05:28:15.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54380/0 (socket says 172.21.15.45:54380) 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 -- 172.21.15.45:0/1156170552 learned_addr learned my addr 172.21.15.45:0/1156170552 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cabfff640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100260 0x7f6cac196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cb0dea640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac19aee0 0x7f6cac19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 -- 172.21.15.45:0/1156170552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100260 msgr2=0x7f6cac196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100260 0x7f6cac196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 -- 172.21.15.45:0/1156170552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac19aee0 msgr2=0x7f6cac19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac19aee0 0x7f6cac19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 -- 172.21.15.45:0/1156170552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009560 con 0x7f6cac105330 2024-03-12T05:28:15.480 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.484+0000 7f6cab7fe640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000d740 tx=0x7f6ca000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.487+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0009c30 con 0x7f6cac105330 2024-03-12T05:28:15.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.487+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac1a3190 con 0x7f6cac105330 2024-03-12T05:28:15.481 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.487+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f6ca000de50 con 0x7f6cac105330 2024-03-12T05:28:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.487+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0004e40 con 0x7f6cac105330 2024-03-12T05:28:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.487+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac1a35f0 con 0x7f6cac105330 2024-03-12T05:28:15.482 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.489+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac067f70 con 0x7f6cac105330 2024-03-12T05:28:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.490+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6ca0004fa0 con 0x7f6cac105330 2024-03-12T05:28:15.484 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.491+0000 7f6ca97fa640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 0x7f6c940a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.491+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f6ca00cd170 con 0x7f6cac105330 2024-03-12T05:28:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.491+0000 7f6cabfff640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 0x7f6c940a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.491+0000 7f6cabfff640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 0x7f6c940a4060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005ab0 tx=0x7f6c9c005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.496+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ca0099bd0 con 0x7f6cac105330 2024-03-12T05:28:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.677+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6cac100c10 con 0x7f6cac105330 2024-03-12T05:28:15.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.677+0000 7f6ca97fa640 1 -- 172.21.15.45:0/1156170552 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ca00200a0 con 0x7f6cac105330 2024-03-12T05:28:15.676 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:28:15.676 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 msgr2=0x7f6c940a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 0x7f6c940a4060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c005ab0 tx=0x7f6c9c005fd0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 msgr2=0x7f6cac196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000d740 tx=0x7f6ca000dc00 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 shutdown_connections 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f6c940a1ba0 0x7f6c940a4060 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6cac105330 0x7f6cac196b60 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac19aee0 0x7f6cac19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 --2- 172.21.15.45:0/1156170552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cac100260 0x7f6cac196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 >> 172.21.15.45:0/1156170552 conn(0x7f6cac077180 msgr2=0x7f6cac0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 shutdown_connections 2024-03-12T05:28:15.677 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.679+0000 7f6cb284d640 1 -- 172.21.15.45:0/1156170552 wait complete. 2024-03-12T05:28:15.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:15 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3326781900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:15.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:28:15.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:15 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3326781900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:15.791 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- 172.21.15.45:0/1236755883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c100c30 msgr2=0x7fac9c104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 --2- 172.21.15.45:0/1236755883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c100c30 0x7fac9c104c70 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fac90009930 tx=0x7fac9002bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- 172.21.15.45:0/1236755883 shutdown_connections 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 --2- 172.21.15.45:0/1236755883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c100c30 0x7fac9c104c70 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 --2- 172.21.15.45:0/1236755883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c105350 0x7fac9c10cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 --2- 172.21.15.45:0/1236755883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c100280 0x7fac9c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- 172.21.15.45:0/1236755883 >> 172.21.15.45:0/1236755883 conn(0x7fac9c077150 msgr2=0x7fac9c077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- 172.21.15.45:0/1236755883 shutdown_connections 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- 172.21.15.45:0/1236755883 wait complete. 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 Processor -- start 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.795+0000 7faca47b5640 1 -- start start 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c100280 0x7fac9c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 0x7fac9c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.792 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c1a3cd0 con 0x7fac9c199050 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c1a3e40 con 0x7fac9c100280 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c1a4000 con 0x7fac9c105350 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54396/0 (socket says 172.21.15.45:54396) 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 -- 172.21.15.45:0/346991623 learned_addr learned my addr 172.21.15.45:0/346991623 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca1d50640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 0x7fac9c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2551640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c100280 0x7fac9c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 -- 172.21.15.45:0/346991623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 msgr2=0x7fac9c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 0x7fac9c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 -- 172.21.15.45:0/346991623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c100280 msgr2=0x7fac9c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c100280 0x7fac9c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 -- 172.21.15.45:0/346991623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac90009560 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca1d50640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 0x7fac9c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca2d52640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009fd0 tx=0x7fac8c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c00bea0 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fac8c009940 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c1a42e0 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.796+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0106c0 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.797+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c1a47c0 con 0x7fac9c199050 2024-03-12T05:28:15.794 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.797+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c101750 con 0x7fac9c199050 2024-03-12T05:28:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.799+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fac8c010820 con 0x7fac9c199050 2024-03-12T05:28:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.800+0000 7fac9b7fe640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 0x7fac840a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.800+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fac8c0cd4f0 con 0x7fac9c199050 2024-03-12T05:28:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.800+0000 7faca2551640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 0x7fac840a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:15.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.800+0000 7faca2551640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 0x7fac840a40e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fac9400bbe0 tx=0x7fac9400b390 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:15.797 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.803+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac8c0963b0 con 0x7fac9c199050 2024-03-12T05:28:15.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:15 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3326781900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:15.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.984+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac9c108680 con 0x7fac9c199050 2024-03-12T05:28:15.979 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.985+0000 7fac9b7fe640 1 -- 172.21.15.45:0/346991623 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac8c0170a0 con 0x7fac9c199050 2024-03-12T05:28:15.979 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 msgr2=0x7fac840a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 0x7fac840a40e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fac9400bbe0 tx=0x7fac9400b390 comp rx=0 tx=0).stop 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 msgr2=0x7fac9c1a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009fd0 tx=0x7fac8c00b260 comp rx=0 tx=0).stop 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 shutdown_connections 2024-03-12T05:28:15.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fac840a1c20 0x7fac840a40e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac9c199050 0x7fac9c1a3600 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac9c105350 0x7fac9c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 --2- 172.21.15.45:0/346991623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac9c100280 0x7fac9c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 >> 172.21.15.45:0/346991623 conn(0x7fac9c077150 msgr2=0x7fac9c0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 shutdown_connections 2024-03-12T05:28:15.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:15.987+0000 7faca47b5640 1 -- 172.21.15.45:0/346991623 wait complete. 2024-03-12T05:28:15.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 -- 172.21.15.45:0/3088722887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900101d30 msgr2=0x7f8900105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 --2- 172.21.15.45:0/3088722887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900101d30 0x7f8900105d70 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009930 tx=0x7f88f402bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 -- 172.21.15.45:0/3088722887 shutdown_connections 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 --2- 172.21.15.45:0/3088722887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900101d30 0x7f8900105d70 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 --2- 172.21.15.45:0/3088722887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8900106450 0x7f890010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 --2- 172.21.15.45:0/3088722887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f8900101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.102+0000 7f890569a640 1 -- 172.21.15.45:0/3088722887 >> 172.21.15.45:0/3088722887 conn(0x7f89000fcc20 msgr2=0x7f89000ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- 172.21.15.45:0/3088722887 shutdown_connections 2024-03-12T05:28:16.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- 172.21.15.45:0/3088722887 wait complete. 2024-03-12T05:28:16.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 Processor -- start 2024-03-12T05:28:16.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- start start 2024-03-12T05:28:16.099 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f890019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 0x7f8900197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8900197f90 con 0x7f8900106450 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8900198100 con 0x7f89001975a0 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f890569a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001982c0 con 0x7f8900101380 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.103+0000 7f88fe7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54414/0 (socket says 172.21.15.45:54414) 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 -- 172.21.15.45:0/4013063630 learned_addr learned my addr 172.21.15.45:0/4013063630 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88ff7fe640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 0x7f8900197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88feffd640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f890019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 -- 172.21.15.45:0/4013063630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 msgr2=0x7f890019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f890019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.100 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 -- 172.21.15.45:0/4013063630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 msgr2=0x7f8900197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 0x7f8900197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 -- 172.21.15.45:0/4013063630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f4009560 con 0x7f8900106450 2024-03-12T05:28:16.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88ff7fe640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 0x7f8900197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:28:16.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88feffd640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f890019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88fe7fc640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f88f4000c00 tx=0x7f88f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f402cb70 con 0x7f8900106450 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f88f402de90 con 0x7f8900106450 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89001a42b0 con 0x7f8900106450 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f40053c0 con 0x7f8900106450 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.104+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001a4760 con 0x7f8900106450 2024-03-12T05:28:16.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.105+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8900067f70 con 0x7f8900106450 2024-03-12T05:28:16.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.106+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f88f40035f0 con 0x7f8900106450 2024-03-12T05:28:16.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.107+0000 7f88e7fff640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 0x7f88e80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.107+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f88f40efb80 con 0x7f8900106450 2024-03-12T05:28:16.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.107+0000 7f88feffd640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 0x7f88e80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.107+0000 7f88feffd640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 0x7f88e80a4060 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f88f80058c0 tx=0x7f88f8005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.111+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88f40b8b10 con 0x7f8900106450 2024-03-12T05:28:16.289 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.295+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8900101d30 con 0x7f8900106450 2024-03-12T05:28:16.290 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.296+0000 7f88e7fff640 1 -- 172.21.15.45:0/4013063630 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88f40bc1d0 con 0x7f8900106450 2024-03-12T05:28:16.290 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 msgr2=0x7f88e80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 0x7f88e80a4060 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f88f80058c0 tx=0x7f88f8005850 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 msgr2=0x7f890019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f88f4000c00 tx=0x7f88f4004270 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 shutdown_connections 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f88e80a1ba0 0x7f88e80a4060 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900106450 0x7f890019d930 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89001975a0 0x7f8900197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.297+0000 7f890569a640 1 --2- 172.21.15.45:0/4013063630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8900101380 0x7f890019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.298+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 >> 172.21.15.45:0/4013063630 conn(0x7f89000fcc20 msgr2=0x7f89000fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.298+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 shutdown_connections 2024-03-12T05:28:16.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.298+0000 7f890569a640 1 -- 172.21.15.45:0/4013063630 wait complete. 2024-03-12T05:28:16.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-12T05:28:16.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- 172.21.15.45:0/2149623770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 msgr2=0x7fc45c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 --2- 172.21.15.45:0/2149623770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c10cf00 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc454009cd0 tx=0x7fc45402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- 172.21.15.45:0/2149623770 shutdown_connections 2024-03-12T05:28:16.408 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 --2- 172.21.15.45:0/2149623770 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c10cf00 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 --2- 172.21.15.45:0/2149623770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c100c10 0x7fc45c104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 --2- 172.21.15.45:0/2149623770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- 172.21.15.45:0/2149623770 >> 172.21.15.45:0/2149623770 conn(0x7fc45c077180 msgr2=0x7fc45c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- 172.21.15.45:0/2149623770 shutdown_connections 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- 172.21.15.45:0/2149623770 wait complete. 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 Processor -- start 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.412+0000 7fc46092c640 1 -- start start 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c199020 0x7fc45c1a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c1a3c70 con 0x7fc45c105330 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c1a3de0 con 0x7fc45c100260 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c1a3fa0 con 0x7fc45c199020 2024-03-12T05:28:16.409 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54432/0 (socket says 172.21.15.45:54432) 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 -- 172.21.15.45:0/3349038447 learned_addr learned my addr 172.21.15.45:0/3349038447 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc45ad9d640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c199020 0x7fc45c1a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc45a59c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 -- 172.21.15.45:0/3349038447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c199020 msgr2=0x7fc45c1a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c199020 0x7fc45c1a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 -- 172.21.15.45:0/3349038447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 msgr2=0x7fc45c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 -- 172.21.15.45:0/3349038447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454009900 con 0x7fc45c105330 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc45a59c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc453fff640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc44400d740 tx=0x7fc44400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444009c30 con 0x7fc45c105330 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc44400de50 con 0x7fc45c105330 2024-03-12T05:28:16.410 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444004e40 con 0x7fc45c105330 2024-03-12T05:28:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c1a4280 con 0x7fc45c105330 2024-03-12T05:28:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.413+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c1a4790 con 0x7fc45c105330 2024-03-12T05:28:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.414+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c067f70 con 0x7fc45c105330 2024-03-12T05:28:16.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.415+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc444004fa0 con 0x7fc45c105330 2024-03-12T05:28:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.416+0000 7fc4537fe640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 0x7fc4480a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.416+0000 7fc45a59c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 0x7fc4480a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.416+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fc4440cd590 con 0x7fc45c105330 2024-03-12T05:28:16.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.416+0000 7fc45a59c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 0x7fc4480a40e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0058c0 tx=0x7fc44c005d30 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.420+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc444096450 con 0x7fc45c105330 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.613+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc45c108580 con 0x7fc45c105330 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.614+0000 7fc4537fe640 1 -- 172.21.15.45:0/3349038447 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v111) v1 ==== 81+0+1002 (secure 0 0 0) 0x7fc444017070 con 0x7fc45c105330 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3527414001 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/1305486259 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4294356868 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/662273801 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:16.608 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4138375301 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2281722728 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2630663966 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2481316973 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/1134390880 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2524215406 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/1305486259 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2524215406 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1231700554 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2345425176 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/1134390880 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4188215761 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.609 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/759469548 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:16.610 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-03-12T05:28:16.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 msgr2=0x7fc4480a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 0x7fc4480a40e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0058c0 tx=0x7fc44c005d30 comp rx=0 tx=0).stop 2024-03-12T05:28:16.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 msgr2=0x7fc45c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc44400d740 tx=0x7fc44400dc00 comp rx=0 tx=0).stop 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 shutdown_connections 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc4480a1c20 0x7fc4480a40e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc45c105330 0x7fc45c198a90 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc45c199020 0x7fc45c1a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 --2- 172.21.15.45:0/3349038447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc45c100260 0x7fc45c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 >> 172.21.15.45:0/3349038447 conn(0x7fc45c077180 msgr2=0x7fc45c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 shutdown_connections 2024-03-12T05:28:16.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.616+0000 7fc46092c640 1 -- 172.21.15.45:0/3349038447 wait complete. 2024-03-12T05:28:16.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:16 smithi045 ceph-mon[32174]: pgmap v547: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:16.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:16 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1156170552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:16 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/346991623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.614 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:16 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/4013063630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.620 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-12T05:28:16.621 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T05:28:16.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:28:16.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:16 smithi116 ceph-mon[35486]: pgmap v547: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:16.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:16 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1156170552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:16 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/346991623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.741 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:16 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/4013063630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/559554290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 msgr2=0x7f4f541007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/559554290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541007e0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009900 tx=0x7f4f4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/559554290 shutdown_connections 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/559554290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541007e0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/559554290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54100d20 0x7f4f5410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/559554290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54106130 0x7f4f54106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.785+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/559554290 >> 172.21.15.45:0/559554290 conn(0x7f4f54077060 msgr2=0x7f4f54077470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/559554290 shutdown_connections 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/559554290 wait complete. 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 Processor -- start 2024-03-12T05:28:16.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- start start 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54100d20 0x7f4f54198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 0x7f4f541a3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54199260 con 0x7f4f54100360 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541a3a80 con 0x7f4f54106130 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f5bb4b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f541a3bc0 con 0x7f4f54100d20 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f598e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f598e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54450/0 (socket says 172.21.15.45:54450) 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.786+0000 7f4f598e7640 1 -- 172.21.15.45:0/2249341228 learned_addr learned my addr 172.21.15.45:0/2249341228 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 -- 172.21.15.45:0/2249341228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54100d20 msgr2=0x7f4f54198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f590e6640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54100d20 0x7f4f54198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f5a0e8640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 0x7f4f541a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54100d20 0x7f4f54198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.783 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 -- 172.21.15.45:0/2249341228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 msgr2=0x7f4f541a3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 0x7f4f541a3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 -- 172.21.15.45:0/2249341228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44009560 con 0x7f4f54100360 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f5a0e8640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 0x7f4f541a3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f598e7640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c890 tx=0x7f4f5000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50004220 con 0x7f4f54100360 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f4f50007e60 con 0x7f4f54100360 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f541a3e20 con 0x7f4f54100360 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50004a00 con 0x7f4f54100360 2024-03-12T05:28:16.784 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.787+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f541a42f0 con 0x7f4f54100360 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.788+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f54101780 con 0x7f4f54100360 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.789+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4f500079b0 con 0x7f4f54100360 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.790+0000 7f4f4affd640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 0x7f4f400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.790+0000 7f4f590e6640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 0x7f4f400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.790+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f4f500cd9e0 con 0x7f4f54100360 2024-03-12T05:28:16.785 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.791+0000 7f4f590e6640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 0x7f4f400a40e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f44036040 tx=0x7f4f44005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:16.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.794+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f500968e0 con 0x7f4f54100360 2024-03-12T05:28:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:16 smithi130 ceph-mon[35398]: pgmap v547: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:16 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1156170552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:16 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/346991623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:16 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/4013063630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:16.969 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.975+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f540683c0 con 0x7f4f54100360 2024-03-12T05:28:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.976+0000 7f4f4affd640 1 -- 172.21.15.45:0/2249341228 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+23236 (secure 0 0 0) 0x7f4f5001d070 con 0x7f4f54100360 2024-03-12T05:28:16.971 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:28:16.971 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":111,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:28:08.461475+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3527414001":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/1231700554":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/2345425176":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/4188215761":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/759469548":"2024-03-13T05:10:52.434958+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:28:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 msgr2=0x7f4f400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 0x7f4f400a40e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f44036040 tx=0x7f4f44005fd0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 msgr2=0x7f4f541984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c890 tx=0x7f4f5000cd50 comp rx=0 tx=0).stop 2024-03-12T05:28:16.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 shutdown_connections 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f4f400a1c20 0x7f4f400a40e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f54106130 0x7f4f541a3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f54100d20 0x7f4f54198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 --2- 172.21.15.45:0/2249341228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f54100360 0x7f4f541984c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 >> 172.21.15.45:0/2249341228 conn(0x7f4f54077060 msgr2=0x7f4f540ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 shutdown_connections 2024-03-12T05:28:16.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:16.979+0000 7f4f5bb4b640 1 -- 172.21.15.45:0/2249341228 wait complete. 2024-03-12T05:28:16.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:04:07.359124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:08:35.335788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:08:36.314618+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 31, 'snap_epoch': 111, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:28:16.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:28:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 -- 172.21.15.45:0/2600239653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc24106450 msgr2=0x7fbc2410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2600239653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc24106450 0x7fbc2410e020 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009930 tx=0x7fbc1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 -- 172.21.15.45:0/2600239653 shutdown_connections 2024-03-12T05:28:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2600239653 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc24106450 0x7fbc2410e020 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2600239653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101d30 0x7fbc24105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2600239653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 0x7fbc24101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.095+0000 7fbc29be2640 1 -- 172.21.15.45:0/2600239653 >> 172.21.15.45:0/2600239653 conn(0x7fbc240fcc20 msgr2=0x7fbc240ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- 172.21.15.45:0/2600239653 shutdown_connections 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- 172.21.15.45:0/2600239653 wait complete. 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 Processor -- start 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- start start 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 0x7fbc2419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24106450 0x7fbc2419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24197fd0 con 0x7fbc241975e0 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24198140 con 0x7fbc24106450 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.096+0000 7fbc29be2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc24198300 con 0x7fbc24101380 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54472/0 (socket says 172.21.15.45:54472) 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 -- 172.21.15.45:0/2599990967 learned_addr learned my addr 172.21.15.45:0/2599990967 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc237fe640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 0x7fbc2419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 -- 172.21.15.45:0/2599990967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 msgr2=0x7fbc2419d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 0x7fbc2419d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 -- 172.21.15.45:0/2599990967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24106450 msgr2=0x7fbc2419d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.094 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24106450 0x7fbc2419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 -- 172.21.15.45:0/2599990967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c009560 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc23fff640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c005940 tx=0x7fbc1c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c004440 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fbc1c02de90 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc241a42a0 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c005280 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.097+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc241a4750 con 0x7fbc241975e0 2024-03-12T05:28:17.095 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.098+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc24067f70 con 0x7fbc241975e0 2024-03-12T05:28:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.099+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbc1c003c10 con 0x7fbc241975e0 2024-03-12T05:28:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.100+0000 7fbc20ff9640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 0x7fbc0c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.100+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fbc1c0effe0 con 0x7fbc241975e0 2024-03-12T05:28:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.100+0000 7fbc237fe640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 0x7fbc0c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.100+0000 7fbc237fe640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 0x7fbc0c0a4060 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbc14005fd0 tx=0x7fbc14009500 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.104+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc1c0b8f70 con 0x7fbc241975e0 2024-03-12T05:28:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.285+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc24101d30 con 0x7fbc241975e0 2024-03-12T05:28:17.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.286+0000 7fbc20ff9640 1 -- 172.21.15.45:0/2599990967 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc1c0bc630 con 0x7fbc241975e0 2024-03-12T05:28:17.280 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 msgr2=0x7fbc0c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 0x7fbc0c0a4060 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbc14005fd0 tx=0x7fbc14009500 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 msgr2=0x7fbc24197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c005940 tx=0x7fbc1c004040 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 shutdown_connections 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fbc0c0a1ba0 0x7fbc0c0a4060 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbc241975e0 0x7fbc24197a90 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24106450 0x7fbc2419d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 --2- 172.21.15.45:0/2599990967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc24101380 0x7fbc2419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.282 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 >> 172.21.15.45:0/2599990967 conn(0x7fbc240fcc20 msgr2=0x7fbc240fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 shutdown_connections 2024-03-12T05:28:17.283 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.288+0000 7fbc29be2640 1 -- 172.21.15.45:0/2599990967 wait complete. 2024-03-12T05:28:17.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:28:17.401 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 -- 172.21.15.45:0/2292586555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 msgr2=0x7fe81c10dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 --2- 172.21.15.45:0/2292586555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c10dc30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009ca0 tx=0x7fe80c02c180 comp rx=0 tx=0).stop 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 -- 172.21.15.45:0/2292586555 shutdown_connections 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 --2- 172.21.15.45:0/2292586555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c10dc30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 --2- 172.21.15.45:0/2292586555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 0x7fe81c10b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 --2- 172.21.15.45:0/2292586555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 0x7fe81c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 -- 172.21.15.45:0/2292586555 >> 172.21.15.45:0/2292586555 conn(0x7fe81c0771e0 msgr2=0x7fe81c0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 -- 172.21.15.45:0/2292586555 shutdown_connections 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.404+0000 7fe821862640 1 -- 172.21.15.45:0/2292586555 wait complete. 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 Processor -- start 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 -- start start 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 0x7fe81c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.402 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 0x7fe81c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c19c4f0 con 0x7fe81c10b7c0 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c19c660 con 0x7fe81c1023f0 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe821862640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c19c820 con 0x7fe81c102d10 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40742/0 (socket says 172.21.15.45:40742) 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81b7fe640 1 -- 172.21.15.45:0/2709134395 learned_addr learned my addr 172.21.15.45:0/2709134395 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81affd640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 0x7fe81c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81a7fc640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 0x7fe81c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.405+0000 7fe81b7fe640 1 -- 172.21.15.45:0/2709134395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 msgr2=0x7fe81c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe81b7fe640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 0x7fe81c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe81b7fe640 1 -- 172.21.15.45:0/2709134395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 msgr2=0x7fe81c197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe81b7fe640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 0x7fe81c197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.403 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe81b7fe640 1 -- 172.21.15.45:0/2709134395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c009900 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe81b7fe640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe80c02c790 tx=0x7fe80c0349e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c003e00 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fe80c032b30 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c19caa0 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c004770 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.406+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c19cd70 con 0x7fe81c10b7c0 2024-03-12T05:28:17.404 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.407+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c0683c0 con 0x7fe81c10b7c0 2024-03-12T05:28:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.408+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe80c03b040 con 0x7fe81c10b7c0 2024-03-12T05:28:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.409+0000 7fe820860640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 0x7fe8040a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.409+0000 7fe81affd640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 0x7fe8040a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.409+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7fe80c0f0280 con 0x7fe81c10b7c0 2024-03-12T05:28:17.405 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.409+0000 7fe81affd640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 0x7fe8040a4060 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe814005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.407 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.413+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe80c0b9210 con 0x7fe81c10b7c0 2024-03-12T05:28:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.594+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe81c1111e0 con 0x7fe81c10b7c0 2024-03-12T05:28:17.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.595+0000 7fe820860640 1 -- 172.21.15.45:0/2709134395 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe80c0bc8d0 con 0x7fe81c10b7c0 2024-03-12T05:28:17.589 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 msgr2=0x7fe8040a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 0x7fe8040a4060 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe814005850 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 msgr2=0x7fe81c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe80c02c790 tx=0x7fe80c0349e0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 shutdown_connections 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fe8040a1ba0 0x7fe8040a4060 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.596+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe81c10b7c0 0x7fe81c19bef0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.597+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe81c102d10 0x7fe81c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.597+0000 7fe821862640 1 --2- 172.21.15.45:0/2709134395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe81c1023f0 0x7fe81c197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.597+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 >> 172.21.15.45:0/2709134395 conn(0x7fe81c0771e0 msgr2=0x7fe81c0fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.597+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 shutdown_connections 2024-03-12T05:28:17.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.597+0000 7fe821862640 1 -- 172.21.15.45:0/2709134395 wait complete. 2024-03-12T05:28:17.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:17 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3349038447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:17.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:17 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2249341228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:17.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:17 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2599990967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:17.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:28:17.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 -- 172.21.15.45:0/4056165821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08106460 msgr2=0x7fad0810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 --2- 172.21.15.45:0/4056165821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08106460 0x7fad0810e030 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fad00009cd0 tx=0x7fad0002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 -- 172.21.15.45:0/4056165821 shutdown_connections 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 --2- 172.21.15.45:0/4056165821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08106460 0x7fad0810e030 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 --2- 172.21.15.45:0/4056165821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08101d40 0x7fad08105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 --2- 172.21.15.45:0/4056165821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad08101390 0x7fad08101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 -- 172.21.15.45:0/4056165821 >> 172.21.15.45:0/4056165821 conn(0x7fad080fcbb0 msgr2=0x7fad080fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 -- 172.21.15.45:0/4056165821 shutdown_connections 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.713+0000 7fad0f19a640 1 -- 172.21.15.45:0/4056165821 wait complete. 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 Processor -- start 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 -- start start 2024-03-12T05:28:17.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 0x7fad08197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 0x7fad0819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0819c920 con 0x7fad08101390 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0819ca90 con 0x7fad0819bf30 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0f19a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0819cc00 con 0x7fad08106460 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40754/0 (socket says 172.21.15.45:40754) 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 -- 172.21.15.45:0/1688717807 learned_addr learned my addr 172.21.15.45:0/1688717807 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0d737640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 0x7fad0819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 -- 172.21.15.45:0/1688717807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 msgr2=0x7fad08197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad07fff640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 0x7fad08197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 0x7fad08197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 -- 172.21.15.45:0/1688717807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 msgr2=0x7fad0819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 0x7fad0819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0cf36640 1 -- 172.21.15.45:0/1688717807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00009900 con 0x7fad08101390 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad0d737640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 0x7fad0819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.714+0000 7fad07fff640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 0x7fad08197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad0cf36640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c920 tx=0x7facfc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc007c70 con 0x7fad08101390 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7facfc007dd0 con 0x7fad08101390 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad081a41e0 con 0x7fad08101390 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facfc0187c0 con 0x7fad08101390 2024-03-12T05:28:17.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.715+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad081a46e0 con 0x7fad08101390 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.716+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad080683c0 con 0x7fad08101390 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.717+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7facfc00f450 con 0x7fad08101390 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.718+0000 7fad05ffb640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 0x7facf80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.718+0000 7fad07fff640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 0x7facf80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.718+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7facfc0cda30 con 0x7fad08101390 2024-03-12T05:28:17.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.718+0000 7fad07fff640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 0x7facf80a40e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fad08107a20 tx=0x7facf4005cb0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:17.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.722+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facfc096870 con 0x7fad08101390 2024-03-12T05:28:17.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:17 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3349038447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:17.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:17 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2249341228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:17.740 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:17 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2599990967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:17.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.904+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad08101d40 con 0x7fad08101390 2024-03-12T05:28:17.898 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.904+0000 7fad05ffb640 1 -- 172.21.15.45:0/1688717807 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7facfc099f30 con 0x7fad08101390 2024-03-12T05:28:17.898 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 64 2024-03-12T05:28:17.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 msgr2=0x7facf80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 0x7facf80a40e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fad08107a20 tx=0x7facf4005cb0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 msgr2=0x7fad081976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c920 tx=0x7facfc00cde0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 shutdown_connections 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7facf80a1c20 0x7facf80a40e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad0819bf30 0x7fad0819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08106460 0x7fad08197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 --2- 172.21.15.45:0/1688717807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad08101390 0x7fad081976c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 >> 172.21.15.45:0/1688717807 conn(0x7fad080fcbb0 msgr2=0x7fad080feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 shutdown_connections 2024-03-12T05:28:17.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:17.906+0000 7fad0f19a640 1 -- 172.21.15.45:0/1688717807 wait complete. 2024-03-12T05:28:17.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-12T05:28:17.965 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:17 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3349038447' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:17.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:17 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2249341228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:17.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:17 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2599990967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 -- 172.21.15.45:0/4030489055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4100c10 msgr2=0x7f02e4104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 --2- 172.21.15.45:0/4030489055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4100c10 0x7f02e4104c50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009930 tx=0x7f02d802bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 -- 172.21.15.45:0/4030489055 shutdown_connections 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 --2- 172.21.15.45:0/4030489055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4100c10 0x7f02e4104c50 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 --2- 172.21.15.45:0/4030489055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4105330 0x7f02e410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 --2- 172.21.15.45:0/4030489055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 0x7f02e4100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 -- 172.21.15.45:0/4030489055 >> 172.21.15.45:0/4030489055 conn(0x7f02e4077180 msgr2=0x7f02e4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:18.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 -- 172.21.15.45:0/4030489055 shutdown_connections 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.020+0000 7f02e9662640 1 -- 172.21.15.45:0/4030489055 wait complete. 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 Processor -- start 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 -- start start 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 0x7f02e41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 0x7f02e41a3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41a3c30 con 0x7f02e4105330 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41a3da0 con 0x7f02e4198fb0 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e9662640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41a3f60 con 0x7f02e4100260 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40782/0 (socket says 172.21.15.45:40782) 2024-03-12T05:28:18.018 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.021+0000 7f02e27fc640 1 -- 172.21.15.45:0/1123056445 learned_addr learned my addr 172.21.15.45:0/1123056445 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e2ffd640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 0x7f02e41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e37fe640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 0x7f02e41a3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 -- 172.21.15.45:0/1123056445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 msgr2=0x7f02e41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 0x7f02e41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 -- 172.21.15.45:0/1123056445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 msgr2=0x7f02e41a3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 0x7f02e41a3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 -- 172.21.15.45:0/1123056445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8009560 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e37fe640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 0x7f02e41a3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e27fc640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f02d8000c00 tx=0x7f02d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d802cb70 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f02d802de90 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e41a41e0 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d80053c0 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.022+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e41a46f0 con 0x7f02e4105330 2024-03-12T05:28:18.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.023+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e4067f70 con 0x7f02e4105330 2024-03-12T05:28:18.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.024+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f02d80035f0 con 0x7f02e4105330 2024-03-12T05:28:18.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.025+0000 7f02cbfff640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 0x7f02cc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:18.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.025+0000 7f02e2ffd640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 0x7f02cc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:18.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.025+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8608+0+0 (secure 0 0 0) 0x7f02d80f0330 con 0x7f02e4105330 2024-03-12T05:28:18.021 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.026+0000 7f02e2ffd640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 0x7f02cc0a4060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005fd0 tx=0x7f02dc005850 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:18.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.029+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02d80b92c0 con 0x7f02e4105330 2024-03-12T05:28:18.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.223+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f02e4108580 con 0x7f02e4105330 2024-03-12T05:28:18.217 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.223+0000 7f02cbfff640 1 -- 172.21.15.45:0/1123056445 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v111) v1 ==== 81+0+1002 (secure 0 0 0) 0x7f02d80bc980 con 0x7f02e4105330 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3527414001 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/1305486259 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4294356868 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/662273801 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4138375301 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2281722728 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2630663966 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2481316973 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:18.218 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/1134390880 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2524215406 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/1305486259 2024-03-13T05:01:16.183699+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2524215406 2024-03-13T05:01:43.021024+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1231700554 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/2345425176 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/1134390880 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/4188215761 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.219 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/759469548 2024-03-13T05:10:52.434958+0000 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 msgr2=0x7f02cc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 0x7f02cc0a4060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f02dc005fd0 tx=0x7f02dc005850 comp rx=0 tx=0).stop 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 msgr2=0x7f02e4198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f02d8000c00 tx=0x7f02d8004270 comp rx=0 tx=0).stop 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 shutdown_connections 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f02cc0a1ba0 0x7f02cc0a4060 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02e4105330 0x7f02e4198a20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4198fb0 0x7f02e41a3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 --2- 172.21.15.45:0/1123056445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02e4100260 0x7f02e41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 >> 172.21.15.45:0/1123056445 conn(0x7f02e4077180 msgr2=0x7f02e40fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.225+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 shutdown_connections 2024-03-12T05:28:18.221 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:18.226+0000 7f02e9662640 1 -- 172.21.15.45:0/1123056445 wait complete. 2024-03-12T05:28:18.230 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi045.front.sepia.ceph.com... 2024-03-12T05:28:18.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:28:18.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo fusermount -u /tmp/cephfs 2024-03-12T05:28:18.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[85020]: fuse finished with error 0 and tester_r 0 2024-03-12T05:28:18.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-12T05:28:18.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:18 smithi130 ceph-mon[35398]: pgmap v548: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:18.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:18 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2709134395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:18 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1688717807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:18 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1123056445' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:18.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:18 smithi045 ceph-mon[32174]: pgmap v548: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:18.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:18 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2709134395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:18 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1688717807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:18 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1123056445' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:18.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:18 smithi116 ceph-mon[35486]: pgmap v548: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-12T05:28:18.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:18 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2709134395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:18 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1688717807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:28:18.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:18 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1123056445' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T05:28:20.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:20 smithi045 ceph-mon[32174]: pgmap v549: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 298 B/s wr, 12 op/s 2024-03-12T05:28:20.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:20 smithi045 ceph-mon[32174]: osdmap e112: 12 total, 12 up, 12 in 2024-03-12T05:28:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:20 smithi130 ceph-mon[35398]: pgmap v549: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 298 B/s wr, 12 op/s 2024-03-12T05:28:20.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:20 smithi130 ceph-mon[35398]: osdmap e112: 12 total, 12 up, 12 in 2024-03-12T05:28:20.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:20 smithi116 ceph-mon[35486]: pgmap v549: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 298 B/s wr, 12 op/s 2024-03-12T05:28:20.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:20 smithi116 ceph-mon[35486]: osdmap e112: 12 total, 12 up, 12 in 2024-03-12T05:28:21.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:21 smithi130 ceph-mon[35398]: pgmap v551: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:21.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:21 smithi116 ceph-mon[35486]: pgmap v551: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:22.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:21 smithi045 ceph-mon[32174]: pgmap v551: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:24.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:23 smithi045 ceph-mon[32174]: pgmap v552: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:23 smithi130 ceph-mon[35398]: pgmap v552: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:24.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:23 smithi116 ceph-mon[35486]: pgmap v552: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-12T05:28:24.474 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:28:24.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:28:24.475 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-03-12T05:28:24.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:28:24.639 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-03-12T05:28:24.701 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-03-12T05:28:24.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:28:24.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-03-12T05:28:24.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:28:24.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-03-12T05:28:24.830 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T05:28:24.842 INFO:tasks.cephadm:Teardown begin 2024-03-12T05:28:24.842 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-12T05:28:24.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:28:24.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 -- 172.21.15.45:0/1938744679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 msgr2=0x7f89781006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:24.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 --2- 172.21.15.45:0/1938744679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f89781006d0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8974009930 tx=0x7f897402bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 -- 172.21.15.45:0/1938744679 shutdown_connections 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 --2- 172.21.15.45:0/1938744679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 0x7f897810cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 --2- 172.21.15.45:0/1938744679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8978100ca0 0x7f8978104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 --2- 172.21.15.45:0/1938744679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f89781006d0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 -- 172.21.15.45:0/1938744679 >> 172.21.15.45:0/1938744679 conn(0x7f8978077180 msgr2=0x7f8978077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 -- 172.21.15.45:0/1938744679 shutdown_connections 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.957+0000 7f897fc55640 1 -- 172.21.15.45:0/1938744679 wait complete. 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 Processor -- start 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 -- start start 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 0x7f89781022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8978102800 0x7f89781a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:24.954 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89781a5f50 con 0x7f89781002f0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89781a60c0 con 0x7f89781053c0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897fc55640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89781a6280 con 0x7f8978102800 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40806/0 (socket says 172.21.15.45:40806) 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 -- 172.21.15.45:0/1112488295 learned_addr learned my addr 172.21.15.45:0/1112488295 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d1f0640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 0x7f89781022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 -- 172.21.15.45:0/1112488295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8978102800 msgr2=0x7f89781a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8978102800 0x7f89781a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 -- 172.21.15.45:0/1112488295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 msgr2=0x7f89781022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 0x7f89781022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.958+0000 7f897d9f1640 1 -- 172.21.15.45:0/1112488295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8974009560 con 0x7f89781002f0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f897d9f1640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8974009230 tx=0x7f89740040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740044c0 con 0x7f89781002f0 2024-03-12T05:28:24.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89781a6500 con 0x7f89781002f0 2024-03-12T05:28:24.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f897402de90 con 0x7f89781002f0 2024-03-12T05:28:24.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974005400 con 0x7f89781002f0 2024-03-12T05:28:24.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.959+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89781a6a10 con 0x7f89781002f0 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.960+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978067f70 con 0x7f89781002f0 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.961+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8974005560 con 0x7f89781002f0 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.962+0000 7f896effd640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 0x7f89640a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.962+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f89740ef850 con 0x7f89781002f0 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.962+0000 7f897d1f0640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 0x7f89640a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.962+0000 7f897d1f0640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 0x7f89640a4090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8968005fd0 tx=0x7f8968005e20 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:24.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:24.966+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f89740b8800 con 0x7f89781002f0 2024-03-12T05:28:25.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.188+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8978109630 con 0x7f89781002f0 2024-03-12T05:28:25.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.189+0000 7f896effd640 1 -- 172.21.15.45:0/1112488295 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f89740bbec0 con 0x7f89781002f0 2024-03-12T05:28:25.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:28:25.183 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:28:25.184 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:28:25.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 msgr2=0x7f89640a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 0x7f89640a4090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8968005fd0 tx=0x7f8968005e20 comp rx=0 tx=0).stop 2024-03-12T05:28:25.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 msgr2=0x7f8978101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f8974009230 tx=0x7f89740040c0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 shutdown_connections 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f89640a1bd0 0x7f89640a4090 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8978102800 0x7f89781a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89781053c0 0x7f89781022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 --2- 172.21.15.45:0/1112488295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89781002f0 0x7f8978101d80 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 >> 172.21.15.45:0/1112488295 conn(0x7f8978077180 msgr2=0x7f89780686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 shutdown_connections 2024-03-12T05:28:25.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.191+0000 7f897fc55640 1 -- 172.21.15.45:0/1112488295 wait complete. 2024-03-12T05:28:25.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:28:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 -- 172.21.15.45:0/903552222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 msgr2=0x7f91b810dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 --2- 172.21.15.45:0/903552222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b810dc60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009ca0 tx=0x7f91a802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 -- 172.21.15.45:0/903552222 shutdown_connections 2024-03-12T05:28:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 --2- 172.21.15.45:0/903552222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b810dc60 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 --2- 172.21.15.45:0/903552222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b810b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 --2- 172.21.15.45:0/903552222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 0x7f91b81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.306+0000 7f91bfd02640 1 -- 172.21.15.45:0/903552222 >> 172.21.15.45:0/903552222 conn(0x7f91b8077180 msgr2=0x7f91b8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- 172.21.15.45:0/903552222 shutdown_connections 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- 172.21.15.45:0/903552222 wait complete. 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 Processor -- start 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- start start 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 0x7f91b819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8197c60 con 0x7f91b810b7c0 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8197dd0 con 0x7f91b81023f0 2024-03-12T05:28:25.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91bfd02640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8197f90 con 0x7f91b8102d10 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91be29f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91be29f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40824/0 (socket says 172.21.15.45:40824) 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.307+0000 7f91be29f640 1 -- 172.21.15.45:0/2819941413 learned_addr learned my addr 172.21.15.45:0/2819941413 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91bda9e640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 0x7f91b819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91bd29d640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 -- 172.21.15.45:0/2819941413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 msgr2=0x7f91b819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b819d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 -- 172.21.15.45:0/2819941413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 msgr2=0x7f91b819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 0x7f91b819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 -- 172.21.15.45:0/2819941413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a8009900 con 0x7f91b810b7c0 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91bd29d640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91be29f640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f91a802c700 tx=0x7f91a8035d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a80329e0 con 0x7f91b810b7c0 2024-03-12T05:28:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b8198210 con 0x7f91b810b7c0 2024-03-12T05:28:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f91a8003e20 con 0x7f91b810b7c0 2024-03-12T05:28:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a803f440 con 0x7f91b810b7c0 2024-03-12T05:28:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.308+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b81a44b0 con 0x7f91b810b7c0 2024-03-12T05:28:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.309+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b8067f70 con 0x7f91b810b7c0 2024-03-12T05:28:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.310+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f91a803b040 con 0x7f91b810b7c0 2024-03-12T05:28:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.311+0000 7f91b6ffd640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 0x7f91a00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.311+0000 7f91bda9e640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 0x7f91a00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.311+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f91a80f0590 con 0x7f91b810b7c0 2024-03-12T05:28:25.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.311+0000 7f91bda9e640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 0x7f91a00a4060 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009890 tx=0x7f91b0008040 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.309 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.315+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f91a80b94e0 con 0x7f91b810b7c0 2024-03-12T05:28:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.534+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b8198af0 con 0x7f91b810b7c0 2024-03-12T05:28:25.528 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.534+0000 7f91b6ffd640 1 -- 172.21.15.45:0/2819941413 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+11397 (secure 0 0 0) 0x7f91a80bcba0 con 0x7f91b810b7c0 2024-03-12T05:28:25.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:28:25.529 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6835/1645552275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":1645552275},{"type":"v1","addr":"172.21.15.45:6835","nonce":1645552275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6837/378571831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":378571831},{"type":"v1","addr":"172.21.15.45:6837","nonce":378571831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:08:37.112985+0000","modified":"2024-03-12T05:13:58.412459+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24472,"mds_3":24467,"mds_4":24460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14619":{"gid":14619,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6839/1651909238","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6838","nonce":1651909238},{"type":"v1","addr":"172.21.15.45:6839","nonce":1651909238}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14631":{"gid":14631,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.45:6841/3025141330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6840","nonce":3025141330},{"type":"v1","addr":"172.21.15.45:6841","nonce":3025141330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24419":{"gid":24419,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/2049389812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2049389812},{"type":"v1","addr":"172.21.15.116:6833","nonce":2049389812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1602847227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1602847227},{"type":"v1","addr":"172.21.15.130:6833","nonce":1602847227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/4238162743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":4238162743},{"type":"v1","addr":"172.21.15.116:6835","nonce":4238162743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24460":{"gid":24460,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/1592150426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":1592150426},{"type":"v1","addr":"172.21.15.130:6835","nonce":1592150426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24467":{"gid":24467,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/2581569461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2581569461},{"type":"v1","addr":"172.21.15.116:6837","nonce":2581569461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.130:6837/2852741522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2852741522},{"type":"v1","addr":"172.21.15.130:6837","nonce":2852741522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":5,"addr":"172.21.15.130:6839/2939102833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2939102833},{"type":"v1","addr":"172.21.15.130:6839","nonce":2939102833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/397791649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":397791649},{"type":"v1","addr":"172.21.15.116:6839","nonce":397791649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:28:25.529 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 33 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 msgr2=0x7f91a00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 0x7f91a00a4060 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009890 tx=0x7f91b0008040 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 msgr2=0x7f91b8197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f91a802c700 tx=0x7f91a8035d20 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 shutdown_connections 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f91a00a1ba0 0x7f91a00a4060 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91b810b7c0 0x7f91b8197610 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91b8102d10 0x7f91b819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 --2- 172.21.15.45:0/2819941413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91b81023f0 0x7f91b819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 >> 172.21.15.45:0/2819941413 conn(0x7f91b8077180 msgr2=0x7f91b80fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.536+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 shutdown_connections 2024-03-12T05:28:25.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.537+0000 7f91bfd02640 1 -- 172.21.15.45:0/2819941413 wait complete. 2024-03-12T05:28:25.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:28:25.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 -- 172.21.15.45:0/516723043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a41023b0 msgr2=0x7fc3a4102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/516723043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a41023b0 0x7fc3a4102790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009900 tx=0x7fc3a002bcd0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 -- 172.21.15.45:0/516723043 shutdown_connections 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/516723043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 0x7fc3a410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/516723043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a4102cd0 0x7fc3a410b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/516723043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a41023b0 0x7fc3a4102790 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 -- 172.21.15.45:0/516723043 >> 172.21.15.45:0/516723043 conn(0x7fc3a4076400 msgr2=0x7fc3a4076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 -- 172.21.15.45:0/516723043 shutdown_connections 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.651+0000 7fc3aba0e640 1 -- 172.21.15.45:0/516723043 wait complete. 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 Processor -- start 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 -- start start 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a41023b0 0x7fc3a4100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 0x7fc3a4100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41a5ac0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41a5c30 con 0x7fc3a410b780 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3aba0e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41a5df0 con 0x7fc3a41023b0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40838/0 (socket says 172.21.15.45:40838) 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 -- 172.21.15.45:0/1221368058 learned_addr learned my addr 172.21.15.45:0/1221368058 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a9fab640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 0x7fc3a4100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 -- 172.21.15.45:0/1221368058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a41023b0 msgr2=0x7fc3a4100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a97aa640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a41023b0 0x7fc3a4100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a41023b0 0x7fc3a4100000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 -- 172.21.15.45:0/1221368058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 msgr2=0x7fc3a4100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 0x7fc3a4100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 -- 172.21.15.45:0/1221368058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0009560 con 0x7fc3a4102cd0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.652+0000 7fc3a8fa9640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c890 tx=0x7fc39400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.653+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394004220 con 0x7fc3a4102cd0 2024-03-12T05:28:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.653+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fc394007e60 con 0x7fc3a4102cd0 2024-03-12T05:28:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.653+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a41a60d0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.653+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394004a00 con 0x7fc3a4102cd0 2024-03-12T05:28:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.653+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a41a6630 con 0x7fc3a4102cd0 2024-03-12T05:28:25.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.654+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4069fd0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.655+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc3940079b0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.655+0000 7fc39a7fc640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 0x7fc3900a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.655+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc3940cd8a0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.656+0000 7fc3a97aa640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 0x7fc3900a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.656+0000 7fc3a97aa640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 0x7fc3900a40e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036040 tx=0x7fc3a0000bc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.659+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc3940967c0 con 0x7fc3a4102cd0 2024-03-12T05:28:25.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.842+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a4106870 con 0x7fc3a4102cd0 2024-03-12T05:28:25.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.844+0000 7fc39a7fc640 1 -- 172.21.15.45:0/1221368058 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+23168 (secure 0 0 0) 0x7fc39401d070 con 0x7fc3a4102cd0 2024-03-12T05:28:25.838 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-12T05:28:25.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":112,"fsid":"52aed436-e02d-11ee-95c8-87774f69a715","created":"2024-03-12T05:00:48.629903+0000","modified":"2024-03-12T05:28:19.603199+0000","last_up_change":"2024-03-12T05:07:49.783061+0000","last_in_change":"2024-03-12T05:07:29.564424+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:04:07.359124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:08:35.335788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:08:36.314618+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":31,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4e690af1-6483-47d3-8fcb-240b9f1bec6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6803","nonce":55762477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6805","nonce":55762477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6809","nonce":55762477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":55762477},{"type":"v1","addr":"172.21.15.45:6807","nonce":55762477}]},"public_addr":"172.21.15.45:6803/55762477","cluster_addr":"172.21.15.45:6805/55762477","heartbeat_back_addr":"172.21.15.45:6809/55762477","heartbeat_front_addr":"172.21.15.45:6807/55762477","state":["exists","up"]},{"osd":1,"uuid":"50765f29-2dd5-499e-aeb8-529a3344ab49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6801","nonce":1071649441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6803","nonce":1071649441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6807","nonce":1071649441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1071649441},{"type":"v1","addr":"172.21.15.116:6805","nonce":1071649441}]},"public_addr":"172.21.15.116:6801/1071649441","cluster_addr":"172.21.15.116:6803/1071649441","heartbeat_back_addr":"172.21.15.116:6807/1071649441","heartbeat_front_addr":"172.21.15.116:6805/1071649441","state":["exists","up"]},{"osd":2,"uuid":"8d3b4de1-376d-4209-bef9-a48e51f77a93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6801","nonce":845680146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6803","nonce":845680146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6807","nonce":845680146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":845680146},{"type":"v1","addr":"172.21.15.130:6805","nonce":845680146}]},"public_addr":"172.21.15.130:6801/845680146","cluster_addr":"172.21.15.130:6803/845680146","heartbeat_back_addr":"172.21.15.130:6807/845680146","heartbeat_front_addr":"172.21.15.130:6805/845680146","state":["exists","up"]},{"osd":3,"uuid":"522e40da-a98f-4f92-bc68-60f665e2f962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6811","nonce":1948936218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6813","nonce":1948936218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6817","nonce":1948936218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1948936218},{"type":"v1","addr":"172.21.15.45:6815","nonce":1948936218}]},"public_addr":"172.21.15.45:6811/1948936218","cluster_addr":"172.21.15.45:6813/1948936218","heartbeat_back_addr":"172.21.15.45:6817/1948936218","heartbeat_front_addr":"172.21.15.45:6815/1948936218","state":["exists","up"]},{"osd":4,"uuid":"c172804f-1d0d-4ab9-9dc9-a67b20f80b52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6809","nonce":3946228497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6811","nonce":3946228497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6815","nonce":3946228497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3946228497},{"type":"v1","addr":"172.21.15.116:6813","nonce":3946228497}]},"public_addr":"172.21.15.116:6809/3946228497","cluster_addr":"172.21.15.116:6811/3946228497","heartbeat_back_addr":"172.21.15.116:6815/3946228497","heartbeat_front_addr":"172.21.15.116:6813/3946228497","state":["exists","up"]},{"osd":5,"uuid":"5bf734df-d459-46bb-a3a2-432bdcf349fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6809","nonce":2846635408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6811","nonce":2846635408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6815","nonce":2846635408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":2846635408},{"type":"v1","addr":"172.21.15.130:6813","nonce":2846635408}]},"public_addr":"172.21.15.130:6809/2846635408","cluster_addr":"172.21.15.130:6811/2846635408","heartbeat_back_addr":"172.21.15.130:6815/2846635408","heartbeat_front_addr":"172.21.15.130:6813/2846635408","state":["exists","up"]},{"osd":6,"uuid":"84d0438a-d41f-43a0-b705-636605037f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6819","nonce":3221804389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6821","nonce":3221804389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6825","nonce":3221804389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3221804389},{"type":"v1","addr":"172.21.15.45:6823","nonce":3221804389}]},"public_addr":"172.21.15.45:6819/3221804389","cluster_addr":"172.21.15.45:6821/3221804389","heartbeat_back_addr":"172.21.15.45:6825/3221804389","heartbeat_front_addr":"172.21.15.45:6823/3221804389","state":["exists","up"]},{"osd":7,"uuid":"da01102a-465e-47b8-bc0a-8d4119260bac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6817","nonce":1067697558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6819","nonce":1067697558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6823","nonce":1067697558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1067697558},{"type":"v1","addr":"172.21.15.116:6821","nonce":1067697558}]},"public_addr":"172.21.15.116:6817/1067697558","cluster_addr":"172.21.15.116:6819/1067697558","heartbeat_back_addr":"172.21.15.116:6823/1067697558","heartbeat_front_addr":"172.21.15.116:6821/1067697558","state":["exists","up"]},{"osd":8,"uuid":"ae6cce03-2fb7-4fe5-a943-33e47113ef19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6817","nonce":1919686935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6819","nonce":1919686935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6823","nonce":1919686935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":1919686935},{"type":"v1","addr":"172.21.15.130:6821","nonce":1919686935}]},"public_addr":"172.21.15.130:6817/1919686935","cluster_addr":"172.21.15.130:6819/1919686935","heartbeat_back_addr":"172.21.15.130:6823/1919686935","heartbeat_front_addr":"172.21.15.130:6821/1919686935","state":["exists","up"]},{"osd":9,"uuid":"8f6399aa-8c7f-4fc3-a9d4-4ddbb70fdd8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6827","nonce":15901149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6829","nonce":15901149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6833","nonce":15901149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":15901149},{"type":"v1","addr":"172.21.15.45:6831","nonce":15901149}]},"public_addr":"172.21.15.45:6827/15901149","cluster_addr":"172.21.15.45:6829/15901149","heartbeat_back_addr":"172.21.15.45:6833/15901149","heartbeat_front_addr":"172.21.15.45:6831/15901149","state":["exists","up"]},{"osd":10,"uuid":"00932bcc-b7f2-4a9c-9421-ce264276091b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6825","nonce":2890731133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6827","nonce":2890731133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6831","nonce":2890731133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2890731133},{"type":"v1","addr":"172.21.15.116:6829","nonce":2890731133}]},"public_addr":"172.21.15.116:6825/2890731133","cluster_addr":"172.21.15.116:6827/2890731133","heartbeat_back_addr":"172.21.15.116:6831/2890731133","heartbeat_front_addr":"172.21.15.116:6829/2890731133","state":["exists","up"]},{"osd":11,"uuid":"57499d1f-cdfc-473e-b09c-9c906168826d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6825","nonce":627874902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6827","nonce":627874902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6831","nonce":627874902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":627874902},{"type":"v1","addr":"172.21.15.130:6829","nonce":627874902}]},"public_addr":"172.21.15.130:6825/627874902","cluster_addr":"172.21.15.130:6827/627874902","heartbeat_back_addr":"172.21.15.130:6831/627874902","heartbeat_front_addr":"172.21.15.130:6829/627874902","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:16.634055+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:03:39.471582+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:03.027503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:27.106236+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:04:51.535389+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:16.319121+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:05:41.498653+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:06.505705+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:31.358810+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:06:56.594012+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:21.802123+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:07:47.923078+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9},{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/3527414001":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/4294356868":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/662273801":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/4138375301":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2281722728":"2024-03-13T05:01:16.183699+0000","172.21.15.45:0/2630663966":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/2481316973":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:6801/1305486259":"2024-03-13T05:01:16.183699+0000","172.21.15.45:6801/2524215406":"2024-03-13T05:01:43.021024+0000","172.21.15.45:0/1231700554":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/2345425176":"2024-03-13T05:10:52.434958+0000","172.21.15.45:6800/1134390880":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/4188215761":"2024-03-13T05:10:52.434958+0000","172.21.15.45:0/759469548":"2024-03-13T05:10:52.434958+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":18,"length":1},{"begin":31,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 msgr2=0x7fc3900a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 0x7fc3900a40e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036040 tx=0x7fc3a0000bc0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 msgr2=0x7fc3a4100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c890 tx=0x7fc39400cd50 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 shutdown_connections 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fc3900a1c20 0x7fc3900a40e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3a4102cd0 0x7fc3a4100540 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3a410b780 0x7fc3a4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 --2- 172.21.15.45:0/1221368058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc3a41023b0 0x7fc3a4100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 >> 172.21.15.45:0/1221368058 conn(0x7fc3a4076400 msgr2=0x7fc3a40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 shutdown_connections 2024-03-12T05:28:25.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.846+0000 7fc3aba0e640 1 -- 172.21.15.45:0/1221368058 wait complete. 2024-03-12T05:28:25.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- 172.21.15.45:0/1293405256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c101d40 msgr2=0x7f800c105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 --2- 172.21.15.45:0/1293405256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c101d40 0x7f800c105d80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8000009930 tx=0x7f800002bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- 172.21.15.45:0/1293405256 shutdown_connections 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 --2- 172.21.15.45:0/1293405256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c101d40 0x7f800c105d80 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 --2- 172.21.15.45:0/1293405256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c106460 0x7f800c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 --2- 172.21.15.45:0/1293405256 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c101390 0x7f800c101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- 172.21.15.45:0/1293405256 >> 172.21.15.45:0/1293405256 conn(0x7f800c0fcc10 msgr2=0x7f800c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- 172.21.15.45:0/1293405256 shutdown_connections 2024-03-12T05:28:25.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- 172.21.15.45:0/1293405256 wait complete. 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 Processor -- start 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.962+0000 7f80108ae640 1 -- start start 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c101390 0x7f800c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c1975f0 0x7f800c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c197fe0 con 0x7f800c106460 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c198150 con 0x7f800c101390 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c198310 con 0x7f800c1975f0 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40854/0 (socket says 172.21.15.45:40854) 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 -- 172.21.15.45:0/3457781376 learned_addr learned my addr 172.21.15.45:0/3457781376 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f800a59c640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c101390 0x7f800c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f800ad9d640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c1975f0 0x7f800c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 -- 172.21.15.45:0/3457781376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c1975f0 msgr2=0x7f800c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c1975f0 0x7f800c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 -- 172.21.15.45:0/3457781376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c101390 msgr2=0x7f800c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c101390 0x7f800c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 -- 172.21.15.45:0/3457781376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000009560 con 0x7f800c106460 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f8009d9b640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8000000c00 tx=0x7f8000004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800002cb70 con 0x7f800c106460 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f800002de90 con 0x7f800c106460 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c1a42b0 con 0x7f800c106460 2024-03-12T05:28:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000053c0 con 0x7f800c106460 2024-03-12T05:28:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.963+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c1a47b0 con 0x7f800c106460 2024-03-12T05:28:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.964+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c067f70 con 0x7f800c106460 2024-03-12T05:28:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.965+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f80000035f0 con 0x7f800c106460 2024-03-12T05:28:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.966+0000 7f7ffb7fe640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 0x7f7ff40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:25.961 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.966+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f80000efa40 con 0x7f800c106460 2024-03-12T05:28:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.966+0000 7f800a59c640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 0x7f7ff40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:25.962 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.966+0000 7f800a59c640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 0x7f7ff40a40e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8004005fd0 tx=0x7f8004009680 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:25.964 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:25.970+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f80000f1e60 con 0x7f800c106460 2024-03-12T05:28:26.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:25 smithi045 ceph-mon[32174]: pgmap v553: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-03-12T05:28:26.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:25 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1112488295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:25 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2819941413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.187 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.193+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f800c109690 con 0x7f800c106460 2024-03-12T05:28:26.189 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.195+0000 7f7ffb7fe640 1 -- 172.21.15.45:0/3457781376 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f80000b89f0 con 0x7f800c106460 2024-03-12T05:28:26.189 INFO:teuthology.orchestra.run.smithi045.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-12T05:28:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 msgr2=0x7f7ff40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 0x7f7ff40a40e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8004005fd0 tx=0x7f8004009680 comp rx=0 tx=0).stop 2024-03-12T05:28:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 msgr2=0x7f800c19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:26.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8000000c00 tx=0x7f8000004270 comp rx=0 tx=0).stop 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 shutdown_connections 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7ff40a1c20 0x7f7ff40a40e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c106460 0x7f800c19d930 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f800c1975f0 0x7f800c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 --2- 172.21.15.45:0/3457781376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c101390 0x7f800c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 >> 172.21.15.45:0/3457781376 conn(0x7f800c0fcc10 msgr2=0x7f800c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.199+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 shutdown_connections 2024-03-12T05:28:26.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.200+0000 7f80108ae640 1 -- 172.21.15.45:0/3457781376 wait complete. 2024-03-12T05:28:26.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-12T05:28:26.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:25 smithi130 ceph-mon[35398]: pgmap v553: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-03-12T05:28:26.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:25 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1112488295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:25 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2819941413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:25 smithi116 ceph-mon[35486]: pgmap v553: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-03-12T05:28:26.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:25 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1112488295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:25 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2819941413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:28:26.314 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3613251562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d140685d0 msgr2=0x7f7d140689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3613251562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d140685d0 0x7f7d140689b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009930 tx=0x7f7d0402bc30 comp rx=0 tx=0).stop 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3613251562 shutdown_connections 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3613251562 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d14106430 0x7f7d1410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3613251562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14068f80 0x7f7d14105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3613251562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d140685d0 0x7f7d140689b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3613251562 >> 172.21.15.45:0/3613251562 conn(0x7f7d14077180 msgr2=0x7f7d14077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3613251562 shutdown_connections 2024-03-12T05:28:26.315 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.320+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3613251562 wait complete. 2024-03-12T05:28:26.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 Processor -- start 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 -- start start 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 0x7f7d14100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 0x7f7d14100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141a5f00 con 0x7f7d14100ac0 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141a6070 con 0x7f7d14106430 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d1a55e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d141a6230 con 0x7f7d140685d0 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d18afb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d18afb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40886/0 (socket says 172.21.15.45:40886) 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d18afb640 1 -- 172.21.15.45:0/3552573167 learned_addr learned my addr 172.21.15.45:0/3552573167 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.321+0000 7f7d137fe640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 0x7f7d14100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d13fff640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 0x7f7d14100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:26.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 -- 172.21.15.45:0/3552573167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 msgr2=0x7f7d14100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 0x7f7d14100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 -- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 msgr2=0x7f7d141a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 -- 172.21.15.45:0/3552573167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d04009560 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d13fff640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 0x7f7d14100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d18afb640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d137fe640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 0x7f7d14100580 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800d6a0 tx=0x7f7d0800db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08005220 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f7d08009cd0 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d141a6510 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08004cc0 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.322+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d141a6a20 con 0x7f7d14106430 2024-03-12T05:28:26.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.323+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d14067f70 con 0x7f7d14106430 2024-03-12T05:28:26.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.325+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7d08009820 con 0x7f7d14106430 2024-03-12T05:28:26.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.326+0000 7f7d117fa640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 0x7f7cfc0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:26.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.326+0000 7f7d13fff640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 0x7f7cfc0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.326+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 8950+0+0 (secure 0 0 0) 0x7f7d080cd260 con 0x7f7d14106430 2024-03-12T05:28:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.326+0000 7f7d13fff640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 0x7f7cfc0a4090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009900 tx=0x7f7d04005fd0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.330+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d08095f80 con 0x7f7d14106430 2024-03-12T05:28:26.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a[32170]: 2024-03-12T05:28:26.194+0000 7f3f4f198700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T05:28:26.487 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.487 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.487 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mds[74631]: -- 172.21.15.45:0/1596199648 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556b6b1921a0 con 0x556b6b168800 2024-03-12T05:28:26.487 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: starting mds.a at 2024-03-12T05:28:26.488 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.488 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.488 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mds[74270]: -- 172.21.15.45:0/2491913938 <== mon.0 v2:172.21.15.45:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56184aed61a0 con 0x56184aead000 2024-03-12T05:28:26.488 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: starting mds.g at 2024-03-12T05:28:26.490 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.490 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mds[65109]: -- 172.21.15.116:0/2478185043 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa1d5961a0 con 0x55fa1d56d000 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: starting mds.k at 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mds[66044]: -- 172.21.15.116:0/2947528999 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563e115921a0 con 0x563e11568800 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: starting mds.b at 2024-03-12T05:28:26.491 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mds[65416]: -- 172.21.15.116:0/1367664070 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5625b2eae1a0 con 0x5625b2e85000 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: starting mds.e at 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mds[65724]: -- 172.21.15.116:0/4153667123 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b491721a0 con 0x559b49148c00 2024-03-12T05:28:26.492 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: starting mds.h at 2024-03-12T05:28:26.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:26.556+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7d14109660 con 0x7f7d14106430 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mds[65769]: -- 172.21.15.130:0/3362821310 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f779841a0 con 0x558f7795ac00 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: starting mds.f at 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i[66372]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.716 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i[66372]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mds[66376]: -- 172.21.15.130:0/2234138876 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568becea1a0 con 0x5568becc0c00 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i[66372]: starting mds.i at 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mds[65461]: -- 172.21.15.130:0/3197765973 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607acb061a0 con 0x5607acadc800 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: starting mds.l at 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: ignoring --setuser ceph since I am not root 2024-03-12T05:28:26.717 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: ignoring --setgroup ceph since I am not root 2024-03-12T05:28:26.718 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mds[66075]: -- 172.21.15.130:0/629050340 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564710a181a0 con 0x5647109ef000 2024-03-12T05:28:26.718 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: starting mds.c at 2024-03-12T05:28:26.718 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: 2024-03-12T05:28:26.266+0000 7f4059ff6700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-12T05:28:27.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1221368058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:27.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T05:28:27.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: osdmap e113: 12 total, 12 up, 12 in 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/3552573167' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:26 smithi045 ceph-mon[32174]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.205+0000 7f7d117fa640 1 -- 172.21.15.45:0/3552573167 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7d08099640 con 0x7f7d14106430 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.207+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 msgr2=0x7f7cfc0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.207+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 0x7f7cfc0a4090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009900 tx=0x7f7d04005fd0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.207+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 msgr2=0x7f7d14100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.207+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 0x7f7d14100580 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800d6a0 tx=0x7f7d0800db60 comp rx=0 tx=0).stop 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 shutdown_connections 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f7cfc0a1bd0 0x7f7cfc0a4090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d14100ac0 0x7f7d141a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d14106430 0x7f7d14100580 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 --2- 172.21.15.45:0/3552573167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d140685d0 0x7f7d14100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 >> 172.21.15.45:0/3552573167 conn(0x7f7d14077180 msgr2=0x7f7d140fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 shutdown_connections 2024-03-12T05:28:27.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.208+0000 7f7d1a55e640 1 -- 172.21.15.45:0/3552573167 wait complete. 2024-03-12T05:28:27.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1221368058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: osdmap e113: 12 total, 12 up, 12 in 2024-03-12T05:28:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-12T05:28:27.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/3552573167' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:26 smithi130 ceph-mon[35398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1221368058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:28:27.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T05:28:27.240 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3457781376' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: osdmap e113: 12 total, 12 up, 12 in 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/3552573167' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:26 smithi116 ceph-mon[35486]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:28:27.324 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1642697089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e0068f80 msgr2=0x7f46e0105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1642697089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e0068f80 0x7f46e0105d50 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009930 tx=0x7f46cc02bc20 comp rx=0 tx=0).stop 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1642697089 shutdown_connections 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1642697089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e0068f80 0x7f46e0105d50 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1642697089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e0106430 0x7f46e010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1642697089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e00685d0 0x7f46e00689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1642697089 >> 172.21.15.45:0/1642697089 conn(0x7f46e0077180 msgr2=0x7f46e0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1642697089 shutdown_connections 2024-03-12T05:28:27.325 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.330+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1642697089 wait complete. 2024-03-12T05:28:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 Processor -- start 2024-03-12T05:28:27.327 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 -- start start 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e00685d0 0x7f46e0197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0106430 0x7f46e0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e019c8b0 con 0x7f46e019bec0 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e019ca20 con 0x7f46e00685d0 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e6c1c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e019cb90 con 0x7f46e0106430 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e51b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.331+0000 7f46e51b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:40936/0 (socket says 172.21.15.45:40936) 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 -- 172.21.15.45:0/1750067850 learned_addr learned my addr 172.21.15.45:0/1750067850 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e49b8640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e00685d0 0x7f46e0197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46dffff640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0106430 0x7f46e0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 -- 172.21.15.45:0/1750067850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0106430 msgr2=0x7f46e0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:27.328 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0106430 0x7f46e0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 -- 172.21.15.45:0/1750067850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e00685d0 msgr2=0x7f46e0197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e00685d0 0x7f46e0197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 -- 172.21.15.45:0/1750067850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009560 con 0x7f46e019bec0 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e51b9640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f46d800cce0 tx=0x7f46d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d800f450 con 0x7f46e019bec0 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7f46d800fa60 con 0x7f46e019bec0 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01a4170 con 0x7f46e019bec0 2024-03-12T05:28:27.329 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8018b10 con 0x7f46e019bec0 2024-03-12T05:28:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.332+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01a45f0 con 0x7f46e019bec0 2024-03-12T05:28:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.333+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e0067f70 con 0x7f46e019bec0 2024-03-12T05:28:27.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.335+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f46d800fbd0 con 0x7f46e019bec0 2024-03-12T05:28:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.336+0000 7f46ddffb640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 0x7f46d00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.336+0000 7f46e49b8640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 0x7f46d00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.336+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 8950+0+0 (secure 0 0 0) 0x7f46d80cdde0 con 0x7f46e019bec0 2024-03-12T05:28:27.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.336+0000 7f46e49b8640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 0x7f46d00a40e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005e10 tx=0x7f46d4005da0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:27.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.340+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f46d8096aa0 con 0x7f46e019bec0 2024-03-12T05:28:27.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:27.523+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46e01096b0 con 0x7f46e019bec0 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: pgmap v555: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: Cluster is now healthy 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6832/1070554416,v1:172.21.15.116:6833/1070554416] up:boot 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6838/2928559160,v1:172.21.15.116:6839/2928559160] up:boot 2024-03-12T05:28:28.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6834/3033418474,v1:172.21.15.116:6835/3033418474] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.116:6836/507767092,v1:172.21.15.116:6837/507767092] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6832/820405621,v1:172.21.15.130:6834/820405621] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6836/1716584597,v1:172.21.15.130:6838/1716584597] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6837/1642922632,v1:172.21.15.130:6839/1642922632] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.130:6833/2789502372,v1:172.21.15.130:6835/2789502372] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6839/3872832103,v1:172.21.15.45:6841/3872832103] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: mds.? [v2:172.21.15.45:6838/3732258060,v1:172.21.15.45:6840/3732258060] up:boot 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: fsmap 12 up:standby 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:28:28.116 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:28:28.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:28:28.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:28:28.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:28:28.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:28:28.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:27 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.203+0000 7f46ddffb640 1 -- 172.21.15.45:0/1750067850 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) v1 ==== 181+0+0 (secure 0 0 0) 0x7f46d809a160 con 0x7f46e019bec0 2024-03-12T05:28:28.198 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_metadata' removed 2024-03-12T05:28:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 msgr2=0x7f46d00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 0x7f46d00a40e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005e10 tx=0x7f46d4005da0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 msgr2=0x7f46e019c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f46d800cce0 tx=0x7f46d800a620 comp rx=0 tx=0).stop 2024-03-12T05:28:28.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 shutdown_connections 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7f46d00a1c20 0x7f46d00a40e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46e019bec0 0x7f46e019c370 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46e0106430 0x7f46e0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 --2- 172.21.15.45:0/1750067850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46e00685d0 0x7f46e0197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 >> 172.21.15.45:0/1750067850 conn(0x7f46e0077180 msgr2=0x7f46e00fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.207+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 shutdown_connections 2024-03-12T05:28:28.202 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.208+0000 7f46e6c1c640 1 -- 172.21.15.45:0/1750067850 wait complete. 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: pgmap v555: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: Cluster is now healthy 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6832/1070554416,v1:172.21.15.116:6833/1070554416] up:boot 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6838/2928559160,v1:172.21.15.116:6839/2928559160] up:boot 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6834/3033418474,v1:172.21.15.116:6835/3033418474] up:boot 2024-03-12T05:28:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.116:6836/507767092,v1:172.21.15.116:6837/507767092] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6832/820405621,v1:172.21.15.130:6834/820405621] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6836/1716584597,v1:172.21.15.130:6838/1716584597] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6837/1642922632,v1:172.21.15.130:6839/1642922632] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.130:6833/2789502372,v1:172.21.15.130:6835/2789502372] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6839/3872832103,v1:172.21.15.45:6841/3872832103] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: mds.? [v2:172.21.15.45:6838/3732258060,v1:172.21.15.45:6840/3732258060] up:boot 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: fsmap 12 up:standby 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:28:28.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:28:28.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:28:28.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:28:28.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:27 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:28.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: pgmap v555: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 KiB/s wr, 12 op/s 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: Cluster is now healthy 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6832/1070554416,v1:172.21.15.116:6833/1070554416] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6838/2928559160,v1:172.21.15.116:6839/2928559160] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6834/3033418474,v1:172.21.15.116:6835/3033418474] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.116:6836/507767092,v1:172.21.15.116:6837/507767092] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6832/820405621,v1:172.21.15.130:6834/820405621] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6836/1716584597,v1:172.21.15.130:6838/1716584597] up:boot 2024-03-12T05:28:28.241 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6837/1642922632,v1:172.21.15.130:6839/1642922632] up:boot 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.130:6833/2789502372,v1:172.21.15.130:6835/2789502372] up:boot 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6839/3872832103,v1:172.21.15.45:6841/3872832103] up:boot 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: mds.? [v2:172.21.15.45:6838/3732258060,v1:172.21.15.45:6840/3732258060] up:boot 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: fsmap 12 up:standby 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:28:28.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:28:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='mgr.25024 172.21.15.45:0/2733722900' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:28:28.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:27 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:28.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 -- 172.21.15.45:0/169526209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 msgr2=0x7fb7781027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:28.335 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/169526209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb7781027d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fb768009900 tx=0x7fb76802bcd0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 -- 172.21.15.45:0/169526209 shutdown_connections 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/169526209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 0x7fb77810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/169526209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 0x7fb77810b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/169526209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb7781027d0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 -- 172.21.15.45:0/169526209 >> 172.21.15.45:0/169526209 conn(0x7fb7780771e0 msgr2=0x7fb7780775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 -- 172.21.15.45:0/169526209 shutdown_connections 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.339+0000 7fb77d2a7640 1 -- 172.21.15.45:0/169526209 wait complete. 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 Processor -- start 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 -- start start 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 0x7fb778100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 0x7fb778100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:28.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781a59b0 con 0x7fb7781023f0 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781a5b20 con 0x7fb778102d10 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb77d2a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781a5ce0 con 0x7fb77810b7c0 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:51442/0 (socket says 172.21.15.45:51442) 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 -- 172.21.15.45:0/2086279597 learned_addr learned my addr 172.21.15.45:0/2086279597 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb7777fe640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 0x7fb778100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb7767fc640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 0x7fb778100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 -- 172.21.15.45:0/2086279597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 msgr2=0x7fb778100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 0x7fb778100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 -- 172.21.15.45:0/2086279597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 msgr2=0x7fb778100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 0x7fb778100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:28.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.340+0000 7fb776ffd640 1 -- 172.21.15.45:0/2086279597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768009560 con 0x7fb7781023f0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb776ffd640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb768006010 tx=0x7fb768004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680035f0 con 0x7fb7781023f0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1102+0+0 (secure 0 0 0) 0x7fb768003c00 con 0x7fb7781023f0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7781a5f60 con 0x7fb7781023f0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7680055b0 con 0x7fb7781023f0 2024-03-12T05:28:28.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.341+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7781a63e0 con 0x7fb7781023f0 2024-03-12T05:28:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.342+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780683c0 con 0x7fb7781023f0 2024-03-12T05:28:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.343+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 27) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb768004dc0 con 0x7fb7781023f0 2024-03-12T05:28:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.343+0000 7fb75ffff640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 0x7fb7600a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:28:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.343+0000 7fb7767fc640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 0x7fb7600a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:28:28.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.344+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 8396+0+0 (secure 0 0 0) 0x7fb7680f0370 con 0x7fb7781023f0 2024-03-12T05:28:28.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.344+0000 7fb7767fc640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 0x7fb7600a4090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb778101a80 tx=0x7fb76c005cc0 comp rx=0 tx=0).ready entity=mgr.25024 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:28:28.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.348+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb7680b9370 con 0x7fb7781023f0 2024-03-12T05:28:28.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:28.547+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb778101520 con 0x7fb7781023f0 2024-03-12T05:28:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.209+0000 7fb75ffff640 1 -- 172.21.15.45:0/2086279597 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v115) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb7680bca30 con 0x7fb7781023f0 2024-03-12T05:28:29.203 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'cephfs_data' removed 2024-03-12T05:28:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.213+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 msgr2=0x7fb7600a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.213+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 0x7fb7600a4090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb778101a80 tx=0x7fb76c005cc0 comp rx=0 tx=0).stop 2024-03-12T05:28:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.213+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 msgr2=0x7fb778100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:28:29.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.213+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb768006010 tx=0x7fb768004270 comp rx=0 tx=0).stop 2024-03-12T05:28:29.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 shutdown_connections 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:6800/869096728,v1:172.21.15.45:6801/869096728] conn(0x7fb7600a1bd0 0x7fb7600a4090 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb77810b7c0 0x7fb778100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb778102d10 0x7fb778100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 --2- 172.21.15.45:0/2086279597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7781023f0 0x7fb778100040 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 >> 172.21.15.45:0/2086279597 conn(0x7fb7780771e0 msgr2=0x7fb7780fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 shutdown_connections 2024-03-12T05:28:29.209 INFO:teuthology.orchestra.run.smithi045.stderr:2024-03-12T05:28:29.215+0000 7fb77d2a7640 1 -- 172.21.15.45:0/2086279597 wait complete. 2024-03-12T05:28:29.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:28:29.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:28:29.317 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:28:29.361 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T05:28:29.362 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T05:28:29.389 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T05:28:29.411 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T05:28:29.428 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T05:28:29.428 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T05:28:29.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a 2024-03-12T05:28:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T05:28:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 ceph-mon[32174]: osdmap e114: 12 total, 12 up, 12 in 2024-03-12T05:28:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 ceph-mon[32174]: from='client.? 172.21.15.45:0/2086279597' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:29.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T05:28:29.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:29 smithi130 ceph-mon[35398]: osdmap e114: 12 total, 12 up, 12 in 2024-03-12T05:28:29.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:29 smithi130 ceph-mon[35398]: from='client.? 172.21.15.45:0/2086279597' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:29.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/1750067850' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T05:28:29.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:29 smithi116 ceph-mon[35486]: osdmap e114: 12 total, 12 up, 12 in 2024-03-12T05:28:29.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:29 smithi116 ceph-mon[35486]: from='client.? 172.21.15.45:0/2086279597' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T05:28:29.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 systemd[1]: Stopping Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:30.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a[32170]: 2024-03-12T05:28:29.908+0000 7f3f549a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:30.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a[32170]: 2024-03-12T05:28:29.908+0000 7f3f549a3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T05:28:30.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:30 smithi045 podman[122190]: 2024-03-12 05:28:30.044186987 +0000 UTC m=+0.267971877 container died 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:28:31.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:31 smithi045 podman[122190]: 2024-03-12 05:28:31.613357321 +0000 UTC m=+1.837142204 container cleanup 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T05:28:31.864 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:31 smithi045 bash[122190]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a 2024-03-12T05:28:34.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T05:28:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:33 smithi045 podman[122202]: 2024-03-12 05:28:33.837536421 +0000 UTC m=+3.787988639 container remove 20415fd31915e7b306e87c302edc5c57cd53278fd655ee11d32103e58ee21e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T05:28:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:34 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T05:28:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:34 smithi045 systemd[1]: Stopped Ceph mon.a for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:28:34.114 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 12 05:28:34 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.a.service: Consumed 26.395s CPU time. 2024-03-12T05:28:34.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:34.273 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T05:28:34.273 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-12T05:28:34.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.b 2024-03-12T05:28:34.602 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:34 smithi116 systemd[1]: Stopping Ceph mon.b for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b[35482]: 2024-03-12T05:28:34.607+0000 7f9526d02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b[35482]: 2024-03-12T05:28:34.607+0000 7f9526d02700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T05:28:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:34 smithi116 podman[72924]: 2024-03-12 05:28:34.812255068 +0000 UTC m=+0.321873375 container died 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-12T05:28:35.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:35 smithi116 podman[72924]: 2024-03-12 05:28:35.199622929 +0000 UTC m=+0.709241250 container cleanup 3136956a43500f9570575d9fffed3e1e8bf49f235f6f89af33d7a322307c9f69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T05:28:35.491 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 12 05:28:35 smithi116 bash[72924]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-b 2024-03-12T05:28:35.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T05:28:35.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:35.779 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-12T05:28:35.779 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-12T05:28:35.779 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.c 2024-03-12T05:28:36.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:35 smithi130 systemd[1]: Stopping Ceph mon.c for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:36.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c[35390]: 2024-03-12T05:28:36.232+0000 7fac0bda1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:36.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c[35390]: 2024-03-12T05:28:36.232+0000 7fac0bda1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T05:28:36.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:36 smithi130 podman[72597]: 2024-03-12 05:28:36.427223029 +0000 UTC m=+0.370172269 container died 2543392497e6caea8f0a317f8016a3010593c0eeb10b2cc6cd8783724672d3ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-12T05:28:38.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 podman[72597]: 2024-03-12 05:28:38.112326278 +0000 UTC m=+2.055275585 container cleanup 2543392497e6caea8f0a317f8016a3010593c0eeb10b2cc6cd8783724672d3ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:28:38.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 bash[72597]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c 2024-03-12T05:28:38.745 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T05:28:38.966 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 podman[72612]: 2024-03-12 05:28:38.614095246 +0000 UTC m=+2.182614995 container remove 2543392497e6caea8f0a317f8016a3010593c0eeb10b2cc6cd8783724672d3ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T05:28:39.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-12T05:28:39.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 systemd[1]: Stopped Ceph mon.c for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:28:39.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Mar 12 05:28:38 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mon.c.service: Consumed 15.155s CPU time. 2024-03-12T05:28:39.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:39.164 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-12T05:28:39.164 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T05:28:39.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x 2024-03-12T05:28:39.493 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:28:39 smithi045 systemd[1]: Stopping Ceph mgr.x for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:39.864 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:28:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x[32410]: 2024-03-12T05:28:39.498+0000 7f527563e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:39.864 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:28:39 smithi045 podman[122310]: 2024-03-12 05:28:39.529425036 +0000 UTC m=+0.159252992 container died fd1d6e2148cb4aa890ce7c3cc76d68deb193049ad616ae905c9ab99127f90112 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-12T05:28:41.036 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:28:40 smithi045 podman[122310]: 2024-03-12 05:28:40.833323812 +0000 UTC m=+1.463151763 container cleanup fd1d6e2148cb4aa890ce7c3cc76d68deb193049ad616ae905c9ab99127f90112 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T05:28:41.037 INFO:journalctl@ceph.mgr.x.smithi045.stdout:Mar 12 05:28:40 smithi045 bash[122310]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-x 2024-03-12T05:28:41.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T05:28:41.294 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:41.294 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T05:28:41.294 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T05:28:41.294 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y 2024-03-12T05:28:41.740 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:41 smithi116 systemd[1]: Stopping Ceph mgr.y for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:42.990 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:42 smithi116 podman[73046]: 2024-03-12 05:28:42.553445682 +0000 UTC m=+1.063440769 container died f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-12T05:28:43.637 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 podman[73046]: 2024-03-12 05:28:43.389622523 +0000 UTC m=+1.899617617 container cleanup f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-12T05:28:43.638 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 bash[73046]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y 2024-03-12T05:28:43.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T05:28:43.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 podman[73060]: 2024-03-12 05:28:43.64118475 +0000 UTC m=+1.074824658 container remove f93e594effb6a0f5bc53feff7dc8d70e174d5e1491f9e3e88484cd314102d9f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T05:28:43.893 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-12T05:28:43.894 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-12T05:28:43.894 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 systemd[1]: Stopped Ceph mgr.y for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:28:43.894 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 12 05:28:43 smithi116 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.y.service: Consumed 21.838s CPU time. 2024-03-12T05:28:43.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:43.906 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T05:28:43.906 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-12T05:28:43.906 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.z 2024-03-12T05:28:44.215 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:28:43 smithi130 systemd[1]: Stopping Ceph mgr.z for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:44.530 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:28:44 smithi130 podman[72732]: 2024-03-12 05:28:44.255110678 +0000 UTC m=+0.140073603 container died 15fd0feb25c73c14f394cccf79c9454da0477abb1065968422b2f0c7a73e677b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T05:28:44.889 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:28:44 smithi130 podman[72732]: 2024-03-12 05:28:44.644272221 +0000 UTC m=+0.529235148 container cleanup 15fd0feb25c73c14f394cccf79c9454da0477abb1065968422b2f0c7a73e677b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-12T05:28:44.890 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Mar 12 05:28:44 smithi130 bash[72732]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mgr-z 2024-03-12T05:28:44.987 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-12T05:28:45.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:45.026 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-12T05:28:45.026 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T05:28:45.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.0 2024-03-12T05:28:45.348 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:45 smithi045 systemd[1]: Stopping Ceph osd.0 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:45.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:28:45.353+0000 7f1c832a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:45.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:28:45.353+0000 7f1c832a0700 -1 osd.0 115 *** Got signal Terminated *** 2024-03-12T05:28:45.615 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0[47383]: 2024-03-12T05:28:45.353+0000 7f1c832a0700 -1 osd.0 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:28:51.267 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 podman[122433]: 2024-03-12 05:28:51.004653072 +0000 UTC m=+5.774152021 container died cb87d553c9b8bf0e7f8fa0a010d920c8d6b76ef02b4cd9c9c4b3079b7035e476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:28:51.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 podman[122433]: 2024-03-12 05:28:51.354838844 +0000 UTC m=+6.124337792 container cleanup cb87d553c9b8bf0e7f8fa0a010d920c8d6b76ef02b4cd9c9c4b3079b7035e476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-12T05:28:51.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 bash[122433]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0 2024-03-12T05:28:51.930 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 podman[122462]: 2024-03-12 05:28:51.639597538 +0000 UTC m=+0.633221662 container remove cb87d553c9b8bf0e7f8fa0a010d920c8d6b76ef02b4cd9c9c4b3079b7035e476 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-12T05:28:52.211 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 podman[122558]: 2024-03-12 05:28:51.84026168 +0000 UTC m=+0.018635902 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:28:52.211 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:51 smithi045 podman[122558]: 2024-03-12 05:28:51.973941288 +0000 UTC m=+0.152315535 container create 97bdc7837e8a115dfb5207f5f4b6ba7b2d0144134381e6a8b817b207f8045231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:28:52.211 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:52 smithi045 podman[122558]: 2024-03-12 05:28:52.218257978 +0000 UTC m=+0.396632193 container init 97bdc7837e8a115dfb5207f5f4b6ba7b2d0144134381e6a8b817b207f8045231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:28:52.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:52 smithi045 podman[122558]: 2024-03-12 05:28:52.221381755 +0000 UTC m=+0.399755961 container start 97bdc7837e8a115dfb5207f5f4b6ba7b2d0144134381e6a8b817b207f8045231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T05:28:52.614 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:52 smithi045 podman[122558]: 2024-03-12 05:28:52.351806947 +0000 UTC m=+0.530181155 container attach 97bdc7837e8a115dfb5207f5f4b6ba7b2d0144134381e6a8b817b207f8045231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-12T05:28:53.114 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 12 05:28:52 smithi045 podman[122558]: 2024-03-12 05:28:52.795815266 +0000 UTC m=+0.974189482 container died 97bdc7837e8a115dfb5207f5f4b6ba7b2d0144134381e6a8b817b207f8045231 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:28:53.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T05:28:53.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:28:53.464 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T05:28:53.464 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T05:28:53.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.1 2024-03-12T05:28:53.990 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:53 smithi116 systemd[1]: Stopping Ceph osd.1 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:28:53.990 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1[40068]: 2024-03-12T05:28:53.742+0000 7fad417aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:28:53.990 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1[40068]: 2024-03-12T05:28:53.742+0000 7fad417aa700 -1 osd.1 115 *** Got signal Terminated *** 2024-03-12T05:28:53.991 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1[40068]: 2024-03-12T05:28:53.742+0000 7fad417aa700 -1 osd.1 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:28:59.530 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:59 smithi116 podman[73172]: 2024-03-12 05:28:59.27629648 +0000 UTC m=+5.615861418 container died 1797fa2b141da34cc78542d29591cab7eb24468114742bde5d19bec246dc31dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:28:59.813 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:59 smithi116 podman[73172]: 2024-03-12 05:28:59.594149399 +0000 UTC m=+5.933714337 container cleanup 1797fa2b141da34cc78542d29591cab7eb24468114742bde5d19bec246dc31dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T05:28:59.814 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:59 smithi116 bash[73172]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1 2024-03-12T05:29:00.240 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:28:59 smithi116 podman[73188]: 2024-03-12 05:28:59.820487378 +0000 UTC m=+0.545144642 container remove 1797fa2b141da34cc78542d29591cab7eb24468114742bde5d19bec246dc31dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-12T05:29:00.496 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.150791428 +0000 UTC m=+0.018573456 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:00.496 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.263234982 +0000 UTC m=+0.131016998 container create a8b718065de787cc7bbb31249b83295af4783d9687a3d1ac572fafc4fb6ed6c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:00.496 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.50264238 +0000 UTC m=+0.370424404 container init a8b718065de787cc7bbb31249b83295af4783d9687a3d1ac572fafc4fb6ed6c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T05:29:00.831 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.506240219 +0000 UTC m=+0.374022234 container start a8b718065de787cc7bbb31249b83295af4783d9687a3d1ac572fafc4fb6ed6c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:29:00.831 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.599350545 +0000 UTC m=+0.467132574 container attach a8b718065de787cc7bbb31249b83295af4783d9687a3d1ac572fafc4fb6ed6c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-12T05:29:01.205 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 12 05:29:00 smithi116 podman[73280]: 2024-03-12 05:29:00.838553997 +0000 UTC m=+0.706336009 container died a8b718065de787cc7bbb31249b83295af4783d9687a3d1ac572fafc4fb6ed6c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:01.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T05:29:01.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:01.579 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T05:29:01.579 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T05:29:01.580 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.2 2024-03-12T05:29:01.892 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:01 smithi130 systemd[1]: Stopping Ceph osd.2 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:02.215 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2[40565]: 2024-03-12T05:29:01.898+0000 7fcdd7d1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:02.215 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2[40565]: 2024-03-12T05:29:01.898+0000 7fcdd7d1c700 -1 osd.2 115 *** Got signal Terminated *** 2024-03-12T05:29:02.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2[40565]: 2024-03-12T05:29:01.898+0000 7fcdd7d1c700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:07.901 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:07 smithi130 podman[72861]: 2024-03-12 05:29:07.618135496 +0000 UTC m=+5.834489361 container died bf548f5da58c388dd2017faec7f6f8b35b7a3e82639d41027a20dbb46672f462 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:08.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72861]: 2024-03-12 05:29:08.015684958 +0000 UTC m=+6.232038819 container cleanup bf548f5da58c388dd2017faec7f6f8b35b7a3e82639d41027a20dbb46672f462 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True) 2024-03-12T05:29:08.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 bash[72861]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2 2024-03-12T05:29:08.606 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72878]: 2024-03-12 05:29:08.275497146 +0000 UTC m=+0.659893307 container remove bf548f5da58c388dd2017faec7f6f8b35b7a3e82639d41027a20dbb46672f462 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T05:29:08.890 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72980]: 2024-03-12 05:29:08.515086806 +0000 UTC m=+0.021484723 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:08.890 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72980]: 2024-03-12 05:29:08.62650464 +0000 UTC m=+0.132902547 container create b09f7d559abc4e5e8299287b4f15a4326a2caac1f5556b8fc4c7bb39268a3386 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T05:29:09.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72980]: 2024-03-12 05:29:08.896166753 +0000 UTC m=+0.402564655 container init b09f7d559abc4e5e8299287b4f15a4326a2caac1f5556b8fc4c7bb39268a3386 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T05:29:09.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:08 smithi130 podman[72980]: 2024-03-12 05:29:08.900609767 +0000 UTC m=+0.407007669 container start b09f7d559abc4e5e8299287b4f15a4326a2caac1f5556b8fc4c7bb39268a3386 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T05:29:09.216 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:09 smithi130 podman[72980]: 2024-03-12 05:29:09.046627177 +0000 UTC m=+0.553025078 container attach b09f7d559abc4e5e8299287b4f15a4326a2caac1f5556b8fc4c7bb39268a3386 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T05:29:10.327 INFO:journalctl@ceph.osd.2.smithi130.stdout:Mar 12 05:29:10 smithi130 podman[72980]: 2024-03-12 05:29:10.01589182 +0000 UTC m=+1.522289725 container died b09f7d559abc4e5e8299287b4f15a4326a2caac1f5556b8fc4c7bb39268a3386 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:29:10.647 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T05:29:10.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:10.689 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T05:29:10.690 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T05:29:10.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.3 2024-03-12T05:29:11.001 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:10 smithi045 systemd[1]: Stopping Ceph osd.3 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:11.364 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3[54225]: 2024-03-12T05:29:11.006+0000 7fb45e64a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:11.364 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3[54225]: 2024-03-12T05:29:11.006+0000 7fb45e64a700 -1 osd.3 115 *** Got signal Terminated *** 2024-03-12T05:29:11.364 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:11 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3[54225]: 2024-03-12T05:29:11.006+0000 7fb45e64a700 -1 osd.3 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:12.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:11.975+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:11.821+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:12.681 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:12 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:12.385+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:13.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:12 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:12.687+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:13.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:12.964+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:12.816+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:13.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:12.985+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:13.702 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:13.390+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:14.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:13 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:13.708+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:14.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:13.968+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:14.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:13.822+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:14.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:14 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:14.007+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:14.614 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:14 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:14.355+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:15.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:14 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:14.725+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:15.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:14.944+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:15.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:14.856+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:15.406 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:15.346+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:15.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:15 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:15.019+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:16.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:15 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:15.717+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:16.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:15.936+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:16.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:16.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:16 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:16.032+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:16.614 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:16 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:16.313+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:16.614 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:16 smithi045 podman[122664]: 2024-03-12 05:29:16.532087285 +0000 UTC m=+5.633805588 container died b35959c6f82ac61a687061a7e529eb96c405e519af4bbf074fac84e732ffd0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:29:17.114 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:16 smithi045 podman[122664]: 2024-03-12 05:29:16.965863475 +0000 UTC m=+6.067581764 container cleanup b35959c6f82ac61a687061a7e529eb96c405e519af4bbf074fac84e732ffd0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T05:29:17.115 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:16 smithi045 bash[122664]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3 2024-03-12T05:29:17.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:16 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:16.745+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:17.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:16.965+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:16.852+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:17.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:17 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:17.010+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:17.614 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:17 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:17.354+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:17.614 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:17 smithi045 podman[122695]: 2024-03-12 05:29:17.200507813 +0000 UTC m=+0.668303917 container remove b35959c6f82ac61a687061a7e529eb96c405e519af4bbf074fac84e732ffd0f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-12T05:29:17.897 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:17 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:17.741+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:17.897 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:17 smithi045 podman[122785]: 2024-03-12 05:29:17.541916853 +0000 UTC m=+0.017891286 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:17.897 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:17 smithi045 podman[122785]: 2024-03-12 05:29:17.668058338 +0000 UTC m=+0.144032758 container create 2978e71463cd68e1f6cb2715e85482bf056a958bdf3f9c303f6b9287aa99a707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:17.897 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:17 smithi045 podman[122785]: 2024-03-12 05:29:17.903842971 +0000 UTC m=+0.379817392 container init 2978e71463cd68e1f6cb2715e85482bf056a958bdf3f9c303f6b9287aa99a707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T05:29:18.196 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:17 smithi045 podman[122785]: 2024-03-12 05:29:17.907073589 +0000 UTC m=+0.383048024 container start 2978e71463cd68e1f6cb2715e85482bf056a958bdf3f9c303f6b9287aa99a707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-12T05:29:18.196 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:18 smithi045 podman[122785]: 2024-03-12 05:29:18.003783248 +0000 UTC m=+0.479757695 container attach 2978e71463cd68e1f6cb2715e85482bf056a958bdf3f9c303f6b9287aa99a707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-12T05:29:18.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:17.970+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:17.898+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:18.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:18 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:18.056+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:18.509 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:18 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:18.394+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:18.509 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 12 05:29:18 smithi045 podman[122785]: 2024-03-12 05:29:18.20359829 +0000 UTC m=+0.679572728 container died 2978e71463cd68e1f6cb2715e85482bf056a958bdf3f9c303f6b9287aa99a707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:18.834 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:18 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:18.724+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:18.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T05:29:18.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:18.908 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T05:29:18.908 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-12T05:29:18.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.4 2024-03-12T05:29:19.200 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:18 smithi116 systemd[1]: Stopping Ceph osd.4 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:19.200 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:19 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:19.049+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:19.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:18.937+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:18.874+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:19.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:19 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:29:19.205+0000 7f18268a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:19.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:19 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:29:19.205+0000 7f18268a2700 -1 osd.4 115 *** Got signal Terminated *** 2024-03-12T05:29:19.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:19 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:29:19.205+0000 7f18268a2700 -1 osd.4 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:19.747 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:19 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:19.424+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:20.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:19 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:19.753+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:20.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:19.888+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:19.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:20.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:20 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:20.196+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:20.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:20 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:20.031+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:20.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:20 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:20.031+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:20.707 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:20 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:20.449+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:20.919 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:20 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:20.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:20.919 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:20 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:20.615+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:21.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:20 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:20.713+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:21.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:20 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:20.925+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:21.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:20 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:20.925+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:21.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:21 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:21.213+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:21.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:21 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:21.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:21.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:21 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:21.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:21.745 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:21.427+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:21.745 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:21.427+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:21.870 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:21 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:21.592+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:22.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:21 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:21.751+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:22.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:21 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:21.924+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:22.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:21 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:21.924+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:22.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:21 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:21.877+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:22.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:21 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:21.877+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:22.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:22.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:22.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:22.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:22.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:22.250+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:22.771 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:22 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:22.465+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:22.771 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:22 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:22.465+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:22.877 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:22 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:22.590+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:23.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:22 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:22.777+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:23.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:22 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:22.777+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:23.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:22 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:22.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:23.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:22 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:22.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:23.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:22 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:22.953+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:23.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:22 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:22.953+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:23.216 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:22 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:29:22.853+0000 7f1822e96700 -1 osd.4 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.452983+0000 front 2024-03-12T05:28:53.453183+0000 (oldest deadline 2024-03-12T05:29:22.853593+0000) 2024-03-12T05:29:23.217 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:23 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:23.049+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:23.217 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:23 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:23.049+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:23.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:23 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:23.223+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:23.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:23.732+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:23.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:23.732+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:23.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:23.495+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:23.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:23.495+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:23.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:23 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:23.568+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:24.184 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:23 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4[46842]: 2024-03-12T05:29:23.890+0000 7f1822e96700 -1 osd.4 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.452983+0000 front 2024-03-12T05:28:53.453183+0000 (oldest deadline 2024-03-12T05:29:22.853593+0000) 2024-03-12T05:29:24.184 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:24 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:24.063+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:24.184 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:24 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:24.063+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:24.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:23 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:23.930+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:24.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:23 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:23.930+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:24.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:23 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:23.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:24.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:23 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:23.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:24.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:24 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:24.189+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:24.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:24 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:24.689+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:24.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:24 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:24.689+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:24.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:24 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:24.513+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:24.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:24 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:24.513+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:24.906 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:24 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:24.588+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:25.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:24 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:24.912+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:25.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:24 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:24.912+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:25.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:24 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:24.914+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:25.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:24 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:24.914+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:25.220 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:24 smithi116 podman[73403]: 2024-03-12 05:29:24.922693918 +0000 UTC m=+5.814225217 container died 3222ef06758fa4fe94f0810a3fa37ba8aad33efe78a3db65c11b466c69b39002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:29:25.220 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:25 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:25.158+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:25.220 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:25 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:25.093+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:25.220 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:25 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:25.093+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:25.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:25 smithi116 podman[73403]: 2024-03-12 05:29:25.333801242 +0000 UTC m=+6.225332536 container cleanup 3222ef06758fa4fe94f0810a3fa37ba8aad33efe78a3db65c11b466c69b39002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:29:25.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:25 smithi116 bash[73403]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4 2024-03-12T05:29:25.855 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:25 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:25.661+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:25.855 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:25 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:25.661+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:25.855 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:25 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:25.558+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:25.855 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:25 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:25.558+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:25.882 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:25 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:25.628+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:25.938 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:25 smithi116 podman[73422]: 2024-03-12 05:29:25.619293138 +0000 UTC m=+0.684159360 container remove 3222ef06758fa4fe94f0810a3fa37ba8aad33efe78a3db65c11b466c69b39002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:29:26.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:25 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:25.889+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:26.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:25 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:25.889+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:26.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:25 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:25.957+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:26.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:25 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:25.957+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:26.223 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:26.122+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:26.128+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:26 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:26.128+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:25 smithi116 podman[73514]: 2024-03-12 05:29:25.945198073 +0000 UTC m=+0.113440237 container create cc31a00919abb965ce22738baa23a57e680b08b515ca1931a4c83182d62b2ea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:25 smithi116 podman[73514]: 2024-03-12 05:29:25.848917309 +0000 UTC m=+0.017159477 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:26 smithi116 podman[73514]: 2024-03-12 05:29:26.139149059 +0000 UTC m=+0.307391222 container init cc31a00919abb965ce22738baa23a57e680b08b515ca1931a4c83182d62b2ea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:26 smithi116 podman[73514]: 2024-03-12 05:29:26.142325165 +0000 UTC m=+0.310567330 container start cc31a00919abb965ce22738baa23a57e680b08b515ca1931a4c83182d62b2ea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-12T05:29:26.224 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:26 smithi116 podman[73514]: 2024-03-12 05:29:26.230879386 +0000 UTC m=+0.399121553 container attach cc31a00919abb965ce22738baa23a57e680b08b515ca1931a4c83182d62b2ea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T05:29:26.490 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 12 05:29:26 smithi116 podman[73514]: 2024-03-12 05:29:26.437123932 +0000 UTC m=+0.605366100 container died cc31a00919abb965ce22738baa23a57e680b08b515ca1931a4c83182d62b2ea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-12T05:29:26.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:26.695+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:26.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:26.695+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:26.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:26.521+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:26.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:26 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:26.521+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:26.892 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:26.594+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:26.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.4.service' 2024-03-12T05:29:27.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:27.011 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-12T05:29:27.011 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-12T05:29:27.011 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.5 2024-03-12T05:29:27.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:26.985+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:27.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:26.985+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:27.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:26.899+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:27.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:26 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:26.899+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:27.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 systemd[1]: Stopping Ceph osd.5 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:27.240 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:27 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:27.077+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:27.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:27 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:27.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:27.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:27 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:27.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:27.613 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:27.336+0000 7fd77c1a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:27.613 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:27.336+0000 7fd77c1a4700 -1 osd.5 115 *** Got signal Terminated *** 2024-03-12T05:29:27.613 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:27.336+0000 7fd77c1a4700 -1 osd.5 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:27.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:27.690+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:27.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:27.690+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:27.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:27.560+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:27.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:27.560+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:27.927 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:27.620+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:28.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:27.933+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:28.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:27.933+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:28.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:27.949+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:28.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:27 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:27.949+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:28.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:28 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:28.108+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:28.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:28 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:28.111+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:28.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:28 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:28.111+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:28.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:28.604+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:28.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:28.604+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:28.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:28.693+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:28.865 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:28.693+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:28.865 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:28 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:28.693+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:28.923 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:28.577+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:28.923 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:28.577+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:29.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:28.929+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:29.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:28.929+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:29.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:28.929+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.443031+0000 front 2024-03-12T05:29:06.442900+0000 (oldest deadline 2024-03-12T05:29:28.143024+0000) 2024-03-12T05:29:29.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:28.976+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:29.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:28.976+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:29.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:28 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:28.976+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:29.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:29 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:29.123+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:29.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:29 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:29.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:29.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:29 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:29.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:29.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:29.664+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:29.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:29.664+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:29.864 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:29.664+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:29.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:29.617+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:29.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:29.617+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:29.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:29 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:29.617+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:29.892 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:29.608+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:29.892 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:29.608+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:30.215 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:29.898+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:30.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:29.898+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:30.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:29.898+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.443031+0000 front 2024-03-12T05:29:06.442900+0000 (oldest deadline 2024-03-12T05:29:28.143024+0000) 2024-03-12T05:29:30.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:29.945+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:30.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:29.945+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:30.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:29 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:29.945+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:30.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:30 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:30.135+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:30.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:30 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:30.037+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:30.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:30 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:30.037+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:30.914 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:30.658+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:30.915 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:30.658+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:30.915 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:30.884+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:30.915 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:30.884+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:30.915 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:30.884+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.443031+0000 front 2024-03-12T05:29:06.442900+0000 (oldest deadline 2024-03-12T05:29:28.143024+0000) 2024-03-12T05:29:31.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:30.704+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:31.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:30.704+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:31.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:30.704+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:31.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:30.637+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:31.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:30.637+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:31.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:30 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:30.637+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:31.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:30.920+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:31.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:30.920+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:31.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:30 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:30.920+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:31.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:31 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:31.171+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:31.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:31 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:31.171+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:31.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:31 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:31.001+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:31.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:31 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:31.001+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:31.870 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:31.622+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:31.870 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:31.622+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:32.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:31.714+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:32.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:31.714+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:32.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:31.715+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:32.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:31.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:32.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:31.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:32.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:31.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:31.922+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.039623+0000 front 2024-03-12T05:28:46.039691+0000 (oldest deadline 2024-03-12T05:29:11.339764+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:31.922+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.241971+0000 front 2024-03-12T05:28:58.242072+0000 (oldest deadline 2024-03-12T05:29:20.542042+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5[47352]: 2024-03-12T05:29:31.922+0000 7fd778798700 -1 osd.5 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.443031+0000 front 2024-03-12T05:29:06.442900+0000 (oldest deadline 2024-03-12T05:29:28.143024+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:31.876+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:31.876+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:32.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:31 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:31.876+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:32.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:32.183+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:32.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:32.183+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:32.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:32.040+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:32.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:32.040+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:32.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:32 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:32.040+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:32.855 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:32.578+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:32.855 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:32.578+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:33.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:32.730+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:33.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:32.730+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:33.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:32.730+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:33.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:32.693+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:33.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:32.693+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:33.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:32 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:32.693+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:33.216 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:33 smithi130 podman[73087]: 2024-03-12 05:29:33.042015973 +0000 UTC m=+5.830960648 container died c45dbddf992504f920f88b41259d8eef1cc4aa114cf1a653e8b6a4e894862039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:33.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:32.859+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:33.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:32.859+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:33.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:32 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:32.859+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:33.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:33 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:33.224+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:33.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:33 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:33.224+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:33.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:33 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:33.064+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:33.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:33 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:33.064+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:33.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:33 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:33.064+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:33.706 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:33 smithi130 podman[73087]: 2024-03-12 05:29:33.43680106 +0000 UTC m=+6.225745744 container cleanup c45dbddf992504f920f88b41259d8eef1cc4aa114cf1a653e8b6a4e894862039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:33.706 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:33 smithi130 bash[73087]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5 2024-03-12T05:29:33.707 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:33 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:33.583+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:33.707 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:33 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:33.583+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:33.965 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:33 smithi130 podman[73105]: 2024-03-12 05:29:33.713227193 +0000 UTC m=+0.664743880 container remove c45dbddf992504f920f88b41259d8eef1cc4aa114cf1a653e8b6a4e894862039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:33.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:33 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:33.828+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:33.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:33 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:33.828+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:33.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:33 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:33.828+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:34.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:33.716+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:34.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:33.716+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:34.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:33.716+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:34.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:33.686+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:34.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:33.686+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:34.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:33 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:33.686+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:34.302 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:33.958497927 +0000 UTC m=+0.018356375 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:34.302 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:34.081008085 +0000 UTC m=+0.140866524 container create 41ff38dfb8724496935483f5ed453a9972671b657e5ebf922f9a93b1742e3bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T05:29:34.302 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:34.308981488 +0000 UTC m=+0.368839928 container init 41ff38dfb8724496935483f5ed453a9972671b657e5ebf922f9a93b1742e3bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:34.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:34.221+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:34.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:34.221+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:34.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:34.114+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:34.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:34.114+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:34.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:34 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:34.114+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:34.592 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:34.31207882 +0000 UTC m=+0.371937262 container start 41ff38dfb8724496935483f5ed453a9972671b657e5ebf922f9a93b1742e3bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-12T05:29:34.592 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:34.408940993 +0000 UTC m=+0.468799440 container attach 41ff38dfb8724496935483f5ed453a9972671b657e5ebf922f9a93b1742e3bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T05:29:34.861 INFO:journalctl@ceph.osd.5.smithi130.stdout:Mar 12 05:29:34 smithi130 podman[73197]: 2024-03-12 05:29:34.599411596 +0000 UTC m=+0.659270027 container died 41ff38dfb8724496935483f5ed453a9972671b657e5ebf922f9a93b1742e3bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:29:34.861 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:34 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:34.597+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:34.861 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:34 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:34.597+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:35.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:34.734+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:35.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:34.734+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:35.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:34.734+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:35.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:34.701+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:35.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:34.701+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:35.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:34 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:34.701+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:35.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:34 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:34.866+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:35.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:34 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:34.866+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:35.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:34 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:34.866+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:35.259 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.5.service' 2024-03-12T05:29:35.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:35.301 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-12T05:29:35.301 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-12T05:29:35.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.6 2024-03-12T05:29:35.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:35 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:35.238+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:35.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:35 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:35.238+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:35.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:35 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:35.087+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:35.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:35 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:35.087+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:35.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:35 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:35.087+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:35.614 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 systemd[1]: Stopping Ceph osd.6 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:35.891 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:35 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:35.567+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:35.891 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:35 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:35.567+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:36.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:35.678+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:35.678+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:35.678+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.634+0000 7fe795cd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.634+0000 7fe795cd1700 -1 osd.6 115 *** Got signal Terminated *** 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.634+0000 7fe795cd1700 -1 osd.6 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.742+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.742+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:36.115 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:35.742+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:36.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:35 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:35.897+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:36.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:35 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:35.897+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:36.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:35 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:35.897+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:36.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:36 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:36.279+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:36.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:36 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:36.279+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:36.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:36 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:36.097+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:36.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:36 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:36.097+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:36.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:36 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:36.097+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:36.909 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:36.547+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:36.909 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:36.547+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:37.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:36.783+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:37.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:36.783+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:37.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:36.783+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:37.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:36.661+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:37.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:36.661+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:37.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:36 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:36.661+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:37.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:36.915+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:37.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:36.915+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:37.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:36 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:36.915+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:37.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:37.234+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:37.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:37.234+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:37.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:37.106+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:37.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:37.106+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:37.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:37.106+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:37.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:37 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:37.106+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:37.881 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:37 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:37.570+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:37.881 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:37 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:37.570+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:38.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:37.832+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:38.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:37.832+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:38.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:37.832+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:38.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:37.676+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:38.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:37.676+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:38.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:37 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:37.676+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:38.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:37 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:37.886+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:38.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:37 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:37.886+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:38.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:37 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:37.886+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:38.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:38.224+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:38.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:38.224+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:38.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:38.104+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:38.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:38.104+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:38.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:38.104+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:38.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:38 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:38.104+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:38.841 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:38 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:38.599+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:38.841 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:38 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:38.599+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:39.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:38.816+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:39.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:38.816+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:39.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:38.816+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:39.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:38.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:39.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:38.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:39.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:38 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:38.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:39.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:38 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:38.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:39.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:38 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:38.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:39.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:38 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:38.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:39.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:39.229+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:39.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:39.229+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:39.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:39.095+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:39.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:39.095+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:39.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:39.095+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:39.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:39 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:39.095+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:39.715 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:39 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:39.568+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:39.715 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:39 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:39.568+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:40.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:39.768+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.610673+0000 front 2024-03-12T05:28:48.610362+0000 (oldest deadline 2024-03-12T05:29:12.110517+0000) 2024-03-12T05:29:40.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:39.768+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.811563+0000 front 2024-03-12T05:28:56.811434+0000 (oldest deadline 2024-03-12T05:29:22.711457+0000) 2024-03-12T05:29:40.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6[61195]: 2024-03-12T05:29:39.768+0000 7fe78e2bd700 -1 osd.6 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:02.712329+0000 front 2024-03-12T05:29:02.712173+0000 (oldest deadline 2024-03-12T05:29:28.612146+0000) 2024-03-12T05:29:40.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:39.622+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:40.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:39.622+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:40.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:39 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:39.622+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:40.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:39 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:39.808+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:40.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:39 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:39.808+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:40.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:39 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:39.808+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:40.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:40.246+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:40.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:40.246+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:40.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:40.246+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:40.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:40.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:40.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:40.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:40.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:40.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:40.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:40.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:40.830 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:40 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:40.528+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:40.831 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:40 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:40.528+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:41.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:40.642+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:41.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:40.642+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:41.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:40 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:40.642+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:41.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:40 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:40.836+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:41.265 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:40 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:40.836+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:41.265 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:40 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:40.836+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:41.412 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:41 smithi045 podman[122888]: 2024-03-12 05:29:41.157285945 +0000 UTC m=+5.654504180 container died d2d7925406d8fcfe3d1605c66966e8078bcce5fc9e289edfd8891781ffb88b0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-12T05:29:41.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:41.263+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:41.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:41.263+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:41.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:41.263+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:41.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:41.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:41.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:41.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:41.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:41.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:41.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:41 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:41.046+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:41.687 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:41 smithi045 podman[122888]: 2024-03-12 05:29:41.501445686 +0000 UTC m=+5.998663962 container cleanup d2d7925406d8fcfe3d1605c66966e8078bcce5fc9e289edfd8891781ffb88b0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T05:29:41.687 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:41 smithi045 bash[122888]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6 2024-03-12T05:29:41.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:41.649+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:41.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:41.649+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:41.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:41.649+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:41.688 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:41 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:41.649+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:41.841 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:41.545+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:41.841 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:41.545+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:41.841 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:41.545+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:42.114 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:41 smithi045 podman[122905]: 2024-03-12 05:29:41.69428692 +0000 UTC m=+0.533633907 container remove d2d7925406d8fcfe3d1605c66966e8078bcce5fc9e289edfd8891781ffb88b0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:42.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:41.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:42.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:41.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:42.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:41 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:41.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:42.233+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:42.233+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:42.233+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:42.088+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:42.088+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:42.088+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:42.392 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:42 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:42.088+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:42.399 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.025297726 +0000 UTC m=+0.017929171 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:42.399 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.145457741 +0000 UTC m=+0.138089238 container create 4212d3b49a013c31b8ce496c7548ce32e68b48344c1b5db15943610bc5f55381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T05:29:42.399 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.405940927 +0000 UTC m=+0.398572385 container init 4212d3b49a013c31b8ce496c7548ce32e68b48344c1b5db15943610bc5f55381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-12T05:29:42.691 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.409216645 +0000 UTC m=+0.401848093 container start 4212d3b49a013c31b8ce496c7548ce32e68b48344c1b5db15943610bc5f55381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-12T05:29:42.691 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.481065778 +0000 UTC m=+0.473697238 container attach 4212d3b49a013c31b8ce496c7548ce32e68b48344c1b5db15943610bc5f55381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:29:42.846 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:42.554+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:42.846 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:42.554+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:42.846 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:42.554+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:42.968 INFO:journalctl@ceph.osd.6.smithi045.stdout:Mar 12 05:29:42 smithi045 podman[123002]: 2024-03-12 05:29:42.705721623 +0000 UTC m=+0.698353069 container died 4212d3b49a013c31b8ce496c7548ce32e68b48344c1b5db15943610bc5f55381 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-12T05:29:42.969 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:42.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:42.969 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:42.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:42.969 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:42.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:42.969 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:42 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:42.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:43.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:42.852+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:43.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:42.852+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:43.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:42 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:42.852+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:43.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.6.service' 2024-03-12T05:29:43.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:43.302 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-12T05:29:43.302 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-12T05:29:43.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.7 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:43.080+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:43.080+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:43.080+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:43.080+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.211+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.211+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:43.328 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.211+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:43.613 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 systemd[1]: Stopping Ceph osd.7 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:43.828 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:43.532+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:43.828 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:43.532+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:43.828 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:43.532+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:43.990 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.628+0000 7fa4d7cd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:43.990 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.628+0000 7fa4d7cd8700 -1 osd.7 115 *** Got signal Terminated *** 2024-03-12T05:29:43.990 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:43.628+0000 7fa4d7cd8700 -1 osd.7 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:44.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:43.688+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:44.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:43.688+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:44.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:43.688+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:44.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:43 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:43.688+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:44.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:43.834+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:44.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:43.834+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:44.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:43 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:43.834+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:44.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:44.257+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:44.257+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:44.257+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:44.127+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:44.127+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:44.127+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:44.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:44 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:44.127+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:44.861 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:44.579+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:44.862 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:44.579+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:44.862 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:44.579+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:45.016 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:44 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:44.675+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:45.016 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:44 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:44.675+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:45.016 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:44 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:44.675+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:45.016 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:44 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:44.675+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:45.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:44.867+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:45.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:44.867+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:45.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:44 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:44.867+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:45.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:45.207+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:45.207+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:45.207+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:45.110+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:45.110+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:45.110+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:45.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:45 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:45.110+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:45.875 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:45.564+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:45.876 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:45.564+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:45.876 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:45.564+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:45.876 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:45.564+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:46.032 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:45.669+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:46.032 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:45.670+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:46.032 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:45.670+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:46.032 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:45.670+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:46.032 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:45 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:45.670+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:46.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:45.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:46.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:45.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:46.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:45 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:45.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:46.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:46.196+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:46.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:46.196+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:46.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:46.196+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:46.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:46.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:46.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:46.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:46.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:46.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:46.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:46 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:46.071+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:46.900 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:46.606+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:46.901 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:46.606+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:46.901 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:46.606+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:46.901 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:46.606+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:47.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:46.687+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:47.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:46.687+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:47.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:46.687+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:47.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:46.687+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:47.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:46 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:46.687+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:47.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:46.906+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:47.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:46.906+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:47.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:46 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:46.906+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:47.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:47.154+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:47.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:47.154+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:47.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:47.154+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:47.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:47.070+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:47.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:47.070+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:47.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:47.070+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:47.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:47.070+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:47.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:47.864+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:47.864+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:47.864+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:47.864+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:47.649+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:47.649+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:47.649+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:47.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:47 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:47.649+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:48.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:47.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:48.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:47.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:48.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:47.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:48.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:47.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:48.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:47 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:47.681+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:48.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:48.146+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:53.970755+0000 front 2024-03-12T05:28:53.970836+0000 (oldest deadline 2024-03-12T05:29:19.870866+0000) 2024-03-12T05:29:48.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:48.146+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.073203+0000 front 2024-03-12T05:29:06.073266+0000 (oldest deadline 2024-03-12T05:29:30.773237+0000) 2024-03-12T05:29:48.490 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7[53797]: 2024-03-12T05:29:48.146+0000 7fa4d42cc700 -1 osd.7 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:13.574978+0000 front 2024-03-12T05:29:13.575166+0000 (oldest deadline 2024-03-12T05:29:39.474868+0000) 2024-03-12T05:29:48.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:48.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:48.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:48.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:48.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:48.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:48.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:48 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:48.077+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:48.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:48.616+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:48.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:48.616+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:48.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:48.616+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:48.872 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:48.616+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:49.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:48.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:49.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:48.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:49.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:48.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:49.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:48.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:49.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:48 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:48.696+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:49.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:48.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:49.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:48.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:49.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:48.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:49.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:48 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:48.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:49.465 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:49 smithi116 podman[73632]: 2024-03-12 05:29:49.165372764 +0000 UTC m=+5.656723631 container died 5e6258d98f321fbc9531929821459b45ba2658d765772e3a60d8baad0bdd2848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:49.466 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:49.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:49.466 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:49.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:49.466 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:49.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:49.466 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:49.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:49.466 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:49 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:49.085+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:49.733 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:49 smithi116 podman[73632]: 2024-03-12 05:29:49.554866258 +0000 UTC m=+6.046217121 container cleanup 5e6258d98f321fbc9531929821459b45ba2658d765772e3a60d8baad0bdd2848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-12T05:29:49.733 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:49 smithi116 bash[73632]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7 2024-03-12T05:29:49.863 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:49.633+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:49.863 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:49.633+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:49.864 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:49.633+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:49.864 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:49.633+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:49.990 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:49 smithi116 podman[73652]: 2024-03-12 05:29:49.740232464 +0000 UTC m=+0.577102381 container remove 5e6258d98f321fbc9531929821459b45ba2658d765772e3a60d8baad0bdd2848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T05:29:50.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:49.663+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:50.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:49.663+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:50.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:49.663+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:50.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:49.663+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:50.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:49 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:49.663+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:50.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:49.869+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:50.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:49.869+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:50.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:49.869+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:50.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:49 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:49.869+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:50.378 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:50.132+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:50.132+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:50.132+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:50.132+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:50 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:50.132+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:49.959909237 +0000 UTC m=+0.018017220 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:50.074451791 +0000 UTC m=+0.132559759 container create f2e8401ee55249c339569d85eb46b4d7b2b68b79b7d003d8672c896f18e1e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:50.294304365 +0000 UTC m=+0.352412343 container init f2e8401ee55249c339569d85eb46b4d7b2b68b79b7d003d8672c896f18e1e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:50.297652556 +0000 UTC m=+0.355760525 container start f2e8401ee55249c339569d85eb46b4d7b2b68b79b7d003d8672c896f18e1e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:50.379 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:50.385513356 +0000 UTC m=+0.443621338 container attach f2e8401ee55249c339569d85eb46b4d7b2b68b79b7d003d8672c896f18e1e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-12T05:29:50.740 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 12 05:29:50 smithi116 podman[73743]: 2024-03-12 05:29:50.588588311 +0000 UTC m=+0.646696271 container died f2e8401ee55249c339569d85eb46b4d7b2b68b79b7d003d8672c896f18e1e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:29:50.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:50.832+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:50.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:50.832+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:50.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:50.832+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:50.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:50.832+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:50.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:50.586+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:50.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:50.586+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:50.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:50.586+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:50.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:50 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:50.586+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:51.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:50.648+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:51.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:50.648+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:51.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:50.648+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:51.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:50.648+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:51.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:50 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:50.648+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:51.241 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:51.158+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:51.241 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:51.158+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:51.241 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:51.158+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:51.241 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:51.158+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:51.241 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:51.158+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:51.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.7.service' 2024-03-12T05:29:51.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:51.373 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-12T05:29:51.373 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-12T05:29:51.374 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.8 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 systemd[1]: Stopping Ceph osd.8 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.549+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.549+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.549+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.549+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.690+0000 7f479e34e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.690+0000 7f479e34e700 -1 osd.8 115 *** Got signal Terminated *** 2024-03-12T05:29:51.685 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:51.690+0000 7f479e34e700 -1 osd.8 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:51.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:51.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:51.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:51.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:51.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:51.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:51.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:51 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:51.878+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:52.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:51.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:52.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:51.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:52.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:51.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:52.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:51.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:52.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:51 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:51.643+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:52.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:52.204+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:52.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:52.204+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:52.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:52.204+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:52.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:52.204+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:52.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:52 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:52.204+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:52.831 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:52.574+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:52.831 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:52.574+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:52.831 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:52.574+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:52.831 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:52.574+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:53.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:52.655+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:53.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:52.655+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:53.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:52.655+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:53.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:52.655+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:53.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:52 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:52.655+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:53.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:52.837+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:53.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:52.837+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:53.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:52.837+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:53.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:52 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:52.837+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:53.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:53.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:53.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:53.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:53.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:53.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:53.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:53.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:53.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:53 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:53.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:53.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:53 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:53.605+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:53.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:53 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:53.605+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:53.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:53 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:53.605+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:53.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:53 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:53.605+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:53.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:53 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:53.605+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:53.965 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:53.618+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:53.965 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:53.618+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:53.618+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:53.618+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:53.833+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:53.833+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:53.833+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:53.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:53 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:53.833+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:54.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:54.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:54.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:54.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:54.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:54.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:54 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:54.144+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:54.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:54 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:54.580+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:54.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:54 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:54.580+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:54.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:54 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:54.580+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:54.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:54 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:54.580+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:54.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:54 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:54.580+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:54.965 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:54.659+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:54.659+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:54.659+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:54.659+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:54.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:54.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:54.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:54.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:54 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:54.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:55.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:55.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:55.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:55.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:55.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:55.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:55 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:55.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:55.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:55.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:55.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:55.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:55.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:55.865 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:55 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:55.606+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:29:55.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:55.654+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:55.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:55.654+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:55.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:55.654+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:55.967 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:55.654+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:55.654+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.122432+0000 front 2024-03-12T05:29:30.122455+0000 (oldest deadline 2024-03-12T05:29:55.422552+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:55.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:55.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:55.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:55.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:55.968 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:55.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:29:56.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:56.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:56.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:56.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:56.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:56.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:56 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:56.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:56.679+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:58.117187+0000 front 2024-03-12T05:28:58.117371+0000 (oldest deadline 2024-03-12T05:29:19.817309+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:56.679+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.119313+0000 front 2024-03-12T05:29:06.119488+0000 (oldest deadline 2024-03-12T05:29:27.819445+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:56.679+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:10.720327+0000 front 2024-03-12T05:29:10.720256+0000 (oldest deadline 2024-03-12T05:29:40.720805+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:56.679+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.721308+0000 front 2024-03-12T05:29:20.721395+0000 (oldest deadline 2024-03-12T05:29:45.421238+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8[54063]: 2024-03-12T05:29:56.679+0000 7f479a942700 -1 osd.8 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.122432+0000 front 2024-03-12T05:29:30.122455+0000 (oldest deadline 2024-03-12T05:29:55.422552+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:56.885+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:56.885+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:56.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:56.885+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:56.967 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:56.885+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:56.967 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:56 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:56.885+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:57.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:56 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:56.644+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:29:57.457 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:57 smithi130 podman[73314]: 2024-03-12 05:29:57.186894838 +0000 UTC m=+5.606202791 container died 016d10eea39a8137d186f06edb8b30871ed097baee76131714a790124687ff83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-12T05:29:57.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:57.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:57.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:57.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:57.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:57.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:57 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:57.160+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:57.716 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:57 smithi130 podman[73314]: 2024-03-12 05:29:57.549050898 +0000 UTC m=+5.968358855 container cleanup 016d10eea39a8137d186f06edb8b30871ed097baee76131714a790124687ff83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T05:29:57.716 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:57 smithi130 bash[73314]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8 2024-03-12T05:29:58.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:58.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:58.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:58.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:58.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:58.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:57 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:57.684+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:29:58.206 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:57 smithi130 podman[73331]: 2024-03-12 05:29:57.858585805 +0000 UTC m=+0.669852601 container remove 016d10eea39a8137d186f06edb8b30871ed097baee76131714a790124687ff83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T05:29:58.207 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:57.888+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:58.207 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:57.888+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:58.207 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:57.888+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:58.207 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:57.888+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:58.207 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:57 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:57.888+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:29:58.465 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.11685562 +0000 UTC m=+0.018266128 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:29:58.466 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.217702386 +0000 UTC m=+0.119112893 container create 15efb6465a74efee7768f129c9a034e0334a8a888c4651fe89c0f6d7b917913c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T05:29:58.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:58.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:58.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:58.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:58.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:58.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:58 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:58.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:58.764 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.479055827 +0000 UTC m=+0.380466337 container init 15efb6465a74efee7768f129c9a034e0334a8a888c4651fe89c0f6d7b917913c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T05:29:58.764 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.482361149 +0000 UTC m=+0.383771658 container start 15efb6465a74efee7768f129c9a034e0334a8a888c4651fe89c0f6d7b917913c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:29:58.764 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.562784427 +0000 UTC m=+0.464194938 container attach 15efb6465a74efee7768f129c9a034e0334a8a888c4651fe89c0f6d7b917913c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8-deactivate, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-12T05:29:59.092 INFO:journalctl@ceph.osd.8.smithi130.stdout:Mar 12 05:29:58 smithi130 podman[73427]: 2024-03-12 05:29:58.771052968 +0000 UTC m=+0.672463474 container died 15efb6465a74efee7768f129c9a034e0334a8a888c4651fe89c0f6d7b917913c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T05:29:59.092 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:58.879+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:59.092 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:58.879+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:59.092 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:58.879+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:59.092 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:58.879+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:59.093 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:58 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:58.879+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:59.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:58 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:58.715+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:29:59.447 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.8.service' 2024-03-12T05:29:59.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:29:59.491 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-12T05:29:59.491 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-12T05:29:59.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.9 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:29:59.492 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:29:59 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:29:59.168+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:29:59.788 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 systemd[1]: Stopping Ceph osd.9 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.697+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.794+0000 7f3e7f98d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.794+0000 7f3e7f98d700 -1 osd.9 115 *** Got signal Terminated *** 2024-03-12T05:29:59.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:29:59 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:29:59.794+0000 7f3e7f98d700 -1 osd.9 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:29:59.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:59.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:29:59.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:59.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:29:59.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:59.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:29:59.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:59.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:29:59.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:29:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:29:59.884+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:00.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:00 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:00.218+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:00.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:00 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:00.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:00.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:00 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:00.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:00.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:00 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:00.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:00.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:00 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:00.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:00.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:00 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:00.849+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:01.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:30:01.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:30:01.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:30:01.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:30:01.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:30:01.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:00 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:00.724+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:30:01.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:01.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:01.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:01.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:01.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:01.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:01 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:01.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:02.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:30:02.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:01 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:01.749+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:37.745372+0000 front 2024-03-12T05:29:37.745322+0000 (oldest deadline 2024-03-12T05:30:01.245354+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:02.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:01 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:01.814+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:02.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:02.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:02.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:02.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:02.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:02.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:02 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:02.164+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:03.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:30:03.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:30:03.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:30:03.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:30:03.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:30:03.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:30:03.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:02 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:02.789+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:37.745372+0000 front 2024-03-12T05:29:37.745322+0000 (oldest deadline 2024-03-12T05:30:01.245354+0000) 2024-03-12T05:30:03.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:03.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:03.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:03.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:03.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:03.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:02 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:02.842+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:03.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:03.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:03 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:03.195+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:04.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:49.437329+0000 front 2024-03-12T05:28:49.437278+0000 (oldest deadline 2024-03-12T05:29:12.337392+0000) 2024-03-12T05:30:04.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.838649+0000 front 2024-03-12T05:28:55.838565+0000 (oldest deadline 2024-03-12T05:29:20.538477+0000) 2024-03-12T05:30:04.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:06.340079+0000 front 2024-03-12T05:29:06.340101+0000 (oldest deadline 2024-03-12T05:29:28.640089+0000) 2024-03-12T05:30:04.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:14.942219+0000 front 2024-03-12T05:29:14.942327+0000 (oldest deadline 2024-03-12T05:29:40.842379+0000) 2024-03-12T05:30:04.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:20.843063+0000 front 2024-03-12T05:29:20.843074+0000 (oldest deadline 2024-03-12T05:29:45.542846+0000) 2024-03-12T05:30:04.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:30.844312+0000 front 2024-03-12T05:29:30.844222+0000 (oldest deadline 2024-03-12T05:29:54.844503+0000) 2024-03-12T05:30:04.115 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:03 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9[67792]: 2024-03-12T05:30:03.773+0000 7f3e77f79700 -1 osd.9 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:37.745372+0000 front 2024-03-12T05:29:37.745322+0000 (oldest deadline 2024-03-12T05:30:01.245354+0000) 2024-03-12T05:30:04.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:04.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:04.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:04.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:04.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:04.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:03.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:04.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:04.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:04.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:04.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:04.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:04.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:04.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:04 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:04.149+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:05.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:05.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:05.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:05.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:05.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:05.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:04 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:04.830+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:05.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:05.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:05.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:05.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:05.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:05.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:05.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:05 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:05.170+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:05.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:05 smithi045 podman[123120]: 2024-03-12 05:30:05.489995391 +0000 UTC m=+5.795846714 container died 0b17c685f6c4229ec052fb32fed5bead871be3399c803ad8e4f6f6a54cb0143d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:30:05.789 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:05 smithi045 podman[123120]: 2024-03-12 05:30:05.796272906 +0000 UTC m=+6.102124228 container cleanup 0b17c685f6c4229ec052fb32fed5bead871be3399c803ad8e4f6f6a54cb0143d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T05:30:06.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:05 smithi045 bash[123120]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9 2024-03-12T05:30:06.114 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:05 smithi045 podman[123136]: 2024-03-12 05:30:05.964209542 +0000 UTC m=+0.472591963 container remove 0b17c685f6c4229ec052fb32fed5bead871be3399c803ad8e4f6f6a54cb0143d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-12T05:30:06.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:06.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:06.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:06.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:06.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:06.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:05 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:05.820+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:06.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:06.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:06.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:06.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:06.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:06.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:06.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:06 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:06.206+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:06.560 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.189087529 +0000 UTC m=+0.017635691 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:30:06.560 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.289863574 +0000 UTC m=+0.118411717 container create 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T05:30:06.560 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.492351378 +0000 UTC m=+0.320899525 container init 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-12T05:30:06.561 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.495346553 +0000 UTC m=+0.323894706 container start 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T05:30:06.561 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.567207288 +0000 UTC m=+0.395755448 container attach 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:30:06.864 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:06 smithi045 podman[123228]: 2024-03-12 05:30:06.795301139 +0000 UTC m=+0.623849293 container died 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:07.117 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:06 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:06.848+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:07.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.9.service' 2024-03-12T05:30:07.336 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:07 smithi045 podman[123254]: 2024-03-12 05:30:07.295498781 +0000 UTC m=+0.495005262 container remove 4ffba4b4c7d6160fd15a052c4a8eb1ae07742d5b91946effb5cca2787bfcd83a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:30:07.336 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:07 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.9.service: Deactivated successfully. 2024-03-12T05:30:07.336 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:07 smithi045 systemd[1]: Stopped Ceph osd.9 for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:30:07.336 INFO:journalctl@ceph.osd.9.smithi045.stdout:Mar 12 05:30:07 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.9.service: Consumed 7min 649ms CPU time. 2024-03-12T05:30:07.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:07.353 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-12T05:30:07.353 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-12T05:30:07.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.10 2024-03-12T05:30:07.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.199+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:07.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 systemd[1]: Stopping Ceph osd.10 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:07.990 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.658+0000 7f2e1ecc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:07.990 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.658+0000 7f2e1ecc7700 -1 osd.10 115 *** Got signal Terminated *** 2024-03-12T05:30:07.991 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:07 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:07.658+0000 7f2e1ecc7700 -1 osd.10 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:08.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:07.880+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:08.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:08.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:08 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:08.162+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:09.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:09.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:09.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:09.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:09.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:09.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:08 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:08.883+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:09.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:09.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:09 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:09.122+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:10.190 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:09 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:09.919+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:10.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:10.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:10.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:10.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:10.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:10.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:10.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:10 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:10.075+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:10.965 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:10.966 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:10 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:10.899+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:11.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:11.490 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:11.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:11.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:11.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:11.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:11.491 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:11 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:11.044+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:12.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:12.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:11 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:11.926+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:12.392 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.026+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:48.834601+0000 front 2024-03-12T05:28:48.834892+0000 (oldest deadline 2024-03-12T05:29:12.334578+0000) 2024-03-12T05:30:12.443 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:56.435355+0000 front 2024-03-12T05:28:56.435382+0000 (oldest deadline 2024-03-12T05:29:19.935474+0000) 2024-03-12T05:30:12.443 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:03.837431+0000 front 2024-03-12T05:29:03.837637+0000 (oldest deadline 2024-03-12T05:29:32.037867+0000) 2024-03-12T05:30:12.443 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6814 osd.3 since back 2024-03-12T05:29:12.038302+0000 front 2024-03-12T05:29:12.038381+0000 (oldest deadline 2024-03-12T05:29:36.638735+0000) 2024-03-12T05:30:12.444 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:24.040973+0000 front 2024-03-12T05:29:24.041012+0000 (oldest deadline 2024-03-12T05:29:48.741071+0000) 2024-03-12T05:30:12.444 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:28.741886+0000 front 2024-03-12T05:29:28.741817+0000 (oldest deadline 2024-03-12T05:29:53.942097+0000) 2024-03-12T05:30:12.444 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:12 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10[60827]: 2024-03-12T05:30:12.027+0000 7f2e1b2bb700 -1 osd.10 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.243054+0000 front 2024-03-12T05:29:39.243170+0000 (oldest deadline 2024-03-12T05:30:02.743056+0000) 2024-03-12T05:30:13.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:13.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:12 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:12.881+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:13.461 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:13 smithi116 podman[73851]: 2024-03-12 05:30:13.203705508 +0000 UTC m=+5.656952968 container died 71a911bf747b650844558451430fb1e89dcf512efda1a266607e1103540b9f83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-12T05:30:13.740 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:13 smithi116 podman[73851]: 2024-03-12 05:30:13.566935845 +0000 UTC m=+6.020183312 container cleanup 71a911bf747b650844558451430fb1e89dcf512efda1a266607e1103540b9f83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-12T05:30:13.740 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:13 smithi116 bash[73851]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10 2024-03-12T05:30:14.204 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:13 smithi116 podman[73879]: 2024-03-12 05:30:13.910860645 +0000 UTC m=+0.705410540 container remove 71a911bf747b650844558451430fb1e89dcf512efda1a266607e1103540b9f83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-12T05:30:14.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:14.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:14.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:14.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:14.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:14.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:14.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:13 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:13.843+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:14.481 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.211154897 +0000 UTC m=+0.104893767 container create 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:30:14.481 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.124792152 +0000 UTC m=+0.018531022 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:30:14.482 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.413639307 +0000 UTC m=+0.307378190 container init 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-12T05:30:14.482 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.416705356 +0000 UTC m=+0.310444229 container start 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-12T05:30:14.482 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.488326201 +0000 UTC m=+0.382065091 container attach 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-12T05:30:14.740 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:14 smithi116 podman[73971]: 2024-03-12 05:30:14.702033528 +0000 UTC m=+0.595772393 container died 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-12T05:30:15.202 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.10.service' 2024-03-12T05:30:15.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:15.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:15.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:15.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:15.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:15.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:15.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:14 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:14.803+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:15.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:15 smithi116 podman[73997]: 2024-03-12 05:30:15.191601036 +0000 UTC m=+0.479401902 container remove 6050a589f5190702b327dc0fbc197f0ed11402dfebf54588c76e3b01c6805d58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T05:30:15.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:15 smithi116 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-12T05:30:15.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:15 smithi116 systemd[1]: Stopped Ceph osd.10 for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:30:15.240 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 12 05:30:15 smithi116 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.10.service: Consumed 5min 55.980s CPU time. 2024-03-12T05:30:15.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:15.243 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-12T05:30:15.243 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-12T05:30:15.243 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.11 2024-03-12T05:30:15.537 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 systemd[1]: Stopping Ceph osd.11 for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.543+0000 7fe728d18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.543+0000 7fe728d18700 -1 osd.11 115 *** Got signal Terminated *** 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.543+0000 7fe728d18700 -1 osd.11 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:15.819 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:16.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:16.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:16.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:15 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:15.825+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:17.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:17.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:16 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:16.846+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:18.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:18.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:17 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:17.826+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:19.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:18 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:18.810+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:20.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-12T05:28:46.567854+0000 front 2024-03-12T05:28:46.567875+0000 (oldest deadline 2024-03-12T05:29:11.267936+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-12T05:28:55.369129+0000 front 2024-03-12T05:28:55.369052+0000 (oldest deadline 2024-03-12T05:29:21.269242+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-03-12T05:29:01.269753+0000 front 2024-03-12T05:29:01.269808+0000 (oldest deadline 2024-03-12T05:29:28.870283+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-12T05:29:22.273121+0000 front 2024-03-12T05:29:22.273075+0000 (oldest deadline 2024-03-12T05:29:47.573209+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-03-12T05:29:31.074237+0000 front 2024-03-12T05:29:31.074265+0000 (oldest deadline 2024-03-12T05:29:55.174386+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.45:6822 osd.6 since back 2024-03-12T05:29:39.875401+0000 front 2024-03-12T05:29:39.875341+0000 (oldest deadline 2024-03-12T05:30:01.476085+0000) 2024-03-12T05:30:20.216 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:19 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11[60664]: 2024-03-12T05:30:19.827+0000 7fe72530c700 -1 osd.11 115 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-12T05:29:47.778109+0000 front 2024-03-12T05:29:47.778164+0000 (oldest deadline 2024-03-12T05:30:10.078065+0000) 2024-03-12T05:30:21.465 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:21 smithi130 podman[73547]: 2024-03-12 05:30:21.219078372 +0000 UTC m=+5.772517827 container died 933ff8d8c4466026ba21bf8c565ab3b36d095253c43b8d7ae4ef5360c563bfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T05:30:21.729 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:21 smithi130 podman[73547]: 2024-03-12 05:30:21.551553251 +0000 UTC m=+6.104992679 container cleanup 933ff8d8c4466026ba21bf8c565ab3b36d095253c43b8d7ae4ef5360c563bfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T05:30:21.729 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:21 smithi130 bash[73547]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11 2024-03-12T05:30:22.152 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:21 smithi130 podman[73564]: 2024-03-12 05:30:21.735878474 +0000 UTC m=+0.516321751 container remove 933ff8d8c4466026ba21bf8c565ab3b36d095253c43b8d7ae4ef5360c563bfe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T05:30:22.457 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.062536237 +0000 UTC m=+0.017995602 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:30:22.457 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.161600983 +0000 UTC m=+0.117060386 container create d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-12T05:30:22.457 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.388944019 +0000 UTC m=+0.344403409 container init d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T05:30:22.457 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.392084878 +0000 UTC m=+0.347544246 container start d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T05:30:22.457 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.463916746 +0000 UTC m=+0.419376116 container attach d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T05:30:22.716 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:22 smithi130 podman[73655]: 2024-03-12 05:30:22.68633004 +0000 UTC m=+0.641789403 container died d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T05:30:23.207 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.11.service' 2024-03-12T05:30:23.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:23 smithi130 podman[73680]: 2024-03-12 05:30:23.19228575 +0000 UTC m=+0.501111822 container remove d92d0157358704d0fb6381f688e0ba8a4b10272644de5712617bc954182617a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-osd-11-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-12T05:30:23.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:23 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-12T05:30:23.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:23 smithi130 systemd[1]: Stopped Ceph osd.11 for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:30:23.215 INFO:journalctl@ceph.osd.11.smithi130.stdout:Mar 12 05:30:23 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@osd.11.service: Consumed 11min 7.268s CPU time. 2024-03-12T05:30:23.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:23.248 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-12T05:30:23.249 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-12T05:30:23.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.a 2024-03-12T05:30:23.541 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:23 smithi045 systemd[1]: Stopping Ceph mds.a for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:23.864 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: 2024-03-12T05:30:23.545+0000 7ff473664700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:23.864 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:23 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a[74627]: 2024-03-12T05:30:23.546+0000 7ff473664700 -1 mds.a *** got signal Terminated *** 2024-03-12T05:30:26.758 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:26 smithi045 podman[123347]: 2024-03-12 05:30:26.382011398 +0000 UTC m=+2.924309835 container died 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-12T05:30:27.114 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:26 smithi045 podman[123347]: 2024-03-12 05:30:26.863637423 +0000 UTC m=+3.405935853 container cleanup 2fc266cb6911ec8639e979a4980a04fc54dd6ad09df32271f42eacd7caeffda5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T05:30:27.114 INFO:journalctl@ceph.mds.a.smithi045.stdout:Mar 12 05:30:26 smithi045 bash[123347]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-a 2024-03-12T05:30:27.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.a.service' 2024-03-12T05:30:27.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:27.301 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-12T05:30:27.301 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-12T05:30:27.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.d 2024-03-12T05:30:27.614 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:27 smithi045 systemd[1]: Stopping Ceph mds.d for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:28.115 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d[73672]: 2024-03-12T05:30:27.658+0000 7f54e88b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:28.115 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:27 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d[73672]: 2024-03-12T05:30:27.658+0000 7f54e88b8700 -1 mds.d *** got signal Terminated *** 2024-03-12T05:30:30.635 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:30 smithi045 podman[123468]: 2024-03-12 05:30:30.376185009 +0000 UTC m=+2.848613480 container died 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-12T05:30:30.903 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:30 smithi045 podman[123468]: 2024-03-12 05:30:30.71739434 +0000 UTC m=+3.189822816 container cleanup 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T05:30:30.903 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:30 smithi045 bash[123468]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d 2024-03-12T05:30:31.120 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:30 smithi045 podman[123482]: 2024-03-12 05:30:30.910313575 +0000 UTC m=+0.526632098 container remove 7f8c61f1d379f7ac410dbd185e3ee72eec62cc97e1b343bea2f9dec0d3bdb043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T05:30:31.120 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:31 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.d.service: Deactivated successfully. 2024-03-12T05:30:31.121 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:31 smithi045 systemd[1]: Stopped Ceph mds.d for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:30:31.121 INFO:journalctl@ceph.mds.d.smithi045.stdout:Mar 12 05:30:31 smithi045 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.d.service: Consumed 1.153s CPU time. 2024-03-12T05:30:31.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.d.service' 2024-03-12T05:30:31.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:31.157 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-12T05:30:31.157 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-12T05:30:31.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.g 2024-03-12T05:30:31.489 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:31 smithi045 systemd[1]: Stopping Ceph mds.g for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:31.864 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: 2024-03-12T05:30:31.495+0000 7f0295fe9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:31.864 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:31 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g[74266]: 2024-03-12T05:30:31.495+0000 7f0295fe9700 -1 mds.g *** got signal Terminated *** 2024-03-12T05:30:34.673 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:34 smithi045 podman[123590]: 2024-03-12 05:30:34.388078206 +0000 UTC m=+3.005474998 container died 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-12T05:30:34.973 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:34 smithi045 podman[123590]: 2024-03-12 05:30:34.762641022 +0000 UTC m=+3.380037813 container cleanup 798b2fd4c2d8414ea3a38590c0494be46f2a0d110a98229f3cdf7c9e5a21b831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-12T05:30:34.973 INFO:journalctl@ceph.mds.g.smithi045.stdout:Mar 12 05:30:34 smithi045 bash[123590]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-g 2024-03-12T05:30:35.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.g.service' 2024-03-12T05:30:35.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:35.121 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-12T05:30:35.121 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-12T05:30:35.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.j 2024-03-12T05:30:35.364 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:35 smithi045 systemd[1]: Stopping Ceph mds.j for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:35.864 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j[73286]: 2024-03-12T05:30:35.474+0000 7f6c852b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:35.864 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:35 smithi045 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j[73286]: 2024-03-12T05:30:35.474+0000 7f6c852b5700 -1 mds.j *** got signal Terminated *** 2024-03-12T05:30:39.767 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:39 smithi045 podman[123709]: 2024-03-12 05:30:39.494857799 +0000 UTC m=+4.148210394 container died 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:30:39.768 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:39 smithi045 podman[123709]: 2024-03-12 05:30:39.774782925 +0000 UTC m=+4.428135482 container cleanup 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T05:30:40.114 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:39 smithi045 bash[123709]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j 2024-03-12T05:30:40.114 INFO:journalctl@ceph.mds.j.smithi045.stdout:Mar 12 05:30:40 smithi045 podman[123723]: 2024-03-12 05:30:40.009075241 +0000 UTC m=+0.506351058 container remove 19c6da4d4b8da2cf7480bcbb396a24f971c64f38170454975823b792a7cc4655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-j, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-12T05:30:40.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.j.service' 2024-03-12T05:30:40.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:40.174 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-12T05:30:40.174 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-12T05:30:40.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.b 2024-03-12T05:30:40.477 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:40 smithi116 systemd[1]: Stopping Ceph mds.b for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:40.740 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:30:40.483+0000 7fc0c863e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:40.740 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:40 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b[66040]: 2024-03-12T05:30:40.483+0000 7fc0c863e700 -1 mds.b *** got signal Terminated *** 2024-03-12T05:30:42.535 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:42 smithi116 podman[74078]: 2024-03-12 05:30:42.266796216 +0000 UTC m=+1.895103381 container died 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-12T05:30:42.869 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:42 smithi116 podman[74078]: 2024-03-12 05:30:42.642043393 +0000 UTC m=+2.270350567 container cleanup 1eb159864d3b4d849745b468b04c94df3307cc2c6d73c23502b3093e94a78c22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:30:42.869 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 12 05:30:42 smithi116 bash[74078]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-b 2024-03-12T05:30:42.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.b.service' 2024-03-12T05:30:43.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:43.019 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-12T05:30:43.020 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-12T05:30:43.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.e 2024-03-12T05:30:43.240 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:43 smithi116 systemd[1]: Stopping Ceph mds.e for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:43.740 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: 2024-03-12T05:30:43.336+0000 7f8537af3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:43.740 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:43 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e[65412]: 2024-03-12T05:30:43.336+0000 7f8537af3700 -1 mds.e *** got signal Terminated *** 2024-03-12T05:30:46.555 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:46 smithi116 podman[74199]: 2024-03-12 05:30:46.276172764 +0000 UTC m=+3.031789497 container died b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-12T05:30:46.555 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:46 smithi116 podman[74199]: 2024-03-12 05:30:46.562168352 +0000 UTC m=+3.317785086 container cleanup b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T05:30:46.894 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:46 smithi116 bash[74199]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e 2024-03-12T05:30:46.894 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 12 05:30:46 smithi116 podman[74225]: 2024-03-12 05:30:46.780160067 +0000 UTC m=+0.505767156 container remove b6df71652c2adadcf0c8579f4d9991532273d50355fd8bcd5ad6e1c5b3d567ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-12T05:30:46.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.e.service' 2024-03-12T05:30:46.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:46.947 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-12T05:30:46.948 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-12T05:30:46.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.h 2024-03-12T05:30:47.240 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:47 smithi116 systemd[1]: Stopping Ceph mds.h for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:47.740 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: 2024-03-12T05:30:47.269+0000 7f47b76a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:47.740 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:47 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h[65719]: 2024-03-12T05:30:47.269+0000 7f47b76a7700 -1 mds.h *** got signal Terminated *** 2024-03-12T05:30:50.542 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:50 smithi116 podman[74331]: 2024-03-12 05:30:50.262477347 +0000 UTC m=+3.093257630 container died 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:30:50.893 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:50 smithi116 podman[74331]: 2024-03-12 05:30:50.623986011 +0000 UTC m=+3.454766295 container cleanup 8f372197e694fb292b786ef1e6a84970b977ac46bb1803a70f5aaf999b3ba176 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T05:30:50.893 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 12 05:30:50 smithi116 bash[74331]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-h 2024-03-12T05:30:51.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.h.service' 2024-03-12T05:30:51.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:51.080 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-12T05:30:51.080 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-12T05:30:51.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.k 2024-03-12T05:30:51.405 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:51 smithi116 systemd[1]: Stopping Ceph mds.k for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:51.740 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: 2024-03-12T05:30:51.410+0000 7fc618747700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:51.740 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:51 smithi116 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k[65105]: 2024-03-12T05:30:51.410+0000 7fc618747700 -1 mds.k *** got signal Terminated *** 2024-03-12T05:30:54.629 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:54 smithi116 podman[74453]: 2024-03-12 05:30:54.279041024 +0000 UTC m=+2.967956642 container died 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-12T05:30:54.629 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:54 smithi116 podman[74453]: 2024-03-12 05:30:54.635956933 +0000 UTC m=+3.324872602 container cleanup 4681eced2e4fe436f23b68c67ec4651c4cdd4cf890f4f6e8dd30a0907218a831 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-12T05:30:54.905 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 12 05:30:54 smithi116 bash[74453]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-k 2024-03-12T05:30:55.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.k.service' 2024-03-12T05:30:55.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:55.039 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-12T05:30:55.039 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-12T05:30:55.040 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.c 2024-03-12T05:30:55.394 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:55 smithi130 systemd[1]: Stopping Ceph mds.c for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:55.677 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: 2024-03-12T05:30:55.400+0000 7f4057ff2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:55.677 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:55 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c[66071]: 2024-03-12T05:30:55.400+0000 7f4057ff2700 -1 mds.c *** got signal Terminated *** 2024-03-12T05:30:58.626 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:58 smithi130 podman[73761]: 2024-03-12 05:30:58.350993399 +0000 UTC m=+3.100887512 container died 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:30:58.965 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:58 smithi130 podman[73761]: 2024-03-12 05:30:58.707807841 +0000 UTC m=+3.457701955 container cleanup 0f87dde9cd09de950a5129bec98db9177743c3fee966bc199b5383d6ef58dfcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:30:58.966 INFO:journalctl@ceph.mds.c.smithi130.stdout:Mar 12 05:30:58 smithi130 bash[73761]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-c 2024-03-12T05:30:59.091 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.c.service' 2024-03-12T05:30:59.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:30:59.131 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-12T05:30:59.131 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-12T05:30:59.131 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.f 2024-03-12T05:30:59.446 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:30:59 smithi130 systemd[1]: Stopping Ceph mds.f for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:30:59.715 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:30:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: 2024-03-12T05:30:59.453+0000 7fed1ba1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:30:59.715 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:30:59 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f[65765]: 2024-03-12T05:30:59.453+0000 7fed1ba1b700 -1 mds.f *** got signal Terminated *** 2024-03-12T05:31:02.539 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:31:02 smithi130 podman[73893]: 2024-03-12 05:31:02.281416209 +0000 UTC m=+2.928742760 container died 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T05:31:02.814 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:31:02 smithi130 podman[73893]: 2024-03-12 05:31:02.620297267 +0000 UTC m=+3.267623816 container cleanup 7e342701e5970aff32226a3f2ffff9821b6508dda3b80457277e4103276cc7b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-12T05:31:02.814 INFO:journalctl@ceph.mds.f.smithi130.stdout:Mar 12 05:31:02 smithi130 bash[73893]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-f 2024-03-12T05:31:03.011 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.f.service' 2024-03-12T05:31:03.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:31:03.046 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-12T05:31:03.046 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-12T05:31:03.047 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.i 2024-03-12T05:31:03.367 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:03 smithi130 systemd[1]: Stopping Ceph mds.i for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:31:03.715 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i[66372]: 2024-03-12T05:31:03.373+0000 7efda7198700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:31:03.715 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:03 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i[66372]: 2024-03-12T05:31:03.373+0000 7efda7198700 -1 mds.i *** got signal Terminated *** 2024-03-12T05:31:06.643 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:06 smithi130 podman[74015]: 2024-03-12 05:31:06.290443995 +0000 UTC m=+3.019860101 container died 651a89d1fe4ae84870643e7a15dc2f3a6918ed8bfae6cd40296afb784fe96cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T05:31:06.909 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:06 smithi130 podman[74015]: 2024-03-12 05:31:06.72342044 +0000 UTC m=+3.452836546 container cleanup 651a89d1fe4ae84870643e7a15dc2f3a6918ed8bfae6cd40296afb784fe96cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:31:06.909 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:06 smithi130 bash[74015]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i 2024-03-12T05:31:07.137 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.i.service' 2024-03-12T05:31:07.161 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:06 smithi130 podman[74028]: 2024-03-12 05:31:06.916299503 +0000 UTC m=+0.619498994 container remove 651a89d1fe4ae84870643e7a15dc2f3a6918ed8bfae6cd40296afb784fe96cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-12T05:31:07.161 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:07 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.i.service: Deactivated successfully. 2024-03-12T05:31:07.161 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:07 smithi130 systemd[1]: Stopped Ceph mds.i for 52aed436-e02d-11ee-95c8-87774f69a715. 2024-03-12T05:31:07.161 INFO:journalctl@ceph.mds.i.smithi130.stdout:Mar 12 05:31:07 smithi130 systemd[1]: ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.i.service: Consumed 5.750s CPU time. 2024-03-12T05:31:07.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:31:07.172 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-12T05:31:07.172 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-12T05:31:07.172 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.l 2024-03-12T05:31:07.465 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:07 smithi130 systemd[1]: Stopping Ceph mds.l for 52aed436-e02d-11ee-95c8-87774f69a715... 2024-03-12T05:31:07.965 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: 2024-03-12T05:31:07.505+0000 7fb58c332700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:31:07.965 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:07 smithi130 ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l[65457]: 2024-03-12T05:31:07.505+0000 7fb58c332700 -1 mds.l *** got signal Terminated *** 2024-03-12T05:31:10.554 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:10 smithi130 podman[74138]: 2024-03-12 05:31:10.302549026 +0000 UTC m=+2.906433154 container died e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-12T05:31:10.821 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:10 smithi130 podman[74138]: 2024-03-12 05:31:10.626964989 +0000 UTC m=+3.230849117 container cleanup e87bf456e869247ad31c8d3c3d5f4e887fa1285bfc92880c22e33d75d588188e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T05:31:10.821 INFO:journalctl@ceph.mds.l.smithi130.stdout:Mar 12 05:31:10 smithi130 bash[74138]: ceph-52aed436-e02d-11ee-95c8-87774f69a715-mds-l 2024-03-12T05:31:11.039 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-52aed436-e02d-11ee-95c8-87774f69a715@mds.l.service' 2024-03-12T05:31:11.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T05:31:11.073 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-12T05:31:11.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T05:31:11.270 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:31:14.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T05:31:14.441 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:31:17.100 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T05:31:17.291 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:31:20.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:31:20.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:31:20.505 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:31:20.534 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T05:31:20.536 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi045/crash 2024-03-12T05:31:20.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash -- . 2024-03-12T05:31:20.581 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T05:31:20.581 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-03-12T05:31:20.583 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi116/crash 2024-03-12T05:31:20.584 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash -- . 2024-03-12T05:31:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T05:31:20.624 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-12T05:31:20.626 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi130/crash 2024-03-12T05:31:20.627 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash -- . 2024-03-12T05:31:20.676 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/52aed436-e02d-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T05:31:20.676 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-03-12T05:31:20.677 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T05:31:20.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-12T05:31:20.760 INFO:teuthology.orchestra.run.smithi045.stdout:2024-03-12T05:02:41.358973+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:31:20.760 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T05:31:20.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[SEC\]' /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-12T05:31:20.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]' /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-12T05:31:20.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[WRN\]' /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-12T05:31:20.941 INFO:teuthology.orchestra.run.smithi045.stdout:2024-03-12T05:02:41.358973+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:31:20.942 INFO:tasks.cephadm:Compressing logs... 2024-03-12T05:31:20.942 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T05:31:20.984 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T05:31:20.986 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T05:31:21.015 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T05:31:21.021 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T05:31:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T05:31:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T05:31:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.10.log 2024-03-12T05:31:21.030 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.g.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log: --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.j.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-12T05:31:21.036 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose --/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.3.log: /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.9.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.d.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi045.stderr:gzip/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.9.log: -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.a.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.b.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.k.log 2024-03-12T05:31:21.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.h.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.e.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T05:31:21.038 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T05:31:21.040 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T05:31:21.045 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-03-12T05:31:21.045 INFO:teuthology.orchestra.run.smithi130.stderr: -- /var/log/ceph/cephadm.log 2024-03-12T05:31:21.050 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.l.log 2024-03-12T05:31:21.050 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T05:31:21.050 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.i.log 2024-03-12T05:31:21.053 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.11.log 2024-03-12T05:31:21.053 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-12T05:31:21.053 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.8.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.f.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.c.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T05:31:21.054 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T05:31:21.087 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.h.log.gz 2024-03-12T05:31:21.091 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.d.log: /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.a.log: 87.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T05:31:21.100 INFO:teuthology.orchestra.run.smithi116.stderr: 83.5% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T05:31:21.138 INFO:teuthology.orchestra.run.smithi116.stderr: 87.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T05:31:21.160 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.c.log: 83.5% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T05:31:21.174 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T05:31:21.190 INFO:teuthology.orchestra.run.smithi116.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T05:31:21.216 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.d.log.gz 2024-03-12T05:31:21.220 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.j.log.gz 2024-03-12T05:31:21.289 INFO:teuthology.orchestra.run.smithi130.stderr: 91.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T05:31:21.294 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T05:31:21.340 INFO:teuthology.orchestra.run.smithi045.stderr: 84.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T05:31:21.368 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T05:31:21.498 INFO:teuthology.orchestra.run.smithi045.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T05:31:21.534 INFO:teuthology.orchestra.run.smithi116.stderr: 93.4% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.e.log.gz 2024-03-12T05:31:21.553 INFO:teuthology.orchestra.run.smithi130.stderr: 92.4% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.f.log.gz 2024-03-12T05:31:21.602 INFO:teuthology.orchestra.run.smithi130.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T05:31:21.602 INFO:teuthology.orchestra.run.smithi130.stderr: 90.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T05:31:21.602 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T05:31:21.607 INFO:teuthology.orchestra.run.smithi130.stderr: 92.3% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-12T05:31:21.779 INFO:teuthology.orchestra.run.smithi045.stderr: 90.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T05:31:21.950 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.b.log.gz 2024-03-12T05:31:21.975 INFO:teuthology.orchestra.run.smithi116.stderr: 92.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T05:31:23.533 INFO:teuthology.orchestra.run.smithi116.stderr: 95.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.k.log.gz 2024-03-12T05:31:24.031 INFO:teuthology.orchestra.run.smithi045.stderr: 94.4% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.g.log.gz 2024-03-12T05:31:24.142 INFO:teuthology.orchestra.run.smithi045.stderr: 94.6% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.a.log.gz 2024-03-12T05:31:24.793 INFO:teuthology.orchestra.run.smithi130.stderr: 95.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.l.log.gz 2024-03-12T05:31:25.478 INFO:teuthology.orchestra.run.smithi130.stderr: 92.1% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T05:31:30.704 INFO:teuthology.orchestra.run.smithi045.stderr: 89.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T05:31:31.918 INFO:teuthology.orchestra.run.smithi130.stderr: 94.8% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.i.log.gz 2024-03-12T05:31:49.337 INFO:teuthology.orchestra.run.smithi045.stderr: 90.3% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T05:32:20.622 INFO:teuthology.orchestra.run.smithi116.stderr: 90.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-12T05:32:20.960 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.10.log.gz 2024-03-12T05:32:23.835 INFO:teuthology.orchestra.run.smithi130.stderr: 91.2% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-12T05:32:27.906 INFO:teuthology.orchestra.run.smithi116.stderr: 91.3% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-12T05:32:35.961 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T05:32:35.963 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-12T05:32:35.963 INFO:teuthology.orchestra.run.smithi116.stderr:real 1m14.962s 2024-03-12T05:32:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:user 3m36.340s 2024-03-12T05:32:35.964 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m7.001s 2024-03-12T05:33:02.352 INFO:teuthology.orchestra.run.smithi130.stderr: 94.3% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-mds.c.log.gz 2024-03-12T05:33:15.258 INFO:teuthology.orchestra.run.smithi130.stderr: 90.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.11.log.gz 2024-03-12T05:34:05.747 INFO:teuthology.orchestra.run.smithi130.stderr: 91.1% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.8.log.gz 2024-03-12T05:34:41.477 INFO:teuthology.orchestra.run.smithi130.stderr: 91.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T05:34:41.479 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-12T05:34:41.479 INFO:teuthology.orchestra.run.smithi130.stderr:real 3m20.478s 2024-03-12T05:34:41.479 INFO:teuthology.orchestra.run.smithi130.stderr:user 4m33.458s 2024-03-12T05:34:41.479 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m9.976s 2024-03-12T05:41:26.309 INFO:teuthology.orchestra.run.smithi045.stderr: 91.0% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.9.log.gz 2024-03-12T05:43:32.799 INFO:teuthology.orchestra.run.smithi045.stderr: 90.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-12T05:45:28.141 INFO:teuthology.orchestra.run.smithi045.stderr: 90.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T05:48:05.227 INFO:teuthology.orchestra.run.smithi045.stderr: 90.9% -- replaced with /var/log/ceph/52aed436-e02d-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-12T05:49:14.592 INFO:teuthology.orchestra.run.smithi045.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-12T05:49:14.605 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-12T05:49:14.605 INFO:teuthology.orchestra.run.smithi045.stderr:real 17m53.601s 2024-03-12T05:49:14.605 INFO:teuthology.orchestra.run.smithi045.stderr:user 6m56.440s 2024-03-12T05:49:14.605 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m18.550s 2024-03-12T05:49:14.606 INFO:tasks.cephadm:Archiving logs... 2024-03-12T05:49:14.607 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi045/log 2024-03-12T05:49:14.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T05:50:19.851 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi116/log 2024-03-12T05:50:19.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T05:50:59.880 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi130/log 2024-03-12T05:50:59.881 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T05:51:50.625 INFO:tasks.cephadm:Removing cluster... 2024-03-12T05:51:50.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force 2024-03-12T05:51:50.869 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:51:51.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force 2024-03-12T05:51:51.707 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:51:52.215 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 52aed436-e02d-11ee-95c8-87774f69a715 --force 2024-03-12T05:51:52.407 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 52aed436-e02d-11ee-95c8-87774f69a715 2024-03-12T05:51:53.053 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T05:51:53.053 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T05:51:53.072 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T05:51:53.089 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T05:51:53.111 INFO:tasks.cephadm:Teardown complete 2024-03-12T05:51:53.111 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-12T05:51:53.129 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-12T05:51:53.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T05:51:53.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T05:51:53.134 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T05:51:53.258 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T05:51:53.272 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T05:51:53.273 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:51:53.276 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:51:53.278 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:51:53.290 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:51:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:51:53.292 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +1869us[+1869us] +/- 105ms 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -1548us[-1554us] +/- 67ms 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +521us[ +521us] +/- 77ms 2024-03-12T05:51:53.316 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:51:53.319 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:51:53.319 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-12T05:51:53.320 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1877us[+1866us] +/- 105ms 2024-03-12T05:51:53.320 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1542us[-1554us] +/- 67ms 2024-03-12T05:51:53.320 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +517us[ +505us] +/- 77ms 2024-03-12T05:51:53.320 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +1903us[+1903us] +/- 105ms 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1547us[-1548us] +/- 67ms 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +536us[ +536us] +/- 77ms 2024-03-12T05:51:53.362 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:51:53.363 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T05:51:53.376 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T05:51:53.376 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T05:51:53.392 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T05:51:53.604 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T05:51:53.723 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T05:51:53.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:51:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:53.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:51:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:53.961 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:51:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:53.992 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T05:51:54.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T05:51:54.021 INFO:teuthology.task.internal:Duration was 3620.218860 seconds 2024-03-12T05:51:54.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T05:51:54.036 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T05:51:54.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T05:51:54.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T05:51:54.042 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T05:51:54.110 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:51:54.119 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:51:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:51:54.536 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T05:51:54.536 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-03-12T05:51:54.536 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T05:51:54.560 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-12T05:51:54.560 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T05:51:54.593 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-03-12T05:51:54.593 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T05:51:54.614 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T05:51:54.614 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T05:51:54.617 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T05:51:54.636 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T05:51:54.682 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T05:51:54.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T05:51:54.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T05:51:54.707 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T05:51:55.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T05:51:55.118 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T05:51:55.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T05:51:55.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T05:51:55.181 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T05:51:55.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T05:51:55.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.228 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.250 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-03-12T05:51:55.261 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-12T05:51:55.278 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-03-12T05:51:55.308 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:55.322 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:55.363 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T05:51:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:51:55.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T05:51:55.394 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T05:51:55.395 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi045 2024-03-12T05:51:55.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T05:51:55.445 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi116 2024-03-12T05:51:55.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T05:51:55.483 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379/remote/smithi130 2024-03-12T05:51:55.483 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T05:51:55.522 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T05:51:55.523 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T05:51:55.525 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T05:51:55.527 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T05:51:55.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T05:51:55.587 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T05:51:55.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T05:51:55.622 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T05:51:55.622 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T05:51:55.625 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T05:51:55.627 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T05:51:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: 266028 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 05:51 /home/ubuntu/cephtest 2024-03-12T05:51:55.644 INFO:teuthology.orchestra.run.smithi130.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 05:51 /home/ubuntu/cephtest 2024-03-12T05:51:55.644 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 05:51 /home/ubuntu/cephtest 2024-03-12T05:51:55.644 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T05:51:55.668 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T05:51:55.762 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T05:51:55.789 DEBUG:teuthology.nuke:shortname: smithi045 2024-03-12T05:51:55.789 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T05:51:55.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.359289', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFUDKe8UrWEjD/7h7It4GWED7M2xqJWeWgD7FPFX0UqaWjfKMS9KCvppBRfDQ35LWkBNqLnDk41PWpelOlWO94k='} 2024-03-12T05:51:55.818 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-12T05:51:55.819 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T05:51:55.841 DEBUG:teuthology.nuke:shortname: smithi130 2024-03-12T05:51:55.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T05:51:55.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.361079', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkwGVDinsQ1NjwoRq7GGQdA4YfQkcmYB4zicYnObp/VXjYy7TVEfujAtazWuh8CHnVqT6kly+oZxLY3VkjR2NE='} 2024-03-12T05:51:55.878 INFO:teuthology.orchestra.console.smithi045:Power off 2024-03-12T05:51:55.879 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T05:51:55.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591379', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 04:40:15.357326', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiHTZqak80E/y9pbSQZkkBJnx5LZluORpfqRWHVXSGqY9+n6fx5xyj2Yxe0NIRZa5MIKI+Kph5ebWAa8jH9kFg='} 2024-03-12T05:51:55.902 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-03-12T05:51:55.903 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:51:55.917 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-12T05:51:55.918 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T05:51:55.930 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-12T05:51:55.941 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-12T05:51:55.941 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:51:55.959 INFO:teuthology.orchestra.console.smithi130:Power off 2024-03-12T05:51:55.959 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T05:51:55.968 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-12T05:51:55.981 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-03-12T05:51:55.981 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:51:56.004 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-12T05:51:59.931 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:51:59.969 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:00.005 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:00.043 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-12T05:52:00.079 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-12T05:52:00.115 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-12T05:52:04.047 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:04.080 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:04.115 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:04.159 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-03-12T05:52:04.190 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-12T05:52:04.225 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-03-12T05:52:04.260 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-03-12T05:52:04.326 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-03-12T05:52:08.194 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T05:52:08.304 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-12T05:52:08.405 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-12T05:52:08.610 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} duration: 3620.218859910965 failure_reason: '"2024-03-12T05:02:41.358973+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-12T05:52:08.610 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T05:52:08.704 INFO:teuthology.run:FAIL